[Cerowrt-devel] Ideas on how to simplify and popularize bufferbloat control for consideration.

[David Lang]

On Sat, 26 Jul 2014, Sebastian Moeller wrote:

> On Jul 26, 2014, at 22:21 , David Lang <david at lang.hm> wrote:
>
>> On Sat, 26 Jul 2014, Sebastian Moeller wrote:
>>
>>>
>>> On Jul 25, 2014, at 22:57 , David Lang <david at lang.hm> wrote:
>>>
>>>> The trouble is that to measure bandwidth, you have to be able to send and 
>>>> receive a lot of traffic.
>>>
>>> 	Well that is what you typically do, but you can get away with less 
>>> measurement traffic: in an ideal quiescent network sending two packets back 
>>> to back should give you the bandwidth (packet size / incoming time 
>>> difference of both packets), or send two packets of different size (needs 
>>> synchronized clocks, then difference of packet sizes / difference of 
>>> transfer times).
>>
>> Except that your ideal network doesn't exist in the real world. You are never 
>> going to have the entire network quiescent, the router you are going to be 
>> talking to is always going to have other things going on, which can affect 
>> it's timing.
>
> 	Sure, the two packets a required per measurement, guess I would 
> calculate the average and confidence interval over several of these 
> (potentially by a moving window) to get a handle on the variability. I have 
> done some RTT measurements on a ADSL link and can say that realistically one 
> needs in the hundreds data points per packet size. This sounds awe full, but 
> at least it does not require to saturate the link and hence works without 
> dedicated receivers on the other end...
>
>>
>>>> unless the router you are connecting to is running some sort of service to support that,
>>>
>>> 	But this still requires some service on the other side. You could try to use ICMP packets, but these will only allow to measure RTT not one-way delays (if you do this on ADSL you will find the RTT dominated by the typically much slower uplink path). If network equipment would be guaranteed to use NTP for decent clock synchronization and would respond to timestamp ICMP messages with timestamp reply measuring bandwidth might be “cheap” enough to keep running in the background, though.
>>> 	Since this looks too simple there must be a simple reason why this would fail. (It would be nice if ping packets with timestamps would have required the echo server top also store its incoming timestamp in the echo, but I digress)
>>> 	I note that gargoyle uses a sparse stream of ping packets to a close host and uses increases in RTT as proxy for congestion and signal to throttle down stream link…
>>
>> As you say, anything that requires symmetrical traffic (like ICMP isn't going to work, and routers do not currently offer any service that will.
>
> 	Well I think the gargoyle idea is feasible given that there is a 
> reference implementation out in the wild ;).

I'm not worried about an implementation existing as much as the question of if 
it's on the routers/switches by default, and if it isn't, is the service simple 
enough to be able to avoid causing load on these devices and to avoid having any 
security vulnerabilities (or DDos potential)

>>>> you can't just test that link, you have to connect to something beyond 
>>>> that.
>>>
>>> 	So it would be sweet if we could use services that are running on the 
>>> machines anyway, like ping. That way the “load” of all the leaf nodes of the 
>>> internet continuously measuring their bandwidth could be handled in a 
>>> distributed fashion avoiding melt-downs by synchronized measurement streams…
>>
>> Well, let's talk about what we would like to have on the router
>>
>> As I see it, we want to have two services
>>
>> 1. a service you send a small amount of data to and it responds by sending 
>> you a large amount of data (preferrably with the most accurate timestamps it 
>> has and the TTL of the packets it received)
>>
>> 2. a service you send a large amount of data to and it responds by sending 
>> you small responses, telling you how much data it has received (with a 
>> timestamp and what the TTL of the packets it received were)
>>
>> questions:
>>
>> A. Protocol: should these be UDP/TCP/SCTP/raw IP packets/???
>>
>> TCP has the problem of slow start so it would need substantially more traffic 
>> to flow to reach steady-state.
>>
>> anything else has the possibility of taking a different path through the 
>> router/switch software and so the performance may not be the same.
>
> 	You thing UDP would not work out?

I don't trust that UDP would go through the same codepaths and delays as TCP

even fw_codel handles TCP differently

so if we measure with UDP, does it really reflect the 'real world' of TCP?

>> B. How much data is needed to be statistically accurate?
>>
>> Too many things can happen for 1-2 packets to tell you the answer. The 
>> systems on both ends are multi-tasking, and at high speeds, scheduling jitter 
>> will throw off your calculations with too few packets.
>
> 	Yeah, but you can (to steal an I idea from Rick Jones netperf) just keep 
> measuring until the confidence interval around the mean of the data falls 
> below a set magnitude. But for the purpose of traffic shaping you do not need 
> the exact link bandwidth anyway just a close enough proxy to start the search 
> for a decent set point from a reasonable position. I think that the actual 
> shaping rates need to be iteratively optimized.
>
>>
>> C. How can this be prevented from being used for DoS attacks, either against 
>> the thing running the service or against someone else via a reflected attack 
>> if it's a forgable protocol (i.e. UDP)
>
> 	Well, if it only requires a sparse packet stream it is not going to be 
> to useful for DOS attacks,

unless it can be requested a lot

>> One thought I have is to require a high TTL on the packets for the services 
>> to respond to them. That way any abuse of the service would have to take 
>> place from very close on the network.
>>
>> Ideally these services would only respond to senders that are directly 
>> connected, but until these services are deployed and enabled by default, 
>> there is going to be a need to be the ability to 'jump over' old equipment. 
>> This need will probably never go away completely.
>
> 	But if we need to modify DSLAMs and CMTSs it would be much nicer if we 
> could just ask nicely what the current negotiated bandwidths are ;)

negotiated bandwith and effective bandwidth are not the same

what if you can't talk to the devices directly connected to the DSL line, but 
only to a router one hop on either side?

for example, I can't buy (at least not for anything close to a reasonable price) 
a router to run at home that has a DSL port on it, so I will always have some 
device between me and the DSL.

If you have a shared media (cable, wireless, etc), the negotiated speed is 
meaningless.

In my other location, I have a wireless link that is ethernet to the dish on the 
roof, I expect the other end is a similar setup, so I can never see the link 
speed directly (not to mention the fact that rain can degrade the effective link 
speed)

>> Other requirements or restrictions?
>
> 	I think the measurement should be fast and continuous…

Fast yes, because we want to impact the network as little as possible

continuous?? I'm not so sure. Do conditions really change that much? And as I 
ask in the other thread, how much does it hurt if your estimates are wrong?

for wireless links the conditions are much more variable, but we don't really 
know what is going to work well there.

David Lang