[Cerowrt-devel] Ideas on how to simplify and popularize bufferbloat control for consideration.

[David Lang]

On Sat, 26 Jul 2014, Sebastian Moeller wrote:

> On Jul 26, 2014, at 23:14 , David Lang <david at lang.hm> wrote:
>
>> On Sat, 26 Jul 2014, Sebastian Moeller wrote:
>>
>>> On Jul 26, 2014, at 22:21 , David Lang <david at lang.hm> wrote:
>>>
>>>> On Sat, 26 Jul 2014, Sebastian Moeller wrote:
>>>>
>>>>>
>>>>> On Jul 25, 2014, at 22:57 , David Lang <david at lang.hm> wrote:
>>>>>
>>>>>> unless the router you are connecting to is running some sort of service to support that,
>>>>>
>>>>> 	But this still requires some service on the other side. You could try to 
>>>>> use ICMP packets, but these will only allow to measure RTT not one-way 
>>>>> delays (if you do this on ADSL you will find the RTT dominated by the 
>>>>> typically much slower uplink path). If network equipment would be 
>>>>> guaranteed to use NTP for decent clock synchronization and would respond 
>>>>> to timestamp ICMP messages with timestamp reply measuring bandwidth might 
>>>>> be “cheap” enough to keep running in the background, though.
>>>>>
>>>>> 	Since this looks too simple there must be a simple reason why this would 
>>>>> fail. (It would be nice if ping packets with timestamps would have 
>>>>> required the echo server top also store its incoming timestamp in the 
>>>>> echo, but I digress)
>>>>>
>>>>> 	I note that gargoyle uses a sparse stream of ping packets to a close 
>>>>> host and uses increases in RTT as proxy for congestion and signal to 
>>>>> throttle down stream link…
>>>>>
>>>> As you say, anything that requires symmetrical traffic (like ICMP isn't 
>>>> going to work, and routers do not currently offer any service that will.
>>>
>>>
>>> 	Well I think the gargoyle idea is feasible given that there is a 
>>> reference implementation out in the wild ;).
>>
>> I'm not worried about an implementation existing as much as the question of 
>> if it's on the routers/switches by default, and if it isn't, is the service 
>> simple enough to be able to avoid causing load on these devices and to avoid 
>> having any security vulnerabilities (or DDos potential)
>
> 	But with gargoyle the idea is to monitor a sparse ping stream to the 
> closest host responding and interpreting a sudden increase in RTT as a sign 
> the the upstreams buffers are filling up and using this as signal to throttle 
> on the home router. My limited experience shows that quite often close hosts 
> will respond to pings...

that measures latency, but how does it tell you bandwidth unless you are the 
only possible thing on the network and you measure what you are receiving?

>>>>>> you can't just test that link, you have to connect to something beyond that.
>>>>>
>>>>> 	So it would be sweet if we could use services that are running on the machines anyway, like ping. That way the “load” of all the leaf nodes of the internet continuously measuring their bandwidth could be handled in a distributed fashion avoiding melt-downs by synchronized measurement streams…
>>>>
>>>> Well, let's talk about what we would like to have on the router
>>>>
>>>> As I see it, we want to have two services
>>>>
>>>> 1. a service you send a small amount of data to and it responds by sending you a large amount of data (preferrably with the most accurate timestamps it has and the TTL of the packets it received)
>>>>
>>>> 2. a service you send a large amount of data to and it responds by sending you small responses, telling you how much data it has received (with a timestamp and what the TTL of the packets it received were)
>>>>
>>>> questions:
>>>>
>>>> A. Protocol: should these be UDP/TCP/SCTP/raw IP packets/???
>>>>
>>>> TCP has the problem of slow start so it would need substantially more traffic to flow to reach steady-state.
>>>>
>>>> anything else has the possibility of taking a different path through the router/switch software and so the performance may not be the same.
>>>
>>> 	You thing UDP would not work out?
>>
>> I don't trust that UDP would go through the same codepaths and delays as TCP
>
> 	Why should a router care
>
>> even fw_codel handles TCP differently
>
> 	Does it? I thought UDP typically reacts differently to fq_codels 
> dropping strategy but fq_codel does not differentiate between protocols (last 
> time I looked at the code I came to that conclusion, but I am not very fluent 
> in C so I might be simply wrong here)

with TCP, the system can tell the difference between different connections to 
the same system, with UDP it needs to infer this from port numbers, this isn't 
as accurate and so the systems (fq_codel and routers) handle them in a slightly 
different way. This does affect the numbers.

>> so if we measure with UDP, does it really reflect the 'real world' of TCP?
>
> 	But we care for UDP as well, no?

Yes, but the reality is that the vast majority of traffic is TCP, and that's 
what the devices are optimized to handle, so if we measure with UDP we may not 
get the same results as if we measure with TCP.

measuing with ICMP is different yet again.

Think of the router ASICs that handle the 'normal' traffic in the ASIC in the 
card, but 'unusual' traffic needs to be sent to the core CPU to be processed and 
is therefor MUCH slower

>>>> One thought I have is to require a high TTL on the packets for the services to respond to them. That way any abuse of the service would have to take place from very close on the network.
>>>>
>>>> Ideally these services would only respond to senders that are directly connected, but until these services are deployed and enabled by default, there is going to be a need to be the ability to 'jump over' old equipment. This need will probably never go away completely.
>>>
>>> 	But if we need to modify DSLAMs and CMTSs it would be much nicer if we could just ask nicely what the current negotiated bandwidths are ;)
>>
>> negotiated bandwith and effective bandwidth are not the same
>>
>> what if you can't talk to the devices directly connected to the DSL line, but only to a router one hop on either side?
>
> 	In my limited experience the typical bottleneck is the DSL line, so if 
> we shape for that we are fine… Assume for a moment the DSLAM uplink is so 
> congested because of oversubscription of the DSLAM, that now this constitutes 
> the bottleneck. Now the available bandwidth for each user depends on the 
> combined traffic of all users, not a situation we can reasonable shape for 
> anyway (I would hope that ISPs monitor this situation and would remedy it by 
> adding uplink capacity, so this hopefully is just a transient event).

for DSL you are correct, it's a point-to-point connection (star network 
topology), but we have other technologies used in homes that are shared-media 
bus topology networks. This includes cablemodems and wireless links.

>>
>> for example, I can't buy (at least not for anything close to a reasonable price) a router to run at home that has a DSL port on it, so I will always have some device between me and the DSL.
>
> 	http://wiki.openwrt.org/toh/tp-link/td-w8970 or

no 5GHz wireless?

> http://www.traverse.com.au/products ?

I couldn't figure out where to buy one through their site.

> If you had the DSL modem in the router 
> under cerowrts control you would not need to use a traffic shaper for your 
> uplink, as you could apply the BQL ideas to the ADSL driver.
>
>>
>> If you have a shared media (cable, wireless, etc), the negotiated speed is meaningless.
>
> 	Not exactly meaningless, if gives you an upper bound...

true, but is an upper bound good enough? How close does the estimate need to be?

and does it matter if both sides are doign fq_codel or is this still in the mode 
of trying to control the far side indirectly?

>> In my other location, I have a wireless link that is ethernet to the dish on the roof, I expect the other end is a similar setup, so I can never see the link speed directly (not to mention the fact that rain can degrade the effective link speed)
>
> 	One more case for measuring the link speed continuously!

at what point does the measuring process interfere with the use of the link? or 
cause other upstream issues.

>>>> Other requirements or restrictions?
>>>
>>> 	I think the measurement should be fast and continuous…
>>
>> Fast yes, because we want to impact the network as little as possible
>>
>> continuous?? I'm not so sure. Do conditions really change that much?
>
> 	You just gave an example above for changing link conditions, by shared media...

but can you really measure fast enough to handle shared media? at some point you 
need to give up measuring because by the time you have your measurement it's 
obsolete.

If you look at networking with a tight enough timeframe, it's either idle or 
100% utilized depending on if a bit is being sent at that instant, however a 
plot at that precision is worthless :-)

>> And as I ask in the other thread, how much does it hurt if your estimates are wrong?
>
> 	I think I sent a plot to that regard.

yep, our mails are crossing

>>
>> for wireless links the conditions are much more variable, but we don't really know what is going to work well there.
>
> 	Wireless as in point 2 point links or in wifi?

both, point-to-point is variable based on weather, trees blowing in the wind, 
interference, etc. Wifi has a lot more congestion, so interference dominates 
everything else.

David Lang