[Cerowrt-devel] Ideas on how to simplify and popularize bufferbloat control for consideration.

[Sebastian Moeller]

On Jul 27, 2014, at 03:04 , David Lang <david at lang.hm> wrote:

> On Sun, 27 Jul 2014, Sebastian Moeller wrote:
[...]
>> 
>>> 
>>> Think of the router ASICs that handle the 'normal' traffic in the ASIC in the card, but 'unusual' traffic needs to be sent to the core CPU to be processed and is therefor MUCH slower
>> 
>> 	Except for my ICMP RTT measurements I still saw quantization steps in accordance with the expected best case RTT for a packet, showing that the slow processing at least is constant and hence easy to get ridd of in measurements...
> 
> yeah, I have to remind myself of the "perfect is the enemy of good enough" frequently as well. I tend to fall into that trap pretty easily, as this discussion has shown :-)
> 
> ping is easy to test. As a thought, is the response time of NTP queries any more or less stable?

	No idea? How would you test this (any command line to try). The good thingg with the ping is that often even the DSLAM responds keeping external sources (i.e. hops further away in the network) of variability out of the measurement...

> 
>>>>>>> One thought I have is to require a high TTL on the packets for the services to respond to them. That way any abuse of the service would have to take place from very close on the network.
>>>>>>> 
>>>>>>> Ideally these services would only respond to senders that are directly connected, but until these services are deployed and enabled by default, there is going to be a need to be the ability to 'jump over' old equipment. This need will probably never go away completely.
>>>>>> 
>>>>>> 	But if we need to modify DSLAMs and CMTSs it would be much nicer if we could just ask nicely what the current negotiated bandwidths are ;)
>>>>> 
>>>>> negotiated bandwith and effective bandwidth are not the same
>>>>> 
>>>>> what if you can't talk to the devices directly connected to the DSL line, but only to a router one hop on either side?
>>>> 
>>>> 	In my limited experience the typical bottleneck is the DSL line, so if we shape for that we are fine… Assume for a moment the DSLAM uplink is so congested because of oversubscription of the DSLAM, that now this constitutes the bottleneck. Now the available bandwidth for each user depends on the combined traffic of all users, not a situation we can reasonable shape for anyway (I would hope that ISPs monitor this situation and would remedy it by adding uplink capacity, so this hopefully is just a transient event).
>>> 
>>> for DSL you are correct, it's a point-to-point connection (star network topology), but we have other technologies used in homes that are shared-media bus topology networks. This includes cablemodems and wireless links.
>> 
>> 	Well, yes I understand, but you again would assume that the cable ISP tries to provision the system so that most users are happy, so congestion is not the rule? Even then I think cable guarantees some minimum rates per user, no? With wireless it is worse in that RF events outside of the ISP and end users control can ruin the day.
> 
> guarantee is too strong a word. It depends on how much competition there is.
> 
> 15 years or so ago I moved from a 3Mb cablemodem to a 128K IDSL line and saw my performance increase significantly.

	I used to think exactly the same, but currently I tend to think that the difference is about how well managed a node is not so much the access technology, with DSL the shared medium is the link connecting the DSLAM to the backbone, if this is congested it is similar to a busy cable node. In both cases the ISP needs to make sure the shared segments congestion is well managed. I might be that DSLAMs are typically better manages as TELCO’s always dealt with interactive (bi-directional) traffic while cable traditionally was a one-directional transport. So I assume both have different traditions about provisioning. I could be off my rocker here ;)


> 
>>>>> for example, I can't buy (at least not for anything close to a reasonable price) a router to run at home that has a DSL port on it, so I will always have some device between me and the DSL.
>>>> 
>>>> 	http://wiki.openwrt.org/toh/tp-link/td-w8970 or
>>> 
>>> no 5GHz wireless?
>> 
>> 	Could be, but definitely reasonable priced, probany cheap enough to use as smart de-bloated DSL modem, so your main router does not need HTB traffic shaping on uplink anymore. I might actually go that route since I really dislike my ISP primary router, but I digress...
>> 
>>> 
>>>> http://www.traverse.com.au/products ?
>>> 
>>> I couldn't figure out where to buy one through their site.
>> 
>> 	Maybe they only sell in AU, I guess I just wanted to be helpful,
>> 
>>> 
>>>> If you had the DSL modem in the router under cerowrts control you would not need to use a traffic shaper for your uplink, as you could apply the BQL ideas to the ADSL driver.
>>>> 
>>>>> 
>>>>> If you have a shared media (cable, wireless, etc), the negotiated speed is meaningless.
>>>> 
>>>> 	Not exactly meaningless, if gives you an upper bound...
>>> 
>>> true, but is an upper bound good enough? How close does the estimate need to be?
>> 
>> 	If we end up recommending people using say binary search to find the best tradeoff (maximizing throughput while keeping the maximum latency under load increase bounded to say 10ms) we should have an idea where to start, so bit to large is fine as a starting point. Traditionally the recommendation was around 85% of link rates, but that never came with a decent justification or data.
> 
> well, if we are doing a binary search, having the initial estimate off by a lot isn't actually going to hurt much, we'll still converge very quickly on the right value

	Yes, but we still need to solve the question what infrastructure to test against ;)
> 
>>> 
>>> and does it matter if both sides are doign fq_codel or is this still in the mode of trying to control the far side indirectly?
>> 
>> 	Yes, this is only relevant as long as both sides of the bottleneck link are not de-bloated. But it does not look like DSLAMs/CMTs will change any time soon from the old ways...
> 
> yep, I had been forgetting this.
> 
>>> 
>>>>> In my other location, I have a wireless link that is ethernet to the dish on the roof, I expect the other end is a similar setup, so I can never see the link speed directly (not to mention the fact that rain can degrade the effective link speed)
>>>> 
>>>> 	One more case for measuring the link speed continuously!
>>> 
>>> at what point does the measuring process interfere with the use of the link? or cause other upstream issues.
>> 
>> 	If my measuring by sparse stream idea works out the answer to both questions is not much ;)
>> 
>>> 
>>>>>>> Other requirements or restrictions?
>>>>>> 
>>>>>> 	I think the measurement should be fast and continuous…
>>>>> 
>>>>> Fast yes, because we want to impact the network as little as possible
>>>>> 
>>>>> continuous?? I'm not so sure. Do conditions really change that much?
>>>> 
>>>> 	You just gave an example above for changing link conditions, by shared media...
>>> 
>>> but can you really measure fast enough to handle shared media? at some point you need to give up measuring because by the time you have your measurement it's obsolete.
>> 
>> 	So this is not going to work well a wifi wlan with wildly fluctuating rates (see Dave’s upcoming project make-wifi-fast) but for typical cable node where congestion changes over the day as a function of people being at home it might be fast enough.
>> 
>>> 
>>> If you look at networking with a tight enough timeframe, it's either idle or 100% utilized depending on if a bit is being sent at that instant, however a plot at that precision is worthless :-)
>> 
>> 	Yes I think a moving average over some time would be required.
>> 
>>> 
>>>>> And as I ask in the other thread, how much does it hurt if your estimates are wrong?
>>>> 
>>>> 	I think I sent a plot to that regard.
>>> 
>>> yep, our mails are crossing
>>> 
>>>>> 
>>>>> for wireless links the conditions are much more variable, but we don't really know what is going to work well there.
>>>> 
>>>> 	Wireless as in point 2 point links or in wifi?
>>> 
>>> both, point-to-point is variable based on weather, trees blowing in the wind, interference, etc. Wifi has a lot more congestion, so interference dominates everything else.
>> 
>> 	So maybe that is a diffent kettle of fish then.
> 
> I think we need to get a simple, repeatable test together and then have people start using it and reporting what they find and the type of connection they are on, otherwise we are speculating from far too little data.

	So Rich Brown’s betterspeedtest.sh is a simple test, at least for the crowd of people involved in the buffer bloat discussion right now. I always love to see more data, especially I would be interested to see data from VDSL1 lines and GPON fiber lines…

Best Regards
	Sebastian
> 
> David Lang