[Bloat] Bloat done correctly?

[Sebastian Moeller]

Hi Benjamin,

On Jun 12, 2015, at 17:33 , Benjamin Cronce <bcronce at gmail.com> wrote:

> > On Fri, Jun 12, 2015 at 4:08 AM, Sebastian Moeller wrote:
> > Hi Benjamin,
> > 
> > To go off onto a tangent:
> > 
> > On Jun 12, 2015, at 06:45 , Benjamin Cronce wrote:
> > 
> > > [...]
> > > Under load while doing P2P(About 80Mb down and 20Mb up just as I started the test)
> > > HFSC: P2P in 20% queue and 80/443/8080 in 40% queue with ACKs going to a 20% realtime queue
> > > http://www.dslreports.com/speedtest/622452
> > 
> >         I know this is not really your question, but I think the ACKs should go into the same queue as the matching data packets. Think about it that way, if the data is delayed due to congestion it does not make too much sense to tell the sender to send more faster (which essentially is what ACK prioritization does) as that will not really reduce the congestion but rather increase it.
> >         There is one caveat though: when ECN is used it might make sense to send out the ACK that will signal the congestion state back to the sender faster… So if you prioritize ACKs only select those with an ECN-Echo flag ;)
> >         @bloat : What do you all think about this refined ACK prioritization scheme?
> > 
> > Best Regards
> >         Sebastian
> 
> Here's a very real problem for many users. If you have a highly asymmetrical connection like what many DSL and cable users have, doing even mild uploading can consume enough bandwidth to affect your ability to upload ACKs, which can negatively affect your downloads.
> A regular offender to many is P2P. If you're uploading while downloading on a 30/3 connection, you may not be able to ACK data fast enough. Of course this is in of itself mostly an issue caused by bufferbloat and/or lack of fair queueing.

	I agree, but I think reducing buffer bloat and using fair queueing is superior to ACK prioritization, that’s all.

> 
> I guess the real question is, should ACKs be prioritized in a system that does not exhibit bufferbloat or has fair queuing that allows the ACKs to not feel the affect of other bandwidth being consumed. With no data to back this up, my first guess would be it will not matter. If you have no bufferbloat or you have fair queuing, the ACKs should effectively be sent nearly immediately.

	Actually fq_codel mildly boosts sparse flows, and since ACK typically are sparse they usually do not suffer. But all sparse or starting flows get the same treatment, so nothing ACK specific just something that typically works well. 

> One case where it could make a difference if where ACKs make up a sizable portion of the upload bandwidth if download is saturated. An example would be a hyper-asymmetrical connection like 60/3.

	Okay, I will bite, at one ACK every 2 full MSS send, ACKs take up roughly 2% of the reverse traffic, so at 60 the ACK will cost 60*0.02 = 1.2 or 100*1.2/3 = 40%, and I agree that is most likely will cause problems for people using smaller priority queues. It is also one reason why I have read recommendations to size priority classes equally (percentage wise) for up- and download, and make sure ACKs get the same priority as the reverse data...

> In this case, having the ACKs in a separate queue would actually do the opposite, it would put an upper bound on how much bandwidth ACKs could consume unless you gave your ACK queue nearly all of the bandwidth.

	Well, as stated above if your incoming traffic class allows 100% bandwidth use the corresponding outgoing class would better allow the same percentage… So i still think not putting ACKs in higher priorities gives better overall system balance. BUT I have no experience with P2P traffic, so things could be different there. I would hope to be able to tell my P2P apps to mark their packets as background priority, so the data and ACKs would move out of the way of more urgent traffic. I understand that you have actual P2P experience so take my input with a grain of salt, please...

Best Regards
	Sebastian

> 
> In the two examples that I could quickly think of, either it made little difference or it did the opposite of what your proposed. Of course it depends on the configuration of the shaping.
> 
> Minor side tangent about bloat and ACKs that doesn't really need a discussion
> 
> A few months back I was downloading some Linux torrents when I noticed I was getting packetloss. I pretty much never get packetloss. I also noticed that I was receiving a fluctuating 100Mb/s-103Mb/s of ingress on my WAN but only seeing about 80Mb/s of egress on my LAN. I did a packet capture and saw something funny. My WAN was sending out a lot of Dup ACK packets.
> I grabbed a few of the dest IPs that my firewall was sending to and did a trace route. Nice low pings for most of the path, then suddenly about 2-3 hops before the seeder in case their ISP's network, I started to see huge jitter and pings, in the 1sec-3sec range. Cox and Comcast were the two main offenders but they do represent a sizable portion of the population.
> My conclusion was bufferbloat was so bad on their networks that the seeders where not getting ACKs from me in a timely fashion, so they resent the segments assuming they were lost. This issue was so bad that about 20Mb/s of the 100Mb/s was duplicate data segments. I was being DDOS'd by bufferbloat.
> 
> P.S. I removed the emails because I was not absolutely sure if they would be scrubbed.