From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 7967D21F1C4 for ; Tue, 25 Feb 2014 08:29:19 -0800 (PST) Received: from u-089-cab204a2.am1.uni-tuebingen.de ([134.2.89.3]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0MVIva-1WkNBl3s7C-00YhD1 for ; Tue, 25 Feb 2014 17:29:16 +0100 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\)) From: Sebastian Moeller In-Reply-To: Date: Tue, 25 Feb 2014 17:29:16 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4E5BC321-2054-4364-BECC-DF34E0D20380@gmail.com> <34C9F2ED-49BD-43DF-8421-67D1D3CBA5E1@gmx.de> To: Dave Taht X-Mailer: Apple Mail (2.1510) X-Provags-ID: V03:K0:IJjwfmJ4uKMGv5bvMD0W663YbAADa33nCp0KtN4VcG3y+r4H5Go sb3nfcxLtFNR/Pw/Ddi1IXgP0AAru+e0dG5KxkeDC3JDpLkxKbIKGMdRWNlA/DjYhPZAlgx IxrJNAnNtf0YQud47OKiWJ6vjj1RxJaxfsq9wuc2DZLdGATgQJFyI0xxLQcvqpSj77lBXlL 720nnhh81OuZ2ZXX1M/Fw== Cc: Stuart Cheshire , cerowrt-devel Subject: Re: [Cerowrt-devel] Equivocal results with using 3.10.28-14 X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Feb 2014 16:29:20 -0000 Hi Dave, On Feb 25, 2014, at 16:54 , Dave Taht wrote: > On Tue, Feb 25, 2014 at 5:37 AM, Sebastian Moeller = wrote: >> Hi Rich, >>=20 >>=20 >> On Feb 25, 2014, at 14:09 , Rich Brown = wrote: >>=20 >>> Thanks everyone for all the good advice. I will summarize my = responses to all your notes now, then I'll go away and run more tests. >>>=20 >>> - Yes, I am using netperf 2.6.0 and netperf-wrapper from Toke's = github repo. >>>=20 >>> - The "sync rate" is the speed with which the DSL modem sends bits = to/from my house. I got this by going into the modem's admin interface = and poking around. (It turns out that I have a very clean line, high = SNR, low attenuation. I'm much less than a km from the central office.) = So actual speed should approach this, except... >>=20 >> I would think of this as the theoretical upper limit ;) >>=20 >>>=20 >>> - Of course, I have to subtract all those overheads that Sebastian = described - ATM 48-in-53, which knocks off 10%; ATM frame overhead which = could add up to 47 bytes padding to any packet, etc.) >>>=20 >>> - I looked at the target calculation in Dave's Home Gateway best = practices. = (http://snapon.lab.bufferbloat.net/~d/draft-taht-home-gateway-best-practic= es-00.html) Am I correct that it sets the target to five 1500-byte = packet transmission time or 5 msec, whichever is greater? >>=20 >> Note, the auto target implementation in ceropackages-3.10 = sqm-scripts uses the following: >>=20 >> adapt_target_to_slow_link() { >> CUR_LINK_KBPS=3D$1 >> CUR_EXTENDED_TARGET_US=3D >> MAX_PAKET_DELAY_IN_US_AT_1KBPS=3D$(( 1000 * 1000 *1540 * 8 / 1000 = )) >> CUR_EXTENDED_TARGET_US=3D$(( ${MAX_PAKET_DELAY_IN_US_AT_1KBPS} / = ${CUR_LINK_KBPS} )) # note this truncates the decimals >> # do not change anything for fast links >> [ "$CUR_EXTENDED_TARGET_US" -lt 5000 ] && = CUR_EXTENDED_TARGET_US=3D5000 >> case ${QDISC} in >> *codel|pie) >> echo "${CUR_EXTENDED_TARGET_US}" >> ;; >> esac >> } >>=20 >> This is modeled after the shell code Dave sent around, and does not = exactly match the free version, because I could not make heads and tails = out of the free version. (Happy to discuss change this in SQM if anybody = has a better idea) >=20 > Really the target calculation doesn't matter so much so long as it's > larger than the MTU. This is somewhat an artifact of htb which buffers > up an extra packet, and the cpu scheduler=85 Ah, so we just leave the implementation as is? Great! >=20 > It is becoming clearer (with the recent description of the pie + rate > limiter + mac compensation stuff in future cablemodems) that we can do > much work to improve the rate limiters, and should probably intertwine > them like what the cable folk just did, in order to get best > performance. >=20 >>>=20 >>> - I was astonished by the calculation of the bandwidth consumed by = acks in the reverse direction. In a 7mbps/768kbps setting, I'm going to = lose one quarter of the reverse bandwidth? Wow! >=20 > Yes. TCP's sending ack requirement was actually the main driver for > having any bandwidth at all on the upstream. Back in the 90s all cable > providers wanted to provide was sufficient bandwidth for a "buy" > button. But, due to > this behavior of TCP, a ratio between down/up of somewhere between 6 > and 12 to 1 was what was needed to make TCP work well. (and even then > they tried hard to make it worse and ack compression is still part of > many cable modem's provisioning) Interesting! >=20 > IPv6 has much larger acks than ipv4=85. Hopefully mostly delayed/stretched? >=20 > I did a lot of work on various forms of hybrid network compression > back in the day (early 90s), back when we had a single 10Mbit radio > feeding hundreds of subscribers, and a 14.4 modem sending back acks=85 I assume one modem per customer? > and data. It turned out that a substantial percentage of subscribers > actually wanted to upload stuff... and that we couldn't achieve 10Mbit > in the lab with a 14.4 return=85 So automatic "fair-ish" queueing, no single customer could hog = all 10Mbit ;) > and that you can only slice up 10Mbit > so many ways before you ran out of bandwidth, and you run out of > bandwidth long before you can turn a profit... >=20 > (and while I remember details of the radio setup and all the crazy > stuff we did to get more data through the modems, I can't remember the > name of the company) >=20 > At the time I was happy, sorta, in that we'd proven that future ISPs > HAD to provide some level of upstream bandwidth bigger than a buy > button, and the original e2e internet was going to be at least > somewhat preserved... >=20 > I didn't grok at the time that NAT was going to be widely deployed... > I mean, at the time, you sold a connection, and asked how big a > network did you want, we defaulted to a /28, and we were still handing > out class C's to everyone that asked. >=20 >> Well, so was I (first time I did that) but here is the kicker = with classical non0-delayed ACKs this actually doubles since each data = packet gets acknowledged (I assume this puts a lower bound on how = asymmetrical a loin an ISP can sell ;) ). But since I figured out that = macosx seems to default to 1 Ack for every 4 packets, so only half the = traffic. And note any truly bi-directional traffic shouldbe able to = piggyback many of those ACKs into normal data packets. >=20 > I doubt your '4'. Take a capture for a few hundred ms. Ah, I took this from a description about current behavior of = macosx that I found on the internet, it sounded about right. I guess = what wanted to say that the ACK imprint on the upload in the given = example could range from ~400kbps (1 ACK per packet) to 66.7 kbps (6 = ACKs per packet), but even that is a considerable fraction of the = available upload bandwidth=85 Silly idea, it would be great if cerowrt could have a real-time = display of the number/size of ACK packets (lightly smoothed in time) = going in and out the WAN link, that way it would be quite easy to better = estimate the traffic components hidden from RRUL. But that is m playing = arm-chair software architect ;) not that I am going to implement that... >=20 > My understanding of how macos works is that after a stream is > sustained for a while, it switches from one ack every 2 packets into > "stretch acks" - to one every 6 (or so). According to = http://rolande.wordpress.com/2010/12/30/performance-tuning-the-network-sta= ck-on-mac-osx-10-6/ net.inet.tcp.delayed_ack has the following meaning: =95 delayed_ack=3D0 responds after every packet (OFF) =95 delayed_ack=3D1 always employs delayed ack, 6 packets can = get 1 ack =95 delayed_ack=3D2 immediate ack after 2nd packet, 2 packets = per ack (Compatibility Mode) =95 delayed_ack=3D3 should auto detect when to employ delayed = ack, 4 packets per ack. (DEFAULT) I have it set at three, so this is why I argued with 4 (just hoping it = would not opt for one ACK per packet), and I am still on 10.8.5 (just in = case someone in the know should happen to read this; what is the true = behavior?) Best Regards Sebastian >=20 > there are some interesting bugs associated with stretch acks, and also > TSO - in one case we observed a full TSO of TCP RSTs being sent > instead of one. >=20 >=20 >=20 >>>=20 >>> - I wasn't entirely clear how to set the target in the SQM GUI. I = believe that "target ##msec" is an acceptable format. Is that correct? >>=20 >> In the new version with the dedicated target field, "40ms" = will work as will "40000us", "40 ms". In the most recent version "auto" = will set the auto adjustment of target (it will also extend interval by = the new-target - 5ms) to avoid the situation where Target gets larger = than interval. >> In the older versions you would put "target 40ms" into the = egress advanced option string. Note I used double quotes in my examples = for clarity, the GUI does not want those... >>=20 >>=20 >>>=20 >>> - There's also a discussion of setting the target with "auto", but = I'm not sure I understand the syntax. >>=20 >> just type in auto. You can check with log read and "tc -d = qdsic" >>=20 >> Best Regards >> Sebastian >>=20 >>>=20 >>> Now to find some time to go back into the measurement lab! I'll = report again when I have more data. Thanks again. >>>=20 >>> Rich >>>=20 >>>=20 >>>=20 >>> On Feb 24, 2014, at 9:56 AM, Aaron Wood wrote: >>>=20 >>>> Do you have the latest (head) version of netperf and = netperf-wrapper? some changes were made to both that give better UDP = results. >>>>=20 >>>> -Aaron >>>>=20 >>>>=20 >>>> On Mon, Feb 24, 2014 at 3:36 PM, Rich Brown = wrote: >>>>=20 >>>> CeroWrt 3.10.28-14 is doing a good job of keeping latency low. = But... it has two other effects: >>>>=20 >>>> - I don't get the full "7 mbps down, 768 kbps up" as touted by my = DSL provider (Fairpoint). In fact, CeroWrt struggles to get above = 6.0/0.6 mbps. >>>>=20 >>>> - When I adjust the SQM parameters to get close to those numbers, I = get increasing levels of packet loss (5-8%) during a concurrent ping = test. >>>>=20 >>>> So my question to the group is whether this behavior makes sense: = that we can have low latency while losing ~10% of the link capacity, or = that getting close to the link capacity should induce large packet = loss... >>>>=20 >>>> Experimental setup: >>>>=20 >>>> I'm using a Comtrend 583-U DSL modem, that has a sync rate of 7616 = kbps down, 864 kbps up. Theoretically, I should be able to tell SQM to = use numbers a bit lower than those values, with an ATM plus header = overhead with default settings. >>>>=20 >>>> I have posted the results of my netperf-wrapper trials at = http://richb-hanover.com - There are a number of RRUL charts, taken with = different link rates configured, and with different link layers. >>>>=20 >>>> I welcome people's thoughts for other tests/adjustments/etc. >>>>=20 >>>> Rich Brown >>>> Hanover, NH USA >>>>=20 >>>> PS I did try the 3.10.28-16, but ran into troubles with wifi and = ethernet connectivity. I must have screwed up my local configuration - I = was doing it quickly - so I rolled back to 3.10.28.14. >>>> _______________________________________________ >>>> Cerowrt-devel mailing list >>>> Cerowrt-devel@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>>>=20 >>>=20 >>> _______________________________________________ >>> Cerowrt-devel mailing list >>> Cerowrt-devel@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>=20 >> _______________________________________________ >> Cerowrt-devel mailing list >> Cerowrt-devel@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cerowrt-devel >=20 >=20 >=20 > --=20 > Dave T=E4ht >=20 > Fixing bufferbloat with cerowrt: = http://www.teklibre.com/cerowrt/subscribe.html