From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 2175021F1F3 for ; Tue, 28 Apr 2015 15:16:41 -0700 (PDT) Received: by igblo3 with SMTP id lo3so102455160igb.1 for ; Tue, 28 Apr 2015 15:16:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Wk7b7OXjYXBPnsoUJ4HYn2Tl1qzemGWqAGeWLUVmp0E=; b=OwiZX3wwWLvfGjPXXJRw1v0+8UGvyPr6aaWuX4zMfnBnNFwxM+bH+mDcCGpdtCKz/6 DIZlyF0SLr/cK07jMI0ScL9CiU51wxoomCZzk4svvuyxhXHSM45VnbBLZnGEHw4ZsLzf X+zEfWzql02Acd3goO9Oy74r6r707Jz/AV3zkTUolO3U27OhHSyHgxpLqvM5FVzU9+YZ 06wfagHf5V/2/GVuWMzfrpL/CIyq1S7o2AExP+dvErXnJgUZDq3F6M6WCcGD66eS+098 mu6aum+TaQFGI2YPcYUv6cVyMKTd2CzMfWRSBl1RQmxIUcZYRLZh22FGR8v2EAsLlg2A ZiWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Wk7b7OXjYXBPnsoUJ4HYn2Tl1qzemGWqAGeWLUVmp0E=; b=Qqg+JjypqN0XTHIdIyYszRE9H82fnozHIez5K5LgQNZByQkukRSFtZnK/bRtSNr4Sa bmasvVsTNPUai/QCrYW3BB1SiXjEkYI1yhaHtczv3ja4spBTa/Y5Si6ACmQwidGbTG9g u8iGy7FbgOia9BZl+WzAQIz6dpINkwj7JSi4IwIYZrhIbloFuUPSPsF0DUb28GKA21VD bHT/XE1IFvQ1++j756sHuvkXx88WdH/1sgabn8F92K+qREUvrkrPVsdAeSXFYoJdBnm8 aO39LEK0Ua9plpLGmF9LUHui20LRA7e5PsWU85YaopIn1BoeQqLgtZFcEOorQf//F+Q0 C9kg== X-Gm-Message-State: ALoCoQmIz2XbIm9dwEhb+J2n+Uy8+EN+uv7OY5U0nip74d4eQgInBuVpul3/rn80Ugdl5zrEfY/6 MIME-Version: 1.0 X-Received: by 10.42.119.2 with SMTP id z2mr138803icq.1.1430259400577; Tue, 28 Apr 2015 15:16:40 -0700 (PDT) Received: by 10.64.171.110 with HTTP; Tue, 28 Apr 2015 15:16:40 -0700 (PDT) In-Reply-To: References: Date: Tue, 28 Apr 2015 15:16:40 -0700 Message-ID: From: Eric Dumazet To: Dave Taht Content-Type: multipart/alternative; boundary=90e6ba6148be8227510514d037ed Cc: cake@lists.bufferbloat.net, "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Peeling GRO X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Apr 2015 22:17:11 -0000 --90e6ba6148be8227510514d037ed Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Adding a limit on number of segments aggregated by GRO is doable. We have such a patch at Google, mainly because we run a 3.11 based kernel, and we increased MAX_SKB_FRAGS from 17 to 45 Doing so, we added a sysctl to be able to quickly 'revert' this increase in case it had some catastrophic consequence. cat /proc/sys/net/core/gro_max_frags 45 On Tue, Apr 28, 2015 at 3:12 PM, Dave Taht wrote: > On Tue, Apr 28, 2015 at 3:04 PM, Eric Dumazet wrote= : > > This what we do in net/sched/sch_tbf.c (search for tbf_segment()) > > Thank you! I had no idea that what I had been calling peeling already > existed. Perhaps in the simplest.qos model in the sqm-scripts (only > one sub qdisc) we should try tbf instead of htb? > > > Although this is quite expensive, this is doable. > > Turning off GRO universally is also expensive (in terms of costs to > other devices that are not rate limited, and in terms of actually > getting it right in the first place) > > the mvneta dropped from 720Mbit each way to 320 with offloads off. We > have had similar trouble with the ubnt edgerouters (8 ports worth of > offloads). > > We were mostly struggling with stuff > at very low rates < 60Mbit, now we are struggling with 100-300Mbit > inbound.... > > > > > > > > > On Tue, Apr 28, 2015 at 2:38 PM, Dave Taht wrote: > >> > >> Dear Eric: > >> > >> how would you go about peeling apart a TSO/GSO/GRO packet in a qdisc, > >> and then serving up one packet (or drr quantum's worth) of packets? > >> > >> I am seeing 64K GRO'd packets from mvneta driver in the linksys 1900ac > >> device in particular. > >> > >> ---------- Forwarded message ---------- > >> From: Dave Taht > >> Date: Tue, Apr 28, 2015 at 11:04 AM > >> Subject: documentation review request and out of tree cake builds for > >> openwrt/etc. > >> To: cake@lists.bufferbloat.net, bloat , > >> "cerowrt-devel@lists.bufferbloat.net" > >> > >> > >> > >> 1) I have burned much of the morning updating this: > >> > >> http://www.bufferbloat.net/projects/codel/wiki/Cake > >> > >> Reviewers wanted. > >> > >> 2) I am not sure of the right configuration to push stuff to the > >> iproute2-cake repo (removing sfq_codel, and there is an important > >> bugfix in the mainline iproute2 that needs to be pushed into it) > >> > >> 3) I have pushed support for building cake out of tree in openwrt to > >> the ceropackages repo (for now), but have not created a new > >> iproute2-cake for it due to item 2. My intent would be to get both > >> into openwrt chaos calmer for further testing... in some repo or > >> another that wasn't ceropackages. the routing repo? > >> > >> 4) http://www.dslreports.com/forum/news,133536~fmode=3Dflat~days=3D200= 0 > >> has some good convo going on also. > >> > >> -- > >> Dave T=C3=A4ht > >> Open Networking needs **Open Source Hardware** > >> > >> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 > >> > >> > >> -- > >> Dave T=C3=A4ht > >> Open Networking needs **Open Source Hardware** > >> > >> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 > > > > > > > > -- > Dave T=C3=A4ht > Open Networking needs **Open Source Hardware** > > https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 > --90e6ba6148be8227510514d037ed Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Adding a limit on number of segments aggregated by GRO is = doable.

We have such a patch at Google, mainly because w= e run a 3.11 based kernel, and we increased MAX_SKB_FRAGS from 17 to 45
Doing so, we added a sysctl to be able to quickly 'reve= rt' this increase in case it had some catastrophic consequence.

cat /proc/sys/net/core/gro_max_frags
45



=
On Tue, Apr 28, 2015 at 3:12 PM, Dave Taht <d= ave.taht@gmail.com> wrote:
= On Tue, Apr 28, 2015 at 3:04 PM, Eric Dumazet <edumazet@google.com> wrote:
> This what we do in net/sched/sch_tbf.c=C2=A0 (search for tbf_segment()= )

Thank you! I had no idea that what I had been calling peeling already
existed. Perhaps in the simplest.qos model in the sqm-scripts (only
one sub qdisc) we should try tbf instead of htb?

> Although this is quite expensive, this is doable.

Turning off GRO universally is also expensive (in terms of costs to
other devices that are not rate limited, and in terms of actually
getting it right in the first place)

the mvneta dropped from 720Mbit each way to 320 with offloads off. We
have had similar trouble with the ubnt edgerouters (8 ports worth of
offloads).

We were mostly struggling with stuff
at very low rates < 60Mbit, now we are struggling with 100-300Mbit inbou= nd....

>
>
>
> On Tue, Apr 28, 2015 at 2:38 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Dear Eric:
>>
>> how would you go about peeling apart a TSO/GSO/GRO packet in a qdi= sc,
>> and then serving up one packet (or drr quantum's worth) of pac= kets?
>>
>> I am seeing 64K GRO'd packets from mvneta driver in the linksy= s 1900ac
>> device in particular.
>>
>> ---------- Forwarded message ----------
>> From: Dave Taht <dave.ta= ht@gmail.com>
>> Date: Tue, Apr 28, 2015 at 11:04 AM
>> Subject: documentation review request and out of tree cake builds = for
>> openwrt/etc.
>> To: cake@lists.buffe= rbloat.net, bloat <bl= oat@lists.bufferbloat.net>,
>> "cerow= rt-devel@lists.bufferbloat.net"
>> <cerowrt= -devel@lists.bufferbloat.net>
>>
>>
>> 1) I have burned much of the morning updating this:
>>
>> http://www.bufferbloat.net/projects/codel/wiki/Cake
>>
>> Reviewers wanted.
>>
>> 2) I am not sure of the right configuration to push stuff to the >> iproute2-cake repo (removing sfq_codel, and there is an important<= br> >> bugfix in the mainline iproute2 that needs to be pushed into it) >>
>> 3) I have pushed support for building cake out of tree in openwrt = to
>> the ceropackages repo (for now), but have not created a new
>> iproute2-cake for it due to item 2. My intent would be to get both=
>> into openwrt chaos calmer for further testing... in some repo or >> another that wasn't ceropackages. the routing repo?
>>
>> 4) http://www.dslreports.com/forum/news,13= 3536~fmode=3Dflat~days=3D2000
>> has some good convo going on also.
>>
>> --
>> Dave T=C3=A4ht
>> Open Networking needs **Open Source Hardware**
>>
>> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe= 2pFr67
>>
>>
>> --
>> Dave T=C3=A4ht
>> Open Networking needs **Open Source Hardware**
>>
>> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe= 2pFr67
>
>



--
Dave T=C3=A4ht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67

--90e6ba6148be8227510514d037ed--