From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x22f.google.com (mail-wr0-x22f.google.com [IPv6:2a00:1450:400c:c0c::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BA6FB3CC00 for ; Thu, 17 Aug 2017 15:27:03 -0400 (EDT) Received: by mail-wr0-x22f.google.com with SMTP id 5so27786953wrz.5 for ; Thu, 17 Aug 2017 12:27:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=jl2yK62vcyQczDEDCqEmGc399lWi2wT1qTbZTb4Fwf0=; b=gIv/KMdl36cxa8Y0GTYDrp6MqkcSMIm8dYF254J4AHSL6Du/Ck2ADyZG4NQVJHg1Jr fGqEXKG7R6mrptnBmfj9f2VjynJo4EZ2cbFLxlriFJvL9dxQ2wPgNeGoZhJH4IzaYamA fCalM+z35nI9dKifAwfRs4f3HR554t8+V0eAWdI7JDmRfuTKOKtiHfNKy5qI76pd58AQ 8FznWRvEYXvspCTEWxjUmoEAYJ3KU+/vbvw8AXhKAfv/TQiQWya5Hbhk5keDVZAAkBm0 Zh+sKUTTdufOxaqM36zJhg5uWyC5tPXkLNmrSDPKNc4jr8XVenWqmE/wTEB1kqRP/ihX 9+XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=jl2yK62vcyQczDEDCqEmGc399lWi2wT1qTbZTb4Fwf0=; b=eyJgPnzC5aUTZheJvUiIeX3uFW1il+O81rjwrSRRiqcttkM9G+DZz0UzshLncjY/Ta oNMuwJTpBLoV1ah+pJJAPtsgaUhZTXkWHJbUbDsjZh8B20YhOne5KBodRtIE2brbBhIn rJyCmkETtJ0Oj713hddXfRp147iJ7Y77jB4xiCujAz79VQx2kwe2v0AobU21Z55IrCuP qGQwalUOdGpk17ovzq8vJ9CWGQUDFwUjxghrUnERlzosKo9uzTefrUYNtZemD0PEHLZA pdLvzv29TATumr7Nj6RZNtUBiVhZZCqWPyNYWSr4Bp7OXXpzaCkIGftNVGK4lPgr8KJS WxlA== X-Gm-Message-State: AHYfb5j5TObccaBRjPJEg1XtolqwRLoiujNSTBav6LmPC+9RyvqjMuOR lbvxKSAdbGmcuujcAI8MZexA7LrwyQ== X-Received: by 10.80.147.12 with SMTP id m12mr2766699eda.248.1502998022804; Thu, 17 Aug 2017 12:27:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.162.70 with HTTP; Thu, 17 Aug 2017 12:27:02 -0700 (PDT) In-Reply-To: <7CEC639E-A3CF-492A-BE4D-CD05558FFC93@gmail.com> References: <7CEC639E-A3CF-492A-BE4D-CD05558FFC93@gmail.com> From: Cong Xu Date: Thu, 17 Aug 2017 14:27:02 -0500 Message-ID: To: Pete Heist Cc: cake@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="94eb2c1a50623f59290556f7ffee" X-Mailman-Approved-At: Thu, 17 Aug 2017 18:35:15 -0400 Subject: Re: [Cake] flow isolation with ipip X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Aug 2017 19:27:04 -0000 --94eb2c1a50623f59290556f7ffee Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Pete, Thanks for your reply. It is helpful. I have fixed my problem with IPIP. I checked the code of sfq in the kernel, it can recognize the IPIP encapsulation. The issue I met is caused by some other reasons, not IPIP. Regards, Cong 2017-08-17 12:38 GMT-05:00 Pete Heist : > I don=E2=80=99t know if this helps, but I think this should work. :) I us= ed IPIP > tunnels (with and without FOU encapsulation) over a point-to-point WiFi > bridge as a way of testing Cake over WiFi without traffic being prioritiz= ed > by the Linux WiFi stack or WMM, for example. The WiFi stack =E2=80=9Csees= " the > outer IPIP packet, and treats it with whatever diffserv marking is on the > outer packet, rather than what=E2=80=99s on the inner packet that=E2=80= =99s encapsulated. I > applied Cake to the tunnel device, which seemed to see the packets before > encapsulation, and it worked well. I think it should also work for flow > isolation. > > I can go through my setup scripts and get more specific if need be, to > make sure I=E2=80=99m not leading anyone astray. I think the important pa= rt is that > Cake be applied to the tunnel device and not just a regular device that= =E2=80=99s > carrying IPIP traffic... > > > Message: 1 > > Date: Thu, 17 Aug 2017 02:55:17 +0300 > > From: Jonathan Morton > > To: Cong Xu > > Cc: Cake List > > Subject: Re: [Cake] flow isolation with ipip > > Message-ID: > > gmail.com> > > Content-Type: text/plain; charset=3D"utf-8" > > > > Cake makes use of Linux' "packet dissecting" infrastructure. If the > latter > > knows about the tunnelling protocol, Cake should naturally see the IP a= nd > > port numbers of the inner payload rather than the outer tunnel. > > > > I don't know, however, precisely what tunnels are supported. At minimum= , > > don't ever expect encrypted tunnels to behave this way! > > > > - Jonathan Morton > > > > On 18 Jun 2017 21:13, "Cong Xu" wrote: > > > >> Hi, > >> > >> I wonder if cake's flow isolation works with the ipip tunnel? I hope t= o > >> guarantee the networking fair-share among containers/VMs in the same > host. > >> Thus, I used sfq/fq to associate with each tc class created in advance > to > >> provide both shaping and scheduling. The scripts roughly look like thi= s > >> (Assume 2 containers hosting iperf client run in the same host. One > >> container sends 100 parallel streams via -P 100 to iperf server runnin= g > in > >> another host, the other one send 10 parallel streams with -P 10.): > >> > >> tc qdisc add dev $NIC root handle 1: htb default 2 > >> tc class add dev $NIC parent 1: classid 1:1 htb rate ${NIC_RATE}mbit > >> burst 1m cburst 1m > >> tc class add dev $NIC parent 1:1 classid 1:2 htb rate ${RATE1}mbit cei= l > >> ${NIC_RATE}mbit burst 1m cburst 1m > >> tc class add dev $NIC parent 1:1 classid 1:3 htb rate ${RATE2}mbit cei= l > >> ${NIC_RATE}mbit burst1m cburst 1m > >> tc qdisc add dev $NIC parent 1:2 handle 2 sfq perturb 10 > >> tc qdisc add dev $NIC parent 1:3 handle 3 sfq perturb 10 > >> tc filter ad ... > >> > >> It works well, each container running iperf gets the almost same > bandwidth > >> regardless of the flows number. (Without the sfq, the container sendin= g > 100 > >> streams acchieves much higher bandwidth than the 10 streams guy.) > >> > >> -------------- simultaneous 2 unlimited (100 conns vs 10 conns) > >> ------------- > >> job "big-unlimited-client" created > >> job "small-unlimited-client" created > >> -------------- unlimited server <-- unlimited client (100 conns) > >> ------------- > >> [SUM] 0.00-50.01 sec 24.9 GBytes 4.22 Gbits/sec 16874 > >> sender > >> [SUM] 0.00-50.01 sec 24.8 GBytes 4.21 Gbits/sec > >> receiver > >> > >> -------------- unlimited server <-- unlimited client (10 conns) > >> ------------- > >> [SUM] 0.00-50.00 sec 24.4 GBytes 4.19 Gbits/sec 13802 > >> sender > >> [SUM] 0.00-50.00 sec 24.4 GBytes 4.19 Gbits/sec > >> receiver > >> > >> However, if the ipip is enabled, sfq dose not work anymore. > >> > >> -------------- simultaneous 2 unlimited (100 conns vs 10 conns) > >> ------------- > >> job "big-unlimited-client" created > >> job "small-unlimited-client" created > >> -------------- unlimited server <-- unlimited client (100 conns) > >> ------------- > >> [SUM] 0.00-50.00 sec 27.2 GBytes 4.67 Gbits/sec 391278 > >> sender > >> [SUM] 0.00-50.00 sec 27.1 GBytes 4.65 Gbits/sec > >> receiver > >> > >> -------------- unlimited server <-- unlimited client (10 conns) > >> ------------- > >> [SUM] 0.00-50.00 sec 6.85 GBytes 1.18 Gbits/sec 64153 > >> sender > >> [SUM] 0.00-50.00 sec 6.82 GBytes 1.17 Gbits/sec > >> receiver > >> > >> The reason behind is that the src/dst ip addresses using ipip tunnel a= re > >> same for all flows which are the src/dst ip of the host NICs instead o= f > >> veth ip of each container/VM, and there is no ports number for the > outside > >> header of ipip packet. I verified this by capturing the traffic on NIC > and > >> analyzing it with wireshark. The real src/dst ip of container/VM is > visible > >> on the tunnel device (e.g. tunl0). Theoretically, this issue can be > solved > >> if I set up tc class and sfq on tunl0 instead of host NIC. I tried it, > >> unfortunately, it did not work either. fq does not work for the same > >> reason, because both sfq and fq use the same flow classifier (src/dst > ips > >> and ports). So, I just wonder if cake works with ipip tunnel or not. > >> > >> I appreciate if you can provide any help based on your expertise. > Thanks. > >> > >> Regards, > >> Cong > >> > >> _______________________________________________ > >> Cake mailing list > >> Cake@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/cake > > --94eb2c1a50623f59290556f7ffee Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Pete,

Thanks for your reply= . It is helpful. I have fixed my problem with IPIP. I checked the code of s= fq in the kernel, it can recognize the IPIP encapsulation. The issue I met = is caused by some other reasons, not IPIP.

Regards,
= Cong=C2=A0

2017-08-17 12:38 GMT-05:00 Pete Heist <peteheist@gmail.com>:
I don=E2=80=99t know if this helps= , but I think this should work. :) I used IPIP tunnels (with and without FO= U encapsulation) over a point-to-point WiFi bridge as a way of testing Cake= over WiFi without traffic being prioritized by the Linux WiFi stack or WMM= , for example. The WiFi stack =E2=80=9Csees" the outer IPIP packet, an= d treats it with whatever diffserv marking is on the outer packet, rather t= han what=E2=80=99s on the inner packet that=E2=80=99s encapsulated. I appli= ed Cake to the tunnel device, which seemed to see the packets before encaps= ulation, and it worked well. I think it should also work for flow isolation= .

I can go through my setup scripts and get more specific if need be, to make= sure I=E2=80=99m not leading anyone astray. I think the important part is = that Cake be applied to the tunnel device and not just a regular device tha= t=E2=80=99s carrying IPIP traffic...

> Message: 1
> Date: Thu, 17 Aug 2017 02:55:17 +0300
> From: Jonathan Morton <chr= omatix99@gmail.com>
> To: Cong Xu <davidxu06@gmail= .com>
> Cc: Cake List <cake@l= ists.bufferbloat.net>
> Subject: Re: [Cake] flow isolation with ipip
> Message-ID:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<CAJq5cE0qSNrbzUufzaup3sZyeKaN=3DR=3DJAfqREojbyK6pFA= yzDw@mail.gmail.com>
> Content-Type: text/plain; charset=3D"utf-8"
>
> Cake makes use of Linux' "packet dissecting" infrastruct= ure.=C2=A0 If the latter
> knows about the tunnelling protocol, Cake should naturally see the IP = and
> port numbers of the inner payload rather than the outer tunnel.
>
> I don't know, however, precisely what tunnels are supported. At mi= nimum,
> don't ever expect encrypted tunnels to behave this way!
>
> - Jonathan Morton
>
> On 18 Jun 2017 21:13, "Cong Xu" <davidxu06@gmail.com> wrote:
>
>> Hi,
>>
>> I wonder if cake's flow isolation works with the ipip tunnel? = I hope to
>> guarantee the networking fair-share among containers/VMs in the sa= me host.
>> Thus, I used sfq/fq to associate with each tc class created in adv= ance to
>> provide both shaping and scheduling. The scripts roughly look like= this
>> (Assume 2 containers hosting iperf client run in the same host. On= e
>> container sends 100 parallel streams via -P 100 to iperf server ru= nning in
>> another host, the other one send 10 parallel streams with -P 10.):=
>>
>> tc qdisc add dev $NIC root handle 1: htb default 2
>> tc class add dev $NIC parent 1: classid 1:1 htb rate ${NIC_RATE}mb= it
>> burst 1m cburst 1m
>> tc class add dev $NIC parent 1:1 classid 1:2 htb rate ${RATE1}mbit= ceil
>> ${NIC_RATE}mbit burst 1m cburst 1m
>> tc class add dev $NIC parent 1:1 classid 1:3 htb rate ${RATE2}mbit= ceil
>> ${NIC_RATE}mbit burst1m cburst 1m
>> tc qdisc add dev $NIC parent 1:2 handle 2 sfq perturb 10
>> tc qdisc add dev $NIC parent 1:3 handle 3 sfq perturb 10
>> tc filter ad ...
>>
>> It works well, each container running iperf gets the almost same b= andwidth
>> regardless of the flows number. (Without the sfq, the container se= nding 100
>> streams acchieves much higher bandwidth than the 10 streams guy.)<= br> >>
>> -------------- simultaneous 2 unlimited (100 conns vs 10 conns) >> -------------
>> job "big-unlimited-client" created
>> job "small-unlimited-client" created
>> -------------- unlimited server <-- unlimited client (100 conns= )
>> -------------
>> [SUM]=C2=A0 =C2=A00.00-50.01=C2=A0 sec=C2=A0 24.9 GBytes=C2=A0 4.2= 2 Gbits/sec=C2=A0 16874
>> sender
>> [SUM]=C2=A0 =C2=A00.00-50.01=C2=A0 sec=C2=A0 24.8 GBytes=C2=A0 4.2= 1 Gbits/sec
>> receiver
>>
>> -------------- unlimited server <-- unlimited client (10 conns)=
>> -------------
>> [SUM]=C2=A0 =C2=A00.00-50.00=C2=A0 sec=C2=A0 24.4 GBytes=C2=A0 4.1= 9 Gbits/sec=C2=A0 13802
>> sender
>> [SUM]=C2=A0 =C2=A00.00-50.00=C2=A0 sec=C2=A0 24.4 GBytes=C2=A0 4.1= 9 Gbits/sec
>> receiver
>>
>> However, if the ipip is enabled, sfq dose not work anymore.
>>
>> -------------- simultaneous 2 unlimited (100 conns vs 10 conns) >> -------------
>> job "big-unlimited-client" created
>> job "small-unlimited-client" created
>> -------------- unlimited server <-- unlimited client (100 conns= )
>> -------------
>> [SUM]=C2=A0 =C2=A00.00-50.00=C2=A0 sec=C2=A0 27.2 GBytes=C2=A0 4.6= 7 Gbits/sec=C2=A0 391278
>> sender
>> [SUM]=C2=A0 =C2=A00.00-50.00=C2=A0 sec=C2=A0 27.1 GBytes=C2=A0 4.6= 5 Gbits/sec
>> receiver
>>
>> -------------- unlimited server <-- unlimited client (10 conns)=
>> -------------
>> [SUM]=C2=A0 =C2=A00.00-50.00=C2=A0 sec=C2=A0 6.85 GBytes=C2=A0 1.1= 8 Gbits/sec=C2=A0 64153
>> sender
>> [SUM]=C2=A0 =C2=A00.00-50.00=C2=A0 sec=C2=A0 6.82 GBytes=C2=A0 1.1= 7 Gbits/sec
>> receiver
>>
>> The reason behind is that the src/dst ip addresses using ipip tunn= el are
>> same for all flows which are the src/dst ip of the host NICs inste= ad of
>> veth ip of each container/VM, and there is no ports number for the= outside
>> header of ipip packet. I verified this by capturing the traffic on= NIC and
>> analyzing it with wireshark. The real src/dst ip of container/VM i= s visible
>> on the tunnel device (e.g. tunl0). Theoretically, this issue can b= e solved
>> if I set up tc class and sfq on tunl0 instead of host NIC. I tried= it,
>> unfortunately, it did not work either. fq does not work for the sa= me
>> reason, because both sfq and fq use the same flow classifier (src/= dst ips
>> and ports). So, I just wonder if cake works with ipip tunnel or no= t.
>>
>> I appreciate if you can provide any help based on your expertise. = Thanks.
>>
>> Regards,
>> Cong
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferblo= at.net
>> https://lists.bufferbloat.net/listinfo/cake=


--94eb2c1a50623f59290556f7ffee--