From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x231.google.com (mail-wr0-x231.google.com [IPv6:2a00:1450:400c:c0c::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 243AE3B29E for ; Mon, 12 Jun 2017 15:13:55 -0400 (EDT) Received: by mail-wr0-x231.google.com with SMTP id v111so109709773wrc.3 for ; Mon, 12 Jun 2017 12:13:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=M8LJU/QLfBdqHXgb06HQEuISa/OyNxbL6I6SnBItLIs=; b=QUsfNh7a/y6YBwQZb/+v8eUCfL5dIVG7IAaXLfwxEdbyHJoLYVHSLH5oap0tmU0oHT xKfosXUPWRJlZrP6A/VoEC70myI6pOKwQ9BIGDzugVi4GmhAxexB8LIACUcKxPYD5uWd N7ROVH2b4XLeO5hKKV4MFHHaR+ozlsDtIFl2ZPPJxxUlQO8pFyBlFOo2YIboepwSGeXu jpVfu2ooFwYpKxFnfbiPex6k/JSmx7w/+mH75PakFvQ1uR+n8soWyu/nIwyd7o+rhAfF xU5nLg1wqEYXXK6BS0IRszac357GKbDxPU7D9n79pidYLtFP5MBejBdBvZUjT8+ry08W X7Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=M8LJU/QLfBdqHXgb06HQEuISa/OyNxbL6I6SnBItLIs=; b=K5d9NB1wG2egdF9fTh9s0gHhaJfWbth2oJw3aIwqLx9e/jOzu1R9FLGrTvjnyKmuQS ucges2b7z7sXxnDRi8TbFaei+Y0ah0kVEYaRdbxylwJd97CVbylMUzZrAdbEEieJvHAd +1S433T5wefUQDiTbOmWuemDEASXhrvjNDjuokwQJJbb+6bfkzGsoX3SdAMntEQGTyED ZGhC27bvOh7eZ0FUPPt+Y1IPxgJdgulB1OJ+M0QpkF95S3Wg3bQ4sVwIN4gqdSW7Uchg Pnvx1Rc6g72ou150uM9UB9E3OlUevMmf4lcheUztIHxU2IIw1yBuHijHZGwcA6KXAqTC ee7g== X-Gm-Message-State: AODbwcBcPQLtwKQBqisylYS1rmkDMI2xiVVf28OByRWJr/px0yCJO6Pt SzmDVcK2NafQEMuaC1ZkmMYCNy9fQY1Y X-Received: by 10.80.163.200 with SMTP id t8mr13679061edb.158.1497294833663; Mon, 12 Jun 2017 12:13:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.175.98 with HTTP; Mon, 12 Jun 2017 12:13:53 -0700 (PDT) From: Cong Xu Date: Mon, 12 Jun 2017 14:13:53 -0500 Message-ID: To: cake@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="f403045c18d2af45690551c81e67" X-Mailman-Approved-At: Sun, 18 Jun 2017 14:13:30 -0400 Subject: [Cake] flow isolation with ipip X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jun 2017 19:13:55 -0000 --f403045c18d2af45690551c81e67 Content-Type: text/plain; charset="UTF-8" Hi, I wonder if cake's flow isolation works with the ipip tunnel? I hope to guarantee the networking fair-share among containers/VMs in the same host. Thus, I used sfq/fq to associate with each tc class created in advance to provide both shaping and scheduling. The scripts roughly look like this (Assume 2 containers hosting iperf client run in the same host. One container sends 100 parallel streams via -P 100 to iperf server running in another host, the other one send 10 parallel streams with -P 10.): tc qdisc add dev $NIC root handle 1: htb default 2 tc class add dev $NIC parent 1: classid 1:1 htb rate ${NIC_RATE}mbit burst 1m cburst 1m tc class add dev $NIC parent 1:1 classid 1:2 htb rate ${RATE1}mbit ceil ${NIC_RATE}mbit burst 1m cburst 1m tc class add dev $NIC parent 1:1 classid 1:3 htb rate ${RATE2}mbit ceil ${NIC_RATE}mbit burst1m cburst 1m tc qdisc add dev $NIC parent 1:2 handle 2 sfq perturb 10 tc qdisc add dev $NIC parent 1:3 handle 3 sfq perturb 10 tc filter ad ... It works well, each container running iperf gets the almost same bandwidth regardless of the flows number. (Without the sfq, the container sending 100 streams acchieves much higher bandwidth than the 10 streams guy.) -------------- simultaneous 2 unlimited (100 conns vs 10 conns) ------------- job "big-unlimited-client" created job "small-unlimited-client" created -------------- unlimited server <-- unlimited client (100 conns) ------------- [SUM] 0.00-50.01 sec 24.9 GBytes 4.22 Gbits/sec 16874 sender [SUM] 0.00-50.01 sec 24.8 GBytes 4.21 Gbits/sec receiver -------------- unlimited server <-- unlimited client (10 conns) ------------- [SUM] 0.00-50.00 sec 24.4 GBytes 4.19 Gbits/sec 13802 sender [SUM] 0.00-50.00 sec 24.4 GBytes 4.19 Gbits/sec receiver However, if the ipip is enabled, sfq dose not work anymore. -------------- simultaneous 2 unlimited (100 conns vs 10 conns) ------------- job "big-unlimited-client" created job "small-unlimited-client" created -------------- unlimited server <-- unlimited client (100 conns) ------------- [SUM] 0.00-50.00 sec 27.2 GBytes 4.67 Gbits/sec 391278 sender [SUM] 0.00-50.00 sec 27.1 GBytes 4.65 Gbits/sec receiver -------------- unlimited server <-- unlimited client (10 conns) ------------- [SUM] 0.00-50.00 sec 6.85 GBytes 1.18 Gbits/sec 64153 sender [SUM] 0.00-50.00 sec 6.82 GBytes 1.17 Gbits/sec receiver The reason behind is that the src/dst ip addresses using ipip tunnel are same for all flows which are the src/dst ip of the host NICs instead of veth ip of each container/VM, and there is no ports number for the outside header of ipip packet. I verified this by capturing the traffic on NIC and analyzing it with wireshark. The real src/dst ip of container/VM is visible on the tunnel device (e.g. tunl0). Theoretically, this issue can be solved if I set up tc class and sfq on tunl0 instead of host NIC. I tried it, unfortunately, it did not work either. fq does not work for the same reason, because both sfq and fq use the same flow classifier (src/dst ips and ports). So, I just wonder if cake works with ipip tunnel or not. I appreciate if you can provide any help based on your expertise. Thanks. Regards, Cong --f403045c18d2af45690551c81e67 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I wonder if cake's flow isolatio= n works with the ipip tunnel? I hope to guarantee the networking fair-share= among containers/VMs in the same host. Thus, I used sfq/fq to associate wi= th each tc class created in advance to provide both shaping and scheduling.= The scripts roughly look like this (Assume 2 containers hosting iperf clie= nt run in the same host. One container sends 100 parallel streams via -P 10= 0 to iperf server running in another host, the other one send 10 parallel s= treams with -P 10.):=C2=A0

tc qdisc add dev $NIC root handle 1: htb default 2tc class add dev = $NIC parent 1: classid 1:1 htb rate ${NIC_RATE}mbit burst 1m cburst 1m
tc cla= ss add dev $NIC parent 1:1 classid 1:2 htb rate ${RATE1}mbit ceil ${NIC_RATE}mbit burst 1m cburst 1m
tc c= lass add dev $NIC parent 1:1 classid 1:3 htb rate ${RATE2}mbit ceil ${NIC_RATE}mbit burst1m cburst 1m
tc q= disc add dev $NIC parent 1:2 handle 2 sfq perturb 10
tc qdisc add dev $NIC parent 1:3 handle = 3 sfq perturb 10
tc filter ad ...

It works well, each container = running iperf gets the almost same bandwidth regardless of the flows number= . (Without the sfq, the container sending 100 streams acchieves much higher= bandwidth than the 10 streams guy.)

-------------- simultaneous 2 u= nlimited (100 conns vs 10 conns) -------------
job "big-unlimited-c= lient" created
job "small-unlimited-client" created
--= ------------ unlimited server <-- unlimited client (100 conns) ---------= ----
[SUM]=C2=A0=C2=A0 0.00-50.01=C2=A0 sec=C2=A0 24.9 GBytes=C2=A0 4.22= Gbits/sec=C2=A0 16874=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 sender
[SUM]=C2=A0=C2=A0 0.00-50.01=C2=A0 sec=C2= =A0 24.8 GBytes=C2=A0 4.21 Gbits/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver
-------------- unlimited server <-- unlimited client (10 conns) ---= ----------
[SUM]=C2=A0=C2=A0 0.00-50.00=C2=A0 sec=C2=A0 24.4 GBytes=C2= =A0 4.19 Gbits/sec=C2=A0 13802=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 sender
[SUM]=C2=A0=C2=A0 0.00-50.00=C2=A0 se= c=C2=A0 24.4 GBytes=C2=A0 4.19 Gbits/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 recei= ver

However, if the ipip is enabled, sfq dose not work anymore.
=
-------------- simultaneous 2 unlimited (100 conns vs 10 conns) -------= ------
job "big-unlimited-client" created
job "small-u= nlimited-client" created
-------------- unlimited server <-- unl= imited client (100 conns) -------------
[SUM]=C2=A0=C2=A0 0.00-50.00=C2= =A0 sec=C2=A0 27.2 GBytes=C2=A0 4.67 Gbits/sec=C2=A0 391278=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sender
[SUM]= =C2=A0=C2=A0 0.00-50.00=C2=A0 sec=C2=A0 27.1 GBytes=C2=A0 4.65 Gbits/sec=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 receiver

-------------- unlimited server &l= t;-- unlimited client (10 conns) -------------
[SUM]=C2=A0=C2=A0 0.00-50= .00=C2=A0 sec=C2=A0 6.85 GBytes=C2=A0 1.18 Gbits/sec=C2=A0 64153=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sender
[= SUM]=C2=A0=C2=A0 0.00-50.00=C2=A0 sec=C2=A0 6.82 GBytes=C2=A0 1.17 Gbits/se= c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver

The reason behind is that th= e src/dst ip addresses using ipip tunnel are same for all flows which are t= he src/dst ip of the host NICs instead of veth ip of each container/VM, and= there is no ports number for the outside header of ipip packet. I verified= this by capturing the traffic on NIC and analyzing it with wireshark. The = real src/dst ip of container/VM is visible on the tunnel device (e.g. tunl0= ). Theoretically, this issue can be solved if I set up tc class and sfq on = tunl0 instead of host NIC. I tried it, unfortunately, it did not work eithe= r. fq does not work for the same reason, because both sfq and fq use the sa= me flow classifier (src/dst ips and ports). So, I just wonder if cake works= with ipip tunnel or not.

I appreciate if you can provide any help = based on your expertise. Thanks.

Regards,
Cong
--f403045c18d2af45690551c81e67--