[Cake] Corrupted sit-tunnelled packets when using skb_gso_segment() on an IFB interface?

[Cake] Corrupted sit-tunnelled packets when using skb_gso_segment() on an IFB interface?

[Toke Høiland-Jørgensen]

Hi everyone

While investigating a bug report on CAKE[0], I've run into the following
behaviour:

When running CAKE as an ingress shaper on an IFB interface, if the GSO
splitting feature is turned on, TCP throughput will drop dramatically on
6in4 (sit) tunnels running over the interface in question. Looking at a
traffic dump, I'm seeing ~15% packet loss on the encapsulated TCP
stream.

IPv4 traffic is fine on the same interface, as is native IPv6 traffic.
And turning off GSO splitting in CAKE makes the packet loss go away. The
issue only seems to appear on IFB interfaces. So I'm wondering if there
is some interaction that corrupts packets when they are being split in
this configuration?

Steps to reproduce (assuming the box you are running on has IP 10.0.0.2
on eth0, and has a peer at 10.0.0.1 with a suitably configured sit
tunnel):

# modprobe ifb
# ip link set dev ifb0 up
# tc qdisc add dev eth0 handle ffff: ingress
# tc filter add dev eth0 parent ffff: protocol all prio 10 matchall action mirred egress redirect dev ifb0
# tc qdisc replace dev ifb0 root cake
# ip link add type sit local 10.0.0.2 remote 10.0.0.1
# ip link set dev sit1 up
# netperf -H fe80::a00:1%sit1 -t TCP_MAERTS

Whereas, in the same setup, this will work fine:

# netperf -H 10.0.0.1 -t TCP_MAERTS

As will this:

# tc qdisc replace dev ifb0 root cake no-split-gso
# netperf -H fe80::a00:1%sit1 -t TCP_MAERTS


Does anyone have any ideas? :)

-Toke

[0] https://github.com/tohojo/sqm-scripts/issues/72

Re: [Cake] Corrupted sit-tunnelled packets when using skb_gso_segment() on an IFB interface?

[Eric Dumazet]

On 09/10/2018 09:04 AM, Toke Høiland-Jørgensen wrote:
> Hi everyone
> 
> While investigating a bug report on CAKE[0], I've run into the following
> behaviour:
> 
> When running CAKE as an ingress shaper on an IFB interface, if the GSO
> splitting feature is turned on, TCP throughput will drop dramatically on
> 6in4 (sit) tunnels running over the interface in question. Looking at a
> traffic dump, I'm seeing ~15% packet loss on the encapsulated TCP
> stream.
> 
> IPv4 traffic is fine on the same interface, as is native IPv6 traffic.
> And turning off GSO splitting in CAKE makes the packet loss go away. The
> issue only seems to appear on IFB interfaces. So I'm wondering if there
> is some interaction that corrupts packets when they are being split in
> this configuration?
> 
> Steps to reproduce (assuming the box you are running on has IP 10.0.0.2
> on eth0, and has a peer at 10.0.0.1 with a suitably configured sit
> tunnel):
> 
> # modprobe ifb
> # ip link set dev ifb0 up
> # tc qdisc add dev eth0 handle ffff: ingress
> # tc filter add dev eth0 parent ffff: protocol all prio 10 matchall action mirred egress redirect dev ifb0
> # tc qdisc replace dev ifb0 root cake
> # ip link add type sit local 10.0.0.2 remote 10.0.0.1
> # ip link set dev sit1 up
> # netperf -H fe80::a00:1%sit1 -t TCP_MAERTS
> 
> Whereas, in the same setup, this will work fine:
> 
> # netperf -H 10.0.0.1 -t TCP_MAERTS
> 
> As will this:
> 
> # tc qdisc replace dev ifb0 root cake no-split-gso
> # netperf -H fe80::a00:1%sit1 -t TCP_MAERTS
> 
> 
> Does anyone have any ideas? :)
> 

My guess is that skb->mac_len is not properly updated in the segments (compared to the original GSO packet)

Re: [Cake] Corrupted sit-tunnelled packets when using skb_gso_segment() on an IFB interface?

[Eric Dumazet]

On 09/10/2018 11:52 AM, Eric Dumazet wrote:
> 
> 
> On 09/10/2018 09:04 AM, Toke Høiland-Jørgensen wrote:
>> Hi everyone
>>
>> While investigating a bug report on CAKE[0], I've run into the following
>> behaviour:
>>
>> When running CAKE as an ingress shaper on an IFB interface, if the GSO
>> splitting feature is turned on, TCP throughput will drop dramatically on
>> 6in4 (sit) tunnels running over the interface in question. Looking at a
>> traffic dump, I'm seeing ~15% packet loss on the encapsulated TCP
>> stream.
>>
>> IPv4 traffic is fine on the same interface, as is native IPv6 traffic.
>> And turning off GSO splitting in CAKE makes the packet loss go away. The
>> issue only seems to appear on IFB interfaces. So I'm wondering if there
>> is some interaction that corrupts packets when they are being split in
>> this configuration?
>>
>> Steps to reproduce (assuming the box you are running on has IP 10.0.0.2
>> on eth0, and has a peer at 10.0.0.1 with a suitably configured sit
>> tunnel):
>>
>> # modprobe ifb
>> # ip link set dev ifb0 up
>> # tc qdisc add dev eth0 handle ffff: ingress
>> # tc filter add dev eth0 parent ffff: protocol all prio 10 matchall action mirred egress redirect dev ifb0
>> # tc qdisc replace dev ifb0 root cake
>> # ip link add type sit local 10.0.0.2 remote 10.0.0.1
>> # ip link set dev sit1 up
>> # netperf -H fe80::a00:1%sit1 -t TCP_MAERTS
>>
>> Whereas, in the same setup, this will work fine:
>>
>> # netperf -H 10.0.0.1 -t TCP_MAERTS
>>
>> As will this:
>>
>> # tc qdisc replace dev ifb0 root cake no-split-gso
>> # netperf -H fe80::a00:1%sit1 -t TCP_MAERTS
>>
>>
>> Does anyone have any ideas? :)
>>
> 
> My guess is that skb->mac_len is not properly updated in the segments (compared to the original GSO packet)

And the skb->mac_len being not properly set is a problem after commit 
f40ae91307c275fc8b17420fa74145e9937c3c0b act_mirred: Fix bogus header when redirecting from VLAN