* [Cake] quick patch for split_gso
@ 2018-07-25 4:11 Dave Taht
2018-07-25 4:32 ` Dave Taht
0 siblings, 1 reply; 6+ messages in thread
From: Dave Taht @ 2018-07-25 4:11 UTC (permalink / raw)
To: Cake List
[-- Attachment #1: Type: text/plain, Size: 301 bytes --]
I was happy to see the API didn't need to get modified.
Not even compile tested patch attached.
also, in looking at the gso code, when enabled, is there any way to pull the
actual packets out in the stats?
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
[-- Attachment #2: cake_split_gso.patch --]
[-- Type: text/x-patch, Size: 2257 bytes --]
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index 539c949..488cfc6 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -80,7 +80,6 @@
#define CAKE_QUEUES (1024)
#define CAKE_FLOW_MASK 63
#define CAKE_FLOW_NAT_FLAG 64
-#define CAKE_SPLIT_GSO_THRESHOLD (125000000) /* 1Gbps */
/* struct cobalt_params - contains codel and blue parameters
* @interval: codel initial drop rate
@@ -164,6 +163,7 @@ struct cake_tin_data {
u16 unresponsive_flow_count;
u32 max_skblen;
+ u32 split_gso;
struct list_head new_flows;
struct list_head old_flows;
@@ -260,7 +260,6 @@ enum {
CAKE_FLAG_AUTORATE_INGRESS = BIT(1),
CAKE_FLAG_INGRESS = BIT(2),
CAKE_FLAG_WASH = BIT(3),
- CAKE_FLAG_SPLIT_GSO = BIT(4)
};
/* COBALT operates the Codel and BLUE algorithms in parallel, in order to
@@ -1651,7 +1650,7 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
if (unlikely(len > b->max_skblen))
b->max_skblen = len;
- if (skb_is_gso(skb) && q->rate_flags & CAKE_FLAG_SPLIT_GSO) {
+ if (skb_is_gso(skb) && q->split_gso < len ) {
struct sk_buff *segs, *nskb;
netdev_features_t features = netif_skb_features(skb);
unsigned int slen = 0;
@@ -2569,10 +2568,8 @@ static int cake_change(struct Qdisc *sch, struct nlattr *opt,
if (tb[TCA_CAKE_MEMORY])
q->buffer_config_limit = nla_get_u32(tb[TCA_CAKE_MEMORY]);
- if (q->rate_bps && q->rate_bps <= CAKE_SPLIT_GSO_THRESHOLD)
- q->rate_flags |= CAKE_FLAG_SPLIT_GSO;
- else
- q->rate_flags &= ~CAKE_FLAG_SPLIT_GSO;
+ if (tb[TCA_CAKE_SPLIT_GSO])
+ q->split_gso = nla_get_u32(tb[TCA_CAKE_SPLIT_GSO]);
if (q->tins) {
sch_tree_lock(sch);
@@ -2611,7 +2608,7 @@ static int cake_init(struct Qdisc *sch, struct nlattr *opt,
q->cur_tin = 0;
q->cur_flow = 0;
-
+ q->split_gso = ~0; /* always split gso packets */
qdisc_watchdog_init(&q->watchdog, sch);
if (opt) {
@@ -2729,8 +2726,7 @@ static int cake_dump(struct Qdisc *sch, struct sk_buff *skb)
if (nla_put_u32(skb, TCA_CAKE_MPU, q->rate_mpu))
goto nla_put_failure;
- if (nla_put_u32(skb, TCA_CAKE_SPLIT_GSO,
- !!(q->rate_flags & CAKE_FLAG_SPLIT_GSO)))
+ if (nla_put_u32(skb, TCA_CAKE_SPLIT_GSO, q->split_gso))
goto nla_put_failure;
return nla_nest_end(skb, opts);
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Cake] quick patch for split_gso
2018-07-25 4:11 [Cake] quick patch for split_gso Dave Taht
@ 2018-07-25 4:32 ` Dave Taht
2018-07-25 4:35 ` Dave Taht
0 siblings, 1 reply; 6+ messages in thread
From: Dave Taht @ 2018-07-25 4:32 UTC (permalink / raw)
To: Cake List
and... whew - even at a gbit, this makes bql's estimator get a smaller value.
root@spaceheater:/sys/class/net/enp6s0/queues/tx-0/byte_queue_limits# cat limit
130554
tc qdisc replace dev enp6s0 root cake bandwidth 1gbit
run netperf a few times
On Tue, Jul 24, 2018 at 9:11 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> I was happy to see the API didn't need to get modified.
>
> Not even compile tested patch attached.
>
> also, in looking at the gso code, when enabled, is there any way to pull the
> actual packets out in the stats?
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Cake] quick patch for split_gso
2018-07-25 4:32 ` Dave Taht
@ 2018-07-25 4:35 ` Dave Taht
2018-07-25 5:01 ` Dave Taht
0 siblings, 1 reply; 6+ messages in thread
From: Dave Taht @ 2018-07-25 4:35 UTC (permalink / raw)
To: Cake List
On Tue, Jul 24, 2018 at 9:32 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> and... whew - even at a gbit, this makes bql's estimator get a smaller value.
>
> root@spaceheater:/sys/class/net/enp6s0/queues/tx-0/byte_queue_limits# cat limit
> 130554
> tc qdisc replace dev enp6s0 root cake bandwidth 1gbit
> run netperf a few times
root@spaceheater:/sys/class/net/enp6s0/queues/tx-0/byte_queue_limits# cat limit
42392
I was puzzled there for a while as to why line rate interpacket
latency was so much worse in the "final" version that went upstream. I
think BQL was even lower when I was testing on a different box than
spaceheater
>
> On Tue, Jul 24, 2018 at 9:11 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > I was happy to see the API didn't need to get modified.
> >
> > Not even compile tested patch attached.
> >
> > also, in looking at the gso code, when enabled, is there any way to pull the
> > actual packets out in the stats?
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Cake] quick patch for split_gso
2018-07-25 4:35 ` Dave Taht
@ 2018-07-25 5:01 ` Dave Taht
2018-07-25 21:03 ` Dave Taht
0 siblings, 1 reply; 6+ messages in thread
From: Dave Taht @ 2018-07-25 5:01 UTC (permalink / raw)
To: Cake List
[-- Attachment #1: Type: text/plain, Size: 187 bytes --]
Ah... 1gbit, BQL at a sane size, 750 usec RTT. I guess I have to go
compile this patch now.....
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
[-- Attachment #2: bql_compared.png --]
[-- Type: image/png, Size: 218464 bytes --]
[-- Attachment #3: thatquestforinterpacketlowlatency.png --]
[-- Type: image/png, Size: 62169 bytes --]
[-- Attachment #4: tcp_nup-2018-07-24T215059.956999.cake-1gbitshaped-bql_128000.flent.gz --]
[-- Type: application/gzip, Size: 35692 bytes --]
[-- Attachment #5: tcp_nup-2018-07-24T215326.386191.cake-1gbitnoshaped-bql_128000.flent.gz --]
[-- Type: application/gzip, Size: 35430 bytes --]
[-- Attachment #6: tcp_nup-2018-07-24T214520.556370.cake-1gbitshaped-bql_43000.flent.gz --]
[-- Type: application/gzip, Size: 35893 bytes --]
[-- Attachment #7: tcp_nup-2018-07-24T214717.134554.cake-1gbitshaped-bql_16000.flent.gz --]
[-- Type: application/gzip, Size: 37052 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Cake] quick patch for split_gso
2018-07-25 5:01 ` Dave Taht
@ 2018-07-25 21:03 ` Dave Taht
2018-07-25 21:21 ` Dave Taht
0 siblings, 1 reply; 6+ messages in thread
From: Dave Taht @ 2018-07-25 21:03 UTC (permalink / raw)
To: Cake List
[-- Attachment #1: Type: text/plain, Size: 2721 bytes --]
so, I went and tested that patch. Turned out len was signed... I'd put
it in the wrong place... and the very act of doing that test at all
permutes the result.
cake unshaped at 1gig, always splitting, ends up with a bql value of
about 42,000, full throughput, e.g.
if (skb_is_gso(skb)) {
...
avg median # data pts
Ping (ms) ICMP : 0.80 0.77 ms 341
TCP upload avg : 235.40 235.39 Mbits/s 301
TCP upload sum : 941.61 941.55 Mbits/s 301
TCP upload::1 : 235.36 235.39 Mbits/s 290
TCP upload::2 : 235.46 235.39 Mbits/s 291
TCP upload::3 : 235.38 235.45 Mbits/s 275
TCP upload::4 : 235.41 235.44 Mbits/s 278
conditionally splitting, either never or always, gives me bql of: 140852
if (skb_is_gso(skb) && len >= q->split_gso) {
avg median # data pts
Ping (ms) ICMP : 1.64 1.71 ms 347
TCP upload avg : 235.40 235.38 Mbits/s 301
TCP upload sum : 941.59 941.52 Mbits/s 301
TCP upload::1 : 235.35 235.44 Mbits/s 280
TCP upload::2 : 235.40 235.39 Mbits/s 276
TCP upload::3 : 235.47 235.43 Mbits/s 276
TCP upload::4 : 235.37 235.45 Mbits/s 282
the cache line miss can't possibly have cost that much. (?) Guess I
have to go check the range of len.
I WANT MY 800usec back!
PS For giggles, I accidentally ended up using pfifo_fast for one run.
I thought I'd broke something.
Summary of tcp_nup test run 'split_gso_test_again_len_512000_bql_auto'
(at 2018-07-25 20:51:10.533339):
avg median # data pts
Ping (ms) ICMP : 3.32 3.59 ms 347
TCP upload avg : 235.38 235.38 Mbits/s 301
TCP upload sum : 941.52 941.54 Mbits/s 301
TCP upload::1 : 152.48 150.93 Mbits/s 281
TCP upload::2 : 152.54 150.35 Mbits/s 286
TCP upload::3 : 483.96 491.12 Mbits/s 284
TCP upload::4 : 152.54 150.49 Mbits/s 284
On Tue, Jul 24, 2018 at 10:01 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> Ah... 1gbit, BQL at a sane size, 750 usec RTT. I guess I have to go
> compile this patch now.....
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
[-- Attachment #2: cake_split_gso.patch --]
[-- Type: application/x-patch, Size: 2257 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Cake] quick patch for split_gso
2018-07-25 21:03 ` Dave Taht
@ 2018-07-25 21:21 ` Dave Taht
0 siblings, 0 replies; 6+ messages in thread
From: Dave Taht @ 2018-07-25 21:21 UTC (permalink / raw)
To: Cake List
remarkable how sensitive this is. I went back to using the rate flags
(thus making this a binary option), consistently get bql of 45000 now.
I want me 50usec back!
Summary of tcp_nup test run 'split_gso_test_again_rate_flag_bql_auto'
(at 2018-07-25 21:13:13.816743):
avg median # data pts
Ping (ms) ICMP : 0.85 0.82 ms 343
TCP upload avg : 235.39 235.39 Mbits/s 301
TCP upload sum : 941.57 941.57 Mbits/s 301
TCP upload::1 : 235.32 235.45 Mbits/s 276
TCP upload::2 : 235.41 235.42 Mbits/s 282
TCP upload::3 : 235.42 235.42 Mbits/s 276
TCP upload::4 : 235.42 235.43 Mbits/s 281
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index 539c949..f77c61f 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -2572,7 +2572,7 @@ static int cake_change(struct Qdisc *sch, struct
nlattr *opt,
if (q->rate_bps && q->rate_bps <= CAKE_SPLIT_GSO_THRESHOLD)
q->rate_flags |= CAKE_FLAG_SPLIT_GSO;
else
- q->rate_flags &= ~CAKE_FLAG_SPLIT_GSO;
+ q->rate_flags |= CAKE_FLAG_SPLIT_GSO;
if (q->tins) {
sch_tree_lock(sch);
On Wed, Jul 25, 2018 at 2:03 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> so, I went and tested that patch. Turned out len was signed... I'd put
> it in the wrong place... and the very act of doing that test at all
> permutes the result.
>
> cake unshaped at 1gig, always splitting, ends up with a bql value of
> about 42,000, full throughput, e.g.
>
> if (skb_is_gso(skb)) {
>
> ...
>
> avg median # data pts
> Ping (ms) ICMP : 0.80 0.77 ms 341
> TCP upload avg : 235.40 235.39 Mbits/s 301
> TCP upload sum : 941.61 941.55 Mbits/s 301
> TCP upload::1 : 235.36 235.39 Mbits/s 290
> TCP upload::2 : 235.46 235.39 Mbits/s 291
> TCP upload::3 : 235.38 235.45 Mbits/s 275
> TCP upload::4 : 235.41 235.44 Mbits/s 278
>
> conditionally splitting, either never or always, gives me bql of: 140852
>
> if (skb_is_gso(skb) && len >= q->split_gso) {
>
> avg median # data pts
> Ping (ms) ICMP : 1.64 1.71 ms 347
> TCP upload avg : 235.40 235.38 Mbits/s 301
> TCP upload sum : 941.59 941.52 Mbits/s 301
> TCP upload::1 : 235.35 235.44 Mbits/s 280
> TCP upload::2 : 235.40 235.39 Mbits/s 276
> TCP upload::3 : 235.47 235.43 Mbits/s 276
> TCP upload::4 : 235.37 235.45 Mbits/s 282
>
> the cache line miss can't possibly have cost that much. (?) Guess I
> have to go check the range of len.
>
> I WANT MY 800usec back!
>
> PS For giggles, I accidentally ended up using pfifo_fast for one run.
> I thought I'd broke something.
>
> Summary of tcp_nup test run 'split_gso_test_again_len_512000_bql_auto'
> (at 2018-07-25 20:51:10.533339):
>
> avg median # data pts
> Ping (ms) ICMP : 3.32 3.59 ms 347
> TCP upload avg : 235.38 235.38 Mbits/s 301
> TCP upload sum : 941.52 941.54 Mbits/s 301
> TCP upload::1 : 152.48 150.93 Mbits/s 281
> TCP upload::2 : 152.54 150.35 Mbits/s 286
> TCP upload::3 : 483.96 491.12 Mbits/s 284
> TCP upload::4 : 152.54 150.49 Mbits/s 284
> On Tue, Jul 24, 2018 at 10:01 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > Ah... 1gbit, BQL at a sane size, 750 usec RTT. I guess I have to go
> > compile this patch now.....
> >
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-07-25 21:21 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-25 4:11 [Cake] quick patch for split_gso Dave Taht
2018-07-25 4:32 ` Dave Taht
2018-07-25 4:35 ` Dave Taht
2018-07-25 5:01 ` Dave Taht
2018-07-25 21:03 ` Dave Taht
2018-07-25 21:21 ` Dave Taht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox