[Cake] Playing with ingredients = ruined the CAKE

[Cake] Playing with ingredients = ruined the CAKE

[Kevin Darbyshire-Bryant]

I’m trying to create a ‘diffserv5’ for the purposes of implementing a 'Least Effort’ class: something like LE=Bittorrent, BK=Backups/long term down/uploads, BE=Best Effort/Normal, VI=Streaming media/facetime/zoom, VO=VOIP/SIP.  Not too hard you’d think, take diffserv4 and add a tin.

I did this with tin allocation: 0=LE, 1=BE, 2=BK, 3=VI, 4=VO.  BW allocation relative to base rate = LE>>8, BE>>0, BK>>4, VI>>1, VO>>2.  Tin display order = 0, 2, 1, 3, 4.  In theory I don’t mind LE being starved hence the above order.  This pretty much ‘jammed' the shaper as soon as any traffic went into LE with other higher priority tins seeing huge latencies, lots of drops and general bad news all over.

I tried again with a slightly different tin allocation: 0=BE, 1=LE, 2=BK, 3=VI, 4=VO more in keeping with the existing arrangement and display order 1, 2, 0, 3, 4. The shaper doesn’t appear to obviously wedge, though I have seen some latency spikes that I don’t normally see, so it feels like there’s still a corner case being hit.

Does anyone have any ideas?

Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A

Re: [Cake] Playing with ingredients = ruined the CAKE

[Kevin Darbyshire-Bryant]

> On 29 May 2020, at 11:06, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> I’m trying to create a ‘diffserv5’ for the purposes of implementing a 'Least Effort’ class: something like LE=Bittorrent, BK=Backups/long term down/uploads, BE=Best Effort/Normal, VI=Streaming media/facetime/zoom, VO=VOIP/SIP.  Not too hard you’d think, take diffserv4 and add a tin.
> 
> I did this with tin allocation: 0=LE, 1=BE, 2=BK, 3=VI, 4=VO.  BW allocation relative to base rate = LE>>8, BE>>0, BK>>4, VI>>1, VO>>2.  Tin display order = 0, 2, 1, 3, 4.  In theory I don’t mind LE being starved hence the above order.  This pretty much ‘jammed' the shaper as soon as any traffic went into LE with other higher priority tins seeing huge latencies, lots of drops and general bad news all over.
> 
> I tried again with a slightly different tin allocation: 0=BE, 1=LE, 2=BK, 3=VI, 4=VO more in keeping with the existing arrangement and display order 1, 2, 0, 3, 4. The shaper doesn’t appear to obviously wedge, though I have seen some latency spikes that I don’t normally see, so it feels like there’s still a corner case being hit.
> 
> Does anyone have any ideas?

Pondering out loud:  Is setting different (i.e. increased) codel intervals & targets sensible for ‘artificially’ reduced bandwidths, especially in ingress mode?  If I have a 100mbit link and I wish to have a minimum reservation for a low bandwidth, low priority tin e.g. 1mbit. Does it make sense to make that tin respond slower as if it were a 1mbit link whereas it’s a minimally reserved portion of a 100mbit link and it could burst up 100 times quicker than I think?  Egress I suspect is less of a problem in that we’ll queue the packets and eventually throw them in the floor, but ingress???

Kevin

Re: [Cake] Playing with ingredients = ruined the CAKE

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 129 bytes --]

This is currently what I’m playing with:


Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A

[-- Attachment #2: 0001-experiment-with-diffserv5-incl-an-LE-class.patch --]
[-- Type: application/octet-stream, Size: 5325 bytes --]

From fe1d1fb237aaa8d5728a81707d1c2af6e89aeb23 Mon Sep 17 00:00:00 2001
From: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
Date: Wed, 27 May 2020 17:05:51 +0100
Subject: [PATCH] experiment with diffserv5 incl an LE class

Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
---
 pkt_sched.h |  3 +-
 sch_cake.c  | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+), 1 deletion(-)

diff --git a/pkt_sched.h b/pkt_sched.h
index 745cbc7..6d6106a 100644
--- a/pkt_sched.h
+++ b/pkt_sched.h
@@ -947,7 +947,7 @@ enum {
 	CAKE_FLOW_DUAL_SRC, /* = CAKE_FLOW_SRC_IP | CAKE_FLOW_FLOWS */
 	CAKE_FLOW_DUAL_DST, /* = CAKE_FLOW_DST_IP | CAKE_FLOW_FLOWS */
 	CAKE_FLOW_TRIPLE,   /* = CAKE_FLOW_HOSTS  | CAKE_FLOW_FLOWS */
-	CAKE_FLOW_MAX,
+	CAKE_FLOW_MAX
 };
 
 enum {
@@ -956,6 +956,7 @@ enum {
 	CAKE_DIFFSERV_DIFFSERV8,
 	CAKE_DIFFSERV_BESTEFFORT,
 	CAKE_DIFFSERV_PRECEDENCE,
+	CAKE_DIFFSERV_DIFFSERV5,
 	CAKE_DIFFSERV_MAX
 };
 
diff --git a/sch_cake.c b/sch_cake.c
index cb9bbf7..524c5a6 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -333,6 +333,17 @@ static const u8 diffserv8[] = {
 	7, 2, 2, 2, 2, 2, 2, 2,
 };
 
+static const u8 diffserv5[] = {
+	0, 1, 0, 0, 3, 0, 0, 0,
+	2, 0, 0, 0, 0, 0, 0, 0,
+	3, 0, 3, 0, 3, 0, 3, 0,
+	3, 0, 3, 0, 3, 0, 3, 0,
+	4, 0, 3, 0, 3, 0, 3, 0,
+	4, 0, 0, 0, 4, 0, 4, 0,
+	4, 0, 0, 0, 0, 0, 0, 0,
+	4, 0, 0, 0, 0, 0, 0, 0,
+};
+
 static const u8 diffserv4[] = {
 	0, 1, 0, 0, 2, 0, 0, 0,
 	1, 0, 0, 0, 0, 0, 0, 0,
@@ -370,6 +381,7 @@ static const u8 besteffort[] = {
 
 static const u8 normal_order[] = {0, 1, 2, 3, 4, 5, 6, 7};
 static const u8 bulk_order[] = {1, 0, 2, 3};
+static const u8 le_order[] = {1, 2, 0, 3, 4};
 
 #define REC_INV_SQRT_CACHE (16)
 static u32 cobalt_rec_inv_sqrt_cache[REC_INV_SQRT_CACHE] = {0};
@@ -2350,6 +2362,17 @@ static void cake_set_rate(struct cake_tin_data *b, u64 rate, u32 mtu,
 	b->cparams.p_dec = 1 << 20; /* 1/4096 */
 }
 
+static void cake_config_ingress(struct cake_sched_data *q)
+{
+	u32 i;
+
+	for (i = 1; i < q->tin_cnt ; i++) {
+		q->tins[i].cparams.target = q->tins[0].cparams.target;
+		q->tins[i].cparams.interval = q->tins[0].cparams.interval;
+		q->tins[i].flow_quantum = q->tins[0].flow_quantum;
+	}
+}
+
 static int cake_config_besteffort(struct Qdisc *sch)
 {
 	struct cake_sched_data *q = qdisc_priv(sch);
@@ -2397,6 +2420,8 @@ static int cake_config_precedence(struct Qdisc *sch)
 		quantum  *= 7;
 		quantum >>= 3;
 	}
+/*	if (q->rate_flags & CAKE_FLAG_INGRESS)*/
+		cake_config_ingress(q);
 
 	return 0;
 }
@@ -2489,6 +2514,58 @@ static int cake_config_diffserv8(struct Qdisc *sch)
 		quantum  *= 7;
 		quantum >>= 3;
 	}
+/*	if (q->rate_flags & CAKE_FLAG_INGRESS)*/
+		cake_config_ingress(q);
+
+	return 0;
+}
+
+static int cake_config_diffserv5(struct Qdisc *sch)
+{
+/*  Further pruned list of traffic classes for four-class system:
+ *
+ *	    Latency Sensitive  (CS7, CS6, EF, VA, CS5, CS4)
+ *	    Streaming Media    (AF4x, AF3x, CS3, AF2x, TOS4, CS2, TOS1)
+ *	    Background Traffic (CS1)
+ *	    Best Effort        (CS0, AF1x, TOS2, and those not specified)
+ *	    Least Effort       (LE)
+ *
+ *		Total 5 traffic classes.
+ */
+
+	struct cake_sched_data *q = qdisc_priv(sch);
+	u32 mtu = psched_mtu(qdisc_dev(sch));
+	u64 rate = q->rate_bps;
+	u32 quantum = 1024;
+	u32 i;
+
+	q->tin_cnt = 5;
+
+	/* codepoint to class mapping */
+	q->tin_index = diffserv5;
+	q->tin_order = le_order;
+
+	/* class characteristics */
+	cake_set_rate(&q->tins[0], rate, mtu,
+		      us_to_ns(q->target), us_to_ns(q->interval));
+	cake_set_rate(&q->tins[1], rate >> 8, mtu,
+		      us_to_ns(q->target), us_to_ns(q->interval));
+	cake_set_rate(&q->tins[2], rate >> 4, mtu,
+		      us_to_ns(q->target), us_to_ns(q->interval));
+	cake_set_rate(&q->tins[3], rate >> 1, mtu,
+		      us_to_ns(q->target), us_to_ns(q->interval));
+	cake_set_rate(&q->tins[4], rate >> 2, mtu,
+		      us_to_ns(q->target), us_to_ns(q->interval));
+
+	/* bandwidth-sharing weights */
+	q->tins[0].tin_quantum = quantum;	/*BE*/
+	q->tins[1].tin_quantum = quantum >> 8;	/*LE*/
+	q->tins[2].tin_quantum = quantum >> 4;	/*BK*/
+	q->tins[3].tin_quantum = quantum >> 1;	/*VI*/
+	q->tins[4].tin_quantum = quantum >> 2;	/*VO*/
+
+/*	if (q->rate_flags & CAKE_FLAG_INGRESS)*/
+		cake_config_ingress(q);
 
 	return 0;
 }
@@ -2509,6 +2586,7 @@ static int cake_config_diffserv4(struct Qdisc *sch)
 	u32 mtu = psched_mtu(qdisc_dev(sch));
 	u64 rate = q->rate_bps;
 	u32 quantum = 1024;
+	u32 i;
 
 	q->tin_cnt = 4;
 
@@ -2532,6 +2610,9 @@ static int cake_config_diffserv4(struct Qdisc *sch)
 	q->tins[2].tin_quantum = quantum >> 1;
 	q->tins[3].tin_quantum = quantum >> 2;
 
+/*	if (q->rate_flags & CAKE_FLAG_INGRESS)*/
+		cake_config_ingress(q);
+
 	return 0;
 }
 
@@ -2566,6 +2647,9 @@ static int cake_config_diffserv3(struct Qdisc *sch)
 	q->tins[1].tin_quantum = quantum >> 4;
 	q->tins[2].tin_quantum = quantum >> 2;
 
+/*	if (q->rate_flags & CAKE_FLAG_INGRESS)*/
+		cake_config_ingress(q);
+
 	return 0;
 }
 
@@ -2587,6 +2671,10 @@ static void cake_reconfigure(struct Qdisc *sch)
 		ft = cake_config_diffserv8(sch);
 		break;
 
+	case CAKE_DIFFSERV_DIFFSERV5:
+		ft = cake_config_diffserv5(sch);
+		break;
+
 	case CAKE_DIFFSERV_DIFFSERV4:
 		ft = cake_config_diffserv4(sch);
 		break;
-- 
2.24.3 (Apple Git-128)

Re: [Cake] Playing with ingredients = ruined the CAKE

[John Yates]

Kevin,

I am curious how this effort relates to Dave Taht's point in his May
20th "not really huge on EF landing where it does in wifi" thread.

/john

Re: [Cake] Playing with ingredients = ruined the CAKE

[Kevin Darbyshire-Bryant]

> On 31 May 2020, at 17:38, John Yates <john@yates-sheets.org> wrote:
> 
> Kevin,
> 
> I am curious how this effort relates to Dave Taht's point in his May
> 20th "not really huge on EF landing where it does in wifi" thread.
> 
> /john

Hi John,

I have absolutely no idea, don’t appear to have that thread :-) My own DSCP/CAKE interests are aligned to exercising CAKE’s built-in classification/bandwidth allocations across my WAN link.  In essence I have traffic types of ‘Least Effort’ (bittorrent - nearly starvable), ‘Bulk’ (backups/long term up/downloads - low minimum b/w), ’Normal’ (Most short term stuff - all bandwith), VI (video conference calls/streaming, long-term more important flows latency important- up to 1/2 b/w), VO (voice/latency critical up to 1/4 b/w).  Or two levels of ’not so important’ and two levels of ‘more important’ around normal/everything.

The classification process happens as a combination of iptables/ipsets rules on the internet router using tc act_ctinfo to preserve the DSCP classification of flows across the WAN.

Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A

Re: [Cake] Playing with ingredients = ruined the CAKE

[John Yates]

On Sun, May 31, 2020 at 1:08 PM Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
> I have absolutely no idea, don’t appear to have that thread :-)

Mea culpa.  Should have included this link to the thread:

https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-May/002860.html

/john

Re: [Cake] Playing with ingredients = ruined the CAKE

[Kevin Darbyshire-Bryant]

> On 31 May 2020, at 18:26, John Yates <john@yates-sheets.org> wrote:
> 
> On Sun, May 31, 2020 at 1:08 PM Kevin Darbyshire-Bryant
> <kevin@darbyshire-bryant.me.uk> wrote:
>> I have absolutely no idea, don’t appear to have that thread :-)
> 
> Mea culpa.  Should have included this link to the thread:
> 
> https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-May/002860.html
> 
> /john

Ah, well after the initial excitement that ‘oh an application actually sets DSCP’ I checked what marking my zoom packets had on the next conference…to find… Best effort.  Crushing disappointment led to this in my firewall box:

#Zoom - connections go to Zoom with dest ports 8801-8810
$IPTABLES -t mangle -A QOS_MARK_F_${IFACE} -p udp -m udp -m set --match-set Zoom4 dst -m multiport --dports 8801:8810 -j DSCP --set-dscp-class CS3 -m comment --comment "Zoom CS3 VI"
$IP6TABLES -t mangle -A QOS_MARK_F_${IFACE} -p udp -m udp -m set --match-set Zoom6 dst -m multiport --dports 8801:8810 -j DSCP --set-dscp-class CS3 -m comment --comment "Zoom CS3 VI”

With dnsmasq configured to fill the Zoom4/6 ipsets with relevant IP addresses

ipset=/zoom.us/Zoom4,Zoom6 

Works a treat.

Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A

Re: [Cake] Playing with ingredients = ruined the CAKE

[Dave Taht]

On Sun, May 31, 2020 at 11:08 AM Kevin Darbyshire-Bryant
<kevin@darbyshire-bryant.me.uk> wrote:
>
>
>
> > On 31 May 2020, at 18:26, John Yates <john@yates-sheets.org> wrote:
> >
> > On Sun, May 31, 2020 at 1:08 PM Kevin Darbyshire-Bryant
> > <kevin@darbyshire-bryant.me.uk> wrote:
> >> I have absolutely no idea, don’t appear to have that thread :-)
> >
> > Mea culpa.  Should have included this link to the thread:
> >
> > https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-May/002860.html
> >
> > /john
>
> Ah, well after the initial excitement that ‘oh an application actually sets DSCP’ I checked what marking my zoom packets had on the next conference…to find… Best effort.  Crushing disappointment led to this in my firewall box:
>
> #Zoom - connections go to Zoom with dest ports 8801-8810
> $IPTABLES -t mangle -A QOS_MARK_F_${IFACE} -p udp -m udp -m set --match-set Zoom4 dst -m multiport --dports 8801:8810 -j DSCP --set-dscp-class CS3 -m comment --comment "Zoom CS3 VI"
> $IP6TABLES -t mangle -A QOS_MARK_F_${IFACE} -p udp -m udp -m set --match-set Zoom6 dst -m multiport --dports 8801:8810 -j DSCP --set-dscp-class CS3 -m comment --comment "Zoom CS3 VI”
>
> With dnsmasq configured to fill the Zoom4/6 ipsets with relevant IP addresses
>
> ipset=/zoom.us/Zoom4,Zoom6
>
> Works a treat.

groovy. nicer than what I did, except that I don't remember where CS3
lands in wifi anymore! CS4 and CS5 land in the VI queue....

As for the "EF" (and for that matter, CS6 and CS7) issue on wifi, it
lands in the VO queue. babel and BGP sets CS6 last I looked, and the
VO queue on 802.11n (which is still quite common on both clients and
APs) cannot aggregate. Given the rise of videoconferencing, mapping
stuff into the VI queue makes the most sense for all forms of wifi,
for both voice and video traffic. I like to think we've conclusively
proven that
packet aggregation is way more efficient in terms of airtime and media
aquisition for both 802.11n and 802.11ac at this point.

Worse than that, EF once meant "expedited forwarding", an early
attempt to create a paid-for "fast lane" on the internet. I'd not use
that
for anything nowadays.

So I could see EF landing in VI, and CS6, at least in the babel case,
being a good candidate for VI also, but the existing usage of CS6 for
BGP (tcp transfers) is a lousy place to put stuff into the VI queue, also.

And all in all, our existing fq_codel for wifi code does not work well
when we saturate all four wifi queues in the first place, and in
general
I think it's better that APs just use the BE queue at all times with
our existing codebase for that. Someday, perhaps, the scheduler
will only feed 1-2 hw queues at a time....

On the other hand, other APs, with massively overbuffered BE queues,
will probably do better with videoconferencing-style traffic landing
in VI,
so long as it's access controlled to a reasonable extent.

Clients SHOULD put videoconferencing (and gaming and latency sensitive
traffic, like interactive ssh and mosh) into the VI queue
to more minimize media acquization delays.

On the gripping hand, the best thing anyone can do for wifi is for all
devices to be located as close to the AP as possible.

>
> Cheers,
>
> Kevin D-B
>
> gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cake] Playing with ingredients = ruined the CAKE

[Sebastian Moeller]

Hi Dave,


> On May 31, 2020, at 21:01, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Sun, May 31, 2020 at 11:08 AM Kevin Darbyshire-Bryant
> <kevin@darbyshire-bryant.me.uk> wrote:
>> 
>> 
>> 
>>> On 31 May 2020, at 18:26, John Yates <john@yates-sheets.org> wrote:
>>> 
>>> On Sun, May 31, 2020 at 1:08 PM Kevin Darbyshire-Bryant
>>> <kevin@darbyshire-bryant.me.uk> wrote:
>>>> I have absolutely no idea, don’t appear to have that thread :-)
>>> 
>>> Mea culpa.  Should have included this link to the thread:
>>> 
>>> https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-May/002860.html
>>> 
>>> /john
>> 
>> Ah, well after the initial excitement that ‘oh an application actually sets DSCP’ I checked what marking my zoom packets had on the next conference…to find… Best effort.  Crushing disappointment led to this in my firewall box:
>> 
>> #Zoom - connections go to Zoom with dest ports 8801-8810
>> $IPTABLES -t mangle -A QOS_MARK_F_${IFACE} -p udp -m udp -m set --match-set Zoom4 dst -m multiport --dports 8801:8810 -j DSCP --set-dscp-class CS3 -m comment --comment "Zoom CS3 VI"
>> $IP6TABLES -t mangle -A QOS_MARK_F_${IFACE} -p udp -m udp -m set --match-set Zoom6 dst -m multiport --dports 8801:8810 -j DSCP --set-dscp-class CS3 -m comment --comment "Zoom CS3 VI”
>> 
>> With dnsmasq configured to fill the Zoom4/6 ipsets with relevant IP addresses
>> 
>> ipset=/zoom.us/Zoom4,Zoom6
>> 
>> Works a treat.
> 
> groovy. nicer than what I did, except that I don't remember where CS3
> lands in wifi anymore! CS4 and CS5 land in the VI queue....
> 
> As for the "EF" (and for that matter, CS6 and CS7) issue on wifi, it
> lands in the VO queue.

	According to https://tools.ietf.org/html/rfc8325#page-10 it actually often lands in AC_VI:
"Voice (EF-101110) will be mapped to UP 5 (101), and treated in the
      Video Access Category (AC_VI) rather than the Voice Access
      Category (AC_VO), for which it is intended"

Which IMHO is the right thing, AC_VO is so ruthless in acquiring airtime when competing with the other AC's that IMHO it should only be kept as the "nuclear option" in case an AP does not get sufficient airtime when competing with AC_VO hogging stations, but not as part as a normal scheme, UNLESS there are only managed APs in the environment, and any airtime hogging is not at the expense of the neighbors...



> babel and BGP sets CS6 last I looked, and the
> VO queue on 802.11n (which is still quite common on both clients and
> APs) cannot aggregate. Given the rise of videoconferencing, mapping
> stuff into the VI queue makes the most sense for all forms of wifi,
> for both voice and video traffic. I like to think we've conclusively
> proven that
> packet aggregation is way more efficient in terms of airtime and media
> aquisition for both 802.11n and 802.11ac at this point.

	+1; not that I can back this up with data...

> 
> Worse than that, EF once meant "expedited forwarding", an early
> attempt to create a paid-for "fast lane" on the internet. I'd not use
> that
> for anything nowadays.
> 
> So I could see EF landing in VI, and CS6, at least in the babel case,
> being a good candidate for VI also, but the existing usage of CS6 for
> BGP (tcp transfers) is a lousy place to put stuff into the VI queue, also.
> 
> And all in all, our existing fq_codel for wifi code does not work well
> when we saturate all four wifi queues in the first place, and in
> general
> I think it's better that APs just use the BE queue at all times with
> our existing codebase for that. Someday, perhaps, the scheduler
> will only feed 1-2 hw queues at a time....

	+1, I would assume the issue is that queued packets in AC_VO will basically stall all other quees until the VO queue clears. I have a macbook, which in rrul_cs8 tests over wifi basically hogs all airtime for its two AC_VO flows, essentially starving all other flows in both directions to a trickle. Hence my assesssment that AC_VO is a tad anti-social.

> 
> On the other hand, other APs, with massively overbuffered BE queues,
> will probably do better with videoconferencing-style traffic landing
> in VI,
> so long as it's access controlled to a reasonable extent.
> 
> Clients SHOULD put videoconferencing (and gaming and latency sensitive
> traffic, like interactive ssh and mosh) into the VI queue
> to more minimize media acquization delays.
> 
> On the gripping hand, the best thing anyone can do for wifi is for all
> devices to be located as close to the AP as possible.

	Puzzled, using zoom, over an ath9k radio with Toke & yours airtime fairness/fq_codel fixes works quite well with competing traffic even with zoom in the default AC_BE. May this is because my nominal 100/40 link, shaped with layer_cake down to 49/36 Mbps does not make the wifi into the real bottleneck in the first place...

Best Regards
	Sebastian


> 
>> 
>> Cheers,
>> 
>> Kevin D-B
>> 
>> gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> -- 
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
> 
> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake