From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wg0-f41.google.com (mail-wg0-f41.google.com [74.125.82.41]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 7B2E3200B33; Thu, 30 Aug 2012 15:59:51 -0700 (PDT) Received: by wgbds1 with SMTP id ds1so434831wgb.4 for ; Thu, 30 Aug 2012 15:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=y0YCgGs1r9g36jZDyI92JrlEe2jNLSj1XChnXZygUMI=; b=zZw8ybMCV/BCW/PW2f/Wk7IBWPZ2etIns+6Ff/sRQ9isB+3a3mhdNrgLRNcoxiMxYM p2JnhPGuZW0vGevlk0kKdm6jPIeYk6h6vQynIfY3wCpm7cnX3LeyA+4Dq//xR2x4aLmp 41cNpPy5p6O+F2oz1hAVP+bSFKrVbLAcqYq3vW6FqZeSWIhEv2enM1tElqJwwaE90Ryl JzBEBP7q6oIxA7VIlIx2H2vJn9oRjUMtYdnnQaILlpQeSai+kwiiX0naR+gwWV58+RpV etSnda/S91bPZt5XIGg+2PqG9FFOue7lt+ft3pnmFPr3Ko4ds0B/TFnGPtbOv8PUk3iU T5Mw== MIME-Version: 1.0 Received: by 10.180.74.33 with SMTP id q1mr248161wiv.4.1346367588851; Thu, 30 Aug 2012 15:59:48 -0700 (PDT) Received: by 10.223.159.134 with HTTP; Thu, 30 Aug 2012 15:59:48 -0700 (PDT) Date: Thu, 30 Aug 2012 15:59:48 -0700 Message-ID: From: Dave Taht To: cerowrt-devel@lists.bufferbloat.net, codel@lists.bufferbloat.net Content-Type: text/plain; charset=ISO-8859-1 Subject: [Codel] better mixing in fq_codel X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2012 22:59:52 -0000 I have finally found the source of the issues I was having with htb + fq_codel at low bandwidths, and it wasn't htb to the extent I thought it was. It was fq_codel's use of byte quantums, which was resulting in head of line blocking for multiple streams. The undesirable behavior (quantum of 1500) A new stream, 15 acks (1500 bytes or so) B new stream, 4 acks (66 bytes each) C new stream, 1 packet, 1500 bytes Each stream would be delivered in an entire quantum's quantity before the next. This is basically something that is nearly unnoticable at higher bandwidths, but down here in the slow moving mud, it's quite significant. At 1 mbit, 1500 bytes is forever... Better mixing behavior results from A 1 packet B 1 packet C 1 packet (if not larger than quantum) A 1 packet "" B 1 packet "" ... While I have a patch for this (all 5 lines of it), which does this sort of mixing, it crashes under load, but I'll get there. Behavior before it crashes is rather nice, where before I was observing something like 36 ms of delay with htb for: small packets, "sparse" or ANT streams under a variety of loads, it drops below 3ms in the general case for those. Both qos-scripts and simple-qos benefit hugely. One of these three sets of changes to fq_codel_dequeue or enqueue is dubious. (well, I have a half dozen other patches and fixups to codel and fq_codel in the queue too, so have to rip out each). But yea! this is the low bandwidth behavior we want! fq_codel_enqueue ... if (list_empty(&flow->flowchain)) { list_add_tail(&flow->flowchain, deprio(skb) ? &q->old_flows : &q->new_flows); // the deprio routine looks at diffserv CS1 and kicks anything marked that way always to the old flows // it would be better if this was policy set in userspace, this is just a hack for now q->new_flow_count++; flow->deficit = min(q->quantum, qdisc_pkt_len(skb)); // Alway deliver 1 packet in a new flow unless it's less than quantum (in which case it too will be kicked to old flows) } ... fq_codel_dequeue() if (!skb) { /* force a pass through old_flows to prevent starvation */ if ((head == &q->new_flows) && !list_empty(&q->old_flows)) list_move_tail(&flow->flowchain, &q->old_flows); else list_del_init(&flow->flowchain); goto begin; } qdisc_bstats_update(sch, skb); flow->deficit -= qdisc_pkt_len(skb); /* do DRR instead */ // if (!list_empty(&flow->flowchain) && !list_empty(head)) // list_move_tail(&flow->flowchain, head); // and various clueless combinations of this go boom. Or I busted something else somewhere.