That makes a lot of sense.

I missed the point the fq separate ICMP and tcp flows.

My current setting is

Ubuntu sender A with codel (htb and fq and fq_codel happens here)
---1G---freebsd ipfw machine inject link delay of 20ms------1G-----Ubuntu
receiver B machine.

What I am trying to do is measure the delay of packets of the same flow
experienced in A under different queuing policies (sfq or fq_codel).

Is there a simple way to do this?

Thanks
-Mo



2013/12/5 Jonathan Morton <chromatix99@gmail.com>

>
> On 5 Dec, 2013, at 9:49 pm, Dong Mo wrote:
>
> > And sending packet from the sender, where qdisc is set, to the receiver,
> the queuing delay is quite small and is approximately the same as when
> fq_codel enabled.
> >
> > However, if I am not using sfq, and send only one flow over this link,
> the delay will explode again.
> >
> > So is sfq here cut off the queue length somehow? How should I make a
> bufferbloat fair queue on linux box using tc qdisc?
>
> How are you measuring the delay?  Ping?
>
> This is relevant because ICMP (ping) occupies a different "flow" than the
> TCP stream(s) you are loading the link with.  SFQ and fq_codel both service
> flows fairly, ensuring that each gets some traffic through continuously.
>  So your pings have low latency even if SFQ's queue is full.
>
> The difference is that fq_codel keeps the queue of each flow short, while
> SFQ allows each queue to grow up to the limit.  The latter behaviour is
> adequate in some situations, but has some bad consequences such as slow
> recovery from packet loss on each flow.  And because SFQ can *only* keep
> the queue within limits by dropping packets, there *will* be packet loss on
> a regular basis unless the receive window of the TCP flow is exhausted
> first.  By contrast fq_codel is ECN-aware, so on ECN-enabled flows it can
> avoid dropping packets altogether while still keeping latency low enough
> for rapid recovery if packet loss occurs anyway.
>
> It also often happens that packets dropped due to a full queue happen to
> be at the end of a TCP connection, for which detection and recovery of the
> problem is much slower (and usually on very human-visible timescales) than
> in the middle of a connection.  In this case SFQ is no help, because it
> also drops from teh tail of the queue, where the last packets of a
> connection arrive.  By contrast fq_codel drops from the head of the queue
> when required (ie. when ECN is not available), so unless two connections
> share a flow (rare but not impossible), the lost packet will always be from
> the middle of a connection and will be retransmitted quickly.
>
> That is the sort of behaviour you should test for when comparing fq_codel
> and SFQ.  A simple ping test under load is satisfied by both qdiscs.
>
>  - Jonathan Morton
>
>