[Cake] Query on ACK

[Cake] Query on ACK

[Avakash bhat]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

Hi all,

We are trying to implement the ACK filtering module of CAKE in ns-3
(Network Simulator).

We had a question on the working of ack filtering.
If an incoming ack which can replace an eligible ack in the queue is about
to be enqueued, do we replace the ack in the queue with the incoming ack
or do we enqueue the ack to the tail of the queue and remove the eligible
ack from the queue?

We would appreciate any help in resolving this query.

Thanks,
Avakash Bhat

[-- Attachment #2: Type: text/html, Size: 613 bytes --]

Re: [Cake] Query on ACK

[Jonathan Morton]

> On 6 May, 2020, at 9:43 pm, Avakash bhat <avakash261@gmail.com> wrote:
> 
> We are trying to implement the ACK filtering module of CAKE in ns-3 (Network Simulator).

Ah yes.  Sorry I didn't respond to the introduction earlier - we were right in the middle of preparing for an IETF virtual meeting.  The debris is still falling from orbit…

> We had a question on the working of ack filtering. 
> If an incoming ack which can replace an eligible ack in the queue is about to be enqueued, do we replace the ack in the queue with the incoming ack 
> or do we enqueue the ack to the tail of the queue and remove the eligible ack from the queue?

That sounds like an implementation detail.  But what we do in Cake is to simply enqueue all the packets, and deal with everything complicated on dequeue.

At that point, we check whether the two packets at the head of the queue are acks for the same flow, and if so, we further check whether the information in the first packet is redundant given the presence of the second packet.  If there is information in the first packet that is not also provided by the second packet, the first packet is delivered.  Otherwise the first packet is dropped, and the second packet moves to the head of the queue.  This process may repeat several times if there are several consecutive, redundant acks in the queue.

The important part is the set of rules determining whether the ack is redundant.

 - Jonathan Morton

Re: [Cake] Query on ACK

[Toke Høiland-Jørgensen]

Avakash bhat <avakash261@gmail.com> writes:

> Hi all,
>
> We are trying to implement the ACK filtering module of CAKE in ns-3
> (Network Simulator).
>
> We had a question on the working of ack filtering.
> If an incoming ack which can replace an eligible ack in the queue is about
> to be enqueued, do we replace the ack in the queue with the incoming ack
> or do we enqueue the ack to the tail of the queue and remove the eligible
> ack from the queue?

Enqueue the new ACK at the tail, drop the eligible one already in the
queue.

-Toke

Re: [Cake] Query on ACK

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 6 May, 2020, at 9:43 pm, Avakash bhat <avakash261@gmail.com> wrote:
>> 
>> We are trying to implement the ACK filtering module of CAKE in ns-3 (Network Simulator).
>
> Ah yes.  Sorry I didn't respond to the introduction earlier - we were right in the middle of preparing for an IETF virtual meeting.  The debris is still falling from orbit…
>
>> We had a question on the working of ack filtering. 
>> If an incoming ack which can replace an eligible ack in the queue is about to be enqueued, do we replace the ack in the queue with the incoming ack 
>> or do we enqueue the ack to the tail of the queue and remove the eligible ack from the queue?
>
> That sounds like an implementation detail. But what we do in Cake is
> to simply enqueue all the packets, and deal with everything
> complicated on dequeue.

The ACK filter is run on enqueue, actually :)

> At that point, we check whether the two packets at the head of the
> queue are acks for the same flow, and if so, we further check whether
> the information in the first packet is redundant given the presence of
> the second packet. If there is information in the first packet that is
> not also provided by the second packet, the first packet is delivered.
> Otherwise the first packet is dropped, and the second packet moves to
> the head of the queue. This process may repeat several times if there
> are several consecutive, redundant acks in the queue.
>
> The important part is the set of rules determining whether the ack is
> redundant.

Yes, indeed. Please feel free to go through cake_ack_filter() in
sch_cake.c and make sure you get all those edge cases in your
eligibility check...

-Toke

Re: [Cake] Query on ACK

[Avakash bhat]

[-- Attachment #1: Type: text/plain, Size: 303 bytes --]

Thanks for the quick response. I also had a followup question.

If the ack filter adds the new ack to the tail of the queue after removing
an ack from the queue, won't it be starving the ack?
The replaced ack was much ahead in the queue than the ack we replaced at
the tail right?

Thanks,
Avakash Bhat

[-- Attachment #2: Type: text/html, Size: 405 bytes --]

Re: [Cake] Query on ACK

[Jonathan Morton]

> On 7 May, 2020, at 9:44 am, Avakash bhat <avakash261@gmail.com> wrote:
> 
> Thanks for the quick response. I also had a followup question. 
> 
> If the ack filter adds the new ack to the tail of the queue after removing an ack from the queue, won't it be starving the ack? 
> The replaced ack was much ahead in the queue than the ack we replaced at the tail right?

No, if you are doing this on enqueue, then you are comparing the new ack with an ack immediately preceding it in the same queue, which will also be at the tail.  And if you are doing it on dequeue then both packets were enqueued some time ago, and both are already due for delivery very soon.

In general, the second packet is delivered sooner, in place of the first one that was removed.  This means it reduces feedback latency to the (forward path) sender.

 - Jonathan Morton

Re: [Cake] Query on ACK

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

I think that you will remove all redundant Backs in one go considerably advancing the new ACK in the queue. And more importantly, in most relevant modes cake will apply one queue per flow stochastically, so almost all packet's in a reverse ACK flow will be ACK with identical 5-tupel....

On 7 May 2020 08:44:59 CEST, Avakash bhat <avakash261@gmail.com> wrote:
>Thanks for the quick response. I also had a followup question.
>
>If the ack filter adds the new ack to the tail of the queue after
>removing
>an ack from the queue, won't it be starving the ack?
>The replaced ack was much ahead in the queue than the ack we replaced
>at
>the tail right?
>
>Thanks,
>Avakash Bhat

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1074 bytes --]

Re: [Cake] Query on ACK

[Avakash bhat]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

Ok thanks so much for the clarifications.
That cleared it up quite a bit.

Thanks,
Avakash Bhat

On Thu, May 7, 2020 at 12:37 PM Sebastian Moeller <moeller0@gmx.de> wrote:

> I think that you will remove all redundant Backs in one go considerably
> advancing the new ACK in the queue. And more importantly, in most relevant
> modes cake will apply one queue per flow stochastically, so almost all
> packet's in a reverse ACK flow will be ACK with identical 5-tupel....
>
> On 7 May 2020 08:44:59 CEST, Avakash bhat <avakash261@gmail.com> wrote:
>>
>>
>> Thanks for the quick response. I also had a followup question.
>>
>> If the ack filter adds the new ack to the tail of the queue after
>> removing an ack from the queue, won't it be starving the ack?
>> The replaced ack was much ahead in the queue than the ack we replaced at
>> the tail right?
>>
>> Thanks,
>> Avakash Bhat
>>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

[-- Attachment #2: Type: text/html, Size: 1622 bytes --]

Re: [Cake] Query on ACK

[Dave Taht]

On Thu, May 7, 2020 at 11:36 PM Avakash bhat <avakash261@gmail.com> wrote:
>
> Ok thanks so much for the clarifications.
> That cleared it up quite a bit.

I note that there was something really subtle that could have been
done to improve cake's ack handling, and for all I know
it actually happened in the final codebase.

so, please, go forth and duplicate the existing implementation, and
ignore me, cause looking at this hairy code gives me a
headache.

anyway, to try and describe what I thought I saw an interaction with
the scheduler back in the day.

The ack-filter runs, deleting all but one packet from the ack queue,
and delivers that.
the scheduler runs, serves a bunch of other flows, then returns to the
ack queue, which has accumulated a couple more packets,
the ack-filter runs, deleting all but one packet from the ack queue,
and delivers that, but doesn't exhaust its qauntum
but now that flow is in the "fast" queue, and we service just a few
other flows, and return to it, delete a couple, service one... and
stay stuck in the fast queue.

better, I thought, was once the ack filter exceeded the quantum of
packets for that flow in that drr round, even if it only delivered one
packet,
that it should always return it to the bulk queue, because tons more
packets would arrive in the interval between servicing
all the rest of the flows, thus more of which could be safely removed,
while maintaining a steadier clock for tcp.

I've already seen cake remove over 25% of all ack packets with no harm
to the other flows. So for all I know (and I'd have to
look) it's already doing it this way.

>
> Thanks,
> Avakash Bhat
>
> On Thu, May 7, 2020 at 12:37 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> I think that you will remove all redundant Backs in one go considerably advancing the new ACK in the queue. And more importantly, in most relevant modes cake will apply one queue per flow stochastically, so almost all packet's in a reverse ACK flow will be ACK with identical 5-tupel....
>>
>> On 7 May 2020 08:44:59 CEST, Avakash bhat <avakash261@gmail.com> wrote:
>>>
>>>
>>> Thanks for the quick response. I also had a followup question.
>>>
>>> If the ack filter adds the new ack to the tail of the queue after removing an ack from the queue, won't it be starving the ack?
>>> The replaced ack was much ahead in the queue than the ack we replaced at the tail right?
>>>
>>> Thanks,
>>> Avakash Bhat
>>
>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] Query on ACK

[Sebastian Moeller]

Hi Dave,


> On May 8, 2020, at 08:50, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Thu, May 7, 2020 at 11:36 PM Avakash bhat <avakash261@gmail.com> wrote:
>> 
>> Ok thanks so much for the clarifications.
>> That cleared it up quite a bit.
> 
> I note that there was something really subtle that could have been
> done to improve cake's ack handling, and for all I know
> it actually happened in the final codebase.
> 
> so, please, go forth and duplicate the existing implementation, and
> ignore me, cause looking at this hairy code gives me a
> headache.
> 
> anyway, to try and describe what I thought I saw an interaction with
> the scheduler back in the day.
> 
> The ack-filter runs, deleting all but one packet from the ack queue,
> and delivers that.
> the scheduler runs, serves a bunch of other flows, then returns to the
> ack queue, which has accumulated a couple more packets,
> the ack-filter runs, deleting all but one packet from the ack queue,
> and delivers that, but doesn't exhaust its qauntum
> but now that flow is in the "fast" queue, and we service just a few
> other flows, and return to it, delete a couple, service one... and
> stay stuck in the fast queue.

	Why would that be a problem? In that case ACKs did not bunch up (otherwise there would be backlog in the queue and it would forfeit its sparseness boost) and hence delivering the only ACK in a timely fashion should preserve the ACK clock, especially for non ABC-ACKs (https://tools.ietf.org/html/rfc3465) that relay on ACK count? Sure, if the single ACK had already matured a bit and immediately after sending it a fresher ACK would have been enqueued that looks suboptimal, but that race seems to exist no matter what? Now if the goal is to weed out ACKs to conserve bandwidth, sure not filtering ACKs is sub-optimal, but for the clocking?
Side-note, whoever invented the term "ACK-clocking" seemingly had a very fuzzy concept of what a clock is and what precision a clock can be expected to deliver ;)


Best Regards
	Sebastian

P.S.: As so often, I might simple be confused about the actual subtlety...


> 
> better, I thought, was once the ack filter exceeded the quantum of
> packets for that flow in that drr round, even if it only delivered one
> packet,
> that it should always return it to the bulk queue, because tons more
> packets would arrive in the interval between servicing
> all the rest of the flows, thus more of which could be safely removed,
> while maintaining a steadier clock for tcp.
> 
> I've already seen cake remove over 25% of all ack packets with no harm
> to the other flows. So for all I know (and I'd have to
> look) it's already doing it this way.
> 
>> 
>> Thanks,
>> Avakash Bhat
>> 
>> On Thu, May 7, 2020 at 12:37 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>>> 
>>> I think that you will remove all redundant Backs in one go considerably advancing the new ACK in the queue. And more importantly, in most relevant modes cake will apply one queue per flow stochastically, so almost all packet's in a reverse ACK flow will be ACK with identical 5-tupel....
>>> 
>>> On 7 May 2020 08:44:59 CEST, Avakash bhat <avakash261@gmail.com> wrote:
>>>> 
>>>> 
>>>> Thanks for the quick response. I also had a followup question.
>>>> 
>>>> If the ack filter adds the new ack to the tail of the queue after removing an ack from the queue, won't it be starving the ack?
>>>> The replaced ack was much ahead in the queue than the ack we replaced at the tail right?
>>>> 
>>>> Thanks,
>>>> Avakash Bhat
>>> 
>>> 
>>> --
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> -- 
> Make Music, Not War
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-435-0729

Re: [Cake] Query on ACK

[Sebastian Moeller]

Dear All,

just as a side-note. I believe that ACK filtering is one more application that directly profits from flow-queueing (as the set of packets to compare with is already separated out from the set of all queued packets), as one needs to collect ACKs according to their 4-Tuples which FQ does naturally. 

Best Regards
	Sebastian




> On May 7, 2020, at 09:07, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> I think that you will remove all redundant Backs in one go considerably advancing the new ACK in the queue. And more importantly, in most relevant modes cake will apply one queue per flow stochastically, so almost all packet's in a reverse ACK flow will be ACK with identical 5-tupel....
> 
> On 7 May 2020 08:44:59 CEST, Avakash bhat <avakash261@gmail.com> wrote:
> 
> Thanks for the quick response. I also had a followup question. 
> 
> If the ack filter adds the new ack to the tail of the queue after removing an ack from the queue, won't it be starving the ack? 
> The replaced ack was much ahead in the queue than the ack we replaced at the tail right?
> 
> Thanks,
> Avakash Bhat
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Query on ACK

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> On Thu, May 7, 2020 at 11:36 PM Avakash bhat <avakash261@gmail.com> wrote:
>>
>> Ok thanks so much for the clarifications.
>> That cleared it up quite a bit.
>
> I note that there was something really subtle that could have been
> done to improve cake's ack handling, and for all I know
> it actually happened in the final codebase.
>
> so, please, go forth and duplicate the existing implementation, and
> ignore me, cause looking at this hairy code gives me a
> headache.
>
> anyway, to try and describe what I thought I saw an interaction with
> the scheduler back in the day.
>
> The ack-filter runs, deleting all but one packet from the ack queue,
> and delivers that.
> the scheduler runs, serves a bunch of other flows, then returns to the
> ack queue, which has accumulated a couple more packets,
> the ack-filter runs, deleting all but one packet from the ack queue,
> and delivers that, but doesn't exhaust its qauntum

The ACK filter runs on enqueue, so if a queue has only ACKs in it, it
will never accumulate anything in the first place...

-Toke

Re: [Cake] Query on ACK

[Dave Taht]

On Fri, May 8, 2020 at 8:08 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Dave Taht <dave.taht@gmail.com> writes:
>
> > On Thu, May 7, 2020 at 11:36 PM Avakash bhat <avakash261@gmail.com> wrote:
> >>
> >> Ok thanks so much for the clarifications.
> >> That cleared it up quite a bit.
> >
> > I note that there was something really subtle that could have been
> > done to improve cake's ack handling, and for all I know
> > it actually happened in the final codebase.
> >
> > so, please, go forth and duplicate the existing implementation, and
> > ignore me, cause looking at this hairy code gives me a
> > headache.
> >
> > anyway, to try and describe what I thought I saw an interaction with
> > the scheduler back in the day.
> >
> > The ack-filter runs, deleting all but one packet from the ack queue,
> > and delivers that.
> > the scheduler runs, serves a bunch of other flows, then returns to the
> > ack queue, which has accumulated a couple more packets,
> > the ack-filter runs, deleting all but one packet from the ack queue,
> > and delivers that, but doesn't exhaust its qauntum
>
> The ACK filter runs on enqueue, so if a queue has only ACKs in it, it
> will never accumulate anything in the first place...

but the side effect is that on dequeue, it flips it into the fast
queue drr rotation, not the slow, so it can't accumulate
as many acks before delivering the one it has left.

Or so I thought, way back when....

>
> -Toke
>


-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] Query on ACK

[Jonathan Morton]

>> The ACK filter runs on enqueue, so if a queue has only ACKs in it, it
>> will never accumulate anything in the first place...
> 
> but the side effect is that on dequeue, it flips it into the fast
> queue drr rotation, not the slow, so it can't accumulate
> as many acks before delivering the one it has left.
> 
> Or so I thought, way back when....

The ack filter converts a stream of acks that might be treated as a bulk flow into a sparse flow, which is delivered promptly.  This is a good thing; an ack should not be held back solely to see whether another one will arrive.

I think of it as an optimisation to reduce delay of the information in the ack stream, not solely as a way to reduce the bandwidth consumed by the ack stream; the latter is a happy side effect.

 - Jonathan Morton

Re: [Cake] Query on ACK

[Dave Taht]

acks at the time you have reached a point of dropping them
significantly have filled the pipe, also.

What I saw here was that the first flow to really get going, and
really get dropped, dominated over the others,
because I thought it was consistently ending up in the priority queue.

http://blog.cerowrt.org/post/ack_filtering/

Look, all I'm proposing is this idea be tried and tested. Cynically...
since there's a new model coming out as
the result of this work, it immediately turns into something a good
paper can hing on.

On Fri, May 8, 2020 at 8:20 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> >> The ACK filter runs on enqueue, so if a queue has only ACKs in it, it
> >> will never accumulate anything in the first place...
> >
> > but the side effect is that on dequeue, it flips it into the fast
> > queue drr rotation, not the slow, so it can't accumulate
> > as many acks before delivering the one it has left.
> >
> > Or so I thought, way back when....
>
> The ack filter converts a stream of acks that might be treated as a bulk flow into a sparse flow, which is delivered promptly.  This is a good thing; an ack should not be held back solely to see whether another one will arrive.
>
> I think of it as an optimisation to reduce delay of the information in the ack stream, not solely as a way to reduce the bandwidth consumed by the ack stream; the latter is a happy side effect.
>
>  - Jonathan Morton



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

[Cake] Curious regarding Cake sensitivity to hardware queue depth

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]


The following question occurred to me. It's empirical, not theoretical (though the theory might be interesting, it's not clear that creating a theoretical model that matches real hardware is realistic.
 
I'm a big fan of Cake, and recommend it to everyone that can deploy it on the router closest to the bottleneck link. But given its complexity (and interactions with the complex control loops in end-to-end TCP, including things like minimum cwnd and the number of TCP sessions that are concurrent, etc), it's hard for me to guess the answer to certain issues that may come up in practice less commonly.
 
Here's an example. What if there is a FIFO queue that can build up queueing delay to a certain extent between Cake and the bottleneck. In a badly setup DOCSIS 2 system, this can happen due to other traffic into the DOCSIS head-end. Cake's basic assumption that by setting the max bitrate on the uplink to slightly less than the claimed "up to" capacity is not fully realistic in such situations - the queue can build up in the path to the head-end.
 
The same can happen in a WiFi situation on the airlink to the AP from the STA, if you were to run Cake with a setting for bandwidth that is set much lower than the achievable shared airlink bitrate. And on the AP side, the path to STA's is often forced to go through a FIFO either in the hardware or in the proprietary driver that can get pretty large (though adding 20 msec. might not be problematic in most user-interaction cases).
 
But it would be interesting to assess whether inserting a FIFO delay between Cake and the actual NIC that could be cranked up in delay causes surprising results.
 
A "surprise" would be that either throughput or lag-under-load wih various traffic mixes (bulk traffic, in particular) would do something other than the simple thing of staying with the same throughput and adding the FIFO delay into the lag under load.
 
Anyone done such an exploration?
 
One of the reasons I wonder is because I wondered whether just using Cake to moderate the flows from an AP to STAs in an access point would be good, even if the hardware doesn't allow one to shorten the queue it builds up inside. Obviously doing a "distributed Cake" among all the STAs trying to send to an AP would be not necessarily good, because the STAs can't cooperate efficiently because they have to go through the AP and get the packets reflected, encountering all kinds of delay at a scale that would make it hard to coordinate the flow towards the AP. (that's why 802.11ax is based on AP polling rather than LBT arbitration, but even in 802.11ax, the polling doesn't provide any information to the STA's that they could use to manage their flows.).
 
PS: Make WiFi fast really needs to address 802.11ax (WiFi6), which I suspect will be *worse* than WiFi in its congestion interactions with TCP and QUIC on UDP. Sad if a "premium" upgrade becomes a downgrade because of congestion mismanagement. Polling helps increase usable airtime, but it doesn't help TCP and QUIC moderate their contributions to creating bufferbloated connections.

[-- Attachment #2: Type: text/html, Size: 4850 bytes --]

Re: [Cake] Query on ACK

[Avakash bhat]

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

Hi all,
We had another query we would like to resolve. We wanted to verify the
working of ack filter in ns-3,
so we decided to replicate the Fig 6 graph in the CAKE paper(
https://ieeexplore.ieee.org/document/8475045).
While trying to build the topology we realized that we do not know the
number of packets or bytes sent from
the source to the destination for each of the TCP connections ( We are
assuming it is a point to point connection with 4 TCP flows).

Could we get a bit more details about how the experiment was conducted?

Also is this the best way to verify the correctness of our implementation?

Thanks,
Avakash Bhat

On Fri, May 8, 2020 at 9:11 PM Dave Taht <dave.taht@gmail.com> wrote:

> acks at the time you have reached a point of dropping them
> significantly have filled the pipe, also.
>
> What I saw here was that the first flow to really get going, and
> really get dropped, dominated over the others,
> because I thought it was consistently ending up in the priority queue.
>
> http://blog.cerowrt.org/post/ack_filtering/
>
> Look, all I'm proposing is this idea be tried and tested. Cynically...
> since there's a new model coming out as
> the result of this work, it immediately turns into something a good
> paper can hing on.
>
> On Fri, May 8, 2020 at 8:20 AM Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> > >> The ACK filter runs on enqueue, so if a queue has only ACKs in it, it
> > >> will never accumulate anything in the first place...
> > >
> > > but the side effect is that on dequeue, it flips it into the fast
> > > queue drr rotation, not the slow, so it can't accumulate
> > > as many acks before delivering the one it has left.
> > >
> > > Or so I thought, way back when....
> >
> > The ack filter converts a stream of acks that might be treated as a bulk
> flow into a sparse flow, which is delivered promptly.  This is a good
> thing; an ack should not be held back solely to see whether another one
> will arrive.
> >
> > I think of it as an optimisation to reduce delay of the information in
> the ack stream, not solely as a way to reduce the bandwidth consumed by the
> ack stream; the latter is a happy side effect.
> >
> >  - Jonathan Morton
>
>
>
> --
> Make Music, Not War
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-435-0729
>

[-- Attachment #2: Type: text/html, Size: 3218 bytes --]

Re: [Cake] Query on ACK

[Jonathan Morton]

> On 25 May, 2020, at 8:17 am, Avakash bhat <avakash261@gmail.com> wrote:
> 
> We had another query we would like to resolve. We wanted to verify the working of ack filter in ns-3, 
> so we decided to replicate the Fig 6 graph in the CAKE paper(https://ieeexplore.ieee.org/document/8475045). 
> While trying to build the topology we realized that we do not know the number of packets or bytes sent from 
> the source to the destination for each of the TCP connections ( We are assuming it is a point to point connection with 4 TCP flows). 
> 
> Could we get a bit more details about how the experiment was conducted?

I believe this was conducted using the RRUL test in Flent.  This opens four saturating TCP flows in each direction, and also sends a small amount of latency measuring traffic.  On this occasion I don't think we added any simulated path delays, and only imposed the quoted asymmetric bandwidth limits (30Mbps down, 1Mbps up).

> Also is this the best way to verify the correctness of our implementation?

Obviously with limited space in our paper, we could only include a small selection of test results.  Many other tests were run in practice, and we have expanded our test repertoire since.

In particular, we now routinely run tests with a simulated typical Internet path delay inserted, eg. 20ms, 80ms, 160ms baseline RTTs to represent reaching a local-ish CDN, across the Atlantic, and from Europe to the US West Coast.  You will also want to include multiple traffic mixes in the analysis, in particular different congestion control algorithms (at least Reno and CUBIC), and running with ECN both enabled and disabled at the endpoints.

A useful torture test we used was to send many bulk flows up the narrow side of the link and a single bulk flow down the wide side.  For example, 50:1 flow counts with 1:10, 1:20 and 1:30 bandwidth asymmetries.  The acks of the single flow then have to compete with the heavy load of the many flows, and the total goodput of that single flow is an important metric, along with both the total goodput and the Jain's fairness of the upload traffic.  This should show a particularly strong effect of the ack filter, as otherwise individual acks have to be dropped by the AQM, which Codel is not very good at adapting to quickly.

In evaluating the above, you will want to be vigilant not only for observed gross performance, but also the extent to which the ack filter preserves or loses information from the ack stream.  This is particularly true in the runs without ECN, in which congestion signals can only be applied through packet loss, and the feedback of that signal is through dup-acks and SACK.  I think you will find that the "aggressive" setting loses some information, and its performance suffers accordingly in some cases.

 - Jonathan Morton

Re: [Cake] Query on ACK

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

>> On 25 May, 2020, at 8:17 am, Avakash bhat <avakash261@gmail.com> wrote:
>> 
>> We had another query we would like to resolve. We wanted to verify the working of ack filter in ns-3, 
>> so we decided to replicate the Fig 6 graph in the CAKE paper(https://ieeexplore.ieee.org/document/8475045). 
>> While trying to build the topology we realized that we do not know the number of packets or bytes sent from 
>> the source to the destination for each of the TCP connections ( We are assuming it is a point to point connection with 4 TCP flows). 
>> 
>> Could we get a bit more details about how the experiment was conducted?
>
> I believe this was conducted using the RRUL test in Flent.  This opens
> four saturating TCP flows in each direction, and also sends a small
> amount of latency measuring traffic.  On this occasion I don't think
> we added any simulated path delays, and only imposed the quoted
> asymmetric bandwidth limits (30Mbps down, 1Mbps up).

See https://www.cs.kau.se/tohojo/cake/ - the link to the data files near
the bottom of that page also contains the Flent batch file and setup
scripts used to run the whole thing.

(And there's no explicit "number of bytes sent", but rather the flows
are capacity-seeking flows running for a limited *time*).

-Toke

Re: [Cake] Query on ACK

[Avakash bhat]

[-- Attachment #1: Type: text/plain, Size: 1820 bytes --]

Hi all,

I wanted another clarification on the results obtained by the Ack filtering
experiment( Fig 6) .
Was the experiment conducted with only ack filtering enabled?
Or was set associative hash and the other modules of Cake enabled along
with Ack filtering while running this experiment ?

Thanks,
Avakash Bhat

On Mon, May 25, 2020, 5:28 PM Toke Høiland-Jørgensen <toke@redhat.com>
wrote:

> Jonathan Morton <chromatix99@gmail.com> writes:
>
> >> On 25 May, 2020, at 8:17 am, Avakash bhat <avakash261@gmail.com> wrote:
> >>
> >> We had another query we would like to resolve. We wanted to verify the
> working of ack filter in ns-3,
> >> so we decided to replicate the Fig 6 graph in the CAKE paper(
> https://ieeexplore.ieee.org/document/8475045).
> >> While trying to build the topology we realized that we do not know the
> number of packets or bytes sent from
> >> the source to the destination for each of the TCP connections ( We are
> assuming it is a point to point connection with 4 TCP flows).
> >>
> >> Could we get a bit more details about how the experiment was conducted?
> >
> > I believe this was conducted using the RRUL test in Flent.  This opens
> > four saturating TCP flows in each direction, and also sends a small
> > amount of latency measuring traffic.  On this occasion I don't think
> > we added any simulated path delays, and only imposed the quoted
> > asymmetric bandwidth limits (30Mbps down, 1Mbps up).
>
> See https://www.cs.kau.se/tohojo/cake/ - the link to the data files near
> the bottom of that page also contains the Flent batch file and setup
> scripts used to run the whole thing.
>
> (And there's no explicit "number of bytes sent", but rather the flows
> are capacity-seeking flows running for a limited *time*).
>
> -Toke
>
>

[-- Attachment #2: Type: text/html, Size: 2791 bytes --]

Re: [Cake] Query on ACK

[Jonathan Morton]

> On 14 Jun, 2020, at 3:43 pm, Avakash bhat <avakash261@gmail.com> wrote:
> 
> I wanted another clarification on the results obtained by the Ack filtering experiment( Fig 6) . 
> Was the experiment conducted with only ack filtering enabled? 
> Or was set associative hash and the other modules of Cake enabled along with Ack filtering while running this experiment ?

The test was run on a complete implementation of Cake, set up in the normal way.  I think we kept the configuration simple for this test, so everything at defaults except for choosing the shaped bandwidth in each direction.

The ack-filter relies on having fairly good flow isolation, so that consecutive packets in the appropriate queue belong to the same ack stream.  So at minimum it is appropriate to have the set-associative flow hash enabled.

The host-fairness and Diffserv features were probably enabled, but did not have relevant effects in this case, since only one pair of hosts and the Best Effort DSCP were used in the traffic.

 - Jonathan Morton

Re: [Cake] Query on ACK

[Avakash bhat]

[-- Attachment #1: Type: text/plain, Size: 1563 bytes --]

Hi all,

Thank you for the clarification. We will try implementing a similar test.

Thanks to the Cake community's continued support we were able to
successfully merge the set-associative flow hash module into ns-3 (
https://gitlab.com/nsnam/ns-3-dev/-/merge_requests/209).

Hopefully, we are able to achieve a similar result with the ack filter
module and we will continue to work to do so.

Thanks,
Avakash Bhat

On Sun, Jun 14, 2020 at 8:13 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 14 Jun, 2020, at 3:43 pm, Avakash bhat <avakash261@gmail.com> wrote:
> >
> > I wanted another clarification on the results obtained by the Ack
> filtering experiment( Fig 6) .
> > Was the experiment conducted with only ack filtering enabled?
> > Or was set associative hash and the other modules of Cake enabled along
> with Ack filtering while running this experiment ?
>
> The test was run on a complete implementation of Cake, set up in the
> normal way.  I think we kept the configuration simple for this test, so
> everything at defaults except for choosing the shaped bandwidth in each
> direction.
>
> The ack-filter relies on having fairly good flow isolation, so that
> consecutive packets in the appropriate queue belong to the same ack
> stream.  So at minimum it is appropriate to have the set-associative flow
> hash enabled.
>
> The host-fairness and Diffserv features were probably enabled, but did not
> have relevant effects in this case, since only one pair of hosts and the
> Best Effort DSCP were used in the traffic.
>
>  - Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 2127 bytes --]

Re: [Cake] Query on ACK

[Dave Taht]

so glad to see that merged! I know how hard it is to make progress
that anyone can (re)use.

Thanks so much!

On Mon, Jun 15, 2020 at 10:22 PM Avakash bhat <avakash261@gmail.com> wrote:
>
> Hi all,
>
> Thank you for the clarification. We will try implementing a similar test.
>
> Thanks to the Cake community's continued support we were able to successfully merge the set-associative flow hash module into ns-3 (https://gitlab.com/nsnam/ns-3-dev/-/merge_requests/209).
>
> Hopefully, we are able to achieve a similar result with the ack filter module and we will continue to work to do so.
>
> Thanks,
> Avakash Bhat
>
> On Sun, Jun 14, 2020 at 8:13 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>> > On 14 Jun, 2020, at 3:43 pm, Avakash bhat <avakash261@gmail.com> wrote:
>> >
>> > I wanted another clarification on the results obtained by the Ack filtering experiment( Fig 6) .
>> > Was the experiment conducted with only ack filtering enabled?
>> > Or was set associative hash and the other modules of Cake enabled along with Ack filtering while running this experiment ?
>>
>> The test was run on a complete implementation of Cake, set up in the normal way.  I think we kept the configuration simple for this test, so everything at defaults except for choosing the shaped bandwidth in each direction.
>>
>> The ack-filter relies on having fairly good flow isolation, so that consecutive packets in the appropriate queue belong to the same ack stream.  So at minimum it is appropriate to have the set-associative flow hash enabled.
>>
>> The host-fairness and Diffserv features were probably enabled, but did not have relevant effects in this case, since only one pair of hosts and the Best Effort DSCP were used in the traffic.
>>
>>  - Jonathan Morton



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cake] Query on ACK

[Dave Taht]

I have an idea for a good paper that compares set associative vs
non-set associative hashing
in an interesting way that nobody has done yet. I'd rather not publish
the idea here, but if anyone
has time for writing two simulation tests and writing yet another kewl
paper, please contact me offlist,
thx!