[Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

Hello Jonathan and Dave,

Thanks for your timely inputs.

We have implemented COBALT in ns-3. The main C++ file is here:

https://github.com/Daipu/COBALT/blob/master/src/traffic-control/model/cobalt-queue-disc.cc

We have also added Queue length and Throughput graphs for COBALT in our
wiki:

https://github.com/Daipu/COBALT/wiki

Thanks and Regards,
Jendaipou Palmei
Shefali Gupta

[-- Attachment #2: Type: text/html, Size: 837 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Dave Taht]

Your light traffic graph is puzzling, with the synchronized drops. Is
it possible for you to do another test run, same setup, with codel or
pie?
On Thu, Nov 22, 2018 at 5:57 AM Jendaipou Palmei
<jendaipoupalmei@gmail.com> wrote:
>
> Hello Jonathan and Dave,
>
> Thanks for your timely inputs.
>
> We have implemented COBALT in ns-3. The main C++ file is here:
>
> https://github.com/Daipu/COBALT/blob/master/src/traffic-control/model/cobalt-queue-disc.cc
>
> We have also added Queue length and Throughput graphs for COBALT in our wiki:
>
> https://github.com/Daipu/COBALT/wiki
>
> Thanks and Regards,
> Jendaipou Palmei
> Shefali Gupta
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

Hi Dave,

We have added the results for CoDel and PIE in the same setup.

Link:
https://github.com/Daipu/COBALT/wiki/Light-Traffic

Thanks for your help.

On Thu, Nov 22, 2018 at 9:02 PM Dave Taht <dave.taht@gmail.com> wrote:

> Your light traffic graph is puzzling, with the synchronized drops. Is
> it possible for you to do another test run, same setup, with codel or
> pie?
> On Thu, Nov 22, 2018 at 5:57 AM Jendaipou Palmei
> <jendaipoupalmei@gmail.com> wrote:
> >
> > Hello Jonathan and Dave,
> >
> > Thanks for your timely inputs.
> >
> > We have implemented COBALT in ns-3. The main C++ file is here:
> >
> >
> https://github.com/Daipu/COBALT/blob/master/src/traffic-control/model/cobalt-queue-disc.cc
> >
> > We have also added Queue length and Throughput graphs for COBALT in our
> wiki:
> >
> > https://github.com/Daipu/COBALT/wiki
> >
> > Thanks and Regards,
> > Jendaipou Palmei
> > Shefali Gupta
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
>


-- 
Yours Faithfully,
Jendaipou Palmei

[-- Attachment #2: Type: text/html, Size: 2441 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Dave Taht]

Jendaipou Palmei <jendaipoupalmei@gmail.com> writes:

> Hi Dave,
>
> We have added the results for CoDel and PIE in the same setup.
>
> Link:
> https://github.com/Daipu/COBALT/wiki/Light-Traffic

Wonderful, thank you. One new thing in cobalt is it responds much
earlier to stuff in slow start, and thus that first spike is much
smaller than it is with codel or pie. Network researchers spend way too
much time looking at stuff in congestion avoidance mode, where what we
see in the real world is most flows never get out of slow start.

As an example, slashdot.org, last I checked, started 78 flows over 2.2
seconds for a page load.

I'm still puzzled as to why queue utilization spikes and drops so
much on all three qdiscs in your simulation. In general we use a much
larger number of packets in our packet limit (1000 or more) and I am
thinking tail drop is controlling your queue more than any of the qdiscs
are at 10mbit with a 50ms delay. I haven't had enough coffee yet this
morning to do the bdp here, but that's another easy param to tweak.

Ns3, btw, also has fq_codel. I happen to be fond of what fq can do. if
you feel like another run, with the packet limit increased and that
added too? 

Care to try 100mbit? 20ms RTT? 80ms? :)

Can you put your test code up somewhere?


>
> Thanks for your help. 
>
> On Thu, Nov 22, 2018 at 9:02 PM Dave Taht <dave.taht@gmail.com> wrote:
>
>     Your light traffic graph is puzzling, with the synchronized drops.
>     Is
>     it possible for you to do another test run, same setup, with codel
>     or
>     pie?
>     On Thu, Nov 22, 2018 at 5:57 AM Jendaipou Palmei
>     <jendaipoupalmei@gmail.com> wrote:
>     >
>     > Hello Jonathan and Dave,
>     >
>     > Thanks for your timely inputs.
>     >
>     > We have implemented COBALT in ns-3. The main C++ file is here:
>     >
>     >
>     https://github.com/Daipu/COBALT/blob/master/src/traffic-control/model/cobalt-queue-disc.cc
>    
>     >
>     > We have also added Queue length and Throughput graphs for COBALT
>     in our wiki:
>     >
>     > https://github.com/Daipu/COBALT/wiki
>     >
>     > Thanks and Regards,
>     > Jendaipou Palmei
>     > Shefali Gupta
>     > _______________________________________________
>     > Cake mailing list
>     > Cake@lists.bufferbloat.net
>     > https://lists.bufferbloat.net/listinfo/cake
>     
>     
>     
>     -- 
>     
>     Dave Täht
>     CTO, TekLibre, LLC
>     http://www.teklibre.com
>     Tel: 1-831-205-9740

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Dave Taht]

On Fri, Nov 23, 2018 at 8:05 AM Dave Taht <dave@taht.net> wrote:
>
> Jendaipou Palmei <jendaipoupalmei@gmail.com> writes:
>
> > Hi Dave,
> >
> > We have added the results for CoDel and PIE in the same setup.
> >
> > Link:
> > https://github.com/Daipu/COBALT/wiki/Light-Traffic
>
> Wonderful, thank you. One new thing in cobalt is it responds much
> earlier to stuff in slow start, and thus that first spike is much
> smaller than it is with codel or pie. Network researchers spend way too
> much time looking at stuff in congestion avoidance mode, where what we
> see in the real world is most flows never get out of slow start.
>
> As an example, slashdot.org, last I checked, started 78 flows over 2.2
> seconds for a page load.
>
> I'm still puzzled as to why queue utilization spikes and drops so
> much on all three qdiscs in your simulation. In general we use a much
> larger number of packets in our packet limit (1000 or more) and I am
> thinking tail drop is controlling your queue more than any of the qdiscs
> are at 10mbit with a 50ms delay. I haven't had enough coffee yet this
> morning to do the bdp here, but that's another easy param to tweak.

this was definiately pre-coffee, you are showing next to no queue
utilization in the first place and good throughput on pie and codel.

It was the throughput graph I was mis-interpreting.

This is possibly a correct result in your simulation!! - the periodic
throughput drop you are showing in cobalt at this bandwidth and rtt.
I'm happy to see cobalt kick in early on slow start but not happy to
see the periodic drop. Jon, do you have time for a code review?

It also looks like we can setup a simple real-world duplicate of this
test in flent (5 flows) + netem. However (sigh), our tc sampler for
queue depth is broken on modern versions of iproute2.

Also the aqm work that I'd done on cake was eliminated in the rewrite,
and that I'd not gone at looked hard at longer rtts again, lacking
resources.

> Ns3, btw, also has fq_codel. I happen to be fond of what fq can do. if
> you feel like another run, with the packet limit increased and that
> added too?
>
> Care to try 100mbit? 20ms RTT? 80ms? :)
>
> Can you put your test code up somewhere?
>
>
> >
> > Thanks for your help.
> >
> > On Thu, Nov 22, 2018 at 9:02 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> >     Your light traffic graph is puzzling, with the synchronized drops.
> >     Is
> >     it possible for you to do another test run, same setup, with codel
> >     or
> >     pie?
> >     On Thu, Nov 22, 2018 at 5:57 AM Jendaipou Palmei
> >     <jendaipoupalmei@gmail.com> wrote:
> >     >
> >     > Hello Jonathan and Dave,
> >     >
> >     > Thanks for your timely inputs.
> >     >
> >     > We have implemented COBALT in ns-3. The main C++ file is here:
> >     >
> >     >
> >     https://github.com/Daipu/COBALT/blob/master/src/traffic-control/model/cobalt-queue-disc.cc
> >
> >     >
> >     > We have also added Queue length and Throughput graphs for COBALT
> >     in our wiki:
> >     >
> >     > https://github.com/Daipu/COBALT/wiki
> >     >
> >     > Thanks and Regards,
> >     > Jendaipou Palmei
> >     > Shefali Gupta
> >     > _______________________________________________
> >     > Cake mailing list
> >     > Cake@lists.bufferbloat.net
> >     > https://lists.bufferbloat.net/listinfo/cake
> >
> >
> >
> >     --
> >
> >     Dave Täht
> >     CTO, TekLibre, LLC
> >     http://www.teklibre.com
> >     Tel: 1-831-205-9740



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 23 Nov, 2018, at 6:43 pm, Dave Taht <dave.taht@gmail.com> wrote:
> 
> This is possibly a correct result in your simulation!! - the periodic
> throughput drop you are showing in cobalt at this bandwidth and rtt.
> I'm happy to see cobalt kick in early on slow start but not happy to
> see the periodic drop. Jon, do you have time for a code review?

I looked at it briefly, but the code structure is different enough that I need to sit down and study it carefully to figure out whether there are any relevant differences.

The throughput drops most likely occur because the TCPs become synchronised and remain so under AQM action.  You can see that the frequency of the system is lower in the later part of the COBALT run than in the Codel run, but the same as Codel in the earlier part where throughput drops don't occur.  But this shouldn't really occur with a Codel-based AQM (as COBALT is), because a single mark is sufficient to tell TCP to back off over one RTT.  An explanation might be if this implementation of COBALT isn't running down correctly when deactivated, so the mark frequency only rises while being turned on and off.  The run-down behaviour is a major intentional difference between COBALT and reference Codel.

I'll look at the code more closely with that in mind.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

>> This is possibly a correct result in your simulation!! - the periodic
>> throughput drop you are showing in cobalt at this bandwidth and rtt.
>> I'm happy to see cobalt kick in early on slow start but not happy to
>> see the periodic drop. Jon, do you have time for a code review?
> 
> I looked at it briefly, but the code structure is different enough that I need to sit down and study it carefully to figure out whether there are any relevant differences.
> 
> The throughput drops most likely occur because the TCPs become synchronised and remain so under AQM action.  You can see that the frequency of the system is lower in the later part of the COBALT run than in the Codel run, but the same as Codel in the earlier part where throughput drops don't occur.  But this shouldn't really occur with a Codel-based AQM (as COBALT is), because a single mark is sufficient to tell TCP to back off over one RTT.  An explanation might be if this implementation of COBALT isn't running down correctly when deactivated, so the mark frequency only rises while being turned on and off.  The run-down behaviour is a major intentional difference between COBALT and reference Codel.
> 
> I'll look at the code more closely with that in mind.

Okay, I've had a look - not quite line by line, but the parts I consider important for the behaviour seen so far.

There are a couple of small behavioural differences between your code and mine, which should be corrected if the model is to accurately reflect the prototype.  These are probably not relevant for the results shown so far, but are likely show up on more aggressive tests involving unresponsive traffic.

 - On queue overflow, a tail drop is used to resolve it.  While not technically part of COBALT, Cake performs head-dropping on queue overflow, doing so from the longest queue, and I consider that to be best practice.  This gets the message to the offending sender ASAP, without having to bubble up through the jammed queue first.  If the packets currently at the head of the queue are smaller than the one being offered, you might need to drop more than one to maintain the size invariant.

 - The hard-drop flag for BLUE is set at the top of the control-law function, and tested in order to bypass the Codel logic if already set.  This is not how the COBALT code operates; the BLUE logic should come last, and the Codel logic run unconditionally.

Everything else looks reasonably correct at first glance (though the amount of boilerplate is epic).  I would recommend verifying that CobaltQueueEmpty() actually gets called when appropriate though.  Without it, I suspect that the run-down logic won't work as intended.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 3368 bytes --]

Thanks a lot for taking out time to review our code, Jonathan.

We'll make the changes according to your suggestions and produce new plots.

Thanks, Dave for the feedback. Yes, we will run the simulations with
fq-codel once we fix the code as suggested by Jonathan, and also run
simulations with higher bandwidths as you suggested.

I'll upload the source code of the programs in the same repo, and give you
the link.

Regards,
Jendaipou Palmei
Shefali Gupta

On Sat, Nov 24, 2018 at 8:29 AM Jonathan Morton <chromatix99@gmail.com>
wrote:

> >> This is possibly a correct result in your simulation!! - the periodic
> >> throughput drop you are showing in cobalt at this bandwidth and rtt.
> >> I'm happy to see cobalt kick in early on slow start but not happy to
> >> see the periodic drop. Jon, do you have time for a code review?
> >
> > I looked at it briefly, but the code structure is different enough that
> I need to sit down and study it carefully to figure out whether there are
> any relevant differences.
> >
> > The throughput drops most likely occur because the TCPs become
> synchronised and remain so under AQM action.  You can see that the
> frequency of the system is lower in the later part of the COBALT run than
> in the Codel run, but the same as Codel in the earlier part where
> throughput drops don't occur.  But this shouldn't really occur with a
> Codel-based AQM (as COBALT is), because a single mark is sufficient to tell
> TCP to back off over one RTT.  An explanation might be if this
> implementation of COBALT isn't running down correctly when deactivated, so
> the mark frequency only rises while being turned on and off.  The run-down
> behaviour is a major intentional difference between COBALT and reference
> Codel.
> >
> > I'll look at the code more closely with that in mind.
>
> Okay, I've had a look - not quite line by line, but the parts I consider
> important for the behaviour seen so far.
>
> There are a couple of small behavioural differences between your code and
> mine, which should be corrected if the model is to accurately reflect the
> prototype.  These are probably not relevant for the results shown so far,
> but are likely show up on more aggressive tests involving unresponsive
> traffic.
>
>  - On queue overflow, a tail drop is used to resolve it.  While not
> technically part of COBALT, Cake performs head-dropping on queue overflow,
> doing so from the longest queue, and I consider that to be best practice.
> This gets the message to the offending sender ASAP, without having to
> bubble up through the jammed queue first.  If the packets currently at the
> head of the queue are smaller than the one being offered, you might need to
> drop more than one to maintain the size invariant.
>
>  - The hard-drop flag for BLUE is set at the top of the control-law
> function, and tested in order to bypass the Codel logic if already set.
> This is not how the COBALT code operates; the BLUE logic should come last,
> and the Codel logic run unconditionally.
>
> Everything else looks reasonably correct at first glance (though the
> amount of boilerplate is epic).  I would recommend verifying that
> CobaltQueueEmpty() actually gets called when appropriate though.  Without
> it, I suspect that the run-down logic won't work as intended.
>
>  - Jonathan Morton
>
>

-- 
Yours Faithfully,
Jendaipou Palmei

[-- Attachment #2: Type: text/html, Size: 4000 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 3989 bytes --]

Hello Jonathan,

We have made the changes in the code as suggested.

Here is the diff after making the changes:

https://github.com/Daipu/COBALT/commit/033db330287e2072bad94ac441f8aed774678a7d

Additionally, we have also updated the values of 'Pincrement' and
'Pdecrement' (parameters of BLUE) in ns-3 to match the ones used in the
Cake implementation in Linux.

Does the diff look correct?

Thanks and Regards,
Jendaipou Palmei
Shefali Gupta

On Sun, Nov 25, 2018 at 11:52 AM Jendaipou Palmei <jendaipoupalmei@gmail.com>
wrote:

> Thanks a lot for taking out time to review our code, Jonathan.
>
> We'll make the changes according to your suggestions and produce new plots.
>
> Thanks, Dave for the feedback. Yes, we will run the simulations with
> fq-codel once we fix the code as suggested by Jonathan, and also run
> simulations with higher bandwidths as you suggested.
>
> I'll upload the source code of the programs in the same repo, and give you
> the link.
>
> Regards,
> Jendaipou Palmei
> Shefali Gupta
>
> On Sat, Nov 24, 2018 at 8:29 AM Jonathan Morton <chromatix99@gmail.com>
> wrote:
>
>> >> This is possibly a correct result in your simulation!! - the periodic
>> >> throughput drop you are showing in cobalt at this bandwidth and rtt.
>> >> I'm happy to see cobalt kick in early on slow start but not happy to
>> >> see the periodic drop. Jon, do you have time for a code review?
>> >
>> > I looked at it briefly, but the code structure is different enough that
>> I need to sit down and study it carefully to figure out whether there are
>> any relevant differences.
>> >
>> > The throughput drops most likely occur because the TCPs become
>> synchronised and remain so under AQM action.  You can see that the
>> frequency of the system is lower in the later part of the COBALT run than
>> in the Codel run, but the same as Codel in the earlier part where
>> throughput drops don't occur.  But this shouldn't really occur with a
>> Codel-based AQM (as COBALT is), because a single mark is sufficient to tell
>> TCP to back off over one RTT.  An explanation might be if this
>> implementation of COBALT isn't running down correctly when deactivated, so
>> the mark frequency only rises while being turned on and off.  The run-down
>> behaviour is a major intentional difference between COBALT and reference
>> Codel.
>> >
>> > I'll look at the code more closely with that in mind.
>>
>> Okay, I've had a look - not quite line by line, but the parts I consider
>> important for the behaviour seen so far.
>>
>> There are a couple of small behavioural differences between your code and
>> mine, which should be corrected if the model is to accurately reflect the
>> prototype.  These are probably not relevant for the results shown so far,
>> but are likely show up on more aggressive tests involving unresponsive
>> traffic.
>>
>>  - On queue overflow, a tail drop is used to resolve it.  While not
>> technically part of COBALT, Cake performs head-dropping on queue overflow,
>> doing so from the longest queue, and I consider that to be best practice.
>> This gets the message to the offending sender ASAP, without having to
>> bubble up through the jammed queue first.  If the packets currently at the
>> head of the queue are smaller than the one being offered, you might need to
>> drop more than one to maintain the size invariant.
>>
>>  - The hard-drop flag for BLUE is set at the top of the control-law
>> function, and tested in order to bypass the Codel logic if already set.
>> This is not how the COBALT code operates; the BLUE logic should come last,
>> and the Codel logic run unconditionally.
>>
>> Everything else looks reasonably correct at first glance (though the
>> amount of boilerplate is epic).  I would recommend verifying that
>> CobaltQueueEmpty() actually gets called when appropriate though.  Without
>> it, I suspect that the run-down logic won't work as intended.
>>
>>  - Jonathan Morton
>>
>>
>
> --
> Yours Faithfully,
> Jendaipou Palmei
>

[-- Attachment #2: Type: text/html, Size: 5128 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 27 Nov, 2018, at 4:10 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
> 
> We have made the changes in the code as suggested.
> 
> Here is the diff after making the changes:
> 
> https://github.com/Daipu/COBALT/commit/033db330287e2072bad94ac441f8aed774678a7d
> 
> Additionally, we have also updated the values of 'Pincrement' and 'Pdecrement' (parameters of BLUE) in ns-3 to match the ones used in the Cake implementation in Linux.
> 
> Does the diff look correct?

It does look like an improvement.  How's the behaviour?

NB: the decimal values used in the constants are not quite precise equivalents of the Linux values, since you have dropped some of the trailing digits.  You could use (1./256) and (1./4096) to both make them precise and more human-readable.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

Hi Jonathan,

Thanks a lot for the quick review and suggestions to improve the code
readability.

We incorporated all the changes but didn't see any improvement in the
results.

However, we finally noticed that it was the packet size used in our
simulations that was affecting the throughput. It was earlier set to 1000
bytes, and after making it 1500 bytes (including headers) we note that the
throughput is not affected.

We have uploaded the new graphs on the same wiki link:
https://github.com/Daipu/COBALT/wiki/Light-Traffic

We're not sure why packet size is affecting the throughput so largely. Is
it the expected behavior?

Thanks and Regards
Jendaipou Palmei
Shefali Gupta

On Tue, Nov 27, 2018 at 8:07 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 27 Nov, 2018, at 4:10 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com>
> wrote:
> >
> > We have made the changes in the code as suggested.
> >
> > Here is the diff after making the changes:
> >
> >
> https://github.com/Daipu/COBALT/commit/033db330287e2072bad94ac441f8aed774678a7d
> >
> > Additionally, we have also updated the values of 'Pincrement' and
> 'Pdecrement' (parameters of BLUE) in ns-3 to match the ones used in the
> Cake implementation in Linux.
> >
> > Does the diff look correct?
>
> It does look like an improvement.  How's the behaviour?
>
> NB: the decimal values used in the constants are not quite precise
> equivalents of the Linux values, since you have dropped some of the
> trailing digits.  You could use (1./256) and (1./4096) to both make them
> precise and more human-readable.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 2424 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 30 Nov, 2018, at 1:53 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
> 
> However, we finally noticed that it was the packet size used in our simulations that was affecting the throughput. It was earlier set to 1000 bytes, and after making it 1500 bytes (including headers) we note that the throughput is not affected.
> 
> We have uploaded the new graphs on the same wiki link:
> https://github.com/Daipu/COBALT/wiki/Light-Traffic
> 
> We're not sure why packet size is affecting the throughput so largely. Is it the expected behavior?

That particular effect is probably accidental.

Is it possible to also show graphs of the packet marking (or dropping) rate?  That may be revealing.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]

Hello Jonathan,

Thanks for the quick reply.

We have uploaded the plots for the 'count' variable of COBALT (with a
segment size of 1500 and 1000 bytes).

Link: https://github.com/Daipu/COBALT/wiki/Cobalt-Drop-Count

We have not yet implemented ECN feature in COBALT, so packets are currently
dropped instead of being marked.

Are these the plots that you were referring to?

Thanks and Regards
Jendaipou Palmei
Shefali Gupta

On Fri, Nov 30, 2018 at 5:28 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 30 Nov, 2018, at 1:53 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com>
> wrote:
> >
> > However, we finally noticed that it was the packet size used in our
> simulations that was affecting the throughput. It was earlier set to 1000
> bytes, and after making it 1500 bytes (including headers) we note that the
> throughput is not affected.
> >
> > We have uploaded the new graphs on the same wiki link:
> > https://github.com/Daipu/COBALT/wiki/Light-Traffic
> >
> > We're not sure why packet size is affecting the throughput so largely.
> Is it the expected behavior?
>
> That particular effect is probably accidental.
>
> Is it possible to also show graphs of the packet marking (or dropping)
> rate?  That may be revealing.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 2041 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Dave Taht]

y'all are well on your way towards writing a paper on this topic. Need
collaborators?
On Tue, Dec 4, 2018 at 2:32 AM Jendaipou Palmei
<jendaipoupalmei@gmail.com> wrote:
>
> Hello Jonathan,
>
> Thanks for the quick reply.
>
> We have uploaded the plots for the 'count' variable of COBALT (with a segment size of 1500 and 1000 bytes).
>
> Link: https://github.com/Daipu/COBALT/wiki/Cobalt-Drop-Count
>
> We have not yet implemented ECN feature in COBALT, so packets are currently dropped instead of being marked.
>
> Are these the plots that you were referring to?
>
> Thanks and Regards
> Jendaipou Palmei
> Shefali Gupta
>
> On Fri, Nov 30, 2018 at 5:28 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>> > On 30 Nov, 2018, at 1:53 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
>> >
>> > However, we finally noticed that it was the packet size used in our simulations that was affecting the throughput. It was earlier set to 1000 bytes, and after making it 1500 bytes (including headers) we note that the throughput is not affected.
>> >
>> > We have uploaded the new graphs on the same wiki link:
>> > https://github.com/Daipu/COBALT/wiki/Light-Traffic
>> >
>> > We're not sure why packet size is affecting the throughput so largely. Is it the expected behavior?
>>
>> That particular effect is probably accidental.
>>
>> Is it possible to also show graphs of the packet marking (or dropping) rate?  That may be revealing.
>>
>>  - Jonathan Morton
>>


-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 4 Dec, 2018, at 12:31 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
> 
> We have uploaded the plots for the 'count' variable of COBALT (with a segment size of 1500 and 1000 bytes).
> 
> Link: https://github.com/Daipu/COBALT/wiki/Cobalt-Drop-Count
> 
> We have not yet implemented ECN feature in COBALT, so packets are currently dropped instead of being marked.
> 
> Are these the plots that you were referring to?

More-or-less, yes, though these actually show an internal state variable of the Codel algorithm rather than the actual number of marks/drops per time interval.  I was hoping to see similar graphs for the reference-Codel and PIE runs, since we can gain more insight from that, and PIE doesn't have an internal "count" variable that corresponds with Codel.  Nevertheless, the view into "count" behaviour is interesting in itself, and I'd like to see the corresponding graphs from reference Codel.

An artefact visible in these graphs is an apparent lack of sampling while not in the dropping state.  Thus you seem to have a gradual ramp from 0 to 1 count over the several seconds interval between activations, though in fact the variable is discrete.  It would be better to show that transition more precisely.

For study, it is also often helpful to zoom in on small time intervals to see the dynamic behaviour of the algorithm, particularly during the transition from slow-start to steady-state, where there is seemingly a big difference between reference Codel and COBALT.

Another interesting graph to produce for each algorithm and traffic type is the instantaneous throughput of each flow.  This offers insight into the relative fairness of each algorithm, and might help to explain the anomaly seen with 1000-byte packets and COBALT.  Usually this graph also reveals, through the shape of each throughput curve, which CC algorithm is in use - currently I'm guessing NewReno.  CUBIC and CTCP, which are also in common use, would behave differently.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Dave Taht]

On Tue, Dec 4, 2018 at 7:02 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 4 Dec, 2018, at 12:31 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
> >
> > We have uploaded the plots for the 'count' variable of COBALT (with a segment size of 1500 and 1000 bytes).
> >
> > Link: https://github.com/Daipu/COBALT/wiki/Cobalt-Drop-Count
> >
> > We have not yet implemented ECN feature in COBALT, so packets are currently dropped instead of being marked.
> >
> > Are these the plots that you were referring to?
>
> More-or-less, yes, though these actually show an internal state variable of the Codel algorithm rather than the actual number of marks/drops per time interval.  I was hoping to see similar graphs for the reference-Codel and PIE runs, since we can gain more insight from that, and PIE doesn't have an internal "count" variable that corresponds with Codel.  Nevertheless, the view into "count" behaviour is interesting in itself, and I'd like to see the corresponding graphs from reference Codel.
>
> An artefact visible in these graphs is an apparent lack of sampling while not in the dropping state.  Thus you seem to have a gradual ramp from 0 to 1 count over the several seconds interval between activations, though in fact the variable is discrete.  It would be better to show that transition more precisely.
>
> For study, it is also often helpful to zoom in on small time intervals to see the dynamic behaviour of the algorithm, particularly during the transition from slow-start to steady-state, where there is seemingly a big difference between reference Codel and COBALT.

I'm loving the slow start result.

>
> Another interesting graph to produce for each algorithm and traffic type is the instantaneous throughput of each flow.  This offers insight into the relative fairness of each algorithm, and might help to explain the anomaly seen with 1000-byte packets and COBALT.  Usually this graph also reveals, through the shape of each throughput curve, which CC algorithm is in use - currently I'm guessing NewReno.  CUBIC and CTCP, which are also in common use, would behave differently.

a file showing the timestamp of each drop would be easier to post process.

>
>  - Jonathan Morton
>


-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 3456 bytes --]

Hello Dave and Jonathan,

Thanks for the feedback!

We have uploaded the corresponding graphs for reference CoDel.

Link: https://github.com/Daipu/COBALT/wiki/Drop-Count-Graph

We have also plotted the instantaneous throughput for all flows in Light
traffic scenario for COBALT and CoDel.
These graphs are plotted for packet size with 1000 bytes and 1500 bytes.

Link: https://github.com/Daipu/COBALT/wiki/Throughput-for-Separate-Flow

We're currently working on the following:

1. plots for the actual number of marks/drops per time interval for COBALT,
CoDel, and PIE.
2. zoomed in plots on small time intervals to show the dynamic behavior of
the algorithm.
3. a file showing the timestamp of each drop.

About collaborating for writing a paper on this work: we'd be glad to do so
:) thanks for your guidance and help!

Thanks and regards
Jendaipou Palmei
Shefali Gupta

On Tue, Dec 4, 2018 at 8:51 PM Dave Taht <dave.taht@gmail.com> wrote:

> On Tue, Dec 4, 2018 at 7:02 AM Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> > > On 4 Dec, 2018, at 12:31 pm, Jendaipou Palmei <
> jendaipoupalmei@gmail.com> wrote:
> > >
> > > We have uploaded the plots for the 'count' variable of COBALT (with a
> segment size of 1500 and 1000 bytes).
> > >
> > > Link: https://github.com/Daipu/COBALT/wiki/Cobalt-Drop-Count
> > >
> > > We have not yet implemented ECN feature in COBALT, so packets are
> currently dropped instead of being marked.
> > >
> > > Are these the plots that you were referring to?
> >
> > More-or-less, yes, though these actually show an internal state variable
> of the Codel algorithm rather than the actual number of marks/drops per
> time interval.  I was hoping to see similar graphs for the reference-Codel
> and PIE runs, since we can gain more insight from that, and PIE doesn't
> have an internal "count" variable that corresponds with Codel.
> Nevertheless, the view into "count" behaviour is interesting in itself, and
> I'd like to see the corresponding graphs from reference Codel.
> >
> > An artefact visible in these graphs is an apparent lack of sampling
> while not in the dropping state.  Thus you seem to have a gradual ramp from
> 0 to 1 count over the several seconds interval between activations, though
> in fact the variable is discrete.  It would be better to show that
> transition more precisely.
> >
> > For study, it is also often helpful to zoom in on small time intervals
> to see the dynamic behaviour of the algorithm, particularly during the
> transition from slow-start to steady-state, where there is seemingly a big
> difference between reference Codel and COBALT.
>
> I'm loving the slow start result.
>
> >
> > Another interesting graph to produce for each algorithm and traffic type
> is the instantaneous throughput of each flow.  This offers insight into the
> relative fairness of each algorithm, and might help to explain the anomaly
> seen with 1000-byte packets and COBALT.  Usually this graph also reveals,
> through the shape of each throughput curve, which CC algorithm is in use -
> currently I'm guessing NewReno.  CUBIC and CTCP, which are also in common
> use, would behave differently.
>
> a file showing the timestamp of each drop would be easier to post process.
>
> >
> >  - Jonathan Morton
> >
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
>

[-- Attachment #2: Type: text/html, Size: 4620 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 5 Dec, 2018, at 2:23 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
> 
> We have uploaded the corresponding graphs for reference CoDel. 
> 
> Link: https://github.com/Daipu/COBALT/wiki/Drop-Count-Graph

Quite a remarkable difference here - just look at the scales for the corresponding graphs!  I'm actually rather surprised to see reference Codel reaching such deep activation states, when COBALT stays very shallowly activated but is still effective.  It makes me wonder whether there's something odd going on with the ns3 version of Codel.

I'm sure more insight will be gained from the actual drop traces.

> We have also plotted the instantaneous throughput for all flows in Light traffic scenario for COBALT and CoDel.
> These graphs are plotted for packet size with 1000 bytes and 1500 bytes.
> 
> Link: https://github.com/Daipu/COBALT/wiki/Throughput-for-Separate-Flow

This isn't quite what I was thinking of, but it's still interesting.  I was looking for all flows from a single run plotted on a single graph, perhaps stacked so that their sum is visible as overall throughput.  That way, the interaction between one flow backing off and others taking over its unused capacity becomes clearer, and it is possible to see if more than one flow backs off at the same time (indicating that both got hit by AQM).

There are also sampling artefacts apparent in these graphs as rapid oscillations around a mean value.  You might want to look into ways to eliminate or otherwise account for those.

> We're currently working on the following:
> 
> 1. plots for the actual number of marks/drops per time interval for COBALT, CoDel, and PIE.
> 2. zoomed in plots on small time intervals to show the dynamic behavior of the algorithm.
> 3. a file showing the timestamp of each drop.

I await these with interest.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

>> We're currently working on the following:
>> 
>> 1. plots for the actual number of marks/drops per time interval for COBALT, CoDel, and PIE.
>> 2. zoomed in plots on small time intervals to show the dynamic behavior of the algorithm.
>> 3. a file showing the timestamp of each drop.
> 
> I await these with interest.

I noticed that some progress has been made here already, in particular I can now see cwnd graphs which make a very interesting datapoint when directly compared with the throughput and queue-occupancy graphs.  It's now definitely clear that the senders are using TCP Reno or some close variant thereof.

In fact, the three graphs are mutually inconsistent.  Aside from the sharp cwnd reduction events, the cwnd of all the flows increases roughly linearly over time, but the throughput remains flat while the queue is almost always empty (for Codel and COBALT).  This can only be explained if there's a hidden queue at the bottleneck, perhaps associated with the simulated NIC immediately downstream of the AQM.

I would suggest checking the simulation setup carefully for hidden queues of this sort.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

Hi Jonathan,

Yes, we are using TCP NewReno at the moment.

There was a typo in labeling the Y-axis; instead of "Throughput" it should
be "Link Utilization" in the following graphs (now corrected):

https://github.com/Daipu/COBALT/wiki/Light-Traffic

throughput graphs for the same scenario are here:

https://github.com/Daipu/COBALT/wiki/Instantaneous-throughput:-Light-Traffic

and cwnd graphs here:

https://github.com/Daipu/COBALT/wiki/cwnd-plots:-Light-traffic

So, now what we see is that although queue occupancy is under control and
link remains fully utilized, the senders cwnd gets synchronized in one
scenario (only when packet size is 1000 bytes and with COBALT). For all
other cases, there is no synchronization of cwnd (including COBALT with
packet size 1500 bytes).

By hidden queues, do you mean the NIC buffers? ns-3 has a Linux-like
traffic control wherein the packets dequeued by a queue discipline are
enqueued into NIC buffer.

The tasks that we're currently working on are listed here:

https://github.com/Daipu/COBALT/issues/1

Thanks a lot for your help. We really appreciate it.

Regards,
Jendaipou Palmei
Shefali Gupta

On Thu, Dec 6, 2018 at 11:06 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> >> We're currently working on the following:
> >>
> >> 1. plots for the actual number of marks/drops per time interval for
> COBALT, CoDel, and PIE.
> >> 2. zoomed in plots on small time intervals to show the dynamic behavior
> of the algorithm.
> >> 3. a file showing the timestamp of each drop.
> >
> > I await these with interest.
>
> I noticed that some progress has been made here already, in particular I
> can now see cwnd graphs which make a very interesting datapoint when
> directly compared with the throughput and queue-occupancy graphs.  It's now
> definitely clear that the senders are using TCP Reno or some close variant
> thereof.
>
> In fact, the three graphs are mutually inconsistent.  Aside from the sharp
> cwnd reduction events, the cwnd of all the flows increases roughly linearly
> over time, but the throughput remains flat while the queue is almost always
> empty (for Codel and COBALT).  This can only be explained if there's a
> hidden queue at the bottleneck, perhaps associated with the simulated NIC
> immediately downstream of the AQM.
>
> I would suggest checking the simulation setup carefully for hidden queues
> of this sort.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 3574 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 9 Dec, 2018, at 10:37 am, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
> 
> By hidden queues, do you mean the NIC buffers? ns-3 has a Linux-like traffic control wherein the packets dequeued by a queue discipline are enqueued into NIC buffer.

That's right.  Linux now uses BQL, which (given compatible NIC drivers) limits the number of packets in the NIC buffers to a very small value - much smaller than is evident from your data.  If you were to measure the end-to-end RTT of each, I'm certain you would see this effect dominating the mere 50ms latency you're trying to model.

Ideally, AQM would applied to the hardware queue anyway.  For simulation purposes, I recommend reducing the NIC buffer on the bottleneck interface to 1 packet, so that the simulated AQM's effects are easiest to measure.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jendaipou Palmei]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

Hello Jonathan,

Thanks for your feedback.

As suggested, we changed the NIC buffer size to 1 packet for the simulation
and also tried these different buffer sizes: 10, 50 and 75.

The default NIC buffer size in ns-3 is 100 packets.

Additionally, we also enabled BQL and tried.

We see that the link utilization gets significantly affected when we keep
the NIC buffer size small.

The results are put up on the following link:

https://github.com/Daipu/COBALT/wiki/Link-Utilization-Graphs-with-Different-NetDeviceQueue-size

Thanks and Regards,
Jendaipou Palmei
Shefali Gupta

On Sun, Dec 9, 2018 at 6:51 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 9 Dec, 2018, at 10:37 am, Jendaipou Palmei <jendaipoupalmei@gmail.com>
> wrote:
> >
> > By hidden queues, do you mean the NIC buffers? ns-3 has a Linux-like
> traffic control wherein the packets dequeued by a queue discipline are
> enqueued into NIC buffer.
>
> That's right.  Linux now uses BQL, which (given compatible NIC drivers)
> limits the number of packets in the NIC buffers to a very small value -
> much smaller than is evident from your data.  If you were to measure the
> end-to-end RTT of each, I'm certain you would see this effect dominating
> the mere 50ms latency you're trying to model.
>
> Ideally, AQM would applied to the hardware queue anyway.  For simulation
> purposes, I recommend reducing the NIC buffer on the bottleneck interface
> to 1 packet, so that the simulated AQM's effects are easiest to measure.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 2246 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 10 Dec, 2018, at 2:30 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com> wrote:
> 
> As suggested, we changed the NIC buffer size to 1 packet for the simulation and also tried these different buffer sizes: 10, 50 and 75.
> 
> The default NIC buffer size in ns-3 is 100 packets.
> 
> Additionally, we also enabled BQL and tried.
> 
> We see that the link utilization gets significantly affected when we keep the NIC buffer size small.

Yes, that's what I'd expect to see from Reno-type congestion control, and is one good reason why alternatives to Reno were developed (eg. Compound, CUBIC, BBR).  You may wish to explore what happens with Compound and CUBIC, once your basic measurement methodology has matured.

I would suggest using BQL, since it's available and represents a realistic deployment.

If you were to add TCP (or parallel UDP/ICMP) RTT measurements, you'd see that the peak latency was correspondingly improved by removing the dumb FIFO hidden within the NIC.  I estimate that a 100-packet buffer accounts for about 120ms of latency at 10Mbps, which should definitely be visible on such a graph (being almost 250% of your baseline 50ms latency).

Since latency is the main point of adding AQM, I'm a little surprised that you haven't already produced graphs of that sort.  They would have identified this problem much earlier.

At present you only have COBALT graphs with the small NIC buffer.  For a fair comparison, Codel and PIE graphs should be (re-)produced with the same conditions.  The older graphs made with the large NIC buffer are potentially misleading, especially with respect to throughput.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Shefali Gupta]

[-- Attachment #1: Type: text/plain, Size: 2468 bytes --]

Hello Jonathan,

Thanks for your feedback.

As suggested, we have produced CoDel and PIE graphs with small NIC buffer
and uploaded the corresponding graphs.

Link:
https://github.com/Daipu/COBALT/wiki/Link-Utilization-Graphs-with-Different-NetDeviceQueue-size

We have also uploaded one way end-to-end dela*y* graphs in Light traffic
scenario for CoDel, COBALT and PIE.
Link: https://github.com/Daipu/COBALT/wiki/End-To-End-Delay-Graphs

Thanks a lot for your help. We really appreciate it.

Regards,
Shefali Gupta
Jendaipou Palme

On Mon, Dec 10, 2018 at 8:45 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 10 Dec, 2018, at 2:30 pm, Jendaipou Palmei <jendaipoupalmei@gmail.com>
> wrote:
> >
> > As suggested, we changed the NIC buffer size to 1 packet for the
> simulation and also tried these different buffer sizes: 10, 50 and 75.
> >
> > The default NIC buffer size in ns-3 is 100 packets.
> >
> > Additionally, we also enabled BQL and tried.
> >
> > We see that the link utilization gets significantly affected when we
> keep the NIC buffer size small.
>
> Yes, that's what I'd expect to see from Reno-type congestion control, and
> is one good reason why alternatives to Reno were developed (eg. Compound,
> CUBIC, BBR).  You may wish to explore what happens with Compound and CUBIC,
> once your basic measurement methodology has matured.
>
> I would suggest using BQL, since it's available and represents a realistic
> deployment.
>
> If you were to add TCP (or parallel UDP/ICMP) RTT measurements, you'd see
> that the peak latency was correspondingly improved by removing the dumb
> FIFO hidden within the NIC.  I estimate that a 100-packet buffer accounts
> for about 120ms of latency at 10Mbps, which should definitely be visible on
> such a graph (being almost 250% of your baseline 50ms latency).
>
> Since latency is the main point of adding AQM, I'm a little surprised that
> you haven't already produced graphs of that sort.  They would have
> identified this problem much earlier.
>
> At present you only have COBALT graphs with the small NIC buffer.  For a
> fair comparison, Codel and PIE graphs should be (re-)produced with the same
> conditions.  The older graphs made with the large NIC buffer are
> potentially misleading, especially with respect to throughput.
>
>  - Jonathan Morton
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 3859 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 3659 bytes --]

Thank you for doing this. I'm now unconvinced the BQL emulation in NS3 is
accurate.

Loved the combined graphs! While it is important to capture that initial
load spike and indeed, draw it out in the paper, being able to see a bit
more detail in steady state would be good. So showing T-0 -> T-3 and T-3
forward would let you use different scales for each.

I'd kind of like to take a step back and try to construct a paper out of
this that could be published at
usenix or acm next year. It's getting towards the holidays but would y'all
(and your advisor(s)) be available to meet via videoconference sometime
next week? I'm in california, jonathon - somewhere in europe - so that
might be hard.




On Sat, Dec 15, 2018 at 11:06 AM Shefali Gupta <shefaligups11@gmail.com>
wrote:

> Hello Jonathan,
>
> Thanks for your feedback.
>
> As suggested, we have produced CoDel and PIE graphs with small NIC buffer
> and uploaded the corresponding graphs.
>
> Link:
> https://github.com/Daipu/COBALT/wiki/Link-Utilization-Graphs-with-Different-NetDeviceQueue-size
>
> We have also uploaded one way end-to-end dela*y* graphs in Light traffic
> scenario for CoDel, COBALT and PIE.
> Link: https://github.com/Daipu/COBALT/wiki/End-To-End-Delay-Graphs
>
> Thanks a lot for your help. We really appreciate it.
>
> Regards,
> Shefali Gupta
> Jendaipou Palme
>
> On Mon, Dec 10, 2018 at 8:45 PM Jonathan Morton <chromatix99@gmail.com>
> wrote:
>
>> > On 10 Dec, 2018, at 2:30 pm, Jendaipou Palmei <
>> jendaipoupalmei@gmail.com> wrote:
>> >
>> > As suggested, we changed the NIC buffer size to 1 packet for the
>> simulation and also tried these different buffer sizes: 10, 50 and 75.
>> >
>> > The default NIC buffer size in ns-3 is 100 packets.
>> >
>> > Additionally, we also enabled BQL and tried.
>> >
>> > We see that the link utilization gets significantly affected when we
>> keep the NIC buffer size small.
>>
>> Yes, that's what I'd expect to see from Reno-type congestion control, and
>> is one good reason why alternatives to Reno were developed (eg. Compound,
>> CUBIC, BBR).  You may wish to explore what happens with Compound and CUBIC,
>> once your basic measurement methodology has matured.
>>
>> I would suggest using BQL, since it's available and represents a
>> realistic deployment.
>>
>> If you were to add TCP (or parallel UDP/ICMP) RTT measurements, you'd see
>> that the peak latency was correspondingly improved by removing the dumb
>> FIFO hidden within the NIC.  I estimate that a 100-packet buffer accounts
>> for about 120ms of latency at 10Mbps, which should definitely be visible on
>> such a graph (being almost 250% of your baseline 50ms latency).
>>
>> Since latency is the main point of adding AQM, I'm a little surprised
>> that you haven't already produced graphs of that sort.  They would have
>> identified this problem much earlier.
>>
>> At present you only have COBALT graphs with the small NIC buffer.  For a
>> fair comparison, Codel and PIE graphs should be (re-)produced with the same
>> conditions.  The older graphs made with the large NIC buffer are
>> potentially misleading, especially with respect to throughput.
>>
>>  - Jonathan Morton
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>


-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

[-- Attachment #2: Type: text/html, Size: 5763 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Shefali Gupta]

[-- Attachment #1: Type: text/plain, Size: 4579 bytes --]

Hi Dave,

Thanks for the feedback.

We will check with the maintainer of traffic-control module in ns-3 about
the correctness of BQL, and also try to obtain plots that show the steady
state behaviour as you suggested.

In the meantime, we have added the following plots to our wiki:

1. Number of packet drops per time interval

Link:
https://github.com/Daipu/COBALT/wiki/Proactive-Drop-Count-per-time-interval-graphs

2. A file showing the timestamp of each drop

Link: https://github.com/Daipu/COBALT/wiki/Drop-Timestamp-Files

I believe our advisor might have already communicated with you and Jonathan
for the timings of the videoconference.

Thanks and Regards,
Shefali Gupta
Jendaipou Palmei

On Sun, Dec 16, 2018 at 1:40 AM Dave Taht <dave.taht@gmail.com> wrote:

> Thank you for doing this. I'm now unconvinced the BQL emulation in NS3 is
> accurate.
>
> Loved the combined graphs! While it is important to capture that initial
> load spike and indeed, draw it out in the paper, being able to see a bit
> more detail in steady state would be good. So showing T-0 -> T-3 and T-3
> forward would let you use different scales for each.
>
> I'd kind of like to take a step back and try to construct a paper out of
> this that could be published at
> usenix or acm next year. It's getting towards the holidays but would y'all
> (and your advisor(s)) be available to meet via videoconference sometime
> next week? I'm in california, jonathon - somewhere in europe - so that
> might be hard.
>
>
>
>
> On Sat, Dec 15, 2018 at 11:06 AM Shefali Gupta <shefaligups11@gmail.com>
> wrote:
>
>> Hello Jonathan,
>>
>> Thanks for your feedback.
>>
>> As suggested, we have produced CoDel and PIE graphs with small NIC buffer
>> and uploaded the corresponding graphs.
>>
>> Link:
>> https://github.com/Daipu/COBALT/wiki/Link-Utilization-Graphs-with-Different-NetDeviceQueue-size
>>
>> We have also uploaded one way end-to-end dela*y* graphs in Light traffic
>> scenario for CoDel, COBALT and PIE.
>> Link: https://github.com/Daipu/COBALT/wiki/End-To-End-Delay-Graphs
>>
>> Thanks a lot for your help. We really appreciate it.
>>
>> Regards,
>> Shefali Gupta
>> Jendaipou Palme
>>
>> On Mon, Dec 10, 2018 at 8:45 PM Jonathan Morton <chromatix99@gmail.com>
>> wrote:
>>
>>> > On 10 Dec, 2018, at 2:30 pm, Jendaipou Palmei <
>>> jendaipoupalmei@gmail.com> wrote:
>>> >
>>> > As suggested, we changed the NIC buffer size to 1 packet for the
>>> simulation and also tried these different buffer sizes: 10, 50 and 75.
>>> >
>>> > The default NIC buffer size in ns-3 is 100 packets.
>>> >
>>> > Additionally, we also enabled BQL and tried.
>>> >
>>> > We see that the link utilization gets significantly affected when we
>>> keep the NIC buffer size small.
>>>
>>> Yes, that's what I'd expect to see from Reno-type congestion control,
>>> and is one good reason why alternatives to Reno were developed (eg.
>>> Compound, CUBIC, BBR).  You may wish to explore what happens with Compound
>>> and CUBIC, once your basic measurement methodology has matured.
>>>
>>> I would suggest using BQL, since it's available and represents a
>>> realistic deployment.
>>>
>>> If you were to add TCP (or parallel UDP/ICMP) RTT measurements, you'd
>>> see that the peak latency was correspondingly improved by removing the dumb
>>> FIFO hidden within the NIC.  I estimate that a 100-packet buffer accounts
>>> for about 120ms of latency at 10Mbps, which should definitely be visible on
>>> such a graph (being almost 250% of your baseline 50ms latency).
>>>
>>> Since latency is the main point of adding AQM, I'm a little surprised
>>> that you haven't already produced graphs of that sort.  They would have
>>> identified this problem much earlier.
>>>
>>> At present you only have COBALT graphs with the small NIC buffer.  For a
>>> fair comparison, Codel and PIE graphs should be (re-)produced with the same
>>> conditions.  The older graphs made with the large NIC buffer are
>>> potentially misleading, especially with respect to throughput.
>>>
>>>  - Jonathan Morton
>>>
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
>

[-- Attachment #2: Type: text/html, Size: 7488 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

> On 21 Dec, 2018, at 12:37 pm, Shefali Gupta <shefaligups11@gmail.com> wrote:
> 
> In the meantime, we have added the following plots to our wiki:
> 
> 1. Number of packet drops per time interval
> 
> Link: https://github.com/Daipu/COBALT/wiki/Proactive-Drop-Count-per-time-interval-graphs
> 
> 2. A file showing the timestamp of each drop
> 
> Link: https://github.com/Daipu/COBALT/wiki/Drop-Timestamp-Files

Interesting - but very very odd.  COBALT is apparently not behaving anything like as designed.

As an immediate point, the first drop occurs 2 whole seconds later than either Codel or PIE, which is completely at odds with the excellent control of the initial slow-start phase seen in other graphs.

I took the timestamp files, dumped them into a spreadsheet column each, and plotted 1/(T(n)-T(n-1)) for each drop event against the raw timestamps.  This yields a view of the instantaneous drop frequency.  Because PIE sometimes drops multiple packets at once, yielding very high values here, I truncated the Y-axis at 1000 Hz.

The Codel implementation shows the expected behaviour of a linear ramp of drop frequency over time during its dropping phases.  The COBALT implementation does not.  Indeed, during a single phase, COBALT's dropping frequency appears to vary chaotically, as if it is implementing random-drop instead of timed drops.



 - Jonathan Morton


[-- Attachment #2.1: Type: text/html, Size: 2295 bytes --]

[-- Attachment #2.2: PastedGraphic-1.png --]
[-- Type: image/png, Size: 92460 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Shefali Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 2105 bytes --]

Hi Jonathan,

Thanks for your inputs.

We re-looked into the COBALT implementation to understand why it drops the
first packet later than CoDel.

There was a bug in the data that was collected in 'drop timestamp files'.
We tried using a different approach to store packet drop times, and now we
see that COBALT indeed drops the first packet prior to CoDel's first packet
drop (image below). So the issue was that our previous approach of storing
the packet drop times in a file was not correct.

Let us know your opinion.
[image: image.png]
Regards,
Shefali Gupta
Jendaipou Palmei

On Fri, Dec 21, 2018 at 6:18 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> On 21 Dec, 2018, at 12:37 pm, Shefali Gupta <shefaligups11@gmail.com>
> wrote:
>
> In the meantime, we have added the following plots to our wiki:
>
> 1. Number of packet drops per time interval
>
> Link:
> https://github.com/Daipu/COBALT/wiki/Proactive-Drop-Count-per-time-interval-graphs
>
> 2. A file showing the timestamp of each drop
>
> Link: https://github.com/Daipu/COBALT/wiki/Drop-Timestamp-Files
>
>
> Interesting - but very very odd.  COBALT is apparently not behaving
> anything like as designed.
>
> As an immediate point, the first drop occurs 2 whole seconds later than
> either Codel or PIE, which is completely at odds with the excellent control
> of the initial slow-start phase seen in other graphs.
>
> I took the timestamp files, dumped them into a spreadsheet column each,
> and plotted 1/(T(n)-T(n-1)) for each drop event against the raw
> timestamps.  This yields a view of the instantaneous drop frequency.
> Because PIE sometimes drops multiple packets at once, yielding very high
> values here, I truncated the Y-axis at 1000 Hz.
>
> The Codel implementation shows the expected behaviour of a linear ramp of
> drop frequency over time during its dropping phases.  The COBALT
> implementation does not.  Indeed, during a single phase, COBALT's dropping
> frequency appears to vary chaotically, as if it is implementing random-drop
> instead of timed drops.
>
> jendaipoupalmei@gmail.com
>
>  - Jonathan Morton
>
>

[-- Attachment #1.2: Type: text/html, Size: 3344 bytes --]

[-- Attachment #2: PastedGraphic-1.png --]
[-- Type: image/png, Size: 92460 bytes --]

[-- Attachment #3: image.png --]
[-- Type: image/png, Size: 53575 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 21 Jan, 2019, at 1:35 pm, Shefali Gupta <shefaligups11@gmail.com> wrote:
> 
> We re-looked into the COBALT implementation to understand why it drops the first packet later than CoDel.
> 
> There was a bug in the data that was collected in 'drop timestamp files'. We tried using a different approach to store packet drop times, and now we see that COBALT indeed drops the first packet prior to CoDel's first packet drop (image below). So the issue was that our previous approach of storing the packet drop times in a file was not correct.
> 
> Let us know your opinion.

Okay, good catch.

But the more serious problem is with the pattern of drops, which presently looks much more like BLUE activity (random) than Codel (ramping frequency).  That seems to be unchanged in your new graph.

Have you made any progress towards finding out why the queue is apparently too short?  Perhaps log the actual length of the queue when overflow drops occur.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Shefali Gupta]

[-- Attachment #1.1: Type: text/plain, Size: 1293 bytes --]

Hi Jonathan,

We believe we have spotted the issue now. The new plot is attached below.

Does it look as expected?
[image: Updated_Graphs.png]

Thanks and regards,
Shefali Gupta
Jendaipou Palmei

On Mon, Jan 21, 2019 at 6:27 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 21 Jan, 2019, at 1:35 pm, Shefali Gupta <shefaligups11@gmail.com>
> wrote:
> >
> > We re-looked into the COBALT implementation to understand why it drops
> the first packet later than CoDel.
> >
> > There was a bug in the data that was collected in 'drop timestamp
> files'. We tried using a different approach to store packet drop times, and
> now we see that COBALT indeed drops the first packet prior to CoDel's first
> packet drop (image below). So the issue was that our previous approach of
> storing the packet drop times in a file was not correct.
> >
> > Let us know your opinion.
>
> Okay, good catch.
>
> But the more serious problem is with the pattern of drops, which presently
> looks much more like BLUE activity (random) than Codel (ramping
> frequency).  That seems to be unchanged in your new graph.
>
> Have you made any progress towards finding out why the queue is apparently
> too short?  Perhaps log the actual length of the queue when overflow drops
> occur.
>
>  - Jonathan Morton
>
>

[-- Attachment #1.2: Type: text/html, Size: 1949 bytes --]

[-- Attachment #2: Updated_Graphs.png --]
[-- Type: image/png, Size: 49922 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 23 Jan, 2019, at 6:19 pm, Shefali Gupta <shefaligups11@gmail.com> wrote:
> 
> We believe we have spotted the issue now. The new plot is attached below.

That does look considerably improved.  What was the problem in the end?

Now would be a good time to regenerate all the test graphs, and make sure there are results for each test for each qdisc tested.  Ideally all graphs should be generated from the same run(s), to ensure their data is mutually consistent.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Shefali Gupta]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]

Hi Jonathan,

Thanks!

First, we noticed that cobalt_cache_init method was not implemented in ns-3
COBALT, so we implemented it (but this wasn't the main reason for the wrong
behaviour).

Later, we found that the data types used in the implementation of COBALT in
ns-3  were different from those used in Linux.

e.g., uint32_t was used instead of int64_t for the variables with
datatype ktime_t in Linux.

Results improved considerably after these changes.

We will update the wiki with latest results.

Regards,
Shefali Gupta
Jendaipou Palmei

On Wed 23 Jan 2019, 9:53 p.m. Jonathan Morton <chromatix99@gmail.com wrote:

> > On 23 Jan, 2019, at 6:19 pm, Shefali Gupta <shefaligups11@gmail.com>
> wrote:
> >
> > We believe we have spotted the issue now. The new plot is attached below.
>
> That does look considerably improved.  What was the problem in the end?
>
> Now would be a good time to regenerate all the test graphs, and make sure
> there are results for each test for each qdisc tested.  Ideally all graphs
> should be generated from the same run(s), to ensure their data is mutually
> consistent.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 2712 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Shefali Gupta]

[-- Attachment #1: Type: text/plain, Size: 1567 bytes --]

Hello Jonathan,

We have updated all the graphs with updated COBALT code and BQL enabled.

Wiki Link:
https://github.com/Daipu/COBALT/wiki

Diff for COBALT code:
https://github.com/Daipu/COBALT/commit/624d0c761b9d1de99499918cb2078195f8b796b5

Thanks and Regards,
Shefali Gupta
Jendaipou Palmei

On Wed 23 Jan 2019, 10:57 p.m. Shefali Gupta <shefaligups11@gmail.com wrote:

> Hi Jonathan,
>
> Thanks!
>
> First, we noticed that cobalt_cache_init method was not implemented in
> ns-3 COBALT, so we implemented it (but this wasn't the main reason for the
> wrong behaviour).
>
> Later, we found that the data types used in the implementation of COBALT
> in ns-3  were different from those used in Linux.
>
> e.g., uint32_t was used instead of int64_t for the variables with
> datatype ktime_t in Linux.
>
> Results improved considerably after these changes.
>
> We will update the wiki with latest results.
>
> Regards,
> Shefali Gupta
> Jendaipou Palmei
>
> On Wed 23 Jan 2019, 9:53 p.m. Jonathan Morton <chromatix99@gmail.com
> wrote:
>
>> > On 23 Jan, 2019, at 6:19 pm, Shefali Gupta <shefaligups11@gmail.com>
>> wrote:
>> >
>> > We believe we have spotted the issue now. The new plot is attached
>> below.
>>
>> That does look considerably improved.  What was the problem in the end?
>>
>> Now would be a good time to regenerate all the test graphs, and make sure
>> there are results for each test for each qdisc tested.  Ideally all graphs
>> should be generated from the same run(s), to ensure their data is mutually
>> consistent.
>>
>>  - Jonathan Morton
>>
>>

[-- Attachment #2: Type: text/html, Size: 4555 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 25 Jan, 2019, at 10:35 am, Shefali Gupta <shefaligups11@gmail.com> wrote:
> 
> We have updated all the graphs with updated COBALT code and BQL enabled.

The graphs available so far do show a big improvement in behaviour.  Could you also add comparative data for Codel and PIE in the Heavy and Mix scenarios?  It seems reasonable to also generate drop-event logs for those scenarios.

 - Jonathan Morton

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Shefali Gupta]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

Hello Jonathan,

We have added CoDel and PIE graphs for Heavy and Mix scenarios.
Link:
https://github.com/Daipu/COBALT/wiki/Heavy-Traffic
https://github.com/Daipu/COBALT/wiki/Mix-Traffic

Also we have added drop-event logs for the same scenarios.
Link:
https://github.com/Daipu/COBALT/wiki/Drop-Timestamp-Files

Thanks and Regards,
Shefali Gupta
Jendaipou Palmei

On Fri 25 Jan 2019, 2:46 p.m. Jonathan Morton <chromatix99@gmail.com wrote:

> > On 25 Jan, 2019, at 10:35 am, Shefali Gupta <shefaligups11@gmail.com>
> wrote:
> >
> > We have updated all the graphs with updated COBALT code and BQL enabled.
>
> The graphs available so far do show a big improvement in behaviour.  Could
> you also add comparative data for Codel and PIE in the Heavy and Mix
> scenarios?  It seems reasonable to also generate drop-event logs for those
> scenarios.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 2580 bytes --]

Re: [Cake] COBALT implementation in ns-3 with results under different traffic scenarios

[Jonathan Morton]

> On 25 Jan, 2019, at 4:48 pm, Shefali Gupta <shefaligups11@gmail.com> wrote:
> 
> We have added CoDel and PIE graphs for Heavy and Mix scenarios.

Excellent!

Here we can clearly see the difference that my approach to recovery from the dropping state makes; COBALT maintains much better control of the queue when heavily loaded.  The difference is much smaller with the light traffic, because both Codel and COBALT have enough time to recover completely to the quiescent state in that case.

This is definitely worth writing about in a paper.  You'll probably want my insight into the precise nature of the difference.  For that purpose, I'll plot out the drop logs as frequency charts, which should give some clue as to whether BLUE is in effect as well as Codel.

 - Jonathan Morton