[Cerowrt-devel] active sensing queue management

[Cerowrt-devel] active sensing queue management

[Dave Taht]

http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf

gargoyle's qos system follows a similar approach, using htb + sfq, and
a short ttl udp flow.

Doing this sort of measured, then floating the rate control with
"cake" would be fairly easy (although it tends to be a bit more
compute intensive not being on a fast path)

What is sort of missing here is trying to figure out which side of the
bottleneck is the bottleneck (up or down).

-- 
Dave Täht
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Sebastian Moeller]

Hi Dave,


On Jun 10, 2015, at 21:53 , Dave Taht <dave.taht@gmail.com> wrote:

> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf
> 
> gargoyle's qos system follows a similar approach, using htb + sfq, and
> a short ttl udp flow.
> 
> Doing this sort of measured, then floating the rate control with
> "cake" would be fairly easy (although it tends to be a bit more
> compute intensive not being on a fast path)
> 
> What is sort of missing here is trying to figure out which side of the
> bottleneck is the bottleneck (up or down).

	Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned. In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.". At least this is what I get from their short description in glossing over the paper.
	Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers. But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs "on steroids", with higher resolution than milliseconds, but for buffer bloat detection ms resolution would probably be sufficient anyways). Currently, I hear that ISP equipment will not treat ICMP requests with priority though.
	Also I am confused what they actually simulated: “The modems and CMTS were equipped with ASQM, CoDel and PIE,” and “However, the problem pop- ularly called bufferbloat can move about among many queues some of which are resistant to traditional AQM such as Layer 2 MAC protocols used in cable/DSL links. We call this problem bufferbloat displacement.” seem to be slightly at odds. If modems and CTMS have decent AQMs all they need to do is not stuff their sub-IP layer queuesand be done with it. The way I understood the cable labs PIE story, they intended to do exactly that, so at least the “buffer displacement” remedy by ASQM reads a bit like a straw man argument. But as I am a) not of the cs field, and b) only glossed over the paper, most likely I am missing something important that is clearly in the paper...

Best Regards
	Sebastian

> 
> -- 
> Dave Täht
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Daniel Havey]

Hmmm, maybe I can help clarify.  Bufferbloat occurs in the slowest
queue on the path.  This is because the other queues are faster and
will drain.  AQM algorithms work only if they are placed where the
packets pile up (e.g. the slowest queue in the path).  This is
documented in Kathy and Van's CoDel paper.

This is usually all well and good because we know where the bottleneck
(the slowest queue in the path) is.  It is the IP layer in the modem
where the ISP implements their rate limiter.  That is why algorithms
such as PIE and CoDel are implemented in the IP layer on the modem.

Suppose the full committed rate of the token bucket rate limiter is 8
Mbps.  This means that the queue at the IP layer in the modem is
capable of emitting packets at 8 Mbps sustained rate.  The problem
occurs during peak hours when the ISP is not providing the full
committed rate of 8 Mbps or that some queue in the system (probably in
the access link) is providing something less than 8 Mbps (say for sake
of discussion that the number is 7.5 Mbps).

We know that (see Kathy and Van's paper) that AQM algorithms only work
when they are placed at the slowest queue.  However, the AQM is placed
at the queue that is capable of providing 8 Mbps and this is not the
slowest queue.  The AQM algorithm will not work in these conditions.

This is what is shown in the paper where the CoDel and PIE performance
goes to hell in a handbasket.  The ASQM algorithm is designed to
address this problem.





On Wed, Jun 10, 2015 at 1:54 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Dave,
>
>
> On Jun 10, 2015, at 21:53 , Dave Taht <dave.taht@gmail.com> wrote:
>
>> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf
>>
>> gargoyle's qos system follows a similar approach, using htb + sfq, and
>> a short ttl udp flow.
>>
>> Doing this sort of measured, then floating the rate control with
>> "cake" would be fairly easy (although it tends to be a bit more
>> compute intensive not being on a fast path)
>>
>> What is sort of missing here is trying to figure out which side of the
>> bottleneck is the bottleneck (up or down).
>

Yeah, we never did figure out how to separate the up from the
downlink.  However, we just consider the access link as a whole (up +
down) and mark/drop according to ratios of queuing time.  Overall it
seems to work well, but, we never did a mathematical analysis.  Kind
of like saying it's not a "bug", it is a feature.  And it this case it
is true since both sides can experience bloat.


>         Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned. In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.". At least this is what I get from their short description in glossing over the paper.
>         Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers. But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs "on steroids", with higher resolution than milliseconds, but for buffer bloat detection ms resolution would probably be sufficient anyways). Currently, I hear that ISP equipment will not treat ICMP requests with priority though.

Not exactly.  We thought this through for some time and considered
many angles.  Each method has its advantages and disadvantages.

We decided not to use ICMP at all because of the reasons you stated
above.  We also decided not to use a "reflector" although as you said
it would allow us to separate upload queue time from download.  We
decided not to use this because it would be difficult to get ISPs to
do this.

Are final choice for the paper was "magic" IP packets.  This consists
of an IP packet header and the timestamp.  The IP packet is "self
addressed" and we trick the iptables to emit the packet on the correct
interface.  This packet will be returned to us as soon as it reaches
another IP layer (typically at the CMTS).

Here's a quick summary:
ICMP -- Simple, but, needs the ISP's cooperation (good luck :)
Reflector  -- Separates upload queue time from download queue time,
but, requires the ISP to cooperate and to build something for us.
(good luck :)
Magic IP packets -- Requires nothing from the ISP (YaY!  We have a
winner!), but, is a little more complex.


>         Also I am confused what they actually simulated: “The modems and CMTS were equipped with ASQM, CoDel and PIE,” and “However, the problem pop- ularly called bufferbloat can move about among many queues some of which are resistant to traditional AQM such as Layer 2 MAC protocols used in cable/DSL links. We call this problem bufferbloat displacement.” seem to be slightly at odds. If modems and CTMS have decent AQMs all they need to do is not stuff their sub-IP layer queuesand be done with it. The way I understood the cable labs PIE story, they intended to do exactly that, so at least the “buffer displacement” remedy by ASQM reads a bit like a straw man argument. But as I am a) not of the cs field, and b) only glossed over the paper, most likely I am missing something important that is clearly in the paper...

Good point!  However, once again it's not quite that simple.  Queues
are necessary to absorb short term variations in packet arrival rate
(or bursts).  The queue required for any flow is given by the
bandwidth delay product.  Since we don't know the delay we can't
predict the queue size in advance.  What I'm getting at is the
equipment manufacturers aren't putting in humongous queues because
they are stupid, they are putting them there because in some cases you
might really need that large of a queue.

Statically sizing the queues is not the answer.  Managing the size of
the queue with an algorithm is the answer.  :)



>
> Best Regards
>         Sebastian
>
>>
>> --
>> Dave Täht
>> What will it take to vastly improve wifi for everyone?
>> https://plus.google.com/u/0/explore/makewififast
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Sebastian Moeller]

Hi Daniel,

thanks for the clarifications.

On Jun 11, 2015, at 02:10 , Daniel Havey <dhavey@gmail.com> wrote:

> Hmmm, maybe I can help clarify.  Bufferbloat occurs in the slowest
> queue on the path.  This is because the other queues are faster and
> will drain.  AQM algorithms work only if they are placed where the
> packets pile up (e.g. the slowest queue in the path).  This is
> documented in Kathy and Van's CoDel paper.

	I am with you so far.

> 
> This is usually all well and good because we know where the bottleneck
> (the slowest queue in the path) is.  It is the IP layer in the modem
> where the ISP implements their rate limiter.  That is why algorithms
> such as PIE and CoDel are implemented in the IP layer on the modem.

	Okay.

> 
> Suppose the full committed rate of the token bucket rate limiter is 8
> Mbps.  This means that the queue at the IP layer in the modem is
> capable of emitting packets at 8 Mbps sustained rate.  The problem
> occurs during peak hours when the ISP is not providing the full
> committed rate of 8 Mbps or that some queue in the system (probably in
> the access link) is providing something less than 8 Mbps (say for sake
> of discussion that the number is 7.5 Mbps).
> 
> We know that (see Kathy and Van's paper) that AQM algorithms only work
> when they are placed at the slowest queue.  However, the AQM is placed
> at the queue that is capable of providing 8 Mbps and this is not the
> slowest queue.  The AQM algorithm will not work in these conditions.
> 
> This is what is shown in the paper where the CoDel and PIE performance
> goes to hell in a handbasket.  The ASQM algorithm is designed to
> address this problem.

	Except that DOCSIS 3.1 pie in the modem does not work that way. As I understand http://www.cablelabs.com/wp-content/uploads/2014/06/DOCSIS-AQM_May2014.pdf section 3.2 MAP PROCESSING REQUIREMENTS, cable-modem pie will not stuff data into the lower layers until it has received a grant to actually send that data, hence no uncontrolled sub-IP layer buffer bloat possible (unless there are severe RF issues that take out data during transmission). So at least for the upstream direction docsis 3.1 pie will not suffer from buffer displacement, if I understand the cable labs white paper correctly. Your solution still might be a valuable add-on to control the downstream buffer bloat in addition. 
	I also believe that free in france had a modified DSL driver for their box that made sure sub-IP buffering was bound to a low number of packets as well, so no displaced buffers there as well. Now it seems that this solution for DSL was unique so far and has not caught on, but once docsis3.1 modems hit the market upstream PIE in the modems will be reality.

> 
> 
> 
> 
> 
> On Wed, Jun 10, 2015 at 1:54 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Dave,
>> 
>> 
>> On Jun 10, 2015, at 21:53 , Dave Taht <dave.taht@gmail.com> wrote:
>> 
>>> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf
>>> 
>>> gargoyle's qos system follows a similar approach, using htb + sfq, and
>>> a short ttl udp flow.
>>> 
>>> Doing this sort of measured, then floating the rate control with
>>> "cake" would be fairly easy (although it tends to be a bit more
>>> compute intensive not being on a fast path)
>>> 
>>> What is sort of missing here is trying to figure out which side of the
>>> bottleneck is the bottleneck (up or down).
>> 
> 
> Yeah, we never did figure out how to separate the up from the
> downlink.  However, we just consider the access link as a whole (up +
> down) and mark/drop according to ratios of queuing time.  

	This is a bit sad; why reduce say the effective uplink bandwidth if only the downstream is contended? Not that I have a good alternative solution that will not require help from outside boxes.

> Overall it
> seems to work well, but, we never did a mathematical analysis.  Kind
> of like saying it's not a "bug", it is a feature.  And it this case it
> is true since both sides can experience bloat.

	Yes, but you only want to throttle traffic on the congested leg of the link, otherwise bandwidth efficiency goes to hell if you look at bi-direction link-saturating traffic.

> 
> 
>>        Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned. In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.". At least this is what I get from their short description in glossing over the paper.
>>        Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers. But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs "on steroids", with higher resolution than milliseconds, but for buffer bloat detection ms resolution would probably be sufficient anyways). Currently, I hear that ISP equipment will not treat ICMP requests with priority though.
> 
> Not exactly.  We thought this through for some time and considered
> many angles.  Each method has its advantages and disadvantages.
> 
> We decided not to use ICMP at all because of the reasons you stated
> above.  We also decided not to use a "reflector" although as you said
> it would allow us to separate upload queue time from download.  We
> decided not to use this because it would be difficult to get ISPs to
> do this.
> 
> Are final choice for the paper was "magic" IP packets.  This consists
> of an IP packet header and the timestamp.  The IP packet is "self
> addressed" and we trick the iptables to emit the packet on the correct
> interface.  This packet will be returned to us as soon as it reaches
> another IP layer (typically at the CMTS).

	Ah, thanks; I did not get this from reading over your paper (but that is probably caused by me being a layman and having read it very quickly). Question how large is that packet on-the-wire? IP header plus 8 byte makes me assume 20+8 = 28, but that is missing the ethernet header, so rather 14+20+8 = 42, but isn’t the shorts ethernet frame 64bytes?

> 
> Here's a quick summary:
> ICMP -- Simple, but, needs the ISP's cooperation (good luck :)
> Reflector  -- Separates upload queue time from download queue time,
> but, requires the ISP to cooperate and to build something for us.
> (good luck :)
> Magic IP packets -- Requires nothing from the ISP (YaY!  We have a
> winner!), but, is a little more complex.

	At the cost that you only get RTT instead of two one-way delays as one ideally would like. But as stated above if you combine your method with say docsis3.1 pie which promises to keep the upstream under tight control, the any RTT changes should (mainly) be caused by downstream over-buffering (effectively allowing you use you method to control the downstream well).

> 
> 
>>        Also I am confused what they actually simulated: “The modems and CMTS were equipped with ASQM, CoDel and PIE,” and “However, the problem pop- ularly called bufferbloat can move about among many queues some of which are resistant to traditional AQM such as Layer 2 MAC protocols used in cable/DSL links. We call this problem bufferbloat displacement.” seem to be slightly at odds. If modems and CTMS have decent AQMs all they need to do is not stuff their sub-IP layer queuesand be done with it. The way I understood the cable labs PIE story, they intended to do exactly that, so at least the “buffer displacement” remedy by ASQM reads a bit like a straw man argument. But as I am a) not of the cs field, and b) only glossed over the paper, most likely I am missing something important that is clearly in the paper...
> 
> Good point!  However, once again it's not quite that simple.  Queues
> are necessary to absorb short term variations in packet arrival rate
> (or bursts).  The queue required for any flow is given by the
> bandwidth delay product.  

	Not a CS person, but that does not ring fully true; this basically assumes a physical medium that will dump all packets into the buffer at one time point and send them out a full delay period later; I think in reality packets will be serialized and hence some packet will most likely have left the buffer already before all have arrived, so the BDP is an more estimate of an upper bound… not that there is anything wrong with designing solutions aim to handle the worst case well.

> Since we don't know the delay we can't
> predict the queue size in advance.  What I'm getting at is the
> equipment manufacturers aren't putting in humongous queues because
> they are stupid, they are putting them there because in some cases you
> might really need that large of a queue.

	I thought our current pet hypothesis is that they aim for BDP at their highest rated speeds or so, and all customers running that (huh speed capable) equipment at lower rates are out of luck.

> 
> Statically sizing the queues is not the answer.  Managing the size of
> the queue with an algorithm is the answer.  :)

	No disagreement here, we just discuss the how not the why ;)

Best Regards
	Sebastian

> 
> 
> 
>> 
>> Best Regards
>>        Sebastian
>> 
>>> 
>>> --
>>> Dave Täht
>>> What will it take to vastly improve wifi for everyone?
>>> https://plus.google.com/u/0/explore/makewififast
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
>> 

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Daniel Havey]

On Thu, Jun 11, 2015 at 12:27 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Daniel,
>
> thanks for the clarifications.
>
> On Jun 11, 2015, at 02:10 , Daniel Havey <dhavey@gmail.com> wrote:
>
>> Hmmm, maybe I can help clarify.  Bufferbloat occurs in the slowest
>> queue on the path.  This is because the other queues are faster and
>> will drain.  AQM algorithms work only if they are placed where the
>> packets pile up (e.g. the slowest queue in the path).  This is
>> documented in Kathy and Van's CoDel paper.
>
>         I am with you so far.
>
>>
>> This is usually all well and good because we know where the bottleneck
>> (the slowest queue in the path) is.  It is the IP layer in the modem
>> where the ISP implements their rate limiter.  That is why algorithms
>> such as PIE and CoDel are implemented in the IP layer on the modem.
>
>         Okay.
>
>>
>> Suppose the full committed rate of the token bucket rate limiter is 8
>> Mbps.  This means that the queue at the IP layer in the modem is
>> capable of emitting packets at 8 Mbps sustained rate.  The problem
>> occurs during peak hours when the ISP is not providing the full
>> committed rate of 8 Mbps or that some queue in the system (probably in
>> the access link) is providing something less than 8 Mbps (say for sake
>> of discussion that the number is 7.5 Mbps).
>>
>> We know that (see Kathy and Van's paper) that AQM algorithms only work
>> when they are placed at the slowest queue.  However, the AQM is placed
>> at the queue that is capable of providing 8 Mbps and this is not the
>> slowest queue.  The AQM algorithm will not work in these conditions.
>>
>> This is what is shown in the paper where the CoDel and PIE performance
>> goes to hell in a handbasket.  The ASQM algorithm is designed to
>> address this problem.
>
>         Except that DOCSIS 3.1 pie in the modem does not work that way. As I understand http://www.cablelabs.com/wp-content/uploads/2014/06/DOCSIS-AQM_May2014.pdf section 3.2 MAP PROCESSING REQUIREMENTS, cable-modem pie will not stuff data into the lower layers until it has received a grant to actually send that data, hence no uncontrolled sub-IP layer buffer bloat possible (unless there are severe RF issues that take out data during transmission). So at least for the upstream direction docsis 3.1 pie will not suffer from buffer displacement, if I understand the cable labs white paper correctly.

Hmmm, interesting.  Are you sure?  I'm a CS not a EE so the PHY layer
is like black magic to me.  However, I still think (although I am
willing to be convinced otherwise by someone with superior knowledge
:)) that the IP layer puts packet into a MAC layer queue.  Then the
MAC layer makes a queue depth based request for bandwidth in order to
serialize and send the data.

If somebody really knows how this works, please help!  :^)  Is the
upload of a docsis 3.1 modem really unbloatable below the IP layer?  I
just want to know for my own edification :)

> Your solution still might be a valuable add-on to control the downstream buffer bloat in addition.
I agree!  If that reading of the cablelabs paper is correct then this
nicely solves the upload vs. download problem and we don't really need
BQL either.  If it is not true, then we use BQL on the egress to solve
the upload bloat problem and ASQM to solve the download bloat problem.
Perfect solution!  I love it when a plan comes together!  :^)

>         I also believe that free in france had a modified DSL driver for their box that made sure sub-IP buffering was bound to a low number of packets as well, so no displaced buffers there as well. Now it seems that this solution for DSL was unique so far and has not caught on, but once docsis3.1 modems hit the market upstream PIE in the modems will be reality.

freefrance?  Dave isn't that your provider?  I thought they were
running fq_CoDel?
In any case, just because PIE in the modems is a reality don't be
tempted to declare the problem solved and go home.  Never
underestimate the ability of the ISPs to do the wrong thing for very
good reasons :^)  What happens if they don't turn it on?  This is
really what I was trying to solve with ASQM.  What if your provider
won't run CoDel or PIE for whatever incomprehensible reason?  Then you
run ASQM and be done with it.

>
>>
>>
>>
>>
>>
>> On Wed, Jun 10, 2015 at 1:54 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> Hi Dave,
>>>
>>>
>>> On Jun 10, 2015, at 21:53 , Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>>> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf
>>>>
>>>> gargoyle's qos system follows a similar approach, using htb + sfq, and
>>>> a short ttl udp flow.
>>>>
>>>> Doing this sort of measured, then floating the rate control with
>>>> "cake" would be fairly easy (although it tends to be a bit more
>>>> compute intensive not being on a fast path)
>>>>
>>>> What is sort of missing here is trying to figure out which side of the
>>>> bottleneck is the bottleneck (up or down).
>>>
>>
>> Yeah, we never did figure out how to separate the up from the
>> downlink.  However, we just consider the access link as a whole (up +
>> down) and mark/drop according to ratios of queuing time.
>
>         This is a bit sad; why reduce say the effective uplink bandwidth if only the downstream is contended? Not that I have a good alternative solution that will not require help from outside boxes.
>
>> Overall it
>> seems to work well, but, we never did a mathematical analysis.  Kind
>> of like saying it's not a "bug", it is a feature.  And it this case it
>> is true since both sides can experience bloat.
>
>         Yes, but you only want to throttle traffic on the congested leg of the link, otherwise bandwidth efficiency goes to hell if you look at bi-direction link-saturating traffic.
>
>>
>>
>>>        Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned. In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.". At least this is what I get from their short description in glossing over the paper.
>>>        Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers. But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs "on steroids", with higher resolution than milliseconds, but for buffer bloat detection ms resolution would probably be sufficient anyways). Currently, I hear that ISP equipment will not treat ICMP requests with priority though.
>>
>> Not exactly.  We thought this through for some time and considered
>> many angles.  Each method has its advantages and disadvantages.
>>
>> We decided not to use ICMP at all because of the reasons you stated
>> above.  We also decided not to use a "reflector" although as you said
>> it would allow us to separate upload queue time from download.  We
>> decided not to use this because it would be difficult to get ISPs to
>> do this.
>>
>> Are final choice for the paper was "magic" IP packets.  This consists
>> of an IP packet header and the timestamp.  The IP packet is "self
>> addressed" and we trick the iptables to emit the packet on the correct
>> interface.  This packet will be returned to us as soon as it reaches
>> another IP layer (typically at the CMTS).
>
>         Ah, thanks; I did not get this from reading over your paper (but that is probably caused by me being a layman and having read it very quickly). Question how large is that packet on-the-wire? IP header plus 8 byte makes me assume 20+8 = 28, but that is missing the ethernet header, so rather 14+20+8 = 42, but isn’t the shorts ethernet frame 64bytes?
>
>>
>> Here's a quick summary:
>> ICMP -- Simple, but, needs the ISP's cooperation (good luck :)
>> Reflector  -- Separates upload queue time from download queue time,
>> but, requires the ISP to cooperate and to build something for us.
>> (good luck :)
>> Magic IP packets -- Requires nothing from the ISP (YaY!  We have a
>> winner!), but, is a little more complex.
>
>         At the cost that you only get RTT instead of two one-way delays as one ideally would like. But as stated above if you combine your method with say docsis3.1 pie which promises to keep the upstream under tight control, the any RTT changes should (mainly) be caused by downstream over-buffering (effectively allowing you use you method to control the downstream well).
>
>>
>>
>>>        Also I am confused what they actually simulated: “The modems and CMTS were equipped with ASQM, CoDel and PIE,” and “However, the problem pop- ularly called bufferbloat can move about among many queues some of which are resistant to traditional AQM such as Layer 2 MAC protocols used in cable/DSL links. We call this problem bufferbloat displacement.” seem to be slightly at odds. If modems and CTMS have decent AQMs all they need to do is not stuff their sub-IP layer queuesand be done with it. The way I understood the cable labs PIE story, they intended to do exactly that, so at least the “buffer displacement” remedy by ASQM reads a bit like a straw man argument. But as I am a) not of the cs field, and b) only glossed over the paper, most likely I am missing something important that is clearly in the paper...
>>
>> Good point!  However, once again it's not quite that simple.  Queues
>> are necessary to absorb short term variations in packet arrival rate
>> (or bursts).  The queue required for any flow is given by the
>> bandwidth delay product.
>
>         Not a CS person, but that does not ring fully true; this basically assumes a physical medium that will dump all packets into the buffer at one time point and send them out a full delay period later; I think in reality packets will be serialized and hence some packet will most likely have left the buffer already before all have arrived, so the BDP is an more estimate of an upper bound… not that there is anything wrong with designing solutions aim to handle the worst case well.
>
>> Since we don't know the delay we can't
>> predict the queue size in advance.  What I'm getting at is the
>> equipment manufacturers aren't putting in humongous queues because
>> they are stupid, they are putting them there because in some cases you
>> might really need that large of a queue.
>
>         I thought our current pet hypothesis is that they aim for BDP at their highest rated speeds or so, and all customers running that (huh speed capable) equipment at lower rates are out of luck.
>
>>
>> Statically sizing the queues is not the answer.  Managing the size of
>> the queue with an algorithm is the answer.  :)
>
>         No disagreement here, we just discuss the how not the why ;)
>
> Best Regards
>         Sebastian
>
>>
>>
>>
>>>
>>> Best Regards
>>>        Sebastian
>>>
>>>>
>>>> --
>>>> Dave Täht
>>>> What will it take to vastly improve wifi for everyone?
>>>> https://plus.google.com/u/0/explore/makewififast
>>>> _______________________________________________
>>>> Cake mailing list
>>>> Cake@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cake
>>>
>

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Sebastian Moeller]

Hi Daniel,

On Jun 12, 2015, at 17:02 , Daniel Havey <dhavey@gmail.com> wrote:
> On Thu, Jun 11, 2015 at 12:27 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> [...]
>>        Except that DOCSIS 3.1 pie in the modem does not work that way. As I understand http://www.cablelabs.com/wp-content/uploads/2014/06/DOCSIS-AQM_May2014.pdf section 3.2 MAP PROCESSING REQUIREMENTS, cable-modem pie will not stuff data into the lower layers until it has received a grant to actually send that data, hence no uncontrolled sub-IP layer buffer bloat possible (unless there are severe RF issues that take out data during transmission). So at least for the upstream direction docsis 3.1 pie will not suffer from buffer displacement, if I understand the cable labs white paper correctly.
> 
> Hmmm, interesting.  Are you sure?  I'm a CS not a EE so the PHY layer
> is like black magic to me.  However, I still think (although I am
> willing to be convinced otherwise by someone with superior knowledge
> :)) that the IP layer puts packet into a MAC layer queue.  Then the
> MAC layer makes a queue depth based request for bandwidth in order to
> serialize and send the data.

	I am not sure, but maybe Greg White (CCd) can help us out here? @Greg, is it right that the docsis3.1 pie implementation will keep a close lid on how many packets/bytes are queued in lower layers of the stack? 


> 
> If somebody really knows how this works, please help!  :^)  Is the
> upload of a docsis 3.1 modem really unbloatable below the IP layer?  I
> just want to know for my own edification :)
> 
>> Your solution still might be a valuable add-on to control the downstream buffer bloat in addition.
> I agree!  If that reading of the cablelabs paper is correct then this
> nicely solves the upload vs. download problem and we don't really need
> BQL either.  If it is not true, then we use BQL on the egress to solve
> the upload bloat problem and ASQM to solve the download bloat problem.
> Perfect solution!  I love it when a plan comes together!  :^)

	ASFAIK, BQL so far is only implemented in ethernet drivers, so if your uplink is egaul or slightly higher than 10, 100, 1000, or 1000Mbps BQL with fq_codel will not need and shaper on egress and should still hold the buffers at bay. Unfortunately often the actual egress rates are quite off of these ethernet sleep tiers. I believe Dave Taeht is trying to convince ethernet drivers to set their egress at non-traditional rates, I could be wrong though...

> 
>>        I also believe that free in france had a modified DSL driver for their box that made sure sub-IP buffering was bound to a low number of packets as well, so no displaced buffers there as well. Now it seems that this solution for DSL was unique so far and has not caught on, but once docsis3.1 modems hit the market upstream PIE in the modems will be reality.
> 
> freefrance?  Dave isn't that your provider?  I thought they were
> running fq_CoDel?
> In any case, just because PIE in the modems is a reality don't be
> tempted to declare the problem solved and go home.  Never
> underestimate the ability of the ISPs to do the wrong thing for very
> good reasons :^)  What happens if they don't turn it on?  This is
> really what I was trying to solve with ASQM.  What if your provider
> won't run CoDel or PIE for whatever incomprehensible reason?  Then you
> run ASQM and be done with it.

	I like your active sensing approach, I just believe that the scenario you set-out n the paper is not fully true, so I tried to voice my concerns. Personally I am on a vdsl-link so docsis pie or no docsis pie, my link is still bloated and I am looking for new solutions. I like your magic packet idea, even though my taste in these matters is debatable ;) but I fear to work this needs to run on the modem, and thee are only few fully opensource modems around (if any) on which to implement your active probe. Plus on a DSL link the congestion typically comes between DSLAM and BRAS (as the DSL link is not shared, unlike the cable situation) and I fear the DSLAM might already return the probe packet…


Best Regards
	Sebastian


> 
>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Wed, Jun 10, 2015 at 1:54 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>> Hi Dave,
>>>> 
>>>> 
>>>> On Jun 10, 2015, at 21:53 , Dave Taht <dave.taht@gmail.com> wrote:
>>>> 
>>>>> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf
>>>>> 
>>>>> gargoyle's qos system follows a similar approach, using htb + sfq, and
>>>>> a short ttl udp flow.
>>>>> 
>>>>> Doing this sort of measured, then floating the rate control with
>>>>> "cake" would be fairly easy (although it tends to be a bit more
>>>>> compute intensive not being on a fast path)
>>>>> 
>>>>> What is sort of missing here is trying to figure out which side of the
>>>>> bottleneck is the bottleneck (up or down).
>>>> 
>>> 
>>> Yeah, we never did figure out how to separate the up from the
>>> downlink.  However, we just consider the access link as a whole (up +
>>> down) and mark/drop according to ratios of queuing time.
>> 
>>        This is a bit sad; why reduce say the effective uplink bandwidth if only the downstream is contended? Not that I have a good alternative solution that will not require help from outside boxes.
>> 
>>> Overall it
>>> seems to work well, but, we never did a mathematical analysis.  Kind
>>> of like saying it's not a "bug", it is a feature.  And it this case it
>>> is true since both sides can experience bloat.
>> 
>>        Yes, but you only want to throttle traffic on the congested leg of the link, otherwise bandwidth efficiency goes to hell if you look at bi-direction link-saturating traffic.
>> 
>>> 
>>> 
>>>>       Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned. In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.". At least this is what I get from their short description in glossing over the paper.
>>>>       Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers. But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs "on steroids", with higher resolution than milliseconds, but for buffer bloat detection ms resolution would probably be sufficient anyways). Currently, I hear that ISP equipment will not treat ICMP requests with priority though.
>>> 
>>> Not exactly.  We thought this through for some time and considered
>>> many angles.  Each method has its advantages and disadvantages.
>>> 
>>> We decided not to use ICMP at all because of the reasons you stated
>>> above.  We also decided not to use a "reflector" although as you said
>>> it would allow us to separate upload queue time from download.  We
>>> decided not to use this because it would be difficult to get ISPs to
>>> do this.
>>> 
>>> Are final choice for the paper was "magic" IP packets.  This consists
>>> of an IP packet header and the timestamp.  The IP packet is "self
>>> addressed" and we trick the iptables to emit the packet on the correct
>>> interface.  This packet will be returned to us as soon as it reaches
>>> another IP layer (typically at the CMTS).
>> 
>>        Ah, thanks; I did not get this from reading over your paper (but that is probably caused by me being a layman and having read it very quickly). Question how large is that packet on-the-wire? IP header plus 8 byte makes me assume 20+8 = 28, but that is missing the ethernet header, so rather 14+20+8 = 42, but isn’t the shorts ethernet frame 64bytes?
>> 
>>> 
>>> Here's a quick summary:
>>> ICMP -- Simple, but, needs the ISP's cooperation (good luck :)
>>> Reflector  -- Separates upload queue time from download queue time,
>>> but, requires the ISP to cooperate and to build something for us.
>>> (good luck :)
>>> Magic IP packets -- Requires nothing from the ISP (YaY!  We have a
>>> winner!), but, is a little more complex.
>> 
>>        At the cost that you only get RTT instead of two one-way delays as one ideally would like. But as stated above if you combine your method with say docsis3.1 pie which promises to keep the upstream under tight control, the any RTT changes should (mainly) be caused by downstream over-buffering (effectively allowing you use you method to control the downstream well).
>> 
>>> 
>>> 
>>>>       Also I am confused what they actually simulated: “The modems and CMTS were equipped with ASQM, CoDel and PIE,” and “However, the problem pop- ularly called bufferbloat can move about among many queues some of which are resistant to traditional AQM such as Layer 2 MAC protocols used in cable/DSL links. We call this problem bufferbloat displacement.” seem to be slightly at odds. If modems and CTMS have decent AQMs all they need to do is not stuff their sub-IP layer queuesand be done with it. The way I understood the cable labs PIE story, they intended to do exactly that, so at least the “buffer displacement” remedy by ASQM reads a bit like a straw man argument. But as I am a) not of the cs field, and b) only glossed over the paper, most likely I am missing something important that is clearly in the paper...
>>> 
>>> Good point!  However, once again it's not quite that simple.  Queues
>>> are necessary to absorb short term variations in packet arrival rate
>>> (or bursts).  The queue required for any flow is given by the
>>> bandwidth delay product.
>> 
>>        Not a CS person, but that does not ring fully true; this basically assumes a physical medium that will dump all packets into the buffer at one time point and send them out a full delay period later; I think in reality packets will be serialized and hence some packet will most likely have left the buffer already before all have arrived, so the BDP is an more estimate of an upper bound… not that there is anything wrong with designing solutions aim to handle the worst case well.
>> 
>>> Since we don't know the delay we can't
>>> predict the queue size in advance.  What I'm getting at is the
>>> equipment manufacturers aren't putting in humongous queues because
>>> they are stupid, they are putting them there because in some cases you
>>> might really need that large of a queue.
>> 
>>        I thought our current pet hypothesis is that they aim for BDP at their highest rated speeds or so, and all customers running that (huh speed capable) equipment at lower rates are out of luck.
>> 
>>> 
>>> Statically sizing the queues is not the answer.  Managing the size of
>>> the queue with an algorithm is the answer.  :)
>> 
>>        No disagreement here, we just discuss the how not the why ;)
>> 
>> Best Regards
>>        Sebastian
>> 
>>> 
>>> 
>>> 
>>>> 
>>>> Best Regards
>>>>       Sebastian
>>>> 
>>>>> 
>>>>> --
>>>>> Dave Täht
>>>>> What will it take to vastly improve wifi for everyone?
>>>>> https://plus.google.com/u/0/explore/makewififast
>>>>> _______________________________________________
>>>>> Cake mailing list
>>>>> Cake@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/cake
>>>> 
>> 

Re: [Cerowrt-devel] [Cake] active sensing queue management

[David Lang]

On Wed, 10 Jun 2015, Daniel Havey wrote:

> We know that (see Kathy and Van's paper) that AQM algorithms only work
> when they are placed at the slowest queue.  However, the AQM is placed
> at the queue that is capable of providing 8 Mbps and this is not the
> slowest queue.  The AQM algorithm will not work in these conditions.

so the answer is that you don't deploy the AQM algorithm only at the perimeter, 
you deploy it much more widely.

Eventually you get to core devices that have multiple routes they can use to get 
to a destination. Those devices should notice that one route is getting 
congested and start sending the packets through alternate paths.

Now, if the problem is that the aggregate of inbound packets to your downstreams 
where you are the only path becomes higher than the available downstream 
bandwidth, you need to be running an AQM to handle things.

David Lang

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Daniel Havey]

On Thu, Jun 11, 2015 at 6:49 PM, David Lang <david@lang.hm> wrote:
> On Wed, 10 Jun 2015, Daniel Havey wrote:
>
>> We know that (see Kathy and Van's paper) that AQM algorithms only work
>> when they are placed at the slowest queue.  However, the AQM is placed
>> at the queue that is capable of providing 8 Mbps and this is not the
>> slowest queue.  The AQM algorithm will not work in these conditions.
>
>
> so the answer is that you don't deploy the AQM algorithm only at the
> perimeter, you deploy it much more widely.
>
> Eventually you get to core devices that have multiple routes they can use to
> get to a destination. Those devices should notice that one route is getting
> congested and start sending the packets through alternate paths.
>
> Now, if the problem is that the aggregate of inbound packets to your
> downstreams where you are the only path becomes higher than the available
> downstream bandwidth, you need to be running an AQM to handle things.
>
> David Lang
>

Hmmmm, that is interesting.  There might be a problem with processing
power at the core though.  It could be difficult to manage all of
those packets flying through the core routers.


David does bring up an interesting point though.  The ASQM algorithm
was originally designed to solve the "Uncooperative ISP" problem.  I
coined the phrase, but, you can fill in your own adjective to fit your
personal favorite ISP :^)

The paper doesn't indicate this because I got roasted by a bunch of
reviewers for it, but, why not use an ASQM like algorithm other places
than the edge.  Suppose you are netflix and your ISP is shaping your
packets?  You cant do anything about the bandwidth reduction, but, you
can at least reduce the queuing...Just food for thought. :^)

Re: [Cerowrt-devel] [Cake] active sensing queue management

[David Lang]

On Fri, 12 Jun 2015, Daniel Havey wrote:

> On Thu, Jun 11, 2015 at 6:49 PM, David Lang <david@lang.hm> wrote:
>> On Wed, 10 Jun 2015, Daniel Havey wrote:
>>
>>> We know that (see Kathy and Van's paper) that AQM algorithms only work
>>> when they are placed at the slowest queue.  However, the AQM is placed
>>> at the queue that is capable of providing 8 Mbps and this is not the
>>> slowest queue.  The AQM algorithm will not work in these conditions.
>>
>>
>> so the answer is that you don't deploy the AQM algorithm only at the
>> perimeter, you deploy it much more widely.
>>
>> Eventually you get to core devices that have multiple routes they can use to
>> get to a destination. Those devices should notice that one route is getting
>> congested and start sending the packets through alternate paths.
>>
>> Now, if the problem is that the aggregate of inbound packets to your
>> downstreams where you are the only path becomes higher than the available
>> downstream bandwidth, you need to be running an AQM to handle things.
>>
>> David Lang
>>
>
> Hmmmm, that is interesting.  There might be a problem with processing
> power at the core though.  It could be difficult to manage all of
> those packets flying through the core routers.

And that is the question that people are looking at.

But part of the practical question is at what speeds do you start to run into 
problems?

the core of the Internet is already doing dynamic routing of packets, spreading 
them across multiple parallel paths (peering points have multiple 10G links 
between peers), so this should be more of the same, with possibly a small 
variation to use more expensive paths if the cheap ones are congested.

But as you move out from there towards the edge, the packet handling 
requirements drop rather quickly, and I'll bet that you don't have to get very 
far out before you can start affording to implement AQM algorithms. I'm betting 
that you reach that point before you get to the point in the network where you 
no longer have multiple paths available

> David does bring up an interesting point though.  The ASQM algorithm
> was originally designed to solve the "Uncooperative ISP" problem.  I
> coined the phrase, but, you can fill in your own adjective to fit your
> personal favorite ISP :^)
>
> The paper doesn't indicate this because I got roasted by a bunch of
> reviewers for it, but, why not use an ASQM like algorithm other places
> than the edge.  Suppose you are netflix and your ISP is shaping your
> packets?  You cant do anything about the bandwidth reduction, but, you
> can at least reduce the queuing...Just food for thought. :^)

unfortunantly if you are trapped by the ISP/netflix peering war, you reducing 
the number of packets in flight for yourself isn't going to help any. It would 
have to happen on the netflix side of the bottleneck.

David Lang

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 4121 bytes --]

On Fri, 12 Jun 2015, Daniel Havey wrote:
>
> > On Thu, Jun 11, 2015 at 6:49 PM, David Lang <david at lang.hm> wrote:
> >> On Wed, 10 Jun 2015, Daniel Havey wrote:
> >>
> >>> We know that (see Kathy and Van's paper) that AQM algorithms only work
> >>> when they are placed at the slowest queue.  However, the AQM is placed
> >>> at the queue that is capable of providing 8 Mbps and this is not the
> >>> slowest queue.  The AQM algorithm will not work in these conditions.
> >>
> >>
> >> so the answer is that you don't deploy the AQM algorithm only at the
> >> perimeter, you deploy it much more widely.
> >>
> >> Eventually you get to core devices that have multiple routes they can
use to
> >> get to a destination. Those devices should notice that one route is
getting
> >> congested and start sending the packets through alternate paths.
> >>
> >> Now, if the problem is that the aggregate of inbound packets to your
> >> downstreams where you are the only path becomes higher than the
available
> >> downstream bandwidth, you need to be running an AQM to handle things.
> >>
> >> David Lang
> >>
> >
> > Hmmmm, that is interesting.  There might be a problem with processing
> > power at the core though.  It could be difficult to manage all of
> > those packets flying through the core routers.
>
> And that is the question that people are looking at.
>
> But part of the practical question is at what speeds do you start to run
into
> problems?
>
> the core of the Internet is already doing dynamic routing of packets,
spreading
> them across multiple parallel paths (peering points have multiple 10G
links
> between peers), so this should be more of the same, with possibly a small
> variation to use more expensive paths if the cheap ones are congested.

Yes and no. Spreading data across parallel links is mostly done at the MAC
layer and does not show up as separate routes, thinking teaming ports for
Ethernet. Routing is dynamic, but typically takes a bit for the route
changes to propagate. For the most part, you can only control where you
send data, but not where you receive it. The core of the Internet typically
only has 2-3 routes to choose from with one primary route and the other
only used for fail over. Load balancing asymmetrical routes is a very messy
issue that you really don't want to do. Most of the time, the cheapest
route is also the fastest. If you had to choose between a $5k/month 100Gb
port at a peering location or a $30k 10Gb transit link, I'm sure you won't
be doing any load balancing over the 10Gb link unless your 100Gb failed.

Routes really don't change that often. You have a default transit route and
a bunch of peering routes. The peering routes take priority because they're
cheaper and the transit route is for when bad things happen or you just
don't have peering for that route. In the case of my ISP, everything is
transit, let Level 3 worry about peering.

> But as you move out from there towards the edge, the packet handling
> requirements drop rather quickly, and I'll bet that you don't have to get
very
> far out before you can start affording to implement AQM algorithms. I'm
betting
> that you reach that point before you get to the point in the network
where you
> no longer have multiple paths available
>
> > David does bring up an interesting point though.  The ASQM algorithm
> > was originally designed to solve the "Uncooperative ISP" problem.  I
> > coined the phrase, but, you can fill in your own adjective to fit your
> > personal favorite ISP :^)
> >
> > The paper doesn't indicate this because I got roasted by a bunch of
> > reviewers for it, but, why not use an ASQM like algorithm other places
> > than the edge.  Suppose you are netflix and your ISP is shaping your
> > packets?  You cant do anything about the bandwidth reduction, but, you
> > can at least reduce the queuing...Just food for thought. :^)
>
> unfortunantly if you are trapped by the ISP/netflix peering war, you
reducing
> the number of packets in flight for yourself isn't going to help any. It
would
> have to happen on the netflix side of the bottleneck.
>
> David Lang
>

[-- Attachment #2: Type: text/html, Size: 6673 bytes --]

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Alan Jenkins]

On 10/06/15 21:54, Sebastian Moeller wrote:
> Hi Dave,
>
>
> On Jun 10, 2015, at 21:53 , Dave Taht <dave.taht@gmail.com> wrote:
>
>> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf
>>
>> gargoyle's qos system follows a similar approach, using htb + sfq, and
>> a short ttl udp flow.
>>
>> Doing this sort of measured, then floating the rate control with
>> "cake" would be fairly easy (although it tends to be a bit more
>> compute intensive not being on a fast path)
>>
>> What is sort of missing here is trying to figure out which side of the
>> bottleneck is the bottleneck (up or down).
> 	Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned.

They copy & frob real IP headers.  They don't _say_ how the reflection 
works, but I guess low TTL -> ICMP TTL exceeded, like traceroute.  Then 
I read Gargoyle also use ICMP TTL exceeded and I thought my guess is 
quite educated 8).

Note the size of the timestamp, a generous 8 bytes.  It "just happens" 
that ICMP responses are required to include the first 8 bytes of the IP 
payload 8).

>   In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.". At least this is what I get from their short description in glossing over the paper.

Ow!  I hadn't noticed that.  You could reduce both rates proportionally 
but the effect is inelegant.  I wonder what Gargoyle does...

2012 gargoyle developer comment says "There are not settings for active 
congestion control on the uplink side. ACC concentrats on the download 
side only."

Random blog post points out this is sufficient to fix prioritization 
v.s. bufferbloat.  "In upstream direction this is not a big problem 
because your router can still prioritize which packet should be sent 
first".  (Yay, I get to classify every application I care about /s and 
still get killed by uploads in http).

One solution would be if ISPs made sure upload is 100% provisioned. 
Could be cheaper than for (the higher rate) download.

> 	Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers.

I think that's a valid concern.  Is "TTL Exceeded" rate-limited like 
Echo (because it may be generated outside the highest-speed forwarding 
path?), and would this work as tested if everyone did it?

>   But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs "on steroids", with higher resolution than milliseconds, but for buffer bloat detection ms resolution would probably be sufficient anyways). Currently, I hear that ISP equipment will not treat ICMP requests with priority though.
> 	Also I am confused what they actually simulated: “The modems and CMTS were equipped with ASQM, CoDel and PIE,” and “However, the problem pop- ularly called bufferbloat can move about among many queues some of which are resistant to traditional AQM such as Layer 2 MAC protocols used in cable/DSL links. We call this problem bufferbloat displacement.” seem to be slightly at odds. If modems and CTMS have decent AQMs all they need to do is not stuff their sub-IP layer queuesand be done with it. The way I understood the cable labs PIE story, they intended to do exactly that, so at least the “buffer displacement” remedy by ASQM reads a bit like a straw man argument. But as I am a) not of the cs field, and b) only glossed over the paper, most likely I am missing something important that is clearly in the paper...
>
> Best Regards
> 	Sebastian

I had your reaction about pie on the modem.

We could say there is likely room for improvement in any paper, that 
claims bufferbloat eliminated with a "target" parameter of 100ms :p.  
Results don't look that bad (why?) but I do see 25ms bloat v.s. 
codel/pie.  It may be inevitable but deserves not to be glossed over 
with comparisons to the unrelated 100ms default parameter of codel, 
which in reality is the one called "interval" not "target" :).  Good QM 
on the modem+cmts has got to be the best solution.

Alan

Re: [Cerowrt-devel] [Cake] active sensing queue management

[Sebastian Moeller]

Hi Alan,

On Jun 11, 2015, at 03:05 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> On 10/06/15 21:54, Sebastian Moeller wrote:
>> Hi Dave,
>> 
>> 
>> On Jun 10, 2015, at 21:53 , Dave Taht <dave.taht@gmail.com> wrote:
>> 
>>> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf
>>> 
>>> gargoyle's qos system follows a similar approach, using htb + sfq, and
>>> a short ttl udp flow.
>>> 
>>> Doing this sort of measured, then floating the rate control with
>>> "cake" would be fairly easy (although it tends to be a bit more
>>> compute intensive not being on a fast path)
>>> 
>>> What is sort of missing here is trying to figure out which side of the
>>> bottleneck is the bottleneck (up or down).
>> 	Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned.
> 
> They copy & frob real IP headers.  They don't _say_ how the reflection works, but I guess low TTL -> ICMP TTL exceeded, like traceroute.  Then I read Gargoyle also use ICMP TTL exceeded and I thought my guess is quite educated 8).

	Daniel elucidated their magic packets: they create self-addressed IP packets at the simulated CPE and inject them in the simulated cable link; the other end will pass the data through its stack and once the sender-self-addressed packet reaches the IP-layer of the simulated CMTS it gets send back, since that IP layer sees the CPE’s IP address as the to address.
	@Daniel, this trick can only work if a) the magic packets are only passed one IP-hop since the first upstream IP-layer will effectively bounce them back (so the injector in the docsis case needs to be the cable modem) b) the CPE actually has an IP that can be reached from the outside and that is known to the person setting up your AQM, is that correct? How does this work if the CPE acts as an ethernet bridge without an external IP?

> 
> Note the size of the timestamp, a generous 8 bytes.  It "just happens" that ICMP responses are required to include the first 8 bytes of the IP payload 8).
> 
>>  In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.". At least this is what I get from their short description in glossing over the paper.
> 
> Ow!  I hadn't noticed that.  You could reduce both rates proportionally but the effect is inelegant.

	I think that it what they do, as long as one only measures uni-directional saturating traffic this approach will work fine as the bandwidth loss in the opposite direction simply does not materialize.

>  I wonder what Gargoyle does...
> 
> 2012 gargoyle developer comment says "There are not settings for active congestion control on the uplink side. ACC concentrats on the download side only."
> 
> Random blog post points out this is sufficient to fix prioritization v.s. bufferbloat.  "In upstream direction this is not a big problem because your router can still prioritize which packet should be sent first".  (Yay, I get to classify every application I care about /s and still get killed by uploads in http).

	Not fully convinced that this is fully sane, as in cable systems the upstream bandwidth can fluctuate significantly depending on how many people are active. Actually scratch the “cable” since most customer links have shared oversubscribed links somewhere between the CPE and the internet that will make static bandwidth shaping mis-behave some of the time. A good ISP just manages the oversubscription well enough that this issue only occurs transiently… (I hope).


> 
> One solution would be if ISPs made sure upload is 100% provisioned. Could be cheaper than for (the higher rate) download.

	Not going to happen, in my opinion, as economically unfeasible for a publicly traded ISP. I would settle for that approach as long as the ISP is willing to fix its provisioning so that oversubscription episodes are reasonable rare, though.

> 
>> 	Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers.
> 
> I think that's a valid concern.  Is "TTL Exceeded" rate-limited like Echo (because it may be generated outside the highest-speed forwarding path?), and would this work as tested if everyone did it?

	I thing Daniel agrees and that is why they came up with the “magic” packet approach (that drags in its own set of challenges as far as I can see).

> 
>>  But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs "on steroids", with higher resolution than milliseconds, but for buffer bloat detection ms resolution would probably be sufficient anyways). Currently, I hear that ISP equipment will not treat ICMP requests with priority though.
>> 	Also I am confused what they actually simulated: “The modems and CMTS were equipped with ASQM, CoDel and PIE,” and “However, the problem pop- ularly called bufferbloat can move about among many queues some of which are resistant to traditional AQM such as Layer 2 MAC protocols used in cable/DSL links. We call this problem bufferbloat displacement.” seem to be slightly at odds. If modems and CTMS have decent AQMs all they need to do is not stuff their sub-IP layer queuesand be done with it. The way I understood the cable labs PIE story, they intended to do exactly that, so at least the “buffer displacement” remedy by ASQM reads a bit like a straw man argument. But as I am a) not of the cs field, and b) only glossed over the paper, most likely I am missing something important that is clearly in the paper...
>> 
>> Best Regards
>> 	Sebastian
> 
> I had your reaction about pie on the modem.
> 
> We could say there is likely room for improvement in any paper, that claims bufferbloat eliminated with a "target" parameter of 100ms :p.  Results don't look that bad (why?) but I do see 25ms bloat v.s. codel/pie.  It may be inevitable but deserves not to be glossed over with comparisons to the unrelated 100ms default parameter of codel, which in reality is the one called "interval" not "target" :).  Good QM on the modem+cmts has got to be the best solution.

	I fully agree. I have a hunch that their method might be used to supplement docsis 3.1 pie so that the CPEs can also meaningfully measure and control downstream buffer bloat in addition to the upstream without the need to fix the CMTSs. As far as I understand cable labs are quite proactive in trying to fix this in CPE’s  while I have heard nothing about the CMTS manufacturers’ plans (I think the Arris paper was about CPEs not CMTS). Maybe cable labs could be convinced to try this in addition to upstream PIE as a solution that will require no CMTS involvement… (I simple assume that the CMTS does not need to cooperate, but note that the paper seems to rely totally on simulated data, in so far as linux pc’s where used to model each of the network components. So "no real CMTS was harmed during the making of this paper")

Best Regards
	Sebastian




> 
> Alan

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[David Lang]

On Thu, 11 Jun 2015, Sebastian Moeller wrote:

>
> On Jun 11, 2015, at 03:05 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>
>> On 10/06/15 21:54, Sebastian Moeller wrote:
>>
>> One solution would be if ISPs made sure upload is 100% provisioned. Could be 
>> cheaper than for (the higher rate) download.
>
> 	Not going to happen, in my opinion, as economically unfeasible for a 
> publicly traded ISP. I would settle for that approach as long as the ISP is 
> willing to fix its provisioning so that oversubscription episodes are 
> reasonable rare, though.

not going to happen on any network, publicly traded or not.

The question is not "can the theoretical max of all downstream devices exceed 
the upstream bandwidth" because that answer is going to be "yes" for every 
network built, LAN or WAN, but rather "does the demand in practice of the 
combined downstream devices exceed the upstream bandwidth for long enough to be 
a problem"

it's not even a matter of what percentage are they oversubscribed.

someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% of 
theoretical requirements) is probably a lot worse than someone with 100 1G lines 
downstream and a 10G upstream (10% of theoretical requirements) because it's far 
less likely that the users of the 1G lines are actually going to saturate them 
(let alone simultaniously for a noticable timeframe), while it's very likely 
that the users of the 1.5M DSL lines are going to saturate their lines for 
extended timeframes.

The problem shows up when either usage changes rapidly, or the network operator 
is not keeping up with required upgrades as gradual usage changes happen 
(including when they are prevented from upgrading because a peer won't 
cooperate)

As for the "100% provisioning" ideal, think through the theoretical aggregate 
and realize that before you get past very many layers, you get to a bandwidh 
requirement that it's not technically possible to provide.

David Lang

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[MUSCARIELLO Luca IMT/OLN]

On 06/12/2015 03:44 AM, David Lang wrote:
> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
>
>>
>> On Jun 11, 2015, at 03:05 , Alan Jenkins 
>> <alan.christopher.jenkins@gmail.com> wrote:
>>
>>> On 10/06/15 21:54, Sebastian Moeller wrote:
>>>
>>> One solution would be if ISPs made sure upload is 100% provisioned. 
>>> Could be cheaper than for (the higher rate) download.
>>
>>     Not going to happen, in my opinion, as economically unfeasible 
>> for a publicly traded ISP. I would settle for that approach as long 
>> as the ISP is willing to fix its provisioning so that 
>> oversubscription episodes are reasonable rare, though.
>
> not going to happen on any network, publicly traded or not.
>
> The question is not "can the theoretical max of all downstream devices 
> exceed the upstream bandwidth" because that answer is going to be 
> "yes" for every network built, LAN or WAN, but rather "does the demand 
> in practice of the combined downstream devices exceed the upstream 
> bandwidth for long enough to be a problem"
>
> it's not even a matter of what percentage are they oversubscribed.
>
> someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% 
> of theoretical requirements) is probably a lot worse than someone with 
> 100 1G lines downstream and a 10G upstream (10% of theoretical 
> requirements) because it's far less likely that the users of the 1G 
> lines are actually going to saturate them (let alone simultaniously 
> for a noticable timeframe), while it's very likely that the users of 
> the 1.5M DSL lines are going to saturate their lines for extended 
> timeframes.
>
> The problem shows up when either usage changes rapidly, or the network 
> operator is not keeping up with required upgrades as gradual usage 
> changes happen (including when they are prevented from upgrading 
> because a peer won't cooperate)

Good points. Let me add a side comment though.
We observe  that fiber users (e.g. 1Gbps/300Mbps access with GPON) are 
changing behavior
w.r.t. DSL users in the way they use the uplink. This is mostly (not 
only) due to personal cloud storage availability
and the fact that everyone today is able to produce tons of  big videos, 
that people are willing to store
outside the home.
As a result it's not unlikely that a backhaul link utilization may get 
out of the network planning, which is made
on long term statistics. These workloads are unpredictable and if on one 
hand it's not feasible to
over provision based on such unpredictable long peeks  on the other hand 
you'd need smart queue
management to cope with such events, where the bottleneck is the backhaul.

Considering the cost of current equipment upgrades, I feel like very 
high speed accesses will impose
smart queue management everywhere from the access up to transit links, 
including the entire backhaul.
Bad news is that no such queuing systems are available in current 
equipment, so I guess the process
will be pretty slow to happen.

>
> As for the "100% provisioning" ideal, think through the theoretical 
> aggregate and realize that before you get past very many layers, you 
> get to a bandwidh requirement that it's not technically possible to 
> provide.
>
> David Lang
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> .
>

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 10658 bytes --]

> On 12/06/15 02:44, David Lang wrote:
> > On Thu, 11 Jun 2015, Sebastian Moeller wrote:
> >
> >>
> >> On Jun 11, 2015, at 03:05 , Alan Jenkins
> >> <alan.christopher.jenkins at gmail.com> wrote:
> >>
> >>> On 10/06/15 21:54, Sebastian Moeller wrote:
> > >>>
> >>> One solution would be if ISPs made sure upload is 100% provisioned.
> >>> Could be cheaper than for (the higher rate) download.
> >>
> >>     Not going to happen, in my opinion, as economically unfeasible
> >> for a publicly traded ISP. I would settle for that approach as long
> >> as the ISP is willing to fix its provisioning so that
> >> oversubscription episodes are reasonable rare, though.
> >
> > not going to happen on any network, publicly traded or not.

Sorry if this is a tangent from where the current discussion has gone, but
I wanted to correct someone saying something is "impossible".

Funny this was discussed. My ISP sells all packages as symmetrical,
"dedicated", and business grade. They define "dedicated" as never
oversubscribed. They do oversubscribe the trunk, obviously, but from the
last mile to the core, no oversubscription.
It's a small ISP, so one of the times that I called on the weekend, I get a
senior network engineer that answered the support line. He told me they did
try oversubscription when they first rolled out DSL, but managing
congestion is a huge waste of time and increased customer support calls.
They eventually went "dedicated" and never looked back. Support calls were
reduced enough to make it worth their while.

In their ToS, they have a net neutrality promise to NEVER block, shape,
degrade, or prioritize any traffic, including their own or their partner's.

I can understand taking this approach with 1Mb DSL or even 50Mb fiber where
your uplinks are magnitudes faster. I don't know if they can continue this
approach into the 1Gb and future 10Gb PON speeds.

Prices: $40 for 20/20, $70 for 70/70, $90 for 100/100, $200 for 250/250,
/29 static block for $10/m available for all packages. And no extra/hidden
fees. Those prices are exactly what you pay, plus sales tax.
My trace route is a single hop from Level 3, which is their exclusive
transit provider. No data caps, "dedicated", told me I should never have
congestion and I should always get my provisioned rates and to call if I
get anything less than perfect.
I did take them up on this once. I was getting a 20ms ping where I normally
got a 13ms ping and I was only getting 45Mb/s on my 50Mb package at the
time. I called them, they put me in direct contact with an engineer. 15
minutes later the problem was fixed, 15 minutes after that the engineer
called back, told me they were under a DDOS attack and had to activate one
of their many failover links to Level 3 to increase their bandwidth.

Another time I was having brief transient packetloss. I called them at 2am
saying I was having an issue, they told me they'd have someone at my house
to fix the issue asap in the morning. 8am sharp, a knock on the door.
Turned out the CAT5e that existed in the house had a bad crimp.

When I signed up for their fiber service, they were still deploying fiber
all around the city. They actually gave me a direct personal contact during
the entire process. I was called the day before I was supposed to get my
connection and was told my dedicated fiber ring had an issue and
installation was going to be pushed out a week.
I told them I need the Internet for my job and I just dropped my other
provider. I was told they'd call me back. A few hours later, the entire
road was filled with a large work crew. They fixed my fiber ring and I got
my internet on the day promised. No installation fee.

I guess I went off on this tangent because "Not going to happen, in my
opinion, as economically unfeasible" and "not going to happen on any
network, publicly traded or not." are too absolute. It can be done, it is
being done, it is being done for cheap, and being done with "business
class" professionalism. Charter Comm is 1/2 the download speed for the same
price and they don't even have symmetrical or dedicated.

P.S. I think I messed up the chain. Sorry. Still getting used to this.

>
> Sure, I'm flailing.  Note this was in the context of AQSM as Daniel
> describes it.  (Possibly misnamed given it only drops.  All the queuing
> is "underneath" AQSM, "in the MAC layer" as the paper says :).
>
> - AQSM isn't distinguishing up/down bloat.  When it detects bloat it has
> to limit both directions in equal proportion.
>
> => if there is upload contention (and your user is uploading), you may
> hurt apps sensitive to download bandwidth (streaming video), when you
> don't need to.
>
> What would the solutions look like?
>
> i) If contention in one direction was negligible, you could limit the
> other direction only.  Consumer connections are highly asymmetric, and
> AQSM is only measuring the first IP hop.  So it's more feasible than
> 100% in both directions.  And this isn't about core networks (with
> larger statistical universes... whether that helps or not).
>
> I'm sure you're right and they're not asymmetric _enough_.
>
>
> ii) Sebastian points out if you implement AQSM in the modem (as the
> paper claims :p), you may as well BQL the modem drivers and run AQM.
> *But that doesn't work on ingress* - ingress requires tbf/htb with a set
> rate - but the achievable rate is lower in peak hours. So run AQSM on
> ingress only!  Point being that download bloat could be improved without
> changing the other end (CMTS).
>
> >
> > The question is not "can the theoretical max of all downstream devices
> > exceed the upstream bandwidth" because that answer is going to be
> > "yes" for every network built, LAN or WAN, but rather "does the demand
> > in practice of the combined downstream devices exceed the upstream
> > bandwidth for long enough to be a problem"
> >
> > it's not even a matter of what percentage are they oversubscribed.
> >
> > someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30%
> > of theoretical requirements) is probably a lot worse than someone with
> > 100 1G lines downstream and a 10G upstream (10% of theoretical
> > requirements) because it's far less likely that the users of the 1G
> > lines are actually going to saturate them (let alone simultaniously
> > for a noticable timeframe), while it's very likely that the users of
> > the 1.5M DSL lines are going to saturate their lines for extended
> > timeframes.
> >
> > The problem shows up when either usage changes rapidly, or the network
> > operator is not keeping up with required upgrades as gradual usage
> > changes happen (including when they are prevented from upgrading
> > because a peer won't cooperate)
> >
> > As for the "100% provisioning" ideal, think through the theoretical
> > aggregate and realize that before you get past very many layers, you
> > get to a bandwidh requirement that it's not technically possible to
> > provide.
> >
> > David Lang


On Fri, Jun 12, 2015 at 4:52 AM, MUSCARIELLO Luca IMT/OLN <
luca.muscariello@orange.com> wrote:

> On 06/12/2015 03:44 AM, David Lang wrote:
>
>> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
>>
>>
>>> On Jun 11, 2015, at 03:05 , Alan Jenkins <
>>> alan.christopher.jenkins@gmail.com> wrote:
>>>
>>>  On 10/06/15 21:54, Sebastian Moeller wrote:
>>>>
>>>> One solution would be if ISPs made sure upload is 100% provisioned.
>>>> Could be cheaper than for (the higher rate) download.
>>>>
>>>
>>>     Not going to happen, in my opinion, as economically unfeasible for a
>>> publicly traded ISP. I would settle for that approach as long as the ISP is
>>> willing to fix its provisioning so that oversubscription episodes are
>>> reasonable rare, though.
>>>
>>
>> not going to happen on any network, publicly traded or not.
>>
>> The question is not "can the theoretical max of all downstream devices
>> exceed the upstream bandwidth" because that answer is going to be "yes" for
>> every network built, LAN or WAN, but rather "does the demand in practice of
>> the combined downstream devices exceed the upstream bandwidth for long
>> enough to be a problem"
>>
>> it's not even a matter of what percentage are they oversubscribed.
>>
>> someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% of
>> theoretical requirements) is probably a lot worse than someone with 100 1G
>> lines downstream and a 10G upstream (10% of theoretical requirements)
>> because it's far less likely that the users of the 1G lines are actually
>> going to saturate them (let alone simultaniously for a noticable
>> timeframe), while it's very likely that the users of the 1.5M DSL lines are
>> going to saturate their lines for extended timeframes.
>>
>> The problem shows up when either usage changes rapidly, or the network
>> operator is not keeping up with required upgrades as gradual usage changes
>> happen (including when they are prevented from upgrading because a peer
>> won't cooperate)
>>
>
> Good points. Let me add a side comment though.
> We observe  that fiber users (e.g. 1Gbps/300Mbps access with GPON) are
> changing behavior
> w.r.t. DSL users in the way they use the uplink. This is mostly (not only)
> due to personal cloud storage availability
> and the fact that everyone today is able to produce tons of  big videos,
> that people are willing to store
> outside the home.
> As a result it's not unlikely that a backhaul link utilization may get out
> of the network planning, which is made
> on long term statistics. These workloads are unpredictable and if on one
> hand it's not feasible to
> over provision based on such unpredictable long peeks  on the other hand
> you'd need smart queue
> management to cope with such events, where the bottleneck is the backhaul.
>
> Considering the cost of current equipment upgrades, I feel like very high
> speed accesses will impose
> smart queue management everywhere from the access up to transit links,
> including the entire backhaul.
> Bad news is that no such queuing systems are available in current
> equipment, so I guess the process
> will be pretty slow to happen.
>
>
>> As for the "100% provisioning" ideal, think through the theoretical
>> aggregate and realize that before you get past very many layers, you get to
>> a bandwidh requirement that it's not technically possible to provide.
>>
>> David Lang
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>> .
>>
>>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 15807 bytes --]

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[David Lang]

On Fri, 12 Jun 2015, Benjamin Cronce wrote:

>> On 12/06/15 02:44, David Lang wrote:
>>> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
>>>
>>>>
>>>> On Jun 11, 2015, at 03:05 , Alan Jenkins
>>>> <alan.christopher.jenkins at gmail.com> wrote:
>>>>
>>>>> On 10/06/15 21:54, Sebastian Moeller wrote:
>>>>>>
>>>>> One solution would be if ISPs made sure upload is 100% provisioned.
>>>>> Could be cheaper than for (the higher rate) download.
>>>>
>>>>     Not going to happen, in my opinion, as economically unfeasible
>>>> for a publicly traded ISP. I would settle for that approach as long
>>>> as the ISP is willing to fix its provisioning so that
>>>> oversubscription episodes are reasonable rare, though.
>>>
>>> not going to happen on any network, publicly traded or not.
>
> Sorry if this is a tangent from where the current discussion has gone, but
> I wanted to correct someone saying something is "impossible".
>
<snip>
>
> I guess I went off on this tangent because "Not going to happen, in my
> opinion, as economically unfeasible" and "not going to happen on any
> network, publicly traded or not." are too absolute. It can be done, it is
> being done, it is being done for cheap, and being done with "business
> class" professionalism. Charter Comm is 1/2 the download speed for the same
> price and they don't even have symmetrical or dedicated.

not being oversubscribed includes the trunk. Who cares if there is no congestion 
within the ISP if you reach the trunk and everything comes to a screeching halt.

The reason I used the word "imossible" is because we only have the ability to 
make links so fast. Right now we have 10G common, 40G in places, and research 
into 100G, if you go back a few years, 1G was the limit. While the fastest 
connections have increased by a factor of 100, the home connections have 
increased by close to a factor of 1000 during that time (1.5Mb theoretical DSL 
vs 1Gb fiber), and 10G is getting cheap enough to be used for the corporate 
networks.

so the ratio between the fastest link that's possible and the subscribers has 
dropped from 1000:1 to 100:1 with 10:1 uncomfortably close.

some of this can be covered up by deploying more lines, but that only goes so 
far.

If you are trying to guarantee no bandwidth limits on your network under any 
conditions, you need that sort of bandwith between all possible sets of 
customers as well, which means that as you scale to more customers, your 
bandwidth requirements are going up O(n^2).

And then the pricing comes into it. 1G fiber to the home is <$100/month (if it 
exists at all) isn't going to pay for that sort of idle bandwidth.

But the good thing is, you don't actually need that much bandwidth to keep your 
customer's happy either.

If the 'guaranteed bandwidth' is written into the contracts properly, there is a 
penalty to the company if they can't provide you the bandwidth. That leaves them 
the possibility of not actually building out the O(n^2) network, just keeping 
ahead of actual reqirements, and occasionally paying a penalty if they don't.

David Lang

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[Sebastian Moeller]

Hi David,

On Jun 12, 2015, at 03:44 , David Lang <david@lang.hm> wrote:

> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
> 
>> 
>> On Jun 11, 2015, at 03:05 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>> 
>>> On 10/06/15 21:54, Sebastian Moeller wrote:
>>> 
>>> One solution would be if ISPs made sure upload is 100% provisioned. Could be cheaper than for (the higher rate) download.
>> 
>> 	Not going to happen, in my opinion, as economically unfeasible for a publicly traded ISP. I would settle for that approach as long as the ISP is willing to fix its provisioning so that oversubscription episodes are reasonable rare, though.
> 
> not going to happen on any network, publicly traded or not.
> 
> The question is not "can the theoretical max of all downstream devices exceed the upstream bandwidth" because that answer is going to be "yes" for every network built, LAN or WAN, but rather "does the demand in practice of the combined downstream devices exceed the upstream bandwidth for long enough to be a problem”

	This is what I meant to convey with “that oversubscription episodes are reasonable rare” oversubscription here as effective or realized as compared to potential. I realize this is not what “over subscribed" usually means ;). I was aiming at the fact that ISPs nned to balance their static oversubscription in a way that congestion periods on the “nodes” (whatever a node is) are are enough that customers do not jump ship.

> 
> it's not even a matter of what percentage are they oversubscribed.
> 
> someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% of theoretical requirements) is probably a lot worse than someone with 100 1G lines downstream and a 10G upstream (10% of theoretical requirements) because it's far less likely that the users of the 1G lines are actually going to saturate them (let alone simultaniously for a noticable timeframe), while it's very likely that the users of the 1.5M DSL lines are going to saturate their lines for extended timeframes.

	I assume that ISPs take such factors into account when desining their access network initially; but mainly I hope that they actually track realized congestion periods and upgrade bottleneck equipment to keep congestion periods rare.

> 
> The problem shows up when either usage changes rapidly, or the network operator is not keeping up with required upgrades as gradual usage changes happen (including when they are prevented from upgrading because a peer won't cooperate)

	Good point, I was too narrowly focussing on the access link but peering is another "hot potato”. Often end users try to use traceroute and friends and VPNs to uncontested peers to discern access-network congestion from “under-peering” even though at the end of the day the effects are similar. Thinking of it I believe that under-peering shows up more as a bandwidth loss as compared to the combined bandwidth loss and latency increase often seen on the access side (but this is conjecture as I have never seen traffic data from a congested peering connection).

> 
> As for the "100% provisioning" ideal, think through the theoretical aggregate and realize that before you get past very many layers, you get to a bandwidh requirement that it's not technically possible to provide.

	Well, I still believe that an ISP is responsible to keep its part of a contract by at least offering a considerable percentage of the sold access bandwidth into its own core network. But 100 is not going to be that percentage, I agree and I am happy to accept congestion as long as it is transiently (and I do not mean every evening it gets bad, but clears up over night, but rather that the ISP increases bandwidth to keep congestion periods rare)…

Best Regards
        Sebastian

> 
> David Lang

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[Alan Jenkins]

On 12/06/15 02:44, David Lang wrote:
> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
>
>>
>> On Jun 11, 2015, at 03:05 , Alan Jenkins 
>> <alan.christopher.jenkins@gmail.com> wrote:
>>
>>> On 10/06/15 21:54, Sebastian Moeller wrote:
>>>
>>> One solution would be if ISPs made sure upload is 100% provisioned. 
>>> Could be cheaper than for (the higher rate) download.
>>
>>     Not going to happen, in my opinion, as economically unfeasible 
>> for a publicly traded ISP. I would settle for that approach as long 
>> as the ISP is willing to fix its provisioning so that 
>> oversubscription episodes are reasonable rare, though.
>
> not going to happen on any network, publicly traded or not.

Sure, I'm flailing.  Note this was in the context of AQSM as Daniel 
describes it.  (Possibly misnamed given it only drops.  All the queuing 
is "underneath" AQSM, "in the MAC layer" as the paper says :).

- AQSM isn't distinguishing up/down bloat.  When it detects bloat it has 
to limit both directions in equal proportion.

=> if there is upload contention (and your user is uploading), you may 
hurt apps sensitive to download bandwidth (streaming video), when you 
don't need to.

What would the solutions look like?

i) If contention in one direction was negligible, you could limit the 
other direction only.  Consumer connections are highly asymmetric, and 
AQSM is only measuring the first IP hop.  So it's more feasible than 
100% in both directions.  And this isn't about core networks (with 
larger statistical universes... whether that helps or not).

I'm sure you're right and they're not asymmetric _enough_.


ii) Sebastian points out if you implement AQSM in the modem (as the 
paper claims :p), you may as well BQL the modem drivers and run AQM.  
*But that doesn't work on ingress* - ingress requires tbf/htb with a set 
rate - but the achievable rate is lower in peak hours. So run AQSM on 
ingress only!  Point being that download bloat could be improved without 
changing the other end (CMTS).

>
> The question is not "can the theoretical max of all downstream devices 
> exceed the upstream bandwidth" because that answer is going to be 
> "yes" for every network built, LAN or WAN, but rather "does the demand 
> in practice of the combined downstream devices exceed the upstream 
> bandwidth for long enough to be a problem"
>
> it's not even a matter of what percentage are they oversubscribed.
>
> someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% 
> of theoretical requirements) is probably a lot worse than someone with 
> 100 1G lines downstream and a 10G upstream (10% of theoretical 
> requirements) because it's far less likely that the users of the 1G 
> lines are actually going to saturate them (let alone simultaniously 
> for a noticable timeframe), while it's very likely that the users of 
> the 1.5M DSL lines are going to saturate their lines for extended 
> timeframes.
>
> The problem shows up when either usage changes rapidly, or the network 
> operator is not keeping up with required upgrades as gradual usage 
> changes happen (including when they are prevented from upgrading 
> because a peer won't cooperate)
>
> As for the "100% provisioning" ideal, think through the theoretical 
> aggregate and realize that before you get past very many layers, you 
> get to a bandwidh requirement that it's not technically possible to 
> provide.
>
> David Lang

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[Daniel Havey]

On Fri, Jun 12, 2015 at 6:00 AM, Alan Jenkins
<alan.christopher.jenkins@gmail.com> wrote:
> On 12/06/15 02:44, David Lang wrote:
>>
>> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
>>
>>>
>>> On Jun 11, 2015, at 03:05 , Alan Jenkins
>>> <alan.christopher.jenkins@gmail.com> wrote:
>>>
>>>> On 10/06/15 21:54, Sebastian Moeller wrote:
>>>>
>>>> One solution would be if ISPs made sure upload is 100% provisioned.
>>>> Could be cheaper than for (the higher rate) download.
>>>
>>>
>>>     Not going to happen, in my opinion, as economically unfeasible for a
>>> publicly traded ISP. I would settle for that approach as long as the ISP is
>>> willing to fix its provisioning so that oversubscription episodes are
>>> reasonable rare, though.
>>
>>
>> not going to happen on any network, publicly traded or not.
>
>
> Sure, I'm flailing.  Note this was in the context of AQSM as Daniel
> describes it.  (Possibly misnamed given it only drops.  All the queuing is
> "underneath" AQSM, "in the MAC layer" as the paper says :).
>
Noooooooooo!  I am a huge supporter of ECN.  ECE Everywhere!  I'm sure
I wrote "mark/drop" in the paper.  I might have dyslexically written
"drop/mark", but, if I ever gave the impression then I categorically
deny that right now and till forever.  ECN everywhere :^)



> - AQSM isn't distinguishing up/down bloat.  When it detects bloat it has to
> limit both directions in equal proportion.
>
> => if there is upload contention (and your user is uploading), you may hurt
> apps sensitive to download bandwidth (streaming video), when you don't need
> to.
>
> What would the solutions look like?
>
> i) If contention in one direction was negligible, you could limit the other
> direction only.  Consumer connections are highly asymmetric, and AQSM is
> only measuring the first IP hop.  So it's more feasible than 100% in both
> directions.  And this isn't about core networks (with larger statistical
> universes... whether that helps or not).
>
> I'm sure you're right and they're not asymmetric _enough_.
>
>
> ii) Sebastian points out if you implement AQSM in the modem (as the paper
> claims :p), you may as well BQL the modem drivers and run AQM.  *But that
> doesn't work on ingress* - ingress requires tbf/htb with a set rate - but
> the achievable rate is lower in peak hours. So run AQSM on ingress only!
> Point being that download bloat could be improved without changing the other
> end (CMTS).
>
This is pretty cool.  I had not considered BQL (though Dave and Jim
were evangelizing about it at the time :).  This solves the
upload/download problem which I was not able to get past in the paper.
BQL on the egress and ASQM for the ingress.  BQL will make sure that
the upload is under control so that ASQM can get a good measurement on
the download side.  Woot!  Woot!  Uncooperative ISP problem solved!

BTW...Why doesn't BQL work on the ingress?

>>
>> The question is not "can the theoretical max of all downstream devices
>> exceed the upstream bandwidth" because that answer is going to be "yes" for
>> every network built, LAN or WAN, but rather "does the demand in practice of
>> the combined downstream devices exceed the upstream bandwidth for long
>> enough to be a problem"
>>
>> it's not even a matter of what percentage are they oversubscribed.
>>
>> someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% of
>> theoretical requirements) is probably a lot worse than someone with 100 1G
>> lines downstream and a 10G upstream (10% of theoretical requirements)
>> because it's far less likely that the users of the 1G lines are actually
>> going to saturate them (let alone simultaniously for a noticable timeframe),
>> while it's very likely that the users of the 1.5M DSL lines are going to
>> saturate their lines for extended timeframes.
>>
>> The problem shows up when either usage changes rapidly, or the network
>> operator is not keeping up with required upgrades as gradual usage changes
>> happen (including when they are prevented from upgrading because a peer
>> won't cooperate)
>>
>> As for the "100% provisioning" ideal, think through the theoretical
>> aggregate and realize that before you get past very many layers, you get to
>> a bandwidh requirement that it's not technically possible to provide.
>>
>> David Lang
>
>

Yuppers!  Dave is right.  The FCC studies (especially the 80/80 study
out of UNC) from 2010 - 2014 (footnoted in the paper) indicate that
during peak hours it is quite common for an ISP not to provide 100% of
the rated throughput.  In fact in 2014 it indicates that a full on 50%
of the ISPs measured provided less than 100%.  The 100% all the time
goal is unreasonable because it implies too much waste.  Many ISPs get
to 90% or above even during peak hours.  This is good!  We could live
with that :)  Providing that last 10% would mean that they would have
to provide for a lot of excess capacity that goes unused during
non-peak hours.  Wasteful.  That money should be allocated for more
important things like providing AQM for all or saving the planet or
something else. :^)

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[Alan Jenkins]

On 12/06/15 15:35, Daniel Havey wrote:
> On Fri, Jun 12, 2015 at 6:00 AM, Alan Jenkins
> <alan.christopher.jenkins@gmail.com> wrote:
>> On 12/06/15 02:44, David Lang wrote:
>>> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
>>>
>>>> On Jun 11, 2015, at 03:05 , Alan Jenkins
>>>> <alan.christopher.jenkins@gmail.com> wrote:
>>>>
>>>>> On 10/06/15 21:54, Sebastian Moeller wrote:
>>>>>
>>>>> One solution would be if ISPs made sure upload is 100% provisioned.
>>>>> Could be cheaper than for (the higher rate) download.
>>>>
>>>>      Not going to happen, in my opinion, as economically unfeasible for a
>>>> publicly traded ISP. I would settle for that approach as long as the ISP is
>>>> willing to fix its provisioning so that oversubscription episodes are
>>>> reasonable rare, though.
>>>
>>> not going to happen on any network, publicly traded or not.
>>
>> Sure, I'm flailing.  Note this was in the context of AQSM as Daniel
>> describes it.  (Possibly misnamed given it only drops.  All the queuing is
>> "underneath" AQSM, "in the MAC layer" as the paper says :).
>>
> Noooooooooo!  I am a huge supporter of ECN.  ECE Everywhere!  I'm sure
> I wrote "mark/drop" in the paper.  I might have dyslexically written
> "drop/mark", but, if I ever gave the impression then I categorically
> deny that right now and till forever.  ECN everywhere :^)

My bad!  Imagine I wrote mark/drop.  It was just a definitional wibble.  
Is it AQM if it's not your Q, is it policing (and does AQM include 
policing)?

>> - AQSM isn't distinguishing up/down bloat.  When it detects bloat it has to
>> limit both directions in equal proportion.
>>
>> => if there is upload contention (and your user is uploading), you may hurt
>> apps sensitive to download bandwidth (streaming video), when you don't need
>> to.
>>
>> What would the solutions look like?
>>
>> i) If contention in one direction was negligible, you could limit the other
>> direction only.  Consumer connections are highly asymmetric, and AQSM is
>> only measuring the first IP hop.  So it's more feasible than 100% in both
>> directions.  And this isn't about core networks (with larger statistical
>> universes... whether that helps or not).
>>
>> I'm sure you're right and they're not asymmetric _enough_.
>>
>>
>> ii) Sebastian points out if you implement AQSM in the modem (as the paper
>> claims :p), you may as well BQL the modem drivers and run AQM.  *But that
>> doesn't work on ingress* - ingress requires tbf/htb with a set rate - but
>> the achievable rate is lower in peak hours. So run AQSM on ingress only!
>> Point being that download bloat could be improved without changing the other
>> end (CMTS).
>>
> This is pretty cool.  I had not considered BQL (though Dave and Jim
> were evangelizing about it at the time :).  This solves the
> upload/download problem which I was not able to get past in the paper.
> BQL on the egress and ASQM for the ingress.  BQL will make sure that
> the upload is under control so that ASQM can get a good measurement on
> the download side.  Woot!  Woot!  Uncooperative ISP problem solved!
>
> BTW...Why doesn't BQL work on the ingress?

Both AQM & BQL manage transmit queues.  It's the natural way to work.

If you don't control the transmit queue(s) at the bottleneck, you have 
to insert an artificial bottleneck that you _do_ control.

The problem is we don't have the source to hack any modems, only Linux 
routers like in your paper :).  It's why, to set up a router with our 
SQM, you have to know the line rate (both directions).  And tell SQM to 
use maybe 95% of it.  And assume the full rate is available even during 
peak hours :(.

So we still want co-operative ISPs to solve this, because that's who 
procures the modems.  That said, once they've been _developed_ it's 
going to be easier to buy a good modem regardless of ISP.


We rate-limit with htb, so we build a queue for fq_codel to work on.  
(Then Johnathan smushed the two qdiscs together and called the resulting 
code "cake").

Regarding ingress rate-limiting, note that

  - it involves creating a virtual network interface (IFB), to get an 
artificial transmit queue to apply AQM to.   (Something like that, I 
don't know the exact syntax).
  - bursts can still pile up at the upstream bottleneck, so you can't 
eliminate latency increases altogether.  (E.g. caused by tcp's initial 
multiplicative increase).  It can still be a big win though, because isp 
queues would often be allowed to grow many times larger than needed 
(expected rtt / ~100ms).

>>> The question is not "can the theoretical max of all downstream devices
>>> exceed the upstream bandwidth" because that answer is going to be "yes" for
>>> every network built, LAN or WAN, but rather "does the demand in practice of
>>> the combined downstream devices exceed the upstream bandwidth for long
>>> enough to be a problem"
>>>
>>> it's not even a matter of what percentage are they oversubscribed.
>>>
>>> someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% of
>>> theoretical requirements) is probably a lot worse than someone with 100 1G
>>> lines downstream and a 10G upstream (10% of theoretical requirements)
>>> because it's far less likely that the users of the 1G lines are actually
>>> going to saturate them (let alone simultaniously for a noticable timeframe),
>>> while it's very likely that the users of the 1.5M DSL lines are going to
>>> saturate their lines for extended timeframes.
>>>
>>> The problem shows up when either usage changes rapidly, or the network
>>> operator is not keeping up with required upgrades as gradual usage changes
>>> happen (including when they are prevented from upgrading because a peer
>>> won't cooperate)
>>>
>>> As for the "100% provisioning" ideal, think through the theoretical
>>> aggregate and realize that before you get past very many layers, you get to
>>> a bandwidh requirement that it's not technically possible to provide.
>>>
>>> David Lang
>>
> Yuppers!  Dave is right.  The FCC studies (especially the 80/80 study
> out of UNC) from 2010 - 2014 (footnoted in the paper) indicate that
> during peak hours it is quite common for an ISP not to provide 100% of
> the rated throughput.

Note the same technical problem applies if they provide 100%-150%.[1]  
If bloat is a problem, you can probably still hack around it as a user, 
but then you lose the _benefits_ of the oversubscription.

[1] Comcast "Powerboost" http://www.dslreports.com/faq/14520

>    In fact in 2014 it indicates that a full on 50%
> of the ISPs measured provided less than 100%.  The 100% all the time
> goal is unreasonable because it implies too much waste.  Many ISPs get
> to 90% or above even during peak hours.  This is good!  We could live
> with that :)  Providing that last 10% would mean that they would have
> to provide for a lot of excess capacity that goes unused during
> non-peak hours.  Wasteful.  That money should be allocated for more
> important things like providing AQM for all or saving the planet or
> something else. :^)

:)

I grant Dave's caveats.  Couldn't resist looking for the graphs. (Around 
page 40 
http://data.fcc.gov/download/measuring-broadband-america/2014/2014-Fixed-Measuring-Broadband-America-Report.pdf).

It looks like cable ISPs happen to hit an average of 100% during peak 
hours (as you might hope from figure of 50% going below, apparently 50% 
are going above).

Alan

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[David Lang]

On Fri, 12 Jun 2015, Daniel Havey wrote:

>> ii) Sebastian points out if you implement AQSM in the modem (as the paper
>> claims :p), you may as well BQL the modem drivers and run AQM.  *But that
>> doesn't work on ingress* - ingress requires tbf/htb with a set rate - but
>> the achievable rate is lower in peak hours. So run AQSM on ingress only!
>> Point being that download bloat could be improved without changing the other
>> end (CMTS).
>>
> This is pretty cool.  I had not considered BQL (though Dave and Jim
> were evangelizing about it at the time :).  This solves the
> upload/download problem which I was not able to get past in the paper.
> BQL on the egress and ASQM for the ingress.  BQL will make sure that
> the upload is under control so that ASQM can get a good measurement on
> the download side.  Woot!  Woot!  Uncooperative ISP problem solved!
>
> BTW...Why doesn't BQL work on the ingress?

for the same reason that AQM doesn't work on inbound connections. you're on the 
wrong side of the link :-)

implementing AQM and BQL on the ISPs device that's sending to the home user is 
a perfectly reasonable thing to do and is really the right answer for the 
problem.

all the rest of this is a matter of "we can't get at where the problem really is 
to fix it, so let's figure out what else we can do to mitigate the problem"

David Lang

Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management

[David Lang]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2811 bytes --]

On Fri, 12 Jun 2015, Sebastian Moeller wrote:

> Hi David,
>
> On Jun 12, 2015, at 03:44 , David Lang <david@lang.hm> wrote:
>
>> The problem shows up when either usage changes rapidly, or the network 
>> operator is not keeping up with required upgrades as gradual usage changes 
>> happen (including when they are prevented from upgrading because a peer won't 
>> cooperate)
>
> 	Good point, I was too narrowly focussing on the access link but peering 
> is another "hot potato”. Often end users try to use traceroute and friends and 
> VPNs to uncontested peers to discern access-network congestion from 
> “under-peering” even though at the end of the day the effects are similar. 
> Thinking of it I believe that under-peering shows up more as a bandwidth loss 
> as compared to the combined bandwidth loss and latency increase often seen on 
> the access side (but this is conjecture as I have never seen traffic data from 
> a congested peering connection).

At the peering point where congestion happens, queues will expand to the max 
avaialble and packets will be dropped. Layer 3 had some posts showing stats of a 
congested peeing point back before netflix caved a year or so ago.

>>
>> As for the "100% provisioning" ideal, think through the theoretical aggregate 
>> and realize that before you get past very many layers, you get to a bandwidh 
>> requirement that it's not technically possible to provide.
>
> 	Well, I still believe that an ISP is responsible to keep its part of a 
> contract by at least offering a considerable percentage of the sold access 
> bandwidth into its own core network. But 100 is not going to be that 
> percentage, I agree and I am happy to accept congestion as long as it is 
> transiently (and I do not mean every evening it gets bad, but clears up over 
> night, but rather that the ISP increases bandwidth to keep congestion periods 
> rare)…

I think that the target the ISP should be striving for is 0 congestion, not 0 
overprovisioning. And I deliberately say "striving for" because it's not going 
to be perfect. And the question of if you are congested or not depends on the 
timescale you look at (at any instant, a given link is either 0% utilized or 
100% utilized, nowhere in between)

Good AQM will make it so that when congestion does happen, all that happens is 
that the bandwidth ends up getting shared. Everyone continues to operate with 
minimal noticable degredation (ideally all suffered by the non-time critical 
bulk data transfers)

After all, if you are streaming a video from netflix, does it really matter if 
the 2-hour movie is entirely delivered to your local box in 1 min instead of 20 
min? If you're downloading it to put it on a mobile device before you leave, 
possibly, but if you're just watching it, not at all.

David Lang

Re: [Cerowrt-devel] [Cake] [Bloat] active sensing queue management

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 5267 bytes --]

> On Fri, 12 Jun 2015, Benjamin Cronce wrote:
>
> >> On 12/06/15 02:44, David Lang wrote:
> >>> On Thu, 11 Jun 2015, Sebastian Moeller wrote:
> >>>
> >>>>
> >>>> On Jun 11, 2015, at 03:05 , Alan Jenkins
> >>>> <alan.christopher.jenkins at gmail.com> wrote:
> >>>>
> >>>>> On 10/06/15 21:54, Sebastian Moeller wrote:
> >>>>>>
> >>>>> One solution would be if ISPs made sure upload is 100% provisioned.
> >>>>> Could be cheaper than for (the higher rate) download.
> >>>>
> >>>>     Not going to happen, in my opinion, as economically unfeasible
> >>>> for a publicly traded ISP. I would settle for that approach as long
> >>>> as the ISP is willing to fix its provisioning so that
> >>>> oversubscription episodes are reasonable rare, though.
> >>>
> >>> not going to happen on any network, publicly traded or not.
> >
> > Sorry if this is a tangent from where the current discussion has gone,
but
> > I wanted to correct someone saying something is "impossible".
> >
> <snip>
> >
> > I guess I went off on this tangent because "Not going to happen, in my
> > opinion, as economically unfeasible" and "not going to happen on any
> > network, publicly traded or not." are too absolute. It can be done, it
is
> > being done, it is being done for cheap, and being done with "business
> > class" professionalism. Charter Comm is 1/2 the download speed for the
same
> > price and they don't even have symmetrical or dedicated.
>
> not being oversubscribed includes the trunk. Who cares if there is no
congestion
> within the ISP if you reach the trunk and everything comes to a
screeching halt.
>
> The reason I used the word "imossible" is because we only have the
ability to
> make links so fast. Right now we have 10G common, 40G in places, and
research
> into 100G, if you go back a few years, 1G was the limit. While the
fastest
> connections have increased by a factor of 100, the home connections have
> increased by close to a factor of 1000 during that time (1.5Mb
theoretical DSL
> vs 1Gb fiber), and 10G is getting cheap enough to be used for the
corporate
> networks.

100Gb ports only cost about $5k and you can purchase, for unlisted prices,
36Tb/s mutliplexers over a single fiber with 1200km ranges. That's enough
bandwidth for 360 100Gb ports. Congestion on trunks are easy because of the
large number of flows, you can have strong guarantees via statistical
multiplexing. The last mile is the main issue because that's where a single
person can make a difference.

>
> so the ratio between the fastest link that's possible and the subscribers
has
> dropped from 1000:1 to 100:1 with 10:1 uncomfortably close.
>
> some of this can be covered up by deploying more lines, but that only
goes so
> far.
>
> If you are trying to guarantee no bandwidth limits on your network under
any
> conditions, you need that sort of bandwith between all possible sets of
> customers as well, which means that as you scale to more customers, your
> bandwidth requirements are going up O(n^2).

O(n^2) is only needed if you desire a fully connected graph. When I have a
100Mb connection, it doesn't mean I get a 100Mb connection to every
customer at the same time, I get 100Mb total. Otherwise with 1000 customer,
that would mean I would have to have 100Gb link at home. You only need O(n)
scaling. Anyway, I don't think you fully appreciate modern 1
petabit(500Tb/s ingress+egress) line rate core routers.

This is all commercially available hardware, but would of course be
incredibly expensive. 1 petabit router, with 1,000 500Gb/s ports, each
500Gb port plugs into a fiber chassis with 500 1Gb ports. Now you have
500,000 people all with fully dedicated 1Gb/s connections. Of course you do
not have any internet access with this setup because all of the router's
ports are used for connecting customers. Cut the customers in half, 250k
customer, that frees up half of the ports. Now you have 500 500Gb ports
freed up. Take those 500Gb ports, plug those into 36Tb/s multiplexers, get
14 strands of fiber, 7 up and 7 down, and run them to your local exchange
points. Mind you this system would have 10x more bandwidth than the entire
world wide internet peak data usage. That includes all CDNs, trunks,
everything.

Of course the average user consumes less than 10Mb/s during peak hours,
even if they have a 1Gb connection. All that matters is peak usage.
Everything about society is based on actual usage, not theoreticals that
are about as likely as a duplicate Earth appearing next to us in space.
Yes, it is possible, just not likely. Our electrical infrastructure could
not handle every user at once, neither can out water, roads, phones,
hospitals, stores, etc.

> And then the pricing comes into it. 1G fiber to the home is <$100/month
(if it
> exists at all) isn't going to pay for that sort of idle bandwidth.
>
> But the good thing is, you don't actually need that much bandwidth to
keep your
> customer's happy either.
>
> If the 'guaranteed bandwidth' is written into the contracts properly,
there is a
> penalty to the company if they can't provide you the bandwidth. That
leaves them
> the possibility of not actually building out the O(n^2) network, just
keeping
> ahead of actual reqirements, and occasionally paying a penalty if they
don't.
>
> David Lang

[-- Attachment #2: Type: text/html, Size: 6726 bytes --]