[Bloat] Testing Queue models

[Bloat] Testing Queue models

[Dave Taht]

Now that I have BQL and the new SFQ, ARED, and SFQRED working in
cerowrt, I have been working on modeling the behavior of existing
shapers (wshaper, openwrt, sfqred, and something more torrent-proof
that doesn't have a name yet)

There are several things that will get in the way of doing it well at
the present time, and I hope to evolve towards something that works
well in nearly all cases. For purposes of this discussion I'll use a
few abbreviations.

"u" = user
"d" = device
"f" = flow

Some general notes:

Accurately determining uplink bandwidth is something that no tool does
quite right today, and even then uplink bandwidth can be dynamic
(notably on cable). Van and Kathie are working on something that can
figure it out better, dynamically, but that code's not ready yet.
Instead I'm trying to pull together a shaperprobe that detects modern
conditions better, statically, at least.

Uplink speed is the principal bottleneck problem that users deal with.

I'm running SFQ on all interfaces. This helps break up streams into
more manageable chunks. It can be really NICE, on ethernet. In fact,
most of my network bottlenecks have moved to my desktops and laptops
in the general case, and to get best results I have to run the debloat
script and enable SFQ everywhere, in this wack-a-mole game.

SFQ has positive effects on wireless as sparse streams move forward in
the queue, and are thus shipped as an aggregate at the earliest
opportunity. Regrettably there is so much rate-insensitive fixed
buffering at present in the Linux wireless stack as to incur huge
penalties at variable rates.

It can have negative side effects on wireless, as it mucks with packet
aggregation across devices. That said, in home use (rather than with
dozens of wireless devices) it should help somewhat. I'm not worrying
about wireless behavior all that much right now, I'm mostly trying to
exercise the new code... as fixing the aggregation problems really
needs to have a per-destination queue somehow.

Developing correct configuration for RED and ARED is something of a dark art.

The new SFQ prioritizes new streams very well. Perhaps overly well.
Eric has a patch in progress
to make it have more of an idea of 'how new' a stream is. It otherwise
is very effective in reducing latency for short, sparse, streams.

SFQRED's mental model is closer to what an ISP would use on it's
downstream link. It is not (quite) right for what a site would use as
an uplink.

QFQ, which is what I was using to get 1/d level behavior in the
prototype I did several months back, still is hanging under load on
cerowrt. It's working on x86.  :( Might try working with classful SFQ
stuff instead, or I may give DRR a shot instead.

Even getting 1/d is hard because the filters can only use IP and IPv6
addresses, so a device issuing traffic on both ipv4 and ipv6 can get
twice what it deserves in terms of bandwidth, or more if it has
multiple ipv6 addresses.

My preferred solution to this (and part of the wireless problem) is to
sort things out by source mac address. How to do that, remains a
mystery. The way I'm leaning is to invent some sort of "src mac-to-id"
filter that is global router-wide.

I tend to view (in the home) as having 1/u network performance as the
ideal. There are exceptions to this, notably video vs. say,
bittorrent. A clever way of getting closer to 1/u might be to sense
for more recent DNS queries and move that to a more interactive class.

I got 1/d level performance, vs bittorrent, several months back, using
a combination of HTB, and a
two tier QFQ, then red model. I was happy with this, but QFQ was
required. Perhaps I can make
SFQRED do this with the right filters, or switch to a combination of
DRR and SFQRED....

Anyway, at the present time, I'm trying a few approaches, the most
promising one is a two or three tier model using SFQRED, with a very
limited number of things prio 1 (dns), 2 (mostly everything), 3
classified and background traffic and torrent, where classifiable.
Classification is a rathole however, and I'd rather be aiming for 1/d
than 1/f.

Regardless openwrt's default AQM does lousy queue depth management at
present, and I expect to
improve it tremendously over the coming week.

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Bloat] Testing Queue models

[Sebastian Moeller]

Hi Dave,


On Feb 1, 2012, at 10:17 AM, Dave Taht wrote:

> can you reply to the lists for details like this?

	Sorry, forgot to reply all, I hope I fixed it now.

> 
> The ADSL-shaper work that went into the stab option was pretty
> fundamental, and needed.
> 
> A) Fix openwrt's shaper to be more right for their distro (which
> doesn't have SFQRED nor QFQ)
	Okay, I will try to coordinate with upstream if that can be somehow included in vanilla openwork (even though I hope that vanilla will switch to your new scripts soon :) ) This might take a while, but It is on my todo list, But if vanilla sticks to its old shaper and shaper setup scripts than sharing this with cerowrt is going to be only in idea not in implementation...

> B) Fix it to be right on cerowrt
	This is I guess were I need your guidance (and where I actually need to switch to the most recent cerowrt first, I am still on ocean city…)

> C) Clean up the web interfaces to make it obvious that DSL can be 'special'
	That is beyond my capabilities, I am a scripting king of person (I might give it a try but no ETA, ...)

> 
> But first up, is models and analysis. Measure twice, cut once….

	Yeah, I guess I will try the current cerowrt, hack stab inside and see whether it still makes a difference (I assume it will), please just ignore me until then…


> 
> I'm puzzled about your mtu setting, however…

from tc-stab man page:
"mtu
maximum packet size we create size table for, assumed 2048 if not specified explicitly"
so this does not affect the real mtu. Opencoding this default was meant as a reminder for me to go back and try understanding and implement the following part of the man page:
"For ATM size tables, 16~bytes sized slots are perfectly enough. The default values of mtu and tsize create 4~bytes sized slots."
But I did not get around to that...

Best
	Sebastian


> 
> On Wed, Feb 1, 2012 at 6:13 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> Hi Dave,
>> 
>> cool work!
>> 
>> One thing I noticed (ocean city on ADSL over ATM) is the I really needed to use tc's stab option in combination with the default openwork shaper to get latencies down to a useful level.
>> My crude hack was adding:
>> tc_stab_string="stab overhead 18 mtu 2048 mpu 53 linklayer atm"
>> to generate.sh; and then modifying the addition of root disc like the following
>> tc qdisc add dev $dev root handle 1: ${tc_stab_string} hfsc default ${class_default}0
>> Then I only needed to reduce the uplink and downlink speed marginally (5%) and got good and stable ping latencies even in the lieu of massive uploads and opening 100 browser tabs at the same time. Without the stab option I had to reduce nominal speeds to around 65%-70% of the line rate and still got worse ping latencies than with the stab option.
>>        Obviously the exact invocation of the stab depends on a number of factors, like the encapsulation used; since I do not know a generic way of discovering those automatically (unless the del modem is part of the router) I think this needs to be user editable...
>> 
>> So it would be sweet if there would be an easy way (either per editor or GUI) to actually include stab options into the shaper to deal with this DSL over ATM specific issue.
>> 
>> 
>> Thanks for deflating the buffers and rescuing decent latencies…
>> 
>> Best
>>        Sebastian
>> 
>> 
>> On Feb 1, 2012, at 9:12 AM, Dave Taht wrote:
>> 
>>> Now that I have BQL and the new SFQ, ARED, and SFQRED working in
>>> cerowrt, I have been working on modeling the behavior of existing
>>> shapers (wshaper, openwrt, sfqred, and something more torrent-proof
>>> that doesn't have a name yet)
>>> 
>>> There are several things that will get in the way of doing it well at
>>> the present time, and I hope to evolve towards something that works
>>> well in nearly all cases. For purposes of this discussion I'll use a
>>> few abbreviations.
>>> 
>>> "u" = user
>>> "d" = device
>>> "f" = flow
>>> 
>>> Some general notes:
>>> 
>>> Accurately determining uplink bandwidth is something that no tool does
>>> quite right today, and even then uplink bandwidth can be dynamic
>>> (notably on cable). Van and Kathie are working on something that can
>>> figure it out better, dynamically, but that code's not ready yet.
>>> Instead I'm trying to pull together a shaperprobe that detects modern
>>> conditions better, statically, at least.
>>> 
>>> Uplink speed is the principal bottleneck problem that users deal with.
>>> 
>>> I'm running SFQ on all interfaces. This helps break up streams into
>>> more manageable chunks. It can be really NICE, on ethernet. In fact,
>>> most of my network bottlenecks have moved to my desktops and laptops
>>> in the general case, and to get best results I have to run the debloat
>>> script and enable SFQ everywhere, in this wack-a-mole game.
>>> 
>>> SFQ has positive effects on wireless as sparse streams move forward in
>>> the queue, and are thus shipped as an aggregate at the earliest
>>> opportunity. Regrettably there is so much rate-insensitive fixed
>>> buffering at present in the Linux wireless stack as to incur huge
>>> penalties at variable rates.
>>> 
>>> It can have negative side effects on wireless, as it mucks with packet
>>> aggregation across devices. That said, in home use (rather than with
>>> dozens of wireless devices) it should help somewhat. I'm not worrying
>>> about wireless behavior all that much right now, I'm mostly trying to
>>> exercise the new code... as fixing the aggregation problems really
>>> needs to have a per-destination queue somehow.
>>> 
>>> Developing correct configuration for RED and ARED is something of a dark art.
>>> 
>>> The new SFQ prioritizes new streams very well. Perhaps overly well.
>>> Eric has a patch in progress
>>> to make it have more of an idea of 'how new' a stream is. It otherwise
>>> is very effective in reducing latency for short, sparse, streams.
>>> 
>>> SFQRED's mental model is closer to what an ISP would use on it's
>>> downstream link. It is not (quite) right for what a site would use as
>>> an uplink.
>>> 
>>> QFQ, which is what I was using to get 1/d level behavior in the
>>> prototype I did several months back, still is hanging under load on
>>> cerowrt. It's working on x86.  :( Might try working with classful SFQ
>>> stuff instead, or I may give DRR a shot instead.
>>> 
>>> Even getting 1/d is hard because the filters can only use IP and IPv6
>>> addresses, so a device issuing traffic on both ipv4 and ipv6 can get
>>> twice what it deserves in terms of bandwidth, or more if it has
>>> multiple ipv6 addresses.
>>> 
>>> My preferred solution to this (and part of the wireless problem) is to
>>> sort things out by source mac address. How to do that, remains a
>>> mystery. The way I'm leaning is to invent some sort of "src mac-to-id"
>>> filter that is global router-wide.
>>> 
>>> I tend to view (in the home) as having 1/u network performance as the
>>> ideal. There are exceptions to this, notably video vs. say,
>>> bittorrent. A clever way of getting closer to 1/u might be to sense
>>> for more recent DNS queries and move that to a more interactive class.
>>> 
>>> I got 1/d level performance, vs bittorrent, several months back, using
>>> a combination of HTB, and a
>>> two tier QFQ, then red model. I was happy with this, but QFQ was
>>> required. Perhaps I can make
>>> SFQRED do this with the right filters, or switch to a combination of
>>> DRR and SFQRED....
>>> 
>>> Anyway, at the present time, I'm trying a few approaches, the most
>>> promising one is a two or three tier model using SFQRED, with a very
>>> limited number of things prio 1 (dns), 2 (mostly everything), 3
>>> classified and background traffic and torrent, where classifiable.
>>> Classification is a rathole however, and I'd rather be aiming for 1/d
>>> than 1/f.
>>> 
>>> Regardless openwrt's default AQM does lousy queue depth management at
>>> present, and I expect to
>>> improve it tremendously over the coming week.
>>> 
>>> --
>>> Dave Täht
>>> SKYPE: davetaht
>>> US Tel: 1-239-829-5608
>>> FR Tel: 0638645374
>>> http://www.bufferbloat.net
>>> _______________________________________________
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>> 
> 
> 
> 
> -- 
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> FR Tel: 0638645374
> http://www.bufferbloat.net

Re: [Bloat] Testing Queue models

[Dave Taht]

On Wed, Feb 1, 2012 at 7:17 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Dave,
>
>
> On Feb 1, 2012, at 10:17 AM, Dave Taht wrote:
>
>> can you reply to the lists for details like this?
>
>        Sorry, forgot to reply all, I hope I fixed it now.
>
>>
>> The ADSL-shaper work that went into the stab option was pretty
>> fundamental, and needed.
>>
>> A) Fix openwrt's shaper to be more right for their distro (which
>> doesn't have SFQRED nor QFQ)
>        Okay, I will try to coordinate with upstream if that can be somehow included in vanilla openwork (even though I hope that vanilla will switch to your new scripts soon :) ) This might take a while, but It is on my todo list, But if vanilla sticks to its old shaper and shaper setup scripts than sharing this with cerowrt is going to be only in idea not in implementation...

Well, there are several 'easy' fixes.

1) openwrt does not do any ipv6 classification at all presently. Easy
to fix by merely calling
ip6tables as well.
2) it's unclear to me how encapsulated packets (gre, 6in4) etc are handled
3) I'd like to campaign for the removal of esfq entirely from the
build. SFQ long ago
incorporated those features.
4) The queue depth for sfq is crazy.

I'll put more details up on the related bug
http://www.bufferbloat.net/issues/331

but first up to me is modeling the existing behavior, and finding
scenarios where
it misbehaves under common current in-home workloads.

If I have any overall drive, it's to get to where it's possible to do
a bakeoff of various models.
I hope to enlist as many as possible (such as yourself!) in the field,
as well as in (maybe a few) labs.

http://www.bufferbloat.net/issues/301

Queuing behavior is not easy to understand in the first place, most
papers use an avpkt of 1000, when about 80 is closer to normal for
home use, the effects of packet aggregation are not well modeled, and
while it has long been noted that head drop, and random drop, and ecn
look like good ideas, nobody has any real experience with them in real
situations.

One of my biggest concerns is with ledbat (bittorrent) behavior, which
has been only modeled with invalid models to date (IMHO). Another
concern is with the observed behavior of things like netflix and
youtube, which tend to send bursty streams....

That's a lot of new variables to play with!

>> B) Fix it to be right on cerowrt
>        This is I guess were I need your guidance (and where I actually need to switch to the most recent cerowrt first, I am still on ocean city…)

Well, bql-30 turns out to have a kernel configuration error or two
that I'm hopefully correcting now... pimv2 multicast isn't working for
some reason, and I forgot to build oprofile and NO_HZ support. That
said, it seems to work...

>
>> C) Clean up the web interfaces to make it obvious that DSL can be 'special'
>        That is beyond my capabilities, I am a scripting king of person (I might give it a try but no ETA, ...)
>

I have spent most of my spare time in the last 2 months learning
enough of lua to make a dent in the gui problem. I am emphatically not
a UI person, however, and anything I design seems likely to be
overcomplex and confusing to the end user.

>>
>> But first up, is models and analysis. Measure twice, cut once….
>
>        Yeah, I guess I will try the current cerowrt, hack stab inside and see whether it still makes a difference (I assume it will), please just ignore me until then…

Please do. BUT, feel free to wait a week, the new stuff is shaping up quickly.

>
>
>>
>> I'm puzzled about your mtu setting, however…
>
> from tc-stab man page:
> "mtu
> maximum packet size we create size table for, assumed 2048 if not specified explicitly"
> so this does not affect the real mtu. Opencoding this default was meant as a reminder for me to go back and try understanding and implement the following part of the man page:
> "For ATM size tables, 16~bytes sized slots are perfectly enough. The default values of mtu and tsize create 4~bytes sized slots."
> But I did not get around to that...
>
> Best
>        Sebastian
>
>
>>
>> On Wed, Feb 1, 2012 at 6:13 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> Hi Dave,
>>>
>>> cool work!
>>>
>>> One thing I noticed (ocean city on ADSL over ATM) is the I really needed to use tc's stab option in combination with the default openwork shaper to get latencies down to a useful level.
>>> My crude hack was adding:
>>> tc_stab_string="stab overhead 18 mtu 2048 mpu 53 linklayer atm"
>>> to generate.sh; and then modifying the addition of root disc like the following
>>> tc qdisc add dev $dev root handle 1: ${tc_stab_string} hfsc default ${class_default}0
>>> Then I only needed to reduce the uplink and downlink speed marginally (5%) and got good and stable ping latencies even in the lieu of massive uploads and opening 100 browser tabs at the same time. Without the stab option I had to reduce nominal speeds to around 65%-70% of the line rate and still got worse ping latencies than with the stab option.
>>>        Obviously the exact invocation of the stab depends on a number of factors, like the encapsulation used; since I do not know a generic way of discovering those automatically (unless the del modem is part of the router) I think this needs to be user editable...
>>>
>>> So it would be sweet if there would be an easy way (either per editor or GUI) to actually include stab options into the shaper to deal with this DSL over ATM specific issue.
>>>
>>>
>>> Thanks for deflating the buffers and rescuing decent latencies…
>>>
>>> Best
>>>        Sebastian
>>>
>>>
>>> On Feb 1, 2012, at 9:12 AM, Dave Taht wrote:
>>>
>>>> Now that I have BQL and the new SFQ, ARED, and SFQRED working in
>>>> cerowrt, I have been working on modeling the behavior of existing
>>>> shapers (wshaper, openwrt, sfqred, and something more torrent-proof
>>>> that doesn't have a name yet)
>>>>
>>>> There are several things that will get in the way of doing it well at
>>>> the present time, and I hope to evolve towards something that works
>>>> well in nearly all cases. For purposes of this discussion I'll use a
>>>> few abbreviations.
>>>>
>>>> "u" = user
>>>> "d" = device
>>>> "f" = flow
>>>>
>>>> Some general notes:
>>>>
>>>> Accurately determining uplink bandwidth is something that no tool does
>>>> quite right today, and even then uplink bandwidth can be dynamic
>>>> (notably on cable). Van and Kathie are working on something that can
>>>> figure it out better, dynamically, but that code's not ready yet.
>>>> Instead I'm trying to pull together a shaperprobe that detects modern
>>>> conditions better, statically, at least.
>>>>
>>>> Uplink speed is the principal bottleneck problem that users deal with.
>>>>
>>>> I'm running SFQ on all interfaces. This helps break up streams into
>>>> more manageable chunks. It can be really NICE, on ethernet. In fact,
>>>> most of my network bottlenecks have moved to my desktops and laptops
>>>> in the general case, and to get best results I have to run the debloat
>>>> script and enable SFQ everywhere, in this wack-a-mole game.
>>>>
>>>> SFQ has positive effects on wireless as sparse streams move forward in
>>>> the queue, and are thus shipped as an aggregate at the earliest
>>>> opportunity. Regrettably there is so much rate-insensitive fixed
>>>> buffering at present in the Linux wireless stack as to incur huge
>>>> penalties at variable rates.
>>>>
>>>> It can have negative side effects on wireless, as it mucks with packet
>>>> aggregation across devices. That said, in home use (rather than with
>>>> dozens of wireless devices) it should help somewhat. I'm not worrying
>>>> about wireless behavior all that much right now, I'm mostly trying to
>>>> exercise the new code... as fixing the aggregation problems really
>>>> needs to have a per-destination queue somehow.
>>>>
>>>> Developing correct configuration for RED and ARED is something of a dark art.
>>>>
>>>> The new SFQ prioritizes new streams very well. Perhaps overly well.
>>>> Eric has a patch in progress
>>>> to make it have more of an idea of 'how new' a stream is. It otherwise
>>>> is very effective in reducing latency for short, sparse, streams.
>>>>
>>>> SFQRED's mental model is closer to what an ISP would use on it's
>>>> downstream link. It is not (quite) right for what a site would use as
>>>> an uplink.
>>>>
>>>> QFQ, which is what I was using to get 1/d level behavior in the
>>>> prototype I did several months back, still is hanging under load on
>>>> cerowrt. It's working on x86.  :( Might try working with classful SFQ
>>>> stuff instead, or I may give DRR a shot instead.
>>>>
>>>> Even getting 1/d is hard because the filters can only use IP and IPv6
>>>> addresses, so a device issuing traffic on both ipv4 and ipv6 can get
>>>> twice what it deserves in terms of bandwidth, or more if it has
>>>> multiple ipv6 addresses.
>>>>
>>>> My preferred solution to this (and part of the wireless problem) is to
>>>> sort things out by source mac address. How to do that, remains a
>>>> mystery. The way I'm leaning is to invent some sort of "src mac-to-id"
>>>> filter that is global router-wide.
>>>>
>>>> I tend to view (in the home) as having 1/u network performance as the
>>>> ideal. There are exceptions to this, notably video vs. say,
>>>> bittorrent. A clever way of getting closer to 1/u might be to sense
>>>> for more recent DNS queries and move that to a more interactive class.
>>>>
>>>> I got 1/d level performance, vs bittorrent, several months back, using
>>>> a combination of HTB, and a
>>>> two tier QFQ, then red model. I was happy with this, but QFQ was
>>>> required. Perhaps I can make
>>>> SFQRED do this with the right filters, or switch to a combination of
>>>> DRR and SFQRED....
>>>>
>>>> Anyway, at the present time, I'm trying a few approaches, the most
>>>> promising one is a two or three tier model using SFQRED, with a very
>>>> limited number of things prio 1 (dns), 2 (mostly everything), 3
>>>> classified and background traffic and torrent, where classifiable.
>>>> Classification is a rathole however, and I'd rather be aiming for 1/d
>>>> than 1/f.
>>>>
>>>> Regardless openwrt's default AQM does lousy queue depth management at
>>>> present, and I expect to
>>>> improve it tremendously over the coming week.
>>>>
>>>> --
>>>> Dave Täht
>>>> SKYPE: davetaht
>>>> US Tel: 1-239-829-5608
>>>> FR Tel: 0638645374
>>>> http://www.bufferbloat.net
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> Bloat@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> FR Tel: 0638645374
>> http://www.bufferbloat.net
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Bloat] Testing Queue models

[Justin McCann]

On Wed, Feb 1, 2012 at 12:12 PM, Dave Taht <dave.taht@gmail.com> wrote:
>...
> Even getting 1/d is hard because the filters can only use IP and IPv6
> addresses, so a device issuing traffic on both ipv4 and ipv6 can get
> twice what it deserves in terms of bandwidth, or more if it has
> multiple ipv6 addresses.
>
> My preferred solution to this (and part of the wireless problem) is to
> sort things out by source mac address. How to do that, remains a
> mystery. The way I'm leaning is to invent some sort of "src mac-to-id"
> filter that is global router-wide.

A friend and I were discussing last week how each device on an IPv6
network can and will have multiple addresses, so classifying becomes a
bit trickier. But I guess most devices are supposed to have one (ha!)
/64, so you can ignore the lower-order bits? I assume that sort of
IPv6 alias resolution is a solved problem already...?

It sounds to me like you want to assign each device its own netfilter
MARK, and mark each packet (by MAC?) on ingress, and then use the mark
plus any other port/protocol selection you want to put it into the
appropriate queue.

> I tend to view (in the home) as having 1/u network performance as the
> ideal. There are exceptions to this, notably video vs. say,
> bittorrent. A clever way of getting closer to 1/u might be to sense
> for more recent DNS queries and move that to a more interactive class.

You might try assigning each device to a user, and then marking as
above, assuming only one user at a time on a device.

I guess that doing this would address (in the local/home network) what
Bob Briscoe was getting at in our earlier thread:

"Since 2004, we now understand that fairness has to involve accounting
over time. That requires per-user state, which is not something you
can do, or that you need to do, within every queue. We should leave
fairness to separate code, probably on machines specialising in this
at the edge of a provider's network domain - where it knows about
users/customers - separate from the AQM code of each specific queue."

     Justin

Re: [Bloat] Testing Queue models

[Dave Taht]

On Wed, Feb 1, 2012 at 9:08 PM, Justin McCann <jneilm@gmail.com> wrote:
> On Wed, Feb 1, 2012 at 12:12 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>...
>> Even getting 1/d is hard because the filters can only use IP and IPv6
>> addresses, so a device issuing traffic on both ipv4 and ipv6 can get
>> twice what it deserves in terms of bandwidth, or more if it has
>> multiple ipv6 addresses.
>>
>> My preferred solution to this (and part of the wireless problem) is to
>> sort things out by source mac address. How to do that, remains a
>> mystery. The way I'm leaning is to invent some sort of "src mac-to-id"
>> filter that is global router-wide.
>
> A friend and I were discussing last week how each device on an IPv6
> network can and will have multiple addresses, so classifying becomes a
> bit trickier. But I guess most devices are supposed to have one (ha!)
> /64, so you can ignore the lower-order bits? I assume that sort of
> IPv6 alias resolution is a solved problem already...?

I've been thinking about this kind of hard because I was irked and
surprised by dibbler-pd's default behavior. When it exits, it withdraws
the ip addresses it gets... and there seems to be an explicit
assumption that there are no other ipv6 addresses on the network.

Now, personally, I kind of expect an IPv6 network to KEEP WORKING
when disconnected from the internet. In fact, I expect it to come up,
working, when disconnected from the internet, at the same time ipv4
does, and stay working, regardless of whether there is a gateway to
the outside world.

some people actually still like, to um, print, or have a file server, etc.

What appears to be the correct way to do ipv6 in the home,
and maybe the general case is to apply RFC4193, which I sat down
to implement yesterday.

These were my notes regarding the proposed unique local address
generation scheme in opening a new router and getting up on the net.

     1) Obtain the current time of day in 64-bit NTP format [NTP].

I'm not on the internet yet, and time will either be the epoch, or
the build date. And ntp may or may not be working yet.

    2) Obtain an EUI-64 identifier from the system running this

Great. I note that fairly often macs are twiddled with on wireless
and the local bit is often set nowadays too...

     3) Concatenate the time of day with the system-specific identifier
        in order to create a key.

The rfc doesn't specify case, or binary, or ascii, or how you concat

     4) Compute an SHA-1 digest on the key as specified in [FIPS, SHA1];
        the resulting value is 160 bits.

And the randomness from the time is non-existent.

     5) Use the least significant 40 bits as the Global ID.

     6) Concatenate FC00::/7, the L bit set to 1, and the 40-bit Global
        ID to create a Local IPv6 address prefix.

So great, we need, in order to generate that first address:

ntp
openssh (for sha), or sha1sum

Which is all encapsulated in an 88 line shell script that I just
pulled off the net.

While I got that working, I was not including the openssl or sha1sum
utility required for it, which eats 256k on flash,
and isn't needed for anything else...

and most embedded distros have no means of getting the 64 bit ntp
value out of their clock, lacking ntp.

Then the RFC skips the all-important step of:

     7) Update DNS to suit

as out of scope, regardless of the fact that there is no way to get
this new address to the user
or device, otherwise.

So having an random arbitrary and likely-to-be-unique address is handy...

fdfd:7c1a:c378::/48 (which takes over a second to generate on cerowrt)

But there is no way to use that sanely until you have dns, and your
provider isn't going to do a ula for you
so you need your own dns server, and that address inserted into your dns

8) then you need to break that up onto a sane mapping for the
interfaces you have.

8a) firewall off your guest and secure networks

9) Make sure you don't route these out your gateway,

10) supply them to clients via dhcpv6 or radvd. And get names for
those into dns, somehow.

11) Then, assuming you get real, static ipv6 addresses, deprecate the
ulas entirely from your network. If however, you don't own your ipv6
addresses, you'll want something stable, so you keep them, leading to
at least two ipv6 addresses per device...

12) as best as I can tell gai.conf doesn't prioritize for ula addrs by
default, so if you remove the addresses from your network dynamically,
long running tcp sessions will fail.... and did I mention the
firewalling problem already?

/me takes long sip of kool-aid...

There's probably another 30 steps here....

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Bloat] Testing Queue models

[Dave Taht]

On Wed, Feb 1, 2012 at 9:08 PM, Justin McCann <jneilm@gmail.com> wrote:
> On Wed, Feb 1, 2012 at 12:12 PM, Dave Taht <dave.taht@gmail.com> wrote:

> It sounds to me like you want to assign each device its own netfilter
> MARK, and mark each packet (by MAC?) on ingress, and then use the mark
> plus any other port/protocol selection you want to put it into the
> appropriate queue.

the mark would be a unique id per mac, pulled from a small pool in a LRU
fashion. 256 - heck, 64, values for it - would be suitable for most homes
and small businesses.

As for the "plus" part - no, want to FQ across device id, and then FQ/AQM
within that, against all the streams coming from the device.
This would give 1/d fairness and (in particular) cope with bittorrent
pretty nicely.

I have a QFQ prototype which worked pretty well, in the debloat repo,
except for the unfairness of the per/ip and per/ipv6 hashing problem.

>
>> I tend to view (in the home) as having 1/u network performance as the
>> ideal. There are exceptions to this, notably video vs. say,
>> bittorrent. A clever way of getting closer to 1/u might be to sense
>> for more recent DNS queries and move that to a more interactive class.
>
> You might try assigning each device to a user, and then marking as
> above, assuming only one user at a time on a device.

requires manual assignment.

I've noted elsewhere that a users'
'attention' and normal usage pattern may be of interest in doing
it more automatically.

> I guess that doing this would address (in the local/home network) what
> Bob Briscoe was getting at in our earlier thread:
>
> "Since 2004, we now understand that fairness has to involve accounting
> over time. That requires per-user state, which is not something you
> can do, or that you need to do, within every queue. We should leave
> fairness to separate code, probably on machines specialising in this
> at the edge of a provider's network domain - where it knows about
> users/customers - separate from the AQM code of each specific queue."

Well, here we go... as mentioned in the conclusion of that thread,
1/d seems the best possible without having per-brain interfaces,
and 1/u seems to require some theoretical work, and in either case,
seems better than 1/f.

I don't have a whole lot of hope for classification. In fact, I'm kind of
upset that the move away from flash means we are seeing more
video streams on port 80, rather than on the macromedia port...

>
>     Justin



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
FR Tel: 0638645374
http://www.bufferbloat.net

Re: [Bloat] Testing Queue models

[Bruce Atherton]

On 2/1/2012 12:46 PM, Dave Taht wrote:
> I don't have a whole lot of hope for classification. In fact, I'm kind 
> of upset that the move away from flash means we are seeing more video 
> streams on port 80, rather than on the macromedia port...

It may be worse than that in the future. Now that Websockets is RFC6455 
the nature of traffic on port 80 may change a lot. Roy Fielding was so 
concerned about it that he asked that a security note be added to the 
draft spec.

No idea what that will mean for your efforts here.

Re: [Bloat] Testing Queue models

[Dave Taht]

On Wed, Feb 15, 2012 at 2:08 AM, Bruce Atherton <bruce@callenish.com> wrote:
>
>
> On 2/1/2012 12:46 PM, Dave Taht wrote:
>>
>> I don't have a whole lot of hope for classification. In fact, I'm kind of
>> upset that the move away from flash means we are seeing more video streams
>> on port 80, rather than on the macromedia port...
>
>
> It may be worse than that in the future. Now that Websockets is RFC6455 the
> nature of traffic on port 80 may change a lot. Roy Fielding was so concerned
> about it that he asked that a security note be added to the draft spec.

What we conventionally think about as the need for firewalling and
threat models is increasingly irrelevant, particularly with the advent
of new - and standardized, even! - tunneling models like this, as well
as devices that live on 3g and wireless at the same time, etc.

> No idea what that will mean for your efforts here.

Running the entire internet through port 80 and 443, and further,
tunneling new applications through that, rather than using specialized
and well defined protocols seems like the wrong thing.

I have a rant on this topic that amusingly dates to around the time
I'd got involved in the bufferbloat effort...

<rant>
http://nex-6.taht.net/posts/Beating_the_speed_of_light_on_the_web/
</rant>

It's something of a consequence of nat, and may well be a wedge to try
and make ipv6 'more right'...

On my very backlogged 'round-to-it' list has been writing an xtables
module for multi-protocol matching, as the current methods for
matching protocols (at least in linux) only support a single match,
and as you add protocols becomes tedious, error prone, and slow.

iptables -I INPUT -p tcp -j ALLOW
iptables -I INPUT -p udp -j ALLOW
iptables -I INPUT -p 41 -j ALLOW
etc.

Better would be something that did a lookup against a 256bit-map

ip6tables -I INPUT -m protocols --protocols
icmp,tcp,udp,igmp,rdp,dccp,rsvp,gre,esp,ah,ospf,ipip,pim,l2p,isis,sctp,udplite,manet,hip,shim6,wesp
-J ALLOW

In the hope that this would improve end-to-end connectivity,
performance, and availability of new stuff in the general case as ipv6
is rolled out.

Regrettably I haven't got around to writing that bit, nor something
similar for diffserv....

>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Bloat] Testing Queue models

[Bruce Atherton]

Well, port 80/443 is only the default for Websockets. People running 
standalone WS servers are encouraged to use unique ports.

There are two main reasons to have those defaults though:

1) Webapps that use both HTTP and Websockets, which is after all the 
most common usecase, benefit from having a single server that routes the 
connection reasonably far down in the stack so that maximum reuse of 
code can occur.

2) System administrators in many companies are incredibly resistant to 
opening up new ports. They have to open port 80 and sometimes 443. This 
makes ports 80 and 443 "magic" ports in these environments, the kind 
that are the only ones that third party webapp developers can rely on. 
Sad but that is the way of the world. I say this as a third party webapp 
developer.

On 2/17/2012 6:22 AM, Dave Taht wrote:
> On Wed, Feb 15, 2012 at 2:08 AM, Bruce Atherton<bruce@callenish.com>  wrote:
>>
>> On 2/1/2012 12:46 PM, Dave Taht wrote:
>>> I don't have a whole lot of hope for classification. In fact, I'm kind of
>>> upset that the move away from flash means we are seeing more video streams
>>> on port 80, rather than on the macromedia port...
>>
>> It may be worse than that in the future. Now that Websockets is RFC6455 the
>> nature of traffic on port 80 may change a lot. Roy Fielding was so concerned
>> about it that he asked that a security note be added to the draft spec.
> What we conventionally think about as the need for firewalling and
> threat models is increasingly irrelevant, particularly with the advent
> of new - and standardized, even! - tunneling models like this, as well
> as devices that live on 3g and wireless at the same time, etc.
>
>> No idea what that will mean for your efforts here.
> Running the entire internet through port 80 and 443, and further,
> tunneling new applications through that, rather than using specialized
> and well defined protocols seems like the wrong thing.
>
> I have a rant on this topic that amusingly dates to around the time
> I'd got involved in the bufferbloat effort...
>
> <rant>
> http://nex-6.taht.net/posts/Beating_the_speed_of_light_on_the_web/
> </rant>
>
> It's something of a consequence of nat, and may well be a wedge to try
> and make ipv6 'more right'...
>
> On my very backlogged 'round-to-it' list has been writing an xtables
> module for multi-protocol matching, as the current methods for
> matching protocols (at least in linux) only support a single match,
> and as you add protocols becomes tedious, error prone, and slow.
>
> iptables -I INPUT -p tcp -j ALLOW
> iptables -I INPUT -p udp -j ALLOW
> iptables -I INPUT -p 41 -j ALLOW
> etc.
>
> Better would be something that did a lookup against a 256bit-map
>
> ip6tables -I INPUT -m protocols --protocols
> icmp,tcp,udp,igmp,rdp,dccp,rsvp,gre,esp,ah,ospf,ipip,pim,l2p,isis,sctp,udplite,manet,hip,shim6,wesp
> -J ALLOW
>
> In the hope that this would improve end-to-end connectivity,
> performance, and availability of new stuff in the general case as ipv6
> is rolled out.
>
> Regrettably I haven't got around to writing that bit, nor something
> similar for diffserv....
>
>
>