[Bloat] ping loss "considered harmful"

[Bloat] ping loss "considered harmful"

[Dave Taht]

On this thread over here, an otherwise pretty clueful user chose
openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
*higher ping loss*.


http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125

(I note that both fq_codel enabled QoS systems outperformed
streamboost by a lot, which I am happy about)

wow. It never registered to me that users might make a value judgement
based on the amount of ping loss, and in looking back in time, I can
think of multiple people that have said things based on their
perception that losing pings was bad, and that sqm-scripts was "worse
than something else because of it."

sqm-scripts explicitly *deprioritizes* ping. In particular, this
reduces the impact of ping floods from ipv6 to your entire /64, or to
your whole ipv4, fairly well. And I had made the point that
prioritizing ping was a bad idea here (including some dripping sarcasm
later in the piece).

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

but wow, it never occurred to me - in all these years - that ping was
the next core metric on simple tests. I can be really dumb.

I use netperf-wrapper and tend to ignore most of the ping data, but
certainly on some benchmarks we have published ping doesn't look as
good as the other stuff, *because it is deprioritized below all the
other traffic*. Not strictly rate limited - as some systems do by
default, including openwrt, which is impossible to get right - just
deprioritized....

How can we fix this user perception, short of re-prioritizing ping in
sqm-scripts?

-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Sun, 1 Mar 2015, Dave Taht wrote:

> but wow, it never occurred to me - in all these years - that ping was 
> the next core metric on simple tests. I can be really dumb.
>
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?

People will make all kinds of judgement calls based on things that are not 
very relevant.

Some people think a traceroute with few hops in it to be better than one 
with more hops in it. So a lot of providers use their MPLS networks and 
make the intermediate nodes not show up in traceroute.

I don't know how many times I have experienced people have used PING and 
opened trouble tickets when a core router had ICMP ping packet loss (even 
though all core router platforms have rate limiters for ICMP).

When you say that you de-prioritize ping, do you mean only ECHO REQUEST 
and ECHO REPLY, or do you mean all ICMP? Perhaps a solution would be to 
assure 5% of the link bandwidth for ICMP?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [Cerowrt-devel] ping loss "considered harmful"

[David Lang]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2503 bytes --]

On Sun, 1 Mar 2015, Dave Taht wrote:

> On this thread over here, an otherwise pretty clueful user chose
> openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
> *higher ping loss*.
>
>
> http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125
>
> (I note that both fq_codel enabled QoS systems outperformed
> streamboost by a lot, which I am happy about)
>
> wow. It never registered to me that users might make a value judgement
> based on the amount of ping loss, and in looking back in time, I can
> think of multiple people that have said things based on their
> perception that losing pings was bad, and that sqm-scripts was "worse
> than something else because of it."

People make the assumption that if ping packets are being dropped, their data is 
getting dropped as well. This is part of the "dropped packets are bad, so buffer 
so you don't drop packets" mentality

To address this, the test results should first show all the stats without 
showing dropped packets, or if you need to show them, add a comment at that 
point that says that data packets are given priority over ping packets, so 
dropped ping packets don't imply that data packets would be dropped.

David Lang


> sqm-scripts explicitly *deprioritizes* ping. In particular, this
> reduces the impact of ping floods from ipv6 to your entire /64, or to
> your whole ipv4, fairly well. And I had made the point that
> prioritizing ping was a bad idea here (including some dripping sarcasm
> later in the piece).
>
> http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die
>
> but wow, it never occurred to me - in all these years - that ping was
> the next core metric on simple tests. I can be really dumb.
>
> I use netperf-wrapper and tend to ignore most of the ping data, but
> certainly on some benchmarks we have published ping doesn't look as
> good as the other stuff, *because it is deprioritized below all the
> other traffic*. Not strictly rate limited - as some systems do by
> default, including openwrt, which is impossible to get right - just
> deprioritized....
>
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?
>
> -- 
> Dave Täht
> Let's make wifi fast, less jittery and reliable again!
>
> https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Bloat] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Mon, 2 Mar 2015, Brian Trammell wrote:

> Gaming protocols do this right - latency measurement is built into the 
> protocol.

I believe this is the only way to do it properly, and the most likely 
easiest way to get this deployed would be to use the TCP stack.

We need to give users an easy-to-understand metric on how well their 
Internet traffic is working. So the problem here is that the users can't 
tell how well it's working without resorting to ICMP PING to try to figure 
out what's going on.

For instance, if their web browser had insight into what the TCP stack was 
doing then it could present information a lot better to the user. Instead 
of telling the user "time to first byte" (which is L4 information), it 
could tell the less novice user about packet loss, PDV, reordering, RTT, 
how well concurrent connections to the same IP address are doing, tell 
more about *why* some connections are slow instead of just saying "it took 
5.3 seconds to load this webpage and here are the connections and how long 
each took". For the novice user there should be some kind of expert system 
that collects data that you can send to the ISP that also has an expert 
system to say "it seems your local connection delays packets", please 
connect to a wired connection and try again". It would know if the problem 
was excessive delay, excessive delay that varied a lot, packet loss, 
reordering, or whatever.

We have a huge amount of information in our TCP stacks that either are 
locked in there and not used properly to help users figure out what's 
going on, and there is basically zero information flow between the 
applications using TCP and the TCP stack itself. Each just tries to do its 
best on its own layer.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [aqm] ping loss "considered harmful"

[Jonathan Morton]

> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Mon, 2 Mar 2015, Brian Trammell wrote:
> 
>> Gaming protocols do this right - latency measurement is built into the protocol.
> 
> I believe this is the only way to do it properly, and the most likely easiest way to get this deployed would be to use the TCP stack.
> 
> We need to give users an easy-to-understand metric on how well their Internet traffic is working. So the problem here is that the users can't tell how well it's working without resorting to ICMP PING to try to figure out what's going on.
> 
> For instance, if their web browser had insight into what the TCP stack was doing then it could present information a lot better to the user. Instead of telling the user "time to first byte" (which is L4 information), it could tell the less novice user about packet loss, PDV, reordering, RTT, how well concurrent connections to the same IP address are doing, tell more about *why* some connections are slow instead of just saying "it took 5.3 seconds to load this webpage and here are the connections and how long each took". For the novice user there should be some kind of expert system that collects data that you can send to the ISP that also has an expert system to say "it seems your local connection delays packets", please connect to a wired connection and try again". It would know if the problem was excessive delay, excessive delay that varied a lot, packet loss, reordering, or whatever.
> 
> We have a huge amount of information in our TCP stacks that either are locked in there and not used properly to help users figure out what's going on, and there is basically zero information flow between the applications using TCP and the TCP stack itself. Each just tries to do its best on its own layer.

This seems like an actually good idea.  Several of those statistics, at least, could be exposed to userspace without incurring any additional overhead in the stack (except for the queries themselves), which is important for high-performance server users.  TCP stacks already track RTT, and sometimes MinRTT - the difference between these values is a reasonable lower-bound estimate of induced latency.

For stacks which don’t already track all the desirable data, a socket option could be used to turn that on, allocating extra space to do so.  To maximise portability, therefore, it might be necessary to require that option before statistics requests will be valid, even on stacks which do collect it all anyway.

Recent versions of Windows, even, have a semi-magic system which gives a little indicator of whether your connection has functioning Internet connectivity or not.  This could be extended, if Microsoft saw fit, to interpret these statistics and notify the user that their connection was behaving badly in the ways we now find interesting.  Whether Microsoft will do such a thing (which would undoubtedly piss off every major ISP on the planet) is another matter, but it’s a concept that can be used by Linux desktops as well, and with less political fallout.

Now, who’s going to knuckle down and implement it?

 - Jonathan Morton

Re: [Bloat] [Cerowrt-devel] [aqm] ping loss "considered harmful"

[dpreed]

On my own web server (running nginx) I provide an HTTP 1.1 accessible statistics service.  It returns a single JSON structure with the underlying packet statistics for the server and the current connection. Since this packet is inserted into the HTTP 1.1 stream upon request, it provides both the statistics and a single-round-trip "ping" that can measure queue delay on demand.

Changing server code is a lot easier than changing browser code.  Anyone who would like to develop a "standards-track" option for HTTP 1.1 is quite welcome to do so...  right now it is a "RESTful" interface, so measurements of the client side must be obtained indirectly, but simple JavaScript code in a web page can access this with standard HTML5 code.

I'm not looking to support this code or enhance it.  I've thought about putting it up on GitHub.

In any case, it's easy to add a little widget to the corner of all the web pages from a site that displays latency and net statistics at various levels in a very pretty user-understandable form.

To see how this works, just go to the alt.reed.com site (preferably using an iPhone or Android phone) to see a trivial test app I put out a number of years ago for my personal zeroth-order diagnostics.

nginx (as most know) is an incredibly tight, low overhead HTTP 1.1 server that lets you add plugins for lots of things other than retrieving web pages from disk.  It's not the pile of spaghetti that Apache has become. It supports WebSockets and HTTPS quite well, too.




On Monday, March 2, 2015 5:54am, "Jonathan Morton" <chromatix99@gmail.com> said:

> 
>> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> On Mon, 2 Mar 2015, Brian Trammell wrote:
>>
>>> Gaming protocols do this right - latency measurement is built into the protocol.
>>
>> I believe this is the only way to do it properly, and the most likely easiest way
>> to get this deployed would be to use the TCP stack.
>>
>> We need to give users an easy-to-understand metric on how well their Internet
>> traffic is working. So the problem here is that the users can't tell how well
>> it's working without resorting to ICMP PING to try to figure out what's going on.
>>
>> For instance, if their web browser had insight into what the TCP stack was doing
>> then it could present information a lot better to the user. Instead of telling
>> the user "time to first byte" (which is L4 information), it could tell the less
>> novice user about packet loss, PDV, reordering, RTT, how well concurrent
>> connections to the same IP address are doing, tell more about *why* some
>> connections are slow instead of just saying "it took 5.3 seconds to load this
>> webpage and here are the connections and how long each took". For the novice user
>> there should be some kind of expert system that collects data that you can send
>> to the ISP that also has an expert system to say "it seems your local connection
>> delays packets", please connect to a wired connection and try again". It would
>> know if the problem was excessive delay, excessive delay that varied a lot,
>> packet loss, reordering, or whatever.
>>
>> We have a huge amount of information in our TCP stacks that either are locked in
>> there and not used properly to help users figure out what's going on, and there
>> is basically zero information flow between the applications using TCP and the TCP
>> stack itself. Each just tries to do its best on its own layer.
> 
> This seems like an actually good idea.  Several of those statistics, at least,
> could be exposed to userspace without incurring any additional overhead in the
> stack (except for the queries themselves), which is important for high-performance
> server users.  TCP stacks already track RTT, and sometimes MinRTT - the difference
> between these values is a reasonable lower-bound estimate of induced latency.
> 
> For stacks which don’t already track all the desirable data, a socket option
> could be used to turn that on, allocating extra space to do so.  To maximise
> portability, therefore, it might be necessary to require that option before
> statistics requests will be valid, even on stacks which do collect it all anyway.
> 
> Recent versions of Windows, even, have a semi-magic system which gives a little
> indicator of whether your connection has functioning Internet connectivity or not.
>  This could be extended, if Microsoft saw fit, to interpret these statistics and
> notify the user that their connection was behaving badly in the ways we now find
> interesting.  Whether Microsoft will do such a thing (which would undoubtedly piss
> off every major ISP on the planet) is another matter, but it’s a concept
> that can be used by Linux desktops as well, and with less political fallout.
> 
> Now, who’s going to knuckle down and implement it?
> 
>  - Jonathan Morton
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 

Re: [Bloat] [Cerowrt-devel] [aqm] ping loss "considered harmful"

[David Lang]

[-- Attachment #1: Type: TEXT/Plain, Size: 287 bytes --]

On Mon, 2 Mar 2015, Andrew Mcgregor wrote:

> So, are you suggesting that, for example, Chrome's rather extensive network
> debugging information get more publicised?  We can probably arrange that.

That would be good. I have no idea what debugging info you are referring to.

David Lang

[-- Attachment #2: Type: TEXT/PLAIN, Size: 164 bytes --]

_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Bloat] [aqm] ping loss "considered harmful"

[Kathleen Nichols]

Somewhat apropos of this discussion, Pollere has been working on passive
techniques to measure latency under a DoE SBIR grant. Although these
can be used on an end-host, the goal is to deploy anywhere. But part of the
goal with SBIR grants is to figure out a revenue stream. As Dave has noted,
this is not all that easy to do for "fix the Internet" kind of things.
Similar to
Dave (though not as dire since my spouse decided to work for the big G and
thus give me the ability to tinker with problems I like), I worked on
CoDel "for
free" and the only revenue I got from it was a small contract with
CableLabs.
Now, I didn't set out to make money from it, it was intially a fun
project to
do as an antidote to nine crappy work months and I kind of got into it when
the aforementioned spouse sort of pulled one of those Tom Sawyer painting
the fence moves on me.

However.

Are these kinds of projects just always going to be done as sidelines or
will
anyone want to pay, either by licensing a finished approach or sponsoring
a new one? I'm about to explore the former soon, but I do wonder if this
isn't quixotic.

	Kathie

On 3/2/15 2:54 AM, Jonathan Morton wrote:
> 
>> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <swmike@swm.pp.se>
>> wrote:
>> 
>> On Mon, 2 Mar 2015, Brian Trammell wrote:
>> 
>>> Gaming protocols do this right - latency measurement is built
>>> into the protocol.
>> 
>> I believe this is the only way to do it properly, and the most
>> likely easiest way to get this deployed would be to use the TCP
>> stack.
>> 
>> We need to give users an easy-to-understand metric on how well
>> their Internet traffic is working. So the problem here is that the
>> users can't tell how well it's working without resorting to ICMP
>> PING to try to figure out what's going on.
>> 
>> For instance, if their web browser had insight into what the TCP
>> stack was doing then it could present information a lot better to
>> the user. Instead of telling the user "time to first byte" (which
>> is L4 information), it could tell the less novice user about packet
>> loss, PDV, reordering, RTT, how well concurrent connections to the
>> same IP address are doing, tell more about *why* some connections
>> are slow instead of just saying "it took 5.3 seconds to load this
>> webpage and here are the connections and how long each took". For
>> the novice user there should be some kind of expert system that
>> collects data that you can send to the ISP that also has an expert
>> system to say "it seems your local connection delays packets",
>> please connect to a wired connection and try again". It would know
>> if the problem was excessive delay, excessive delay that varied a
>> lot, packet loss, reordering, or whatever.
>> 
>> We have a huge amount of information in our TCP stacks that either
>> are locked in there and not used properly to help users figure out
>> what's going on, and there is basically zero information flow
>> between the applications using TCP and the TCP stack itself. Each
>> just tries to do its best on its own layer.
> 
> This seems like an actually good idea.  Several of those statistics,
> at least, could be exposed to userspace without incurring any
> additional overhead in the stack (except for the queries themselves),
> which is important for high-performance server users.  TCP stacks
> already track RTT, and sometimes MinRTT - the difference between
> these values is a reasonable lower-bound estimate of induced
> latency.
> 
> For stacks which don’t already track all the desirable data, a socket
> option could be used to turn that on, allocating extra space to do
> so.  To maximise portability, therefore, it might be necessary to
> require that option before statistics requests will be valid, even on
> stacks which do collect it all anyway.
> 
> Recent versions of Windows, even, have a semi-magic system which
> gives a little indicator of whether your connection has functioning
> Internet connectivity or not.  This could be extended, if Microsoft
> saw fit, to interpret these statistics and notify the user that their
> connection was behaving badly in the ways we now find interesting.
> Whether Microsoft will do such a thing (which would undoubtedly piss
> off every major ISP on the planet) is another matter, but it’s a
> concept that can be used by Linux desktops as well, and with less
> political fallout.
> 
> Now, who’s going to knuckle down and implement it?
> 
> - Jonathan Morton
> 
> _______________________________________________ aqm mailing list 
> aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
> 

Re: [Bloat] [aqm] ping loss "considered harmful"

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

Probably the least cynical answer to that I can come up with is the hope
that big vendors get a clue that this stuff is needed, and start hiring
field experts as consultants to help them get it right.

You can stop snickering now.

Another possibility is that some of us have to club together and make our
own hardware that Does The Right Thing, possibly still based on one of the
generic reference hardware platforms, if we can find one that we like and
is cheap enough to compete. Which, once again, would take care of the CPE
problem but practically nothing else, unless we somehow managed to get a
foot in the door of the DSLAM market. Which is about as likely as the
Prince of Darkness taking a skiing holiday at home.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 819 bytes --]

Re: [Bloat] [aqm] ping loss "considered harmful"

[Dave Taht]

On Mon, Mar 2, 2015 at 12:06 PM, Wes Felter <wmf@felter.org> wrote:
> What about a token bucket policer, so more than N ICMP/second gets penalized
> but a normal ping won't be.

If I haven't expressed this too many times before, I want to thank you
all for existing.

It is the quality and caliber of all the folk on these mailing lists
coming up with new ideas, that keeps me going.

I've had a really rough could days (bronchitus, the virgin controversy
here: https://plus.google.com/u/0/107942175615993706558/posts/E1yMgbWW81C
), and I will try to write more later, but I thought I would add a bit
of light to the debate by showing you the default openwrt firewall
rules:

http://git.openwrt.org/?p=openwrt.git;a=blob;f=package/network/config/firewall/files/firewall.config;h=d149e77957de1c80358c1e8131920867a193c9ed;hb=HEAD

Problems: They do a (very low) syn limiter (25 syns/sec I think) by
default, which made sense 10 years ago, but not now - and I had run
into trouble with it while attempting to benchmark the alexa top 1
million in parallel through two routers at isc (at 300mbit) a few
years ago, so I ripped that out of cerowrt. It was a really puzzling
set of results, I had no idea why so many syn requests were dropping
on the floor til I inspected the actual firewall rules generated.

Similarly, they rate limit icmp at 1000/sec, which is too high at low
rates and too low at others. So I ripped that out of cerowrt, and in
sqm-scripts, I just held tossed it into the background bucket for for
a min of 30% of the background bandwidth.

I note that there are conflicting definitions of CS1 (background).
Comcast, re-marks about 90% I see to CS1 from whatever it was
originally, in the hope that it is treated as background. The ancient
firmware in commercial home routers *prioritizes* CS1 on etheret and
*deprioritizes it* on wifi, into the 802.11e background queue, when
enabled. CeroWrt tries to cons

I haven't seen ANY shipped open source FQ/AQM/QoS system that squashes
inbound dscp values, either, til cero, and despite the various rfcs
out there recommending that.

Openwrt's firewall system is, sadly, one of the better and saner
firewall rule sets that exists. Other rulesets I have seen in
commercial deployments are amazingly more complex, antiquated, and
frequently wrong.

as for fq/aqm/QoS I have pointed out the flaws in the comments on
wondershaper here:

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

and those flaws, exist, in copy-pasted code in nearly every subsequent
shaper I have examined *and had the time to fix*.

EVERY open source shaper I have seen, also (aside from sqm-scripts)
does not handle ipv6 right, nor do they handle dsl/atm framing issues
correctly. Notably streamboost got it terribly, terribly wrong, as did
Netgear's new "Dynamic QoS", and d-links DIR-5500. Openwrt's
qos-scripts has similar problems. So does gargoyle's stuff, and
dd-wrts... and whatever they were using at nznog on my last talk.

In the original design for sqm-scripts, I'd tried to use 3 tiers of
DRR + fq_codel - for classification, and failed due to in some
circumstances DRR getting "stuck" on a queue when codel did a drop on
it. (a bug that may only have existed at the time). So we went with
HTB, with a max of 30% for prio, a min of 30% for background (due to
all the mismarked traffic, 5% wasn't enough) and background and BE
being able to use up all the bandwidth.

That said, however free.fr did DRR + fq_codel in their implemention
benchmarks out quite nicely,  tho I can't find the relevant comparison
benchmarks at the moment.

particularly the prio queue helps for multicast IPTV traffic. I tried
to discuss some of these issues here, but the wg was not interested at
the time for a document like this.

http://snapon.lab.bufferbloat.net/~d/draft-taht-home-gateway-best-practices-00.html


Since then I specified the combined rate limiter + fq_codel idea
called "cake" with up to 8 progressive levels of DRR for
classification, which jonathon did a GREAT first attempt at, but he
didn't believe me that strict rate limits were not the right thing
until he got a bit of user feedback....

I should write up all the design decisions that went into sqm-scripts
as it exists today - it came out of me using varous shapers in various
environments for 12+ years and for example, I had completely forgotten
why I had deprioritized icmp the way I did! - and rip out a few test
bits (like prioritizing AF42 for mosh) that are not correct for the
real world... whenever I manage to get out of bed.

>
> --
> Wes Felter
>
>
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] [aqm] ping loss "considered harmful"

[David Lang]

On Mon, 2 Mar 2015, Dave Dolson wrote:

> Would you do that to TCP or UDP traffic?
>
> At IETF I often hear laments about middle-boxes breaking the internet by being "clever" with certain types of traffic.
> It seems that policing ICMP falls into that category.
>
> There may have been bugs in the past, but I'm not aware that ICMP packets are any more dangerous than UDP or TCP. And if the RFCs can be believed, ICMPv6 is critical to determining Path-MTU. Don't drop those.
>
> One may wish to rate-limit ICMP (or DNS or TCP) flows as a matter of network policy, but in my opinion this should be kept orthogonal to solving buffer bloat.
>
> Taken to the extreme, a network should support full utilization of a link doing only ping. If I wish to use my connection to the internet to ping hosts at full line rate, why not?

what's going on here isn't that pings are being rate limited, but rather that 
the TCP/UDP traffic is being given priority over the ping traffic. This means 
that when you max out the pipe, pings will suffer.

David Lang

Re: [Bloat] [aqm] ping loss "considered harmful"

[Bill Ver Steeg (versteb)]

[-- Attachment #1: Type: text/plain, Size: 3170 bytes --]


While I do not formally speak for “a big vendor” in this (or any) context, I do happen to work for one.

There are several efforts underway within this particular big vendor to address bloat. Are these efforts crash programs to get code out the door as fast as humanly possible? No. There are efforts underway, though. These things take time…… To be frank, the best way to drive feature development/deployment/adoption in most big companies is to have customers ask for them.

We are actually starting to see broader awareness of the problem at the SP network edge, and this is probably where we need solutions the fastest. There have been some gap-filler changes in the HFC space recently, and the next generation CMTS products will have a quite robust solution. As noted previously, the CPE and the DSLAMs will also need to be addressed. Unfortunately (perhaps fortunately to some folks on this list…. but I digress), the company I work for is no longer in these parts of the business, as we sold off the Linksys group off a few years ago and got out of the DSLAM business many years ago. I do still have some contacts in the chipset vendor community in the CPE space, and could help drive awareness if it would help. As most of us understand, the chipset vendors have a lot clout in this area - as they supply most of the heavy lifting code to the OEM/ODM CPE folks for their commercial offers. The open source community is currently ahead of the chipset folks and the OEM/ODM folks in this space.

So, IMHO the best way to make progress in the CPE arena is to get the chipset vendors to provide a robust AQM scheme in their default drivers. There may have to be some tweaks to get a given algorithm to work in a given chipset/memory/CPU architecture. In fact, there may be room for algorithms that take advantage of features of a given chipset, or to design a new chipset that “does the right thing” in uCode.  If I were a consultant – this is where I would focus my efforts……….

I am not sure how to crack the DSLAM egg, though. Different set of players, different set of initial conditions.

Bill VerSteeg











From: bloat-bounces@lists.bufferbloat.net [mailto:bloat-bounces@lists.bufferbloat.net] On Behalf Of Jonathan Morton
Sent: Monday, March 02, 2015 2:42 PM
To: Kathleen Nichols
Cc: bloat
Subject: Re: [Bloat] [aqm] ping loss "considered harmful"


Probably the least cynical answer to that I can come up with is the hope that big vendors get a clue that this stuff is needed, and start hiring field experts as consultants to help them get it right.

You can stop snickering now.

Another possibility is that some of us have to club together and make our own hardware that Does The Right Thing, possibly still based on one of the generic reference hardware platforms, if we can find one that we like and is cheap enough to compete. Which, once again, would take care of the CPE problem but practically nothing else, unless we somehow managed to get a foot in the door of the DSLAM market. Which is about as likely as the Prince of Darkness taking a skiing holiday at home.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 9167 bytes --]

Re: [Bloat] [aqm] ping loss "considered harmful"

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 6759 bytes --]

On Mon, Mar 2, 2015 at 12:48 PM, Bill Ver Steeg (versteb) <versteb@cisco.com
> wrote:

>
>
> While I do not formally speak for “a big vendor” in this (or any) context,
> I do happen to work for one.
>
>
>
> There are several efforts underway within this particular big vendor to
> address bloat. Are these efforts crash programs to get code out the door as
> fast as humanly possible? No. There are efforts underway, though. These
> things take time…… To be frank, the best way to drive feature
> development/deployment/adoption in most big companies is to have customers
> ask for them.
>

Creating understanding and demand has been my nearly f/t project for
several years now. I hope it is finally starting to work!

However, along the way - in trying to work with everybody in all parts of
the industry, and to "get along" - I found myself in a deep moral and
mental hole where I realized I was no longer being true to myself or being
effective in what I had really set out to do by attempting to create this
open, shared project, where I had hoped we all would be working together
for a common goal.

It was probably the gogo-in-flight test results with a *total lack of ping
loss* and *12 minutes of latency* that set me off in the beginning... here
are some pings from the middle of the test:

64 bytes from :elided: icmp_seq=5108 ttl=42 time=144333 ms
64 bytes from :elided: icmp_seq=5109 ttl=42 time=143335 ms
64 bytes from :elided: icmp_seq=5110 ttl=42 time=142327 ms
64 bytes from :elided: icmp_seq=5111 ttl=42 time=141319 ms
64 bytes from :elided: icmp_seq=5113 ttl=42 time=139311 ms

netperf-wrapper results and the ping log of the flight as it went towards
landing are here:

http://snapon.lab.bufferbloat.net/~d/gogo-in-flight-interplanetary-latencies.tgz

of the actual remaining throughput and latency on the aircraft, after a
test had completed (sort of), and another started:
http://snapon.lab.bufferbloat.net/~d/GoGoSFO/gogo_in_flight_interplanetary_latencies.png

(netperf-wrapper was not designed to deal with RTTs greater than from here
to mars, so
the plot doesn't make much sense! The pings don't fit right! But the
bandwidth is *gone*)

Gogo-in-flight, at least, could use a crash program to fix their
bufferbloat! I know that adding a 50 dollar openwrt box or upgrading their
software would probably cost a lot in regulatory approvals, but gawd!!! 12
minutes of latency! A total collapse of their network! From a simple test!
And despite trying through every channel I could, I have not found anyone
there with an IQ above 80 to talk to about how to dramatically fix their
own services. They make their money from selling the service, not,
actually, in providing it, in a totally captive market.

And at some point, you just have to laugh - as I did when I called them
out, and gave them the finger, at nznog.

And my inspiration, for what basically has caused me to rear back, and
start saying and doing exactly what I think, *all the time*, a few weeks
ago, here and on the internet, in words that that were utterly clear, short
and simple, was George Carlin's piece, here:

https://www.youtube.com/watch?v=vuEQixrBKCc

It is NSFW, but I do find clearing the air, has been useful, and I hope
everyone here, laughs their collective arses off at this!

I will write more tomorrow, I'm done today. I have *severe trust issues*
where your company is concerned, and I will detail why, tomorrow, publicly,
if I can filter out the cuss words and be at least reasonably polite about
it.



> We are actually starting to see broader awareness of the problem at the SP
> network edge, and this is probably where we need solutions the fastest.
> There have been some gap-filler changes in the HFC space recently, and the
> next generation CMTS products will have a quite robust solution. As noted
> previously, the CPE and the DSLAMs will also need to be addressed.
> Unfortunately (perhaps fortunately to some folks on this list…. but I
> digress), the company I work for is no longer in these parts of the
> business, as we sold off the Linksys group off a few years ago and got out
> of the DSLAM business many years ago. I do still have some contacts in the
> chipset vendor community in the CPE space, and could help drive awareness
> if it would help. As most of us understand, the chipset vendors have a lot
> clout in this area - as they supply most of the heavy lifting code to the
> OEM/ODM CPE folks for their commercial offers. The open source community is
> currently ahead of the chipset folks and the OEM/ODM folks in this space.
>
>
>
> So, IMHO the best way to make progress in the CPE arena is to get the
> chipset vendors to provide a robust AQM scheme in their default drivers.
> There may have to be some tweaks to get a given algorithm to work in a
> given chipset/memory/CPU architecture. In fact, there may be room for
> algorithms that take advantage of features of a given chipset, or to design
> a new chipset that “does the right thing” in uCode.  If I were a consultant
> – this is where I would focus my efforts……….
>
>
>
> I am not sure how to crack the DSLAM egg, though. Different set of
> players, different set of initial conditions.
>
>
>
> Bill VerSteeg
>
>
>
>
>
>
>
>
>
>
>
>
>
> *From:* bloat-bounces@lists.bufferbloat.net [mailto:
> bloat-bounces@lists.bufferbloat.net] *On Behalf Of *Jonathan Morton
> *Sent:* Monday, March 02, 2015 2:42 PM
> *To:* Kathleen Nichols
> *Cc:* bloat
> *Subject:* Re: [Bloat] [aqm] ping loss "considered harmful"
>
>
>
> Probably the least cynical answer to that I can come up with is the hope
> that big vendors get a clue that this stuff is needed, and start hiring
> field experts as consultants to help them get it right.
>
> You can stop snickering now.
>
> Another possibility is that some of us have to club together and make our
> own hardware that Does The Right Thing, possibly still based on one of the
> generic reference hardware platforms, if we can find one that we like and
> is cheap enough to compete. Which, once again, would take care of the CPE
> problem but practically nothing else, unless we somehow managed to get a
> foot in the door of the DSLAM market. Which is about as likely as the
> Prince of Darkness taking a skiing holiday at home.
>
> - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>


-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

[-- Attachment #2: Type: text/html, Size: 12738 bytes --]

Re: [Bloat] [aqm] ping loss "considered harmful"

[David Lang]

On Mon, 2 Mar 2015, Joe Touch wrote:

> On 3/2/2015 1:40 AM, Brian Trammell wrote:
> ...
>> The real solution is to create a utility called "ping" that uses
>> traffic that gets prioritized the same way as the traffic you care
>> about instead of ICMP echo request/reply. Users don't care about
>> the packets on the wire so much as they do that you're supposed to
>> ping things.
>
> There are three separate problems:
>
> 1. a ping that doesn't use ICMP
> 	there are dozens of these
>
> 2. needing a reflector
> 	ping gets around this only because the reflector is widely
> 	deployed (and integrated into most OSes)
>
> 3. using the same port as the traffic you care about
> 	transport protocol is only one problem (ICMP being a "transport
> 	protocol" by virtue of using the IP protocol number field)
>
> 	the other is differential prioritization based on port number
>
> 	there's no easy solution to that;
> 	every service would need an integrated
> 	ping reflector
>
> I suspect #3 is the ultimate killer of this idea.

The service you are trying to contact acts as a reflector for TCP traffic. If 
you send a syn you will get back a syn-ack from the TCP stack of the receiving 
system.

For UDP systems, it gets more interesting and service specific. But for TCP 
systems it works today.

David Lang

Re: [Bloat] [aqm] ping loss "considered harmful"

[David Lang]

On Mon, 2 Mar 2015, Joe Touch wrote:

> On 3/2/2015 3:14 PM, David Lang wrote:
>> On Mon, 2 Mar 2015, Joe Touch wrote:
>>
>>> On 3/2/2015 1:40 AM, Brian Trammell wrote:
>>> ...
>>>> The real solution is to create a utility called "ping" that uses
>>>> traffic that gets prioritized the same way as the traffic you care
>>>> about instead of ICMP echo request/reply. Users don't care about
>>>> the packets on the wire so much as they do that you're supposed to
>>>> ping things.
>>>
>>> There are three separate problems:
>>>
>>> 1. a ping that doesn't use ICMP
>>>     there are dozens of these
>>>
>>> 2. needing a reflector
>>>     ping gets around this only because the reflector is widely
>>>     deployed (and integrated into most OSes)
>>>
>>> 3. using the same port as the traffic you care about
>>>     transport protocol is only one problem (ICMP being a "transport
>>>     protocol" by virtue of using the IP protocol number field)
>>>
>>>     the other is differential prioritization based on port number
>>>
>>>     there's no easy solution to that;
>>>     every service would need an integrated
>>>     ping reflector
>>>
>>> I suspect #3 is the ultimate killer of this idea.
>>
>> The service you are trying to contact acts as a reflector for TCP
>> traffic. If you send a syn you will get back a syn-ack from the TCP
>> stack of the receiving system.
>
> Sure, but SYNs and SYN-ACKs don't get prioritized the same as
> non-control TCP segments. And they could have been spoofed by a middlebox.

well, this is exactly the sort of thing that http heartbeat (the core of 
heartbleed) was designed to provide.

It's not perfect, you can see where you really get the response from (vi the 
traceroute), and if you are going through a MITM (i.e. transparent proxy), all 
you are ever going to be able to test is the network between yourself and the 
proxy. It's up to the people who own the proxy to test beyond that.

David Lang

Re: [Bloat] [aqm] ping loss "considered harmful"

[Fred Baker (fred)]

[-- Attachment #1.1: Type: text/plain, Size: 458 bytes --]


> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?

IMHO, ping should go at the same priority as general traffic - the default class, DSCP=0. When I send one, I am asking whether a random packet can get to a given address and get a response back. I can imagine having a command-line parameter to set the DSCP to another value of my choosing.

[-- Attachment #1.2: Type: text/html, Size: 1976 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] [aqm] ping loss "considered harmful"

[Wesley Eddy]

On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
> 
>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>> <mailto:dave.taht@gmail.com>> wrote:
>>
>> How can we fix this user perception, short of re-prioritizing ping in
>> sqm-scripts?
> 
> IMHO, ping should go at the same priority as general traffic - the
> default class, DSCP=0. When I send one, I am asking whether a random
> packet can get to a given address and get a response back. I can imagine
> having a command-line parameter to set the DSCP to another value of my
> choosing.
> 


I generally agree, however ...

The DSCP of the response isn't controllable though, and likely the DSCP
that is ultimately received will not be the one that was sent, so it
can't be as simple as echoing back the same one.  Ping doesn't tell you
latency components in the forward or return path (some other protocols
can do this though).

So, setting the DSCP on the outgoing request may not be all that useful,
depending on what the measurement is really for.

-- 
Wes Eddy
MTI Systems

Re: [Bloat] [aqm] ping loss "considered harmful"

[Fred Baker (fred)]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]


> On Mar 3, 2015, at 9:29 AM, Wesley Eddy <wes@mti-systems.com> wrote:
> 
> On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
>> 
>>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>>> <mailto:dave.taht@gmail.com>> wrote:
>>> 
>>> How can we fix this user perception, short of re-prioritizing ping in
>>> sqm-scripts?
>> 
>> IMHO, ping should go at the same priority as general traffic - the
>> default class, DSCP=0. When I send one, I am asking whether a random
>> packet can get to a given address and get a response back. I can imagine
>> having a command-line parameter to set the DSCP to another value of my
>> choosing.
> 
> I generally agree, however ...
> 
> The DSCP of the response isn't controllable though, and likely the DSCP
> that is ultimately received will not be the one that was sent, so it
> can't be as simple as echoing back the same one.  Ping doesn't tell you
> latency components in the forward or return path (some other protocols
> can do this though).
> 
> So, setting the DSCP on the outgoing request may not be all that useful,
> depending on what the measurement is really for.

Note that I didn’t say “I demand”… :-)

I share the perception that ping is useful when it’s useful, and that it is at best an approximation. If I can get a packet to the destination and a response back, and I know the time I sent it and the time I received the response, I know exactly that - messages went out and back and took some amount of total time. I don’t know anything about the specifics of the path, of buffers en route, or delay time in the target. Traceroute tells me a little more, at the cost of a more intense process. In places I use ping, I tend to send a number of them over a period of time and observe on the statistics that result, not a single ping result.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] [aqm] ping loss "considered harmful"

[Dave Taht]

On Tue, Mar 3, 2015 at 10:00 AM, Fred Baker (fred) <fred@cisco.com> wrote:
>
>> On Mar 3, 2015, at 9:29 AM, Wesley Eddy <wes@mti-systems.com> wrote:
>>
>> On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
>>>
>>>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>>>> <mailto:dave.taht@gmail.com>> wrote:
>>>>
>>>> How can we fix this user perception, short of re-prioritizing ping in
>>>> sqm-scripts?
>>>
>>> IMHO, ping should go at the same priority as general traffic - the
>>> default class, DSCP=0. When I send one, I am asking whether a random
>>> packet can get to a given address and get a response back. I can imagine
>>> having a command-line parameter to set the DSCP to another value of my
>>> choosing.
>>
>> I generally agree, however ...
>>
>> The DSCP of the response isn't controllable though, and likely the DSCP
>> that is ultimately received will not be the one that was sent, so it
>> can't be as simple as echoing back the same one.  Ping doesn't tell you
>> latency components in the forward or return path (some other protocols
>> can do this though).
>>
>> So, setting the DSCP on the outgoing request may not be all that useful,
>> depending on what the measurement is really for.
>
> Note that I didn’t say “I demand”… :-)

My point was A), I have seen tons of shapers out there that actually
prioritize ping over other traffic. I figure everyone here will agree
that is a terrible practice, but I can certainly say it exists, as it
is a dumb mistake replicated in tons of shapers I have seen... that
makes people in marketing happy.

Already put up extensive commentary on that bit of foolishness on
"wondershaper must die".

Please feel free to review any shapers or firewall code you might have
access to for the same sort of BS and/or post the code somewhere for
public review. A BCP for these two things would be nice.

And B) Deprioritizing ping (slightly) as I do came from what has
happened to me multiple times when hit by a bot that ping floods the
network. One time, 30+ virtual windows boxes in a lab got infected by
something that went nuts pinging the entire 10/8 network we were on.
It actually DID melt the switch - and merely getting to isolating that
network off from the rest was a PITA, as getting to the (SFQ-ing)
router involved was nearly impossible via ssh. (like, 2 minutes
between keystrokes).

Thus, ping, deprioritized. I tend to feel deprioritizing it slightly
is much more important in the post ipv6 world.


> I share the perception that ping is useful when it’s useful, and that it is at best an approximation. If I can get a packet to the destination and a response back, and I know the time I sent it and the time I received the response, I know exactly that - messages went out and back and took some amount of total time. I don’t know anything about the specifics of the path, of buffers en route, or delay time in the target. Traceroute tells me a little more, at the cost of a more intense process. In places I use ping, I tend to send a number of them over a period of time and observe on the statistics that result, not a single ping result.



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Mon, 2 Mar 2015, Dave Taht wrote:

> I note that there are conflicting definitions of CS1 (background). 
> Comcast, re-marks about 90% I see to CS1 from whatever it was 
> originally, in the hope that it is treated as background. The ancient 
> firmware in commercial home routers *prioritizes* CS1 on etheret and 
> *deprioritizes it* on wifi, into the 802.11e background queue, when 
> enabled. CeroWrt tries to cons

This is the default I have seen in quite a few 4 queue L2 devices, I 
believe it comes from IEEE recommendations:

https://community.extremenetworks.com/extreme/topics/default_802_1p_priority_to_transmit_queue_mapping

     Product           802.1p Priority/CoS    Transmit Queue

Matrix N; non-Policy Priority, 'show port priority-queue'
  -4 or 8 queues-              0      4&8 Qs-> 1
     Fast Ethernet ports       1               0
                               2               0
                               3               1
                               4               2
                               5               2
                               6               3
                               7               3

Since a lot of products will mark IP PREC part of TOS directly into .1p 
bits, this means CS0 and CS3 goes into higher priority queues compared to 
CS1 and CS2.

http://www.hp.com/rnd/device_help/help/hpwnd/webhelp/HPJ4121A/qos_priority_map.html 
seems to indicate HP does the same.

http://alliedtelesis.com/manuals/GS900M_Series_Web_Browser_User_Guide_revA/aw1001299.html 
says the same.

However, I find devices that do differently by default as you have already 
discovered.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Tue, 3 Mar 2015, Valdis.Kletnieks@vt.edu wrote:

> On Mon, 02 Mar 2015 11:17:33 +0100, Mikael Abrahamsson said:
>
>> We have a huge amount of information in our TCP stacks that either are
>> locked in there and not used properly to help users figure out what's
>> going on, and there is basically zero information flow between the
>> applications using TCP and the TCP stack itself. Each just tries to do its
>> best on its own layer.
>
> You might want to touch base with the 'web10g' crew, who are working on
> instrumenting the Linux network stack to expose more of the inner variables
> for analysis and tuning.
>
> http://web10g.org/

I know about them. I have been pitching my idea since 2008 or so. We need 
more people working on this.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [Cerowrt-devel] [aqm] ping loss "considered harmful"

[dpreed]

It's a heavy burden to place on ICMP ping to say that it should tell you about all aspects of its path through all the networks between source and destination.

On the other hand, I'll suggest that Fred's point - treat ICMP Ping like any other IP datagram with the same header options is the essence of Ping's function.

I'd suggest that a more flexible rule would be for the echo reply to set header options (including DSCP) based on the ping packet's content tells it to be set to.

DSCP should not be changed en route, so the receiver of the echo reply should be able to know what DSCP was used on the reply packet.

Clearly the value of Ping is its standardized form and its ubiquity.  Being able to control all header options from the sender is useful for that function.  If the receiver cannot satisfy the request (e.g. it doesn't support the DSCP mechanism), it can just refuse to set it. That way, Ping acquires an option, but the option is upward compatible if not supported.

(I specifically talk about all header options here, rather than DSCP in particular.  For example, one could request ECN marking in the same way, with the same rules. I'm not a big fan of DSCP because I think the code points are poorly defined and so forth, but that's irrelevant to the thinking about Ping vs. "envelope" option - fully end-to-end modular services like ECN clearly should be testable in this way, and in the case of ECN, the notion of just not doing it if you can't do it fits into the Ping conceptual framework).




On Tuesday, March 3, 2015 1:00pm, "Fred Baker (fred)" <fred@cisco.com> said:

> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
>> On Mar 3, 2015, at 9:29 AM, Wesley Eddy <wes@mti-systems.com> wrote:
>>
>> On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
>>>
>>>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>>>> <mailto:dave.taht@gmail.com>> wrote:
>>>>
>>>> How can we fix this user perception, short of re-prioritizing ping in
>>>> sqm-scripts?
>>>
>>> IMHO, ping should go at the same priority as general traffic - the
>>> default class, DSCP=0. When I send one, I am asking whether a random
>>> packet can get to a given address and get a response back. I can imagine
>>> having a command-line parameter to set the DSCP to another value of my
>>> choosing.
>>
>> I generally agree, however ...
>>
>> The DSCP of the response isn't controllable though, and likely the DSCP
>> that is ultimately received will not be the one that was sent, so it
>> can't be as simple as echoing back the same one.  Ping doesn't tell you
>> latency components in the forward or return path (some other protocols
>> can do this though).
>>
>> So, setting the DSCP on the outgoing request may not be all that useful,
>> depending on what the measurement is really for.
> 
> Note that I didn’t say “I demand”… :-)
> 
> I share the perception that ping is useful when it’s useful, and that it is
> at best an approximation. If I can get a packet to the destination and a response
> back, and I know the time I sent it and the time I received the response, I know
> exactly that - messages went out and back and took some amount of total time. I
> don’t know anything about the specifics of the path, of buffers en route, or
> delay time in the target. Traceroute tells me a little more, at the cost of a more
> intense process. In places I use ping, I tend to send a number of them over a
> period of time and observe on the statistics that result, not a single ping
> result.
> 

Re: [Bloat] [aqm] [Cerowrt-devel] ping loss "considered harmful"

[Mikael Abrahamsson]

On Wed, 4 Mar 2015, dpreed@reed.com wrote:

> DSCP should not be changed en route, so the receiver of the echo reply 
> should be able to know what DSCP was used on the reply packet.

Changing the DSCP bits by the ISP is definitely not unheard of, so don't 
count on it.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] [aqm] ping loss "considered harmful"

[Curtis Villamizar]

In message <CAA93jw4F7iffbTRUt5RFsF0wgoOAXPUVHdu7JESVq4uM17cm7A@mail.gmail.com>
Dave Taht writes:
 
> My point was A), I have seen tons of shapers out there that actually
> prioritize ping over other traffic. I figure everyone here will agree
> that is a terrible practice, but I can certainly say it exists, as it
> is a dumb mistake replicated in tons of shapers I have seen... that
> makes people in marketing happy.
>  
> Already put up extensive commentary on that bit of foolishness on
> "wondershaper must die".


Its possible to detect such a shaper prioritizing ICMP echo/reply by
doing a an HTTP fetch concurrent with a ping and then and see if the
TCP data packet get significantly delayed relative to the ICMP echo
and echo reply packets.  You'd have to do a tcpdump and match the ICMP
echo to the echo reply and see if later the ICMP RTT looks very
different from the TCP RTT.  It might be that the SYN and SYN ACK are
not delayed but the plain old TCP date packets are.

If anyone has a small amount of spare time and wants to put together a
shell script its certainly doable.

Curtis

Re: [Bloat] [aqm] ping loss "considered harmful"

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]


On Mar 5, 2015, at 1:56 PM, Curtis Villamizar <curtis@ipv6.occnc.com> wrote:

> In message <CAA93jw4F7iffbTRUt5RFsF0wgoOAXPUVHdu7JESVq4uM17cm7A@mail.gmail.com>
> Dave Taht writes:
> 
>> My point was A), I have seen tons of shapers out there that actually
>> prioritize ping over other traffic. I figure everyone here will agree
>> that is a terrible practice, but I can certainly say it exists, as it
>> is a dumb mistake replicated in tons of shapers I have seen... that
>> makes people in marketing happy.
>> 
>> Already put up extensive commentary on that bit of foolishness on
>> "wondershaper must die".
> 
> 
> Its possible to detect such a shaper prioritizing ICMP echo/reply by
> doing a an HTTP fetch concurrent with a ping... 

For an easy (but imprecise) way test the HTTP response times, try Blip - http://gfblip.appspot.com/ (or read about it on github: https://github.com/apenwarr/blip) Blip sends short http requests to a couple hosts and measures the response time of the error pages. 

> and then and see if the
> TCP data packet get significantly delayed relative to the ICMP echo
> and echo reply packets.  You'd have to do a tcpdump and match the ICMP
> echo to the echo reply and see if later the ICMP RTT looks very
> different from the TCP RTT.  It might be that the SYN and SYN ACK are
> not delayed but the plain old TCP date packets are.
> 
> If anyone has a small amount of spare time and wants to put together a
> shell script its certainly doable.
> 
> Curtis
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [Bloat] [Cerowrt-devel] ping loss "considered harmful"

[Matt Taggart]

Dave Taht writes:

> wow. It never registered to me that users might make a value judgement
> based on the amount of ping loss, and in looking back in time, I can
> think of multiple people that have said things based on their
> perception that losing pings was bad, and that sqm-scripts was "worse
> than something else because of it."

This thread makes me realize that my standard method of measuring latency 
over time might have issues. I use smokeping

  http://oss.oetiker.ch/smokeping/

which is a really nice way of measuring and visualizing packet loss and 
variations in latency. I am using the default probe type which uses fping 
(ICMP http://www.fping.org/ ).

It has been working well, I set it up for a site in advance of setting up 
SQM and then afterwards I can see the changes and determine if more tuning 
is needed.  But if ICMP is having it's priority adjusted (up or down), then 
the results might not reflect the latency of other services.

Fortunately the nice thing is that many other probe types exist 

  http://oss.oetiker.ch/smokeping/probe/index.en.html

So which probe types would be good to use for bufferbloat measurement? I 
guess the answer is "whatever is important to you", but I also suspect 
there is a set of things that ISPs are known to mess with.
HTTP? But also maybe HTTPS in case they are doing some sort of transparent 
proxy?
DNS?
SIP?
I suppose you could even do explicit checks for things like Netflix (but 
then it's easy to go off on a tangent of building a net neutrality 
observatory).

On a somewhat related note, I was once using smokeping to measure a fiber 
link to a bandwidth provider and had it configured to ping the router IP on 
the other side of the link. In talking to one of their engineers, I learned 
that they deprioritize ICMP when talking _with_ their routers, so my 
measurement weren't valid. (I don't know if they deprioritize ICMP traffic 
going _through_ their routers)

-- 
Matt Taggart
matt@lackof.org

Re: [Bloat] [Cerowrt-devel] ping loss "considered harmful"

[Dave Taht]

I had spoken to someone at nznog that promised to combine mrtg +
smokeping or cacti + smokeping so as to be able to get long term
latency and bandwidth numbers on one graph. cc added.

On Thu, Mar 5, 2015 at 12:38 PM, Matt Taggart <matt@lackof.org> wrote:
> Dave Taht writes:
>
>> wow. It never registered to me that users might make a value judgement
>> based on the amount of ping *loss*, rather than latency, and in looking back in time, I can
>> think of multiple people that have said things based on their
>> perception that losing pings was bad, and that sqm-scripts was "worse
>> than something else because of it."
>
> This thread makes me realize that my standard method of measuring latency
> over time might have issues. I use smokeping
>
>   http://oss.oetiker.ch/smokeping/


in sqm-scripts's case, possibly, all you have been collecting is
largely worst case behavior, which I don't mind collecting as it tends
to be pretty good. :)

However, I have been unclear. In the main (modern - I don't know what
version you have) sqm code, IF you enable dscp squashing on inbound
(the default), you do end up with a single fq_codel queue, not 3, no
classification or ping prioritization. (it is the default because of
all the re-marking I have seen from comcast)

So if you are, as I am, monitoring your boxes from the outside, there
is no classification and prioritization present for ping.

do a tc -s qdisc show ifbwhatever (varies by platform) to see how many
queues you have. Example of a single queued inbound rate limiter +
fq_codel (yea! packet drop AND ecn working great!)

root@lorna-gw:~# tc -s qdisc show dev ifb4ge00
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0
direct_qlen 32
 Sent 168443514948 bytes 334370551 pkt (dropped 0, overlimits
143273498 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 110: parent 1:10 limit 1001p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
 Sent 168443514948 bytes 334370551 pkt (dropped 17480, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 125872421 ecn_mark 1044
  new_flows_len 0 old_flows_len 1

root@lorna-gw:~# uptime
 12:45:35 up 54 days, 22:33,  load average: 0.05, 0.05, 0.04

dscp classification in general, is only useful from within your own
network, going outside.

> which is a really nice way of measuring and visualizing packet loss and
> variations in latency. I am using the default probe type which uses fping
> (ICMP http://www.fping.org/ ).

I LOVE smokeping and wish very much we had a way to combine it with
mrtg data to see latency AND bandwidth at the same time.

>
> It has been working well, I set it up for a site in advance of setting up
> SQM and then afterwards I can see the changes and determine if more tuning
> is needed.  But if ICMP is having it's priority adjusted (up or down), then
> the results might not reflect the latency of other services.
>
> Fortunately the nice thing is that many other probe types exist
>
>   http://oss.oetiker.ch/smokeping/probe/index.en.html
>
> So which probe types would be good to use for bufferbloat measurement? I
> guess the answer is "whatever is important to you", but I also suspect
> there is a set of things that ISPs are known to mess with.
> HTTP? But also maybe HTTPS in case they are doing some sort of transparent
> proxy?
> DNS?
> SIP?
> I suppose you could even do explicit checks for things like Netflix (but
> then it's easy to go off on a tangent of building a net neutrality
> observatory).
>
> On a somewhat related note, I was once using smokeping to measure a fiber
> link to a bandwidth provider and had it configured to ping the router IP on
> the other side of the link. In talking to one of their engineers, I learned
> that they deprioritize ICMP when talking _with_ their routers, so my
> measurement weren't valid. (I don't know if they deprioritize ICMP traffic
> going _through_ their routers)

I do strongly recomend deprioritizing ping slightly, and as I noted, I
have seen many a borken
script that actually prioritized it, which is foolish, at best.

I keep hoping multiple (many!) someones here will go have lunch with
their company's oft lonely, oft starving sysadmin(s), to ask them what
they are doing as to firewalling, QoS and traffic shaping. Most of the
ones I have talked are quite eager to show off their work, which is
unfortunately often of wildly varying quality and complexity.

I find that an offer of saki and sushi are most conducive to getting
that conversation started.

I certainly would like to see more default corporate
firewall/QoS/shaping rules than I have personally, for various
platforms. Someone's got to have some good ideas in them... and it
would be nice to know how far the bad ones, have propagated.

> --
> Matt Taggart
> matt@lackof.org
>
>



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Bloat] [aqm] [Cerowrt-devel] ping loss "considered harmful"

[Sam Silvester]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Thu, Mar 5, 2015 at 6:15 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Wed, 4 Mar 2015, dpreed@reed.com wrote:
>
>  DSCP should not be changed en route, so the receiver of the echo reply
>> should be able to know what DSCP was used on the reply packet.
>>
>
> Changing the DSCP bits by the ISP is definitely not unheard of, so don't
> count on it.
>
>
Out of curiousity, why would you assume DSCP should not be changed en route?

My understanding is within a given DSCP administrative domain, the DSCP
value is expected to be passed unchanged, but at administrative boundaries
(common ones would be between the customer and their ISP, or between two
ISPs at a peering point or private cross connect) it's common to
(re)classify or simply rewrite to a known value, unless there is a specific
agreement negotiated either between ISPs or between the customer and the
ISP.

I'm no RFC expert, but this seems to be discussed in a bit more detail in
RFC 3260, Section 6 (Unknown/Improperly Mapped DSCPs).

Certainly RFC aside, I wouldn't be trusting DSCP markings of IX peers, and
in the case of interconnects used for VoIP, I'd be reclassifying based on
appropriate access lists negotiated between parties so that only traffic
destined between SIP SBCs would be accepted into the higher classes.
Everything else I would expect to rewrite to a known value.

I guess my point is I wouldn't personally be setting DSCP values from my
home DSL router and expecting them to last past my ISPs upstream router.
Perhaps that's just me though?

Regards,

Sam

[-- Attachment #2: Type: text/html, Size: 2278 bytes --]