[Cerowrt-devel] ping loss "considered harmful"

[Cerowrt-devel] ping loss "considered harmful"

[Dave Taht]

On this thread over here, an otherwise pretty clueful user chose
openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
*higher ping loss*.


http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125

(I note that both fq_codel enabled QoS systems outperformed
streamboost by a lot, which I am happy about)

wow. It never registered to me that users might make a value judgement
based on the amount of ping loss, and in looking back in time, I can
think of multiple people that have said things based on their
perception that losing pings was bad, and that sqm-scripts was "worse
than something else because of it."

sqm-scripts explicitly *deprioritizes* ping. In particular, this
reduces the impact of ping floods from ipv6 to your entire /64, or to
your whole ipv4, fairly well. And I had made the point that
prioritizing ping was a bad idea here (including some dripping sarcasm
later in the piece).

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

but wow, it never occurred to me - in all these years - that ping was
the next core metric on simple tests. I can be really dumb.

I use netperf-wrapper and tend to ignore most of the ping data, but
certainly on some benchmarks we have published ping doesn't look as
good as the other stuff, *because it is deprioritized below all the
other traffic*. Not strictly rate limited - as some systems do by
default, including openwrt, which is impossible to get right - just
deprioritized....

How can we fix this user perception, short of re-prioritizing ping in
sqm-scripts?

-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Andrew Mcgregor]

[-- Attachment #1: Type: text/plain, Size: 2266 bytes --]

Hash all pings into one fq_codel bucket reserved for the purpose, so if
we're getting DoSed we drop them, otherwise if they stay thin they get
prioritised?

On 2 March 2015 at 14:57, Dave Taht <dave.taht@gmail.com> wrote:

> On this thread over here, an otherwise pretty clueful user chose
> openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
> *higher ping loss*.
>
>
> http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125
>
> (I note that both fq_codel enabled QoS systems outperformed
> streamboost by a lot, which I am happy about)
>
> wow. It never registered to me that users might make a value judgement
> based on the amount of ping loss, and in looking back in time, I can
> think of multiple people that have said things based on their
> perception that losing pings was bad, and that sqm-scripts was "worse
> than something else because of it."
>
> sqm-scripts explicitly *deprioritizes* ping. In particular, this
> reduces the impact of ping floods from ipv6 to your entire /64, or to
> your whole ipv4, fairly well. And I had made the point that
> prioritizing ping was a bad idea here (including some dripping sarcasm
> later in the piece).
>
> http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die
>
> but wow, it never occurred to me - in all these years - that ping was
> the next core metric on simple tests. I can be really dumb.
>
> I use netperf-wrapper and tend to ignore most of the ping data, but
> certainly on some benchmarks we have published ping doesn't look as
> good as the other stuff, *because it is deprioritized below all the
> other traffic*. Not strictly rate limited - as some systems do by
> default, including openwrt, which is impossible to get right - just
> deprioritized....
>
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?
>
> --
> Dave Täht
> Let's make wifi fast, less jittery and reliable again!
>
> https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>



-- 
Andrew McGregor | SRE | andrewmcgr@google.com | +61 4 1071 2221

[-- Attachment #2: Type: text/html, Size: 4098 bytes --]

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Sun, 1 Mar 2015, Dave Taht wrote:

> but wow, it never occurred to me - in all these years - that ping was 
> the next core metric on simple tests. I can be really dumb.
>
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?

People will make all kinds of judgement calls based on things that are not 
very relevant.

Some people think a traceroute with few hops in it to be better than one 
with more hops in it. So a lot of providers use their MPLS networks and 
make the intermediate nodes not show up in traceroute.

I don't know how many times I have experienced people have used PING and 
opened trouble tickets when a core router had ICMP ping packet loss (even 
though all core router platforms have rate limiters for ICMP).

When you say that you de-prioritize ping, do you mean only ECHO REQUEST 
and ECHO REPLY, or do you mean all ICMP? Perhaps a solution would be to 
assure 5% of the link bandwidth for ICMP?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] ping loss "considered harmful"

[David Lang]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2503 bytes --]

On Sun, 1 Mar 2015, Dave Taht wrote:

> On this thread over here, an otherwise pretty clueful user chose
> openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
> *higher ping loss*.
>
>
> http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125
>
> (I note that both fq_codel enabled QoS systems outperformed
> streamboost by a lot, which I am happy about)
>
> wow. It never registered to me that users might make a value judgement
> based on the amount of ping loss, and in looking back in time, I can
> think of multiple people that have said things based on their
> perception that losing pings was bad, and that sqm-scripts was "worse
> than something else because of it."

People make the assumption that if ping packets are being dropped, their data is 
getting dropped as well. This is part of the "dropped packets are bad, so buffer 
so you don't drop packets" mentality

To address this, the test results should first show all the stats without 
showing dropped packets, or if you need to show them, add a comment at that 
point that says that data packets are given priority over ping packets, so 
dropped ping packets don't imply that data packets would be dropped.

David Lang


> sqm-scripts explicitly *deprioritizes* ping. In particular, this
> reduces the impact of ping floods from ipv6 to your entire /64, or to
> your whole ipv4, fairly well. And I had made the point that
> prioritizing ping was a bad idea here (including some dripping sarcasm
> later in the piece).
>
> http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die
>
> but wow, it never occurred to me - in all these years - that ping was
> the next core metric on simple tests. I can be really dumb.
>
> I use netperf-wrapper and tend to ignore most of the ping data, but
> certainly on some benchmarks we have published ping doesn't look as
> good as the other stuff, *because it is deprioritized below all the
> other traffic*. Not strictly rate limited - as some systems do by
> default, including openwrt, which is impossible to get right - just
> deprioritized....
>
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?
>
> -- 
> Dave Täht
> Let's make wifi fast, less jittery and reliable again!
>
> https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Brian Trammell]

[-- Attachment #1: Type: text/plain, Size: 1839 bytes --]

hi Dave,

> On 02 Mar 2015, at 04:57, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On this thread over here, an otherwise pretty clueful user chose
> openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
> *higher ping loss*.

I am not proud of the fact that I am not surprised by this.

> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?

The real solution is to create a utility called "ping" that uses traffic that gets prioritized the same way as the traffic you care about instead of ICMP echo request/reply. Users don't care about the packets on the wire so much as they do that you're supposed to ping things.

We have a protocol for this <ippm-chair-hat> called TWAMP </ippm-chair-hat> which shares the unfortunate property with all such approaches that it requires the far end to be running a reflector (the advantage of ICMP is the reflector is built into the kernel everywhere) or worse a control protocol for setting reflectors up.

Gaming protocols do this right - latency measurement is built into the protocol. For the web we already have tools and reasonably well accepted metrics (time to first byte). Wrap $YOUR_FAVORITE_BROWSER's perftools in a ping -w <url> utility and this problem goes away in that space. I presume that rtcweb clients in the browser will take advantage of these tools and provide measurements the same way games do.

Andrew's hack is probably easier to actually get deployed in a reasonable amount of time though. :)

Cheers,

Brian

> --
> Dave Täht
> Let's make wifi fast, less jittery and reliable again!
> 
> https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
> 
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Mon, 2 Mar 2015, Brian Trammell wrote:

> Gaming protocols do this right - latency measurement is built into the 
> protocol.

I believe this is the only way to do it properly, and the most likely 
easiest way to get this deployed would be to use the TCP stack.

We need to give users an easy-to-understand metric on how well their 
Internet traffic is working. So the problem here is that the users can't 
tell how well it's working without resorting to ICMP PING to try to figure 
out what's going on.

For instance, if their web browser had insight into what the TCP stack was 
doing then it could present information a lot better to the user. Instead 
of telling the user "time to first byte" (which is L4 information), it 
could tell the less novice user about packet loss, PDV, reordering, RTT, 
how well concurrent connections to the same IP address are doing, tell 
more about *why* some connections are slow instead of just saying "it took 
5.3 seconds to load this webpage and here are the connections and how long 
each took". For the novice user there should be some kind of expert system 
that collects data that you can send to the ISP that also has an expert 
system to say "it seems your local connection delays packets", please 
connect to a wired connection and try again". It would know if the problem 
was excessive delay, excessive delay that varied a lot, packet loss, 
reordering, or whatever.

We have a huge amount of information in our TCP stacks that either are 
locked in there and not used properly to help users figure out what's 
going on, and there is basically zero information flow between the 
applications using TCP and the TCP stack itself. Each just tries to do its 
best on its own layer.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] [Bloat] [aqm] ping loss "considered harmful"

[Jonathan Morton]

> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Mon, 2 Mar 2015, Brian Trammell wrote:
> 
>> Gaming protocols do this right - latency measurement is built into the protocol.
> 
> I believe this is the only way to do it properly, and the most likely easiest way to get this deployed would be to use the TCP stack.
> 
> We need to give users an easy-to-understand metric on how well their Internet traffic is working. So the problem here is that the users can't tell how well it's working without resorting to ICMP PING to try to figure out what's going on.
> 
> For instance, if their web browser had insight into what the TCP stack was doing then it could present information a lot better to the user. Instead of telling the user "time to first byte" (which is L4 information), it could tell the less novice user about packet loss, PDV, reordering, RTT, how well concurrent connections to the same IP address are doing, tell more about *why* some connections are slow instead of just saying "it took 5.3 seconds to load this webpage and here are the connections and how long each took". For the novice user there should be some kind of expert system that collects data that you can send to the ISP that also has an expert system to say "it seems your local connection delays packets", please connect to a wired connection and try again". It would know if the problem was excessive delay, excessive delay that varied a lot, packet loss, reordering, or whatever.
> 
> We have a huge amount of information in our TCP stacks that either are locked in there and not used properly to help users figure out what's going on, and there is basically zero information flow between the applications using TCP and the TCP stack itself. Each just tries to do its best on its own layer.

This seems like an actually good idea.  Several of those statistics, at least, could be exposed to userspace without incurring any additional overhead in the stack (except for the queries themselves), which is important for high-performance server users.  TCP stacks already track RTT, and sometimes MinRTT - the difference between these values is a reasonable lower-bound estimate of induced latency.

For stacks which don’t already track all the desirable data, a socket option could be used to turn that on, allocating extra space to do so.  To maximise portability, therefore, it might be necessary to require that option before statistics requests will be valid, even on stacks which do collect it all anyway.

Recent versions of Windows, even, have a semi-magic system which gives a little indicator of whether your connection has functioning Internet connectivity or not.  This could be extended, if Microsoft saw fit, to interpret these statistics and notify the user that their connection was behaving badly in the ways we now find interesting.  Whether Microsoft will do such a thing (which would undoubtedly piss off every major ISP on the planet) is another matter, but it’s a concept that can be used by Linux desktops as well, and with less political fallout.

Now, who’s going to knuckle down and implement it?

 - Jonathan Morton

Re: [Cerowrt-devel] [Bloat] [aqm] ping loss "considered harmful"

[dpreed]

On my own web server (running nginx) I provide an HTTP 1.1 accessible statistics service.  It returns a single JSON structure with the underlying packet statistics for the server and the current connection. Since this packet is inserted into the HTTP 1.1 stream upon request, it provides both the statistics and a single-round-trip "ping" that can measure queue delay on demand.

Changing server code is a lot easier than changing browser code.  Anyone who would like to develop a "standards-track" option for HTTP 1.1 is quite welcome to do so...  right now it is a "RESTful" interface, so measurements of the client side must be obtained indirectly, but simple JavaScript code in a web page can access this with standard HTML5 code.

I'm not looking to support this code or enhance it.  I've thought about putting it up on GitHub.

In any case, it's easy to add a little widget to the corner of all the web pages from a site that displays latency and net statistics at various levels in a very pretty user-understandable form.

To see how this works, just go to the alt.reed.com site (preferably using an iPhone or Android phone) to see a trivial test app I put out a number of years ago for my personal zeroth-order diagnostics.

nginx (as most know) is an incredibly tight, low overhead HTTP 1.1 server that lets you add plugins for lots of things other than retrieving web pages from disk.  It's not the pile of spaghetti that Apache has become. It supports WebSockets and HTTPS quite well, too.




On Monday, March 2, 2015 5:54am, "Jonathan Morton" <chromatix99@gmail.com> said:

> 
>> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> On Mon, 2 Mar 2015, Brian Trammell wrote:
>>
>>> Gaming protocols do this right - latency measurement is built into the protocol.
>>
>> I believe this is the only way to do it properly, and the most likely easiest way
>> to get this deployed would be to use the TCP stack.
>>
>> We need to give users an easy-to-understand metric on how well their Internet
>> traffic is working. So the problem here is that the users can't tell how well
>> it's working without resorting to ICMP PING to try to figure out what's going on.
>>
>> For instance, if their web browser had insight into what the TCP stack was doing
>> then it could present information a lot better to the user. Instead of telling
>> the user "time to first byte" (which is L4 information), it could tell the less
>> novice user about packet loss, PDV, reordering, RTT, how well concurrent
>> connections to the same IP address are doing, tell more about *why* some
>> connections are slow instead of just saying "it took 5.3 seconds to load this
>> webpage and here are the connections and how long each took". For the novice user
>> there should be some kind of expert system that collects data that you can send
>> to the ISP that also has an expert system to say "it seems your local connection
>> delays packets", please connect to a wired connection and try again". It would
>> know if the problem was excessive delay, excessive delay that varied a lot,
>> packet loss, reordering, or whatever.
>>
>> We have a huge amount of information in our TCP stacks that either are locked in
>> there and not used properly to help users figure out what's going on, and there
>> is basically zero information flow between the applications using TCP and the TCP
>> stack itself. Each just tries to do its best on its own layer.
> 
> This seems like an actually good idea.  Several of those statistics, at least,
> could be exposed to userspace without incurring any additional overhead in the
> stack (except for the queries themselves), which is important for high-performance
> server users.  TCP stacks already track RTT, and sometimes MinRTT - the difference
> between these values is a reasonable lower-bound estimate of induced latency.
> 
> For stacks which don’t already track all the desirable data, a socket option
> could be used to turn that on, allocating extra space to do so.  To maximise
> portability, therefore, it might be necessary to require that option before
> statistics requests will be valid, even on stacks which do collect it all anyway.
> 
> Recent versions of Windows, even, have a semi-magic system which gives a little
> indicator of whether your connection has functioning Internet connectivity or not.
>  This could be extended, if Microsoft saw fit, to interpret these statistics and
> notify the user that their connection was behaving badly in the ways we now find
> interesting.  Whether Microsoft will do such a thing (which would undoubtedly piss
> off every major ISP on the planet) is another matter, but it’s a concept
> that can be used by Linux desktops as well, and with less political fallout.
> 
> Now, who’s going to knuckle down and implement it?
> 
>  - Jonathan Morton
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 

Re: [Cerowrt-devel] [Bloat] [aqm] ping loss "considered harmful"

[Brian Trammell]

[-- Attachment #1: Type: text/plain, Size: 4636 bytes --]


> On 02 Mar 2015, at 11:54, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> 
>> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> 
>> On Mon, 2 Mar 2015, Brian Trammell wrote:
>> 
>>> Gaming protocols do this right - latency measurement is built into the protocol.
>> 
>> I believe this is the only way to do it properly, and the most likely easiest way to get this deployed would be to use the TCP stack.
>> 
>> We need to give users an easy-to-understand metric on how well their Internet traffic is working. So the problem here is that the users can't tell how well it's working without resorting to ICMP PING to try to figure out what's going on.
>> 
>> For instance, if their web browser had insight into what the TCP stack was doing then it could present information a lot better to the user. Instead of telling the user "time to first byte" (which is L4 information), it could tell the less novice user about packet loss, PDV, reordering, RTT, how well concurrent connections to the same IP address are doing, tell more about *why* some connections are slow instead of just saying "it took 5.3 seconds to load this webpage and here are the connections and how long each took". For the novice user there should be some kind of expert system that collects data that you can send to the ISP that also has an expert system to say "it seems your local connection delays packets", please connect to a wired connection and try again". It would know if the problem was excessive delay, excessive delay that varied a lot, packet loss, reordering, or whatever.
>> 
>> We have a huge amount of information in our TCP stacks that either are locked in there and not used properly to help users figure out what's going on, and there is basically zero information flow between the applications using TCP and the TCP stack itself. Each just tries to do its best on its own layer.
> 
> This seems like an actually good idea.  Several of those statistics, at least, could be exposed to userspace without incurring any additional overhead in the stack (except for the queries themselves), which is important for high-performance server users.  TCP stacks already track RTT, and sometimes MinRTT - the difference between these values is a reasonable lower-bound estimate of induced latency.
> 
> For stacks which don’t already track all the desirable data, a socket option could be used to turn that on, allocating extra space to do so.  To maximise portability, therefore, it might be necessary to require that option before statistics requests will be valid, even on stacks which do collect it all anyway.

So there seem to me to be three separate but related problems we want to solve here:

(1) How to get users who don't care what ping is useful information about their applications' network performance.

(2) How to get users who do care what ping is information that actually reflects application performance when they type ping (or, more generally, how to make sure that the common diagnostic tools neither (a) provide misleading information or (b) require network misconfiguration to ensure "proper" operation of the diagnostic tools (cf. speedtest.net and its ilk).

(3) How to get application developers tools they can use to integrate network measurement into their apps without having to roll their own (i.e., helping them to solve (1), and enabling the creation of tools for (2) ).

This is an approach to (3)... but (as with many things) the key to getting it deployed and used on endpoints would be defining a universal interface to it. "Yet another API to learn on every platform" == "I'm just going to use ping, thank you very much."

> Recent versions of Windows, even, have a semi-magic system which gives a little indicator of whether your connection has functioning Internet connectivity or not.  This could be extended, if Microsoft saw fit, to interpret these statistics and notify the user that their connection was behaving badly in the ways we now find interesting.  Whether Microsoft will do such a thing (which would undoubtedly piss off every major ISP on the planet) is another matter, but it’s a concept that can be used by Linux desktops as well, and with less political fallout.

This is, I'm afraid, the kicker. As long as everyone has an interest in pointing the finger at everyone else, people will choose to interpret the metrics how they like, and fail to respond to metrics they don't, no matter how good they are.

> Now, who’s going to knuckle down and implement it?

web10g.org is a start, for Linux anyway.

Cheers,

Brian



[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [Cerowrt-devel] [Bloat] [aqm] ping loss "considered harmful"

[David Lang]

 On Mon, 2 Mar 2015 15:45:10 +0100, Brian Trammell wrote:
>> On 02 Mar 2015, at 11:54, Jonathan Morton <chromatix99@gmail.com> 
>> wrote:
>>
>>
>>> On 2 Mar, 2015, at 12:17, Mikael Abrahamsson <swmike@swm.pp.se> 
>>> wrote:
>>>
>>> On Mon, 2 Mar 2015, Brian Trammell wrote:
>>>
>>>> Gaming protocols do this right - latency measurement is built into 
>>>> the protocol.
>>>
>>> I believe this is the only way to do it properly, and the most 
>>> likely easiest way to get this deployed would be to use the TCP 
>>> stack.
>>>
>>> We need to give users an easy-to-understand metric on how well 
>>> their Internet traffic is working. So the problem here is that the 
>>> users can't tell how well it's working without resorting to ICMP PING 
>>> to try to figure out what's going on.
>>>
>>> For instance, if their web browser had insight into what the TCP 
>>> stack was doing then it could present information a lot better to the 
>>> user. Instead of telling the user "time to first byte" (which is L4 
>>> information), it could tell the less novice user about packet loss, 
>>> PDV, reordering, RTT, how well concurrent connections to the same IP 
>>> address are doing, tell more about *why* some connections are slow 
>>> instead of just saying "it took 5.3 seconds to load this webpage and 
>>> here are the connections and how long each took". For the novice user 
>>> there should be some kind of expert system that collects data that 
>>> you can send to the ISP that also has an expert system to say "it 
>>> seems your local connection delays packets", please connect to a 
>>> wired connection and try again". It would know if the problem was 
>>> excessive delay, excessive delay that varied a lot, packet loss, 
>>> reordering, or whatever.

 There is hping3 that lets you do 'ping' and 'traceroute' on TCP and UDP 
 to any port (with different flags set if needed). Unfortunantly many 
 people are not familiar with it. I've also run into problems that the 
 stock build doesn't work right if you have too many interfaces (IIRC I 
 ran into problems around 8 interfaces, but since I was working with 
 firewalls that had up to 22 interfaces in use, I could be off by quite a 
 bit on that number)

 David Lang

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

On Mon, 02 Mar 2015 11:17:33 +0100, Mikael Abrahamsson said:

> We have a huge amount of information in our TCP stacks that either are
> locked in there and not used properly to help users figure out what's
> going on, and there is basically zero information flow between the
> applications using TCP and the TCP stack itself. Each just tries to do its
> best on its own layer.

You might want to touch base with the 'web10g' crew, who are working on
instrumenting the Linux network stack to expose more of the inner variables
for analysis and tuning.

http://web10g.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Tue, 3 Mar 2015, Valdis.Kletnieks@vt.edu wrote:

> On Mon, 02 Mar 2015 11:17:33 +0100, Mikael Abrahamsson said:
>
>> We have a huge amount of information in our TCP stacks that either are
>> locked in there and not used properly to help users figure out what's
>> going on, and there is basically zero information flow between the
>> applications using TCP and the TCP stack itself. Each just tries to do its
>> best on its own layer.
>
> You might want to touch base with the 'web10g' crew, who are working on
> instrumenting the Linux network stack to expose more of the inner variables
> for analysis and tuning.
>
> http://web10g.org/

I know about them. I have been pitching my idea since 2008 or so. We need 
more people working on this.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Joe Touch]

On 3/2/2015 1:40 AM, Brian Trammell wrote:
...
> The real solution is to create a utility called "ping" that uses 
> traffic that gets prioritized the same way as the traffic you care
> about instead of ICMP echo request/reply. Users don't care about
> the packets on the wire so much as they do that you're supposed to
> ping things.

There are three separate problems:

1. a ping that doesn't use ICMP
	there are dozens of these

2. needing a reflector
	ping gets around this only because the reflector is widely
	deployed (and integrated into most OSes)

3. using the same port as the traffic you care about
	transport protocol is only one problem (ICMP being a "transport
	protocol" by virtue of using the IP protocol number field)

	the other is differential prioritization based on port number

	there's no easy solution to that;
	every service would need an integrated
	ping reflector

I suspect #3 is the ultimate killer of this idea.

Joe

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[David Lang]

On Mon, 2 Mar 2015, Joe Touch wrote:

> On 3/2/2015 1:40 AM, Brian Trammell wrote:
> ...
>> The real solution is to create a utility called "ping" that uses
>> traffic that gets prioritized the same way as the traffic you care
>> about instead of ICMP echo request/reply. Users don't care about
>> the packets on the wire so much as they do that you're supposed to
>> ping things.
>
> There are three separate problems:
>
> 1. a ping that doesn't use ICMP
> 	there are dozens of these
>
> 2. needing a reflector
> 	ping gets around this only because the reflector is widely
> 	deployed (and integrated into most OSes)
>
> 3. using the same port as the traffic you care about
> 	transport protocol is only one problem (ICMP being a "transport
> 	protocol" by virtue of using the IP protocol number field)
>
> 	the other is differential prioritization based on port number
>
> 	there's no easy solution to that;
> 	every service would need an integrated
> 	ping reflector
>
> I suspect #3 is the ultimate killer of this idea.

The service you are trying to contact acts as a reflector for TCP traffic. If 
you send a syn you will get back a syn-ack from the TCP stack of the receiving 
system.

For UDP systems, it gets more interesting and service specific. But for TCP 
systems it works today.

David Lang

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Joe Touch]

On 3/2/2015 3:14 PM, David Lang wrote:
> On Mon, 2 Mar 2015, Joe Touch wrote:
> 
>> On 3/2/2015 1:40 AM, Brian Trammell wrote:
>> ...
>>> The real solution is to create a utility called "ping" that uses
>>> traffic that gets prioritized the same way as the traffic you care
>>> about instead of ICMP echo request/reply. Users don't care about
>>> the packets on the wire so much as they do that you're supposed to
>>> ping things.
>>
>> There are three separate problems:
>>
>> 1. a ping that doesn't use ICMP
>>     there are dozens of these
>>
>> 2. needing a reflector
>>     ping gets around this only because the reflector is widely
>>     deployed (and integrated into most OSes)
>>
>> 3. using the same port as the traffic you care about
>>     transport protocol is only one problem (ICMP being a "transport
>>     protocol" by virtue of using the IP protocol number field)
>>
>>     the other is differential prioritization based on port number
>>
>>     there's no easy solution to that;
>>     every service would need an integrated
>>     ping reflector
>>
>> I suspect #3 is the ultimate killer of this idea.
> 
> The service you are trying to contact acts as a reflector for TCP
> traffic. If you send a syn you will get back a syn-ack from the TCP
> stack of the receiving system.

Sure, but SYNs and SYN-ACKs don't get prioritized the same as
non-control TCP segments. And they could have been spoofed by a middlebox.

Joe

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[David Lang]

On Mon, 2 Mar 2015, Joe Touch wrote:

> On 3/2/2015 3:14 PM, David Lang wrote:
>> On Mon, 2 Mar 2015, Joe Touch wrote:
>>
>>> On 3/2/2015 1:40 AM, Brian Trammell wrote:
>>> ...
>>>> The real solution is to create a utility called "ping" that uses
>>>> traffic that gets prioritized the same way as the traffic you care
>>>> about instead of ICMP echo request/reply. Users don't care about
>>>> the packets on the wire so much as they do that you're supposed to
>>>> ping things.
>>>
>>> There are three separate problems:
>>>
>>> 1. a ping that doesn't use ICMP
>>>     there are dozens of these
>>>
>>> 2. needing a reflector
>>>     ping gets around this only because the reflector is widely
>>>     deployed (and integrated into most OSes)
>>>
>>> 3. using the same port as the traffic you care about
>>>     transport protocol is only one problem (ICMP being a "transport
>>>     protocol" by virtue of using the IP protocol number field)
>>>
>>>     the other is differential prioritization based on port number
>>>
>>>     there's no easy solution to that;
>>>     every service would need an integrated
>>>     ping reflector
>>>
>>> I suspect #3 is the ultimate killer of this idea.
>>
>> The service you are trying to contact acts as a reflector for TCP
>> traffic. If you send a syn you will get back a syn-ack from the TCP
>> stack of the receiving system.
>
> Sure, but SYNs and SYN-ACKs don't get prioritized the same as
> non-control TCP segments. And they could have been spoofed by a middlebox.

well, this is exactly the sort of thing that http heartbeat (the core of 
heartbleed) was designed to provide.

It's not perfect, you can see where you really get the response from (vi the 
traceroute), and if you are going through a MITM (i.e. transparent proxy), all 
you are ever going to be able to test is the network between yourself and the 
proxy. It's up to the people who own the proxy to test beyond that.

David Lang

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Andrew Mcgregor]

[-- Attachment #1: Type: text/plain, Size: 2486 bytes --]

Maybe I should mention https://github.com/apenwarr/blip

HTTP ping, using deliberate 204 responses.  Will run over whatever version
of HTTP/SPDY/QUIC your browser happens to be using at the time.

On 3 March 2015 at 10:34, David Lang <david@lang.hm> wrote:

> On Mon, 2 Mar 2015, Joe Touch wrote:
>
>  On 3/2/2015 3:14 PM, David Lang wrote:
>>
>>> On Mon, 2 Mar 2015, Joe Touch wrote:
>>>
>>>  On 3/2/2015 1:40 AM, Brian Trammell wrote:
>>>> ...
>>>>
>>>>> The real solution is to create a utility called "ping" that uses
>>>>> traffic that gets prioritized the same way as the traffic you care
>>>>> about instead of ICMP echo request/reply. Users don't care about
>>>>> the packets on the wire so much as they do that you're supposed to
>>>>> ping things.
>>>>>
>>>>
>>>> There are three separate problems:
>>>>
>>>> 1. a ping that doesn't use ICMP
>>>>     there are dozens of these
>>>>
>>>> 2. needing a reflector
>>>>     ping gets around this only because the reflector is widely
>>>>     deployed (and integrated into most OSes)
>>>>
>>>> 3. using the same port as the traffic you care about
>>>>     transport protocol is only one problem (ICMP being a "transport
>>>>     protocol" by virtue of using the IP protocol number field)
>>>>
>>>>     the other is differential prioritization based on port number
>>>>
>>>>     there's no easy solution to that;
>>>>     every service would need an integrated
>>>>     ping reflector
>>>>
>>>> I suspect #3 is the ultimate killer of this idea.
>>>>
>>>
>>> The service you are trying to contact acts as a reflector for TCP
>>> traffic. If you send a syn you will get back a syn-ack from the TCP
>>> stack of the receiving system.
>>>
>>
>> Sure, but SYNs and SYN-ACKs don't get prioritized the same as
>> non-control TCP segments. And they could have been spoofed by a middlebox.
>>
>
> well, this is exactly the sort of thing that http heartbeat (the core of
> heartbleed) was designed to provide.
>
> It's not perfect, you can see where you really get the response from (vi
> the traceroute), and if you are going through a MITM (i.e. transparent
> proxy), all you are ever going to be able to test is the network between
> yourself and the proxy. It's up to the people who own the proxy to test
> beyond that.
>
> David Lang
>
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>



-- 
Andrew McGregor | SRE | andrewmcgr@google.com | +61 4 1071 2221

[-- Attachment #2: Type: text/html, Size: 4663 bytes --]

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Dave Dolson]

I'm rather new to the aqm community, but IMHO, it is wrong to deprioritize the ping traffic by default. I would not have expected a forwarding agent to do this. 

And I think measuring ping times and loss is a reasonable thing to do, never expecting forwarding agents along the path to place more value on some IP packets than others. (Especially in my own network/lab when I did not configure such a policy)

There aren't many tools available to an end user. Ping, traceroute, speed test... The network is a black box to most users. 

As for the flood attack aspect, of course a flood of pings should wait their turn in a queue and be dropped as the queue fills.

It would be appropriate if this was fair to different ping flows in the same way TCP SYN packets are treated fairly. Treat ping flood like TCP SYN flood.  

My 2cents. 
-Dave Dolson



----- Original Message -----
From: Dave Taht [mailto:dave.taht@gmail.com]
Sent: Sunday, March 01, 2015 10:57 PM
To: cerowrt-devel@lists.bufferbloat.net <cerowrt-devel@lists.bufferbloat.net>; aqm@ietf.org <aqm@ietf.org>; bloat <bloat@lists.bufferbloat.net>
Subject: [aqm] ping loss "considered harmful"

On this thread over here, an otherwise pretty clueful user chose
openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
*higher ping loss*.


http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125

(I note that both fq_codel enabled QoS systems outperformed
streamboost by a lot, which I am happy about)

wow. It never registered to me that users might make a value judgement
based on the amount of ping loss, and in looking back in time, I can
think of multiple people that have said things based on their
perception that losing pings was bad, and that sqm-scripts was "worse
than something else because of it."

sqm-scripts explicitly *deprioritizes* ping. In particular, this
reduces the impact of ping floods from ipv6 to your entire /64, or to
your whole ipv4, fairly well. And I had made the point that
prioritizing ping was a bad idea here (including some dripping sarcasm
later in the piece).

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

but wow, it never occurred to me - in all these years - that ping was
the next core metric on simple tests. I can be really dumb.

I use netperf-wrapper and tend to ignore most of the ping data, but
certainly on some benchmarks we have published ping doesn't look as
good as the other stuff, *because it is deprioritized below all the
other traffic*. Not strictly rate limited - as some systems do by
default, including openwrt, which is impossible to get right - just
deprioritized....

How can we fix this user perception, short of re-prioritizing ping in
sqm-scripts?

-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

_______________________________________________
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Andrew Mcgregor]

[-- Attachment #1: Type: text/plain, Size: 3634 bytes --]

So, are you suggesting that, for example, Chrome's rather extensive network
debugging information get more publicised?  We can probably arrange that.

On 2 March 2015 at 21:47, Dave Dolson <ddolson@sandvine.com> wrote:

> I'm rather new to the aqm community, but IMHO, it is wrong to deprioritize
> the ping traffic by default. I would not have expected a forwarding agent
> to do this.
>
> And I think measuring ping times and loss is a reasonable thing to do,
> never expecting forwarding agents along the path to place more value on
> some IP packets than others. (Especially in my own network/lab when I did
> not configure such a policy)
>
> There aren't many tools available to an end user. Ping, traceroute, speed
> test... The network is a black box to most users.
>
> As for the flood attack aspect, of course a flood of pings should wait
> their turn in a queue and be dropped as the queue fills.
>
> It would be appropriate if this was fair to different ping flows in the
> same way TCP SYN packets are treated fairly. Treat ping flood like TCP SYN
> flood.
>
> My 2cents.
> -Dave Dolson
>
>
>
> ----- Original Message -----
> From: Dave Taht [mailto:dave.taht@gmail.com]
> Sent: Sunday, March 01, 2015 10:57 PM
> To: cerowrt-devel@lists.bufferbloat.net <
> cerowrt-devel@lists.bufferbloat.net>; aqm@ietf.org <aqm@ietf.org>; bloat <
> bloat@lists.bufferbloat.net>
> Subject: [aqm] ping loss "considered harmful"
>
> On this thread over here, an otherwise pretty clueful user chose
> openwrt's qos-scripts over the sqm-scripts, because sqm-scripts had
> *higher ping loss*.
>
>
> http://forums.dlink.com/index.php?topic=61634.msg251125#msg251125
>
> (I note that both fq_codel enabled QoS systems outperformed
> streamboost by a lot, which I am happy about)
>
> wow. It never registered to me that users might make a value judgement
> based on the amount of ping loss, and in looking back in time, I can
> think of multiple people that have said things based on their
> perception that losing pings was bad, and that sqm-scripts was "worse
> than something else because of it."
>
> sqm-scripts explicitly *deprioritizes* ping. In particular, this
> reduces the impact of ping floods from ipv6 to your entire /64, or to
> your whole ipv4, fairly well. And I had made the point that
> prioritizing ping was a bad idea here (including some dripping sarcasm
> later in the piece).
>
> http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die
>
> but wow, it never occurred to me - in all these years - that ping was
> the next core metric on simple tests. I can be really dumb.
>
> I use netperf-wrapper and tend to ignore most of the ping data, but
> certainly on some benchmarks we have published ping doesn't look as
> good as the other stuff, *because it is deprioritized below all the
> other traffic*. Not strictly rate limited - as some systems do by
> default, including openwrt, which is impossible to get right - just
> deprioritized....
>
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?
>
> --
> Dave Täht
> Let's make wifi fast, less jittery and reliable again!
>
> https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>



-- 
Andrew McGregor | SRE | andrewmcgr@google.com | +61 4 1071 2221

[-- Attachment #2: Type: text/html, Size: 5958 bytes --]

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[David Lang]

[-- Attachment #1: Type: TEXT/Plain, Size: 287 bytes --]

On Mon, 2 Mar 2015, Andrew Mcgregor wrote:

> So, are you suggesting that, for example, Chrome's rather extensive network
> debugging information get more publicised?  We can probably arrange that.

That would be good. I have no idea what debugging info you are referring to.

David Lang

[-- Attachment #2: Type: TEXT/PLAIN, Size: 164 bytes --]

_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Dave Dolson]

Would you do that to TCP or UDP traffic?

At IETF I often hear laments about middle-boxes breaking the internet by being "clever" with certain types of traffic.
It seems that policing ICMP falls into that category.

There may have been bugs in the past, but I'm not aware that ICMP packets are any more dangerous than UDP or TCP. And if the RFCs can be believed, ICMPv6 is critical to determining Path-MTU. Don't drop those.

One may wish to rate-limit ICMP (or DNS or TCP) flows as a matter of network policy, but in my opinion this should be kept orthogonal to solving buffer bloat.

Taken to the extreme, a network should support full utilization of a link doing only ping. If I wish to use my connection to the internet to ping hosts at full line rate, why not?


David Dolson
Senior Software Architect, Sandvine Inc.



-----Original Message-----
From: aqm [mailto:aqm-bounces@ietf.org] On Behalf Of Wes Felter
Sent: Monday, March 02, 2015 3:07 PM
To: aqm@ietf.org
Cc: cerowrt-devel@lists.bufferbloat.net; bloat@lists.bufferbloat.net
Subject: Re: [aqm] ping loss "considered harmful"

What about a token bucket policer, so more than N ICMP/second gets 
penalized but a normal ping won't be.

-- 
Wes Felter


_______________________________________________
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[David Lang]

On Mon, 2 Mar 2015, Dave Dolson wrote:

> Would you do that to TCP or UDP traffic?
>
> At IETF I often hear laments about middle-boxes breaking the internet by being "clever" with certain types of traffic.
> It seems that policing ICMP falls into that category.
>
> There may have been bugs in the past, but I'm not aware that ICMP packets are any more dangerous than UDP or TCP. And if the RFCs can be believed, ICMPv6 is critical to determining Path-MTU. Don't drop those.
>
> One may wish to rate-limit ICMP (or DNS or TCP) flows as a matter of network policy, but in my opinion this should be kept orthogonal to solving buffer bloat.
>
> Taken to the extreme, a network should support full utilization of a link doing only ping. If I wish to use my connection to the internet to ping hosts at full line rate, why not?

what's going on here isn't that pings are being rate limited, but rather that 
the TCP/UDP traffic is being given priority over the ping traffic. This means 
that when you max out the pipe, pings will suffer.

David Lang

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Dave Taht]

On Mon, Mar 2, 2015 at 12:06 PM, Wes Felter <wmf@felter.org> wrote:
> What about a token bucket policer, so more than N ICMP/second gets penalized
> but a normal ping won't be.

If I haven't expressed this too many times before, I want to thank you
all for existing.

It is the quality and caliber of all the folk on these mailing lists
coming up with new ideas, that keeps me going.

I've had a really rough could days (bronchitus, the virgin controversy
here: https://plus.google.com/u/0/107942175615993706558/posts/E1yMgbWW81C
), and I will try to write more later, but I thought I would add a bit
of light to the debate by showing you the default openwrt firewall
rules:

http://git.openwrt.org/?p=openwrt.git;a=blob;f=package/network/config/firewall/files/firewall.config;h=d149e77957de1c80358c1e8131920867a193c9ed;hb=HEAD

Problems: They do a (very low) syn limiter (25 syns/sec I think) by
default, which made sense 10 years ago, but not now - and I had run
into trouble with it while attempting to benchmark the alexa top 1
million in parallel through two routers at isc (at 300mbit) a few
years ago, so I ripped that out of cerowrt. It was a really puzzling
set of results, I had no idea why so many syn requests were dropping
on the floor til I inspected the actual firewall rules generated.

Similarly, they rate limit icmp at 1000/sec, which is too high at low
rates and too low at others. So I ripped that out of cerowrt, and in
sqm-scripts, I just held tossed it into the background bucket for for
a min of 30% of the background bandwidth.

I note that there are conflicting definitions of CS1 (background).
Comcast, re-marks about 90% I see to CS1 from whatever it was
originally, in the hope that it is treated as background. The ancient
firmware in commercial home routers *prioritizes* CS1 on etheret and
*deprioritizes it* on wifi, into the 802.11e background queue, when
enabled. CeroWrt tries to cons

I haven't seen ANY shipped open source FQ/AQM/QoS system that squashes
inbound dscp values, either, til cero, and despite the various rfcs
out there recommending that.

Openwrt's firewall system is, sadly, one of the better and saner
firewall rule sets that exists. Other rulesets I have seen in
commercial deployments are amazingly more complex, antiquated, and
frequently wrong.

as for fq/aqm/QoS I have pointed out the flaws in the comments on
wondershaper here:

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershaper_Must_Die

and those flaws, exist, in copy-pasted code in nearly every subsequent
shaper I have examined *and had the time to fix*.

EVERY open source shaper I have seen, also (aside from sqm-scripts)
does not handle ipv6 right, nor do they handle dsl/atm framing issues
correctly. Notably streamboost got it terribly, terribly wrong, as did
Netgear's new "Dynamic QoS", and d-links DIR-5500. Openwrt's
qos-scripts has similar problems. So does gargoyle's stuff, and
dd-wrts... and whatever they were using at nznog on my last talk.

In the original design for sqm-scripts, I'd tried to use 3 tiers of
DRR + fq_codel - for classification, and failed due to in some
circumstances DRR getting "stuck" on a queue when codel did a drop on
it. (a bug that may only have existed at the time). So we went with
HTB, with a max of 30% for prio, a min of 30% for background (due to
all the mismarked traffic, 5% wasn't enough) and background and BE
being able to use up all the bandwidth.

That said, however free.fr did DRR + fq_codel in their implemention
benchmarks out quite nicely,  tho I can't find the relevant comparison
benchmarks at the moment.

particularly the prio queue helps for multicast IPTV traffic. I tried
to discuss some of these issues here, but the wg was not interested at
the time for a document like this.

http://snapon.lab.bufferbloat.net/~d/draft-taht-home-gateway-best-practices-00.html


Since then I specified the combined rate limiter + fq_codel idea
called "cake" with up to 8 progressive levels of DRR for
classification, which jonathon did a GREAT first attempt at, but he
didn't believe me that strict rate limits were not the right thing
until he got a bit of user feedback....

I should write up all the design decisions that went into sqm-scripts
as it exists today - it came out of me using varous shapers in various
environments for 12+ years and for example, I had completely forgotten
why I had deprioritized icmp the way I did! - and rip out a few test
bits (like prioritizing AF42 for mosh) that are not correct for the
real world... whenever I manage to get out of bed.

>
> --
> Wes Felter
>
>
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Mikael Abrahamsson]

On Mon, 2 Mar 2015, Dave Taht wrote:

> I note that there are conflicting definitions of CS1 (background). 
> Comcast, re-marks about 90% I see to CS1 from whatever it was 
> originally, in the hope that it is treated as background. The ancient 
> firmware in commercial home routers *prioritizes* CS1 on etheret and 
> *deprioritizes it* on wifi, into the 802.11e background queue, when 
> enabled. CeroWrt tries to cons

This is the default I have seen in quite a few 4 queue L2 devices, I 
believe it comes from IEEE recommendations:

https://community.extremenetworks.com/extreme/topics/default_802_1p_priority_to_transmit_queue_mapping

     Product           802.1p Priority/CoS    Transmit Queue

Matrix N; non-Policy Priority, 'show port priority-queue'
  -4 or 8 queues-              0      4&8 Qs-> 1
     Fast Ethernet ports       1               0
                               2               0
                               3               1
                               4               2
                               5               2
                               6               3
                               7               3

Since a lot of products will mark IP PREC part of TOS directly into .1p 
bits, this means CS0 and CS3 goes into higher priority queues compared to 
CS1 and CS2.

http://www.hp.com/rnd/device_help/help/hpwnd/webhelp/HPJ4121A/qos_priority_map.html 
seems to indicate HP does the same.

http://alliedtelesis.com/manuals/GS900M_Series_Web_Browser_User_Guide_revA/aw1001299.html 
says the same.

However, I find devices that do differently by default as you have already 
discovered.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] [aqm] ping loss "considered harmful"

[Fred Baker (fred)]

[-- Attachment #1.1: Type: text/plain, Size: 458 bytes --]


> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> How can we fix this user perception, short of re-prioritizing ping in
> sqm-scripts?

IMHO, ping should go at the same priority as general traffic - the default class, DSCP=0. When I send one, I am asking whether a random packet can get to a given address and get a response back. I can imagine having a command-line parameter to set the DSCP to another value of my choosing.

[-- Attachment #1.2: Type: text/html, Size: 1976 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Cerowrt-devel] [Bloat] [aqm] ping loss "considered harmful"

[Wesley Eddy]

On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
> 
>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>> <mailto:dave.taht@gmail.com>> wrote:
>>
>> How can we fix this user perception, short of re-prioritizing ping in
>> sqm-scripts?
> 
> IMHO, ping should go at the same priority as general traffic - the
> default class, DSCP=0. When I send one, I am asking whether a random
> packet can get to a given address and get a response back. I can imagine
> having a command-line parameter to set the DSCP to another value of my
> choosing.
> 


I generally agree, however ...

The DSCP of the response isn't controllable though, and likely the DSCP
that is ultimately received will not be the one that was sent, so it
can't be as simple as echoing back the same one.  Ping doesn't tell you
latency components in the forward or return path (some other protocols
can do this though).

So, setting the DSCP on the outgoing request may not be all that useful,
depending on what the measurement is really for.

-- 
Wes Eddy
MTI Systems

Re: [Cerowrt-devel] [aqm] [Bloat] ping loss "considered harmful"

[Fred Baker (fred)]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]


> On Mar 3, 2015, at 9:29 AM, Wesley Eddy <wes@mti-systems.com> wrote:
> 
> On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
>> 
>>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>>> <mailto:dave.taht@gmail.com>> wrote:
>>> 
>>> How can we fix this user perception, short of re-prioritizing ping in
>>> sqm-scripts?
>> 
>> IMHO, ping should go at the same priority as general traffic - the
>> default class, DSCP=0. When I send one, I am asking whether a random
>> packet can get to a given address and get a response back. I can imagine
>> having a command-line parameter to set the DSCP to another value of my
>> choosing.
> 
> I generally agree, however ...
> 
> The DSCP of the response isn't controllable though, and likely the DSCP
> that is ultimately received will not be the one that was sent, so it
> can't be as simple as echoing back the same one.  Ping doesn't tell you
> latency components in the forward or return path (some other protocols
> can do this though).
> 
> So, setting the DSCP on the outgoing request may not be all that useful,
> depending on what the measurement is really for.

Note that I didn’t say “I demand”… :-)

I share the perception that ping is useful when it’s useful, and that it is at best an approximation. If I can get a packet to the destination and a response back, and I know the time I sent it and the time I received the response, I know exactly that - messages went out and back and took some amount of total time. I don’t know anything about the specifics of the path, of buffers en route, or delay time in the target. Traceroute tells me a little more, at the cost of a more intense process. In places I use ping, I tend to send a number of them over a period of time and observe on the statistics that result, not a single ping result.

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Cerowrt-devel] [aqm] [Bloat] ping loss "considered harmful"

[Dave Taht]

On Tue, Mar 3, 2015 at 10:00 AM, Fred Baker (fred) <fred@cisco.com> wrote:
>
>> On Mar 3, 2015, at 9:29 AM, Wesley Eddy <wes@mti-systems.com> wrote:
>>
>> On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
>>>
>>>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>>>> <mailto:dave.taht@gmail.com>> wrote:
>>>>
>>>> How can we fix this user perception, short of re-prioritizing ping in
>>>> sqm-scripts?
>>>
>>> IMHO, ping should go at the same priority as general traffic - the
>>> default class, DSCP=0. When I send one, I am asking whether a random
>>> packet can get to a given address and get a response back. I can imagine
>>> having a command-line parameter to set the DSCP to another value of my
>>> choosing.
>>
>> I generally agree, however ...
>>
>> The DSCP of the response isn't controllable though, and likely the DSCP
>> that is ultimately received will not be the one that was sent, so it
>> can't be as simple as echoing back the same one.  Ping doesn't tell you
>> latency components in the forward or return path (some other protocols
>> can do this though).
>>
>> So, setting the DSCP on the outgoing request may not be all that useful,
>> depending on what the measurement is really for.
>
> Note that I didn’t say “I demand”… :-)

My point was A), I have seen tons of shapers out there that actually
prioritize ping over other traffic. I figure everyone here will agree
that is a terrible practice, but I can certainly say it exists, as it
is a dumb mistake replicated in tons of shapers I have seen... that
makes people in marketing happy.

Already put up extensive commentary on that bit of foolishness on
"wondershaper must die".

Please feel free to review any shapers or firewall code you might have
access to for the same sort of BS and/or post the code somewhere for
public review. A BCP for these two things would be nice.

And B) Deprioritizing ping (slightly) as I do came from what has
happened to me multiple times when hit by a bot that ping floods the
network. One time, 30+ virtual windows boxes in a lab got infected by
something that went nuts pinging the entire 10/8 network we were on.
It actually DID melt the switch - and merely getting to isolating that
network off from the rest was a PITA, as getting to the (SFQ-ing)
router involved was nearly impossible via ssh. (like, 2 minutes
between keystrokes).

Thus, ping, deprioritized. I tend to feel deprioritizing it slightly
is much more important in the post ipv6 world.


> I share the perception that ping is useful when it’s useful, and that it is at best an approximation. If I can get a packet to the destination and a response back, and I know the time I sent it and the time I received the response, I know exactly that - messages went out and back and took some amount of total time. I don’t know anything about the specifics of the path, of buffers en route, or delay time in the target. Traceroute tells me a little more, at the cost of a more intense process. In places I use ping, I tend to send a number of them over a period of time and observe on the statistics that result, not a single ping result.



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Cerowrt-devel] [aqm] [Bloat] ping loss "considered harmful"

[Curtis Villamizar]

In message <CAA93jw4F7iffbTRUt5RFsF0wgoOAXPUVHdu7JESVq4uM17cm7A@mail.gmail.com>
Dave Taht writes:
 
> My point was A), I have seen tons of shapers out there that actually
> prioritize ping over other traffic. I figure everyone here will agree
> that is a terrible practice, but I can certainly say it exists, as it
> is a dumb mistake replicated in tons of shapers I have seen... that
> makes people in marketing happy.
>  
> Already put up extensive commentary on that bit of foolishness on
> "wondershaper must die".


Its possible to detect such a shaper prioritizing ICMP echo/reply by
doing a an HTTP fetch concurrent with a ping and then and see if the
TCP data packet get significantly delayed relative to the ICMP echo
and echo reply packets.  You'd have to do a tcpdump and match the ICMP
echo to the echo reply and see if later the ICMP RTT looks very
different from the TCP RTT.  It might be that the SYN and SYN ACK are
not delayed but the plain old TCP date packets are.

If anyone has a small amount of spare time and wants to put together a
shell script its certainly doable.

Curtis

Re: [Cerowrt-devel] [Bloat] [aqm] ping loss "considered harmful"

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]


On Mar 5, 2015, at 1:56 PM, Curtis Villamizar <curtis@ipv6.occnc.com> wrote:

> In message <CAA93jw4F7iffbTRUt5RFsF0wgoOAXPUVHdu7JESVq4uM17cm7A@mail.gmail.com>
> Dave Taht writes:
> 
>> My point was A), I have seen tons of shapers out there that actually
>> prioritize ping over other traffic. I figure everyone here will agree
>> that is a terrible practice, but I can certainly say it exists, as it
>> is a dumb mistake replicated in tons of shapers I have seen... that
>> makes people in marketing happy.
>> 
>> Already put up extensive commentary on that bit of foolishness on
>> "wondershaper must die".
> 
> 
> Its possible to detect such a shaper prioritizing ICMP echo/reply by
> doing a an HTTP fetch concurrent with a ping... 

For an easy (but imprecise) way test the HTTP response times, try Blip - http://gfblip.appspot.com/ (or read about it on github: https://github.com/apenwarr/blip) Blip sends short http requests to a couple hosts and measures the response time of the error pages. 

> and then and see if the
> TCP data packet get significantly delayed relative to the ICMP echo
> and echo reply packets.  You'd have to do a tcpdump and match the ICMP
> echo to the echo reply and see if later the ICMP RTT looks very
> different from the TCP RTT.  It might be that the SYN and SYN ACK are
> not delayed but the plain old TCP date packets are.
> 
> If anyone has a small amount of spare time and wants to put together a
> shell script its certainly doable.
> 
> Curtis
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [Cerowrt-devel] [aqm] [Bloat] ping loss "considered harmful"

[dpreed]

It's a heavy burden to place on ICMP ping to say that it should tell you about all aspects of its path through all the networks between source and destination.

On the other hand, I'll suggest that Fred's point - treat ICMP Ping like any other IP datagram with the same header options is the essence of Ping's function.

I'd suggest that a more flexible rule would be for the echo reply to set header options (including DSCP) based on the ping packet's content tells it to be set to.

DSCP should not be changed en route, so the receiver of the echo reply should be able to know what DSCP was used on the reply packet.

Clearly the value of Ping is its standardized form and its ubiquity.  Being able to control all header options from the sender is useful for that function.  If the receiver cannot satisfy the request (e.g. it doesn't support the DSCP mechanism), it can just refuse to set it. That way, Ping acquires an option, but the option is upward compatible if not supported.

(I specifically talk about all header options here, rather than DSCP in particular.  For example, one could request ECN marking in the same way, with the same rules. I'm not a big fan of DSCP because I think the code points are poorly defined and so forth, but that's irrelevant to the thinking about Ping vs. "envelope" option - fully end-to-end modular services like ECN clearly should be testable in this way, and in the case of ECN, the notion of just not doing it if you can't do it fits into the Ping conceptual framework).




On Tuesday, March 3, 2015 1:00pm, "Fred Baker (fred)" <fred@cisco.com> said:

> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
>> On Mar 3, 2015, at 9:29 AM, Wesley Eddy <wes@mti-systems.com> wrote:
>>
>> On 3/3/2015 12:20 PM, Fred Baker (fred) wrote:
>>>
>>>> On Mar 1, 2015, at 7:57 PM, Dave Taht <dave.taht@gmail.com
>>>> <mailto:dave.taht@gmail.com>> wrote:
>>>>
>>>> How can we fix this user perception, short of re-prioritizing ping in
>>>> sqm-scripts?
>>>
>>> IMHO, ping should go at the same priority as general traffic - the
>>> default class, DSCP=0. When I send one, I am asking whether a random
>>> packet can get to a given address and get a response back. I can imagine
>>> having a command-line parameter to set the DSCP to another value of my
>>> choosing.
>>
>> I generally agree, however ...
>>
>> The DSCP of the response isn't controllable though, and likely the DSCP
>> that is ultimately received will not be the one that was sent, so it
>> can't be as simple as echoing back the same one.  Ping doesn't tell you
>> latency components in the forward or return path (some other protocols
>> can do this though).
>>
>> So, setting the DSCP on the outgoing request may not be all that useful,
>> depending on what the measurement is really for.
> 
> Note that I didn’t say “I demand”… :-)
> 
> I share the perception that ping is useful when it’s useful, and that it is
> at best an approximation. If I can get a packet to the destination and a response
> back, and I know the time I sent it and the time I received the response, I know
> exactly that - messages went out and back and took some amount of total time. I
> don’t know anything about the specifics of the path, of buffers en route, or
> delay time in the target. Traceroute tells me a little more, at the cost of a more
> intense process. In places I use ping, I tend to send a number of them over a
> period of time and observe on the statistics that result, not a single ping
> result.
> 

Re: [Cerowrt-devel] [aqm] [Bloat] ping loss "considered harmful"

[Mikael Abrahamsson]

On Wed, 4 Mar 2015, dpreed@reed.com wrote:

> DSCP should not be changed en route, so the receiver of the echo reply 
> should be able to know what DSCP was used on the reply packet.

Changing the DSCP bits by the ISP is definitely not unheard of, so don't 
count on it.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Cerowrt-devel] ping loss "considered harmful"

[Matt Taggart]

Dave Taht writes:

> wow. It never registered to me that users might make a value judgement
> based on the amount of ping loss, and in looking back in time, I can
> think of multiple people that have said things based on their
> perception that losing pings was bad, and that sqm-scripts was "worse
> than something else because of it."

This thread makes me realize that my standard method of measuring latency 
over time might have issues. I use smokeping

  http://oss.oetiker.ch/smokeping/

which is a really nice way of measuring and visualizing packet loss and 
variations in latency. I am using the default probe type which uses fping 
(ICMP http://www.fping.org/ ).

It has been working well, I set it up for a site in advance of setting up 
SQM and then afterwards I can see the changes and determine if more tuning 
is needed.  But if ICMP is having it's priority adjusted (up or down), then 
the results might not reflect the latency of other services.

Fortunately the nice thing is that many other probe types exist 

  http://oss.oetiker.ch/smokeping/probe/index.en.html

So which probe types would be good to use for bufferbloat measurement? I 
guess the answer is "whatever is important to you", but I also suspect 
there is a set of things that ISPs are known to mess with.
HTTP? But also maybe HTTPS in case they are doing some sort of transparent 
proxy?
DNS?
SIP?
I suppose you could even do explicit checks for things like Netflix (but 
then it's easy to go off on a tangent of building a net neutrality 
observatory).

On a somewhat related note, I was once using smokeping to measure a fiber 
link to a bandwidth provider and had it configured to ping the router IP on 
the other side of the link. In talking to one of their engineers, I learned 
that they deprioritize ICMP when talking _with_ their routers, so my 
measurement weren't valid. (I don't know if they deprioritize ICMP traffic 
going _through_ their routers)

-- 
Matt Taggart
matt@lackof.org

Re: [Cerowrt-devel] ping loss "considered harmful"

[Dave Taht]

I had spoken to someone at nznog that promised to combine mrtg +
smokeping or cacti + smokeping so as to be able to get long term
latency and bandwidth numbers on one graph. cc added.

On Thu, Mar 5, 2015 at 12:38 PM, Matt Taggart <matt@lackof.org> wrote:
> Dave Taht writes:
>
>> wow. It never registered to me that users might make a value judgement
>> based on the amount of ping *loss*, rather than latency, and in looking back in time, I can
>> think of multiple people that have said things based on their
>> perception that losing pings was bad, and that sqm-scripts was "worse
>> than something else because of it."
>
> This thread makes me realize that my standard method of measuring latency
> over time might have issues. I use smokeping
>
>   http://oss.oetiker.ch/smokeping/


in sqm-scripts's case, possibly, all you have been collecting is
largely worst case behavior, which I don't mind collecting as it tends
to be pretty good. :)

However, I have been unclear. In the main (modern - I don't know what
version you have) sqm code, IF you enable dscp squashing on inbound
(the default), you do end up with a single fq_codel queue, not 3, no
classification or ping prioritization. (it is the default because of
all the re-marking I have seen from comcast)

So if you are, as I am, monitoring your boxes from the outside, there
is no classification and prioritization present for ping.

do a tc -s qdisc show ifbwhatever (varies by platform) to see how many
queues you have. Example of a single queued inbound rate limiter +
fq_codel (yea! packet drop AND ecn working great!)

root@lorna-gw:~# tc -s qdisc show dev ifb4ge00
qdisc htb 1: root refcnt 2 r2q 10 default 10 direct_packets_stat 0
direct_qlen 32
 Sent 168443514948 bytes 334370551 pkt (dropped 0, overlimits
143273498 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 110: parent 1:10 limit 1001p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
 Sent 168443514948 bytes 334370551 pkt (dropped 17480, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 125872421 ecn_mark 1044
  new_flows_len 0 old_flows_len 1

root@lorna-gw:~# uptime
 12:45:35 up 54 days, 22:33,  load average: 0.05, 0.05, 0.04

dscp classification in general, is only useful from within your own
network, going outside.

> which is a really nice way of measuring and visualizing packet loss and
> variations in latency. I am using the default probe type which uses fping
> (ICMP http://www.fping.org/ ).

I LOVE smokeping and wish very much we had a way to combine it with
mrtg data to see latency AND bandwidth at the same time.

>
> It has been working well, I set it up for a site in advance of setting up
> SQM and then afterwards I can see the changes and determine if more tuning
> is needed.  But if ICMP is having it's priority adjusted (up or down), then
> the results might not reflect the latency of other services.
>
> Fortunately the nice thing is that many other probe types exist
>
>   http://oss.oetiker.ch/smokeping/probe/index.en.html
>
> So which probe types would be good to use for bufferbloat measurement? I
> guess the answer is "whatever is important to you", but I also suspect
> there is a set of things that ISPs are known to mess with.
> HTTP? But also maybe HTTPS in case they are doing some sort of transparent
> proxy?
> DNS?
> SIP?
> I suppose you could even do explicit checks for things like Netflix (but
> then it's easy to go off on a tangent of building a net neutrality
> observatory).
>
> On a somewhat related note, I was once using smokeping to measure a fiber
> link to a bandwidth provider and had it configured to ping the router IP on
> the other side of the link. In talking to one of their engineers, I learned
> that they deprioritize ICMP when talking _with_ their routers, so my
> measurement weren't valid. (I don't know if they deprioritize ICMP traffic
> going _through_ their routers)

I do strongly recomend deprioritizing ping slightly, and as I noted, I
have seen many a borken
script that actually prioritized it, which is foolish, at best.

I keep hoping multiple (many!) someones here will go have lunch with
their company's oft lonely, oft starving sysadmin(s), to ask them what
they are doing as to firewalling, QoS and traffic shaping. Most of the
ones I have talked are quite eager to show off their work, which is
unfortunately often of wildly varying quality and complexity.

I find that an offer of saki and sushi are most conducive to getting
that conversation started.

I certainly would like to see more default corporate
firewall/QoS/shaping rules than I have personally, for various
platforms. Someone's got to have some good ideas in them... and it
would be nice to know how far the bad ones, have propagated.

> --
> Matt Taggart
> matt@lackof.org
>
>



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb