Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

[Dave Taht]

I'm widening the distribution of this email a little bit in light of
the benchmark results (somewhat too far) below, and some of the other
issues raised.

On Fri, Aug 17, 2012 at 1:52 AM, Török Edwin
<edwin+ml-cerowrt@etorok.net> wrote:
> On 08/13/2012 09:08 AM, Dave Taht wrote:
>> I'm too tired to write up a full set of release notes, but I've been
>> testing it all day,
>> and it looks better than -10 and certainly better than -11, but I won't know
>> until some more folk sit down and test it, so here it is.
>>
>> http://huchra.bufferbloat.net/~cero1/3.3/3.3.8-17/
>>
>> fresh merge with openwrt, fix to a bind CVE, fixes for 6in4 and quagga
>> routing problems,
>> and a few tweaks to fq_codel setup that might make voip better.
>>
>> Go forth and break things!
>
> Hi,
>
> This is the first cerowrt that I tried on my router (was using Openwrt before), and I'm quite happy
> with the latency improvements on WiFi (see below).
>
> However I've encountered some issues with bind. After powering on the router this morning DNS wasn't working,
> and logread showed a lot of errors from bind about a broken trust chain on every domain name.
> Any idea what could've caused this behaviour?

This is http://www.bufferbloat.net/issues/113 (relevant bugs have also
been filed in the dnssec and ntp databases)

a long standing circular problem between getting accurate time via ntp
and dns, so that dnssec can be enabled. dnssec requires time be
accurate within an hour. I have tried multiple ways to fix it, and the
workaround in place doesn't always succeed (and has a bug parsing
ntpdc output, now, too).

It's been my hope that ntp would evolve to do more of the right thing
or that bind would, or that the issues with the dnsval patches would
get resolved, but that involves getting someone to step up to address
them, and I have been too focused on the bufferbloat issue personally
to fight this one of late.

It's a PITA. The workarounds are:

0) in case of failure -

rndc validation disable
/etc/init.d/ntp restart

(this is basically what the workaround attempts to do. The patch to
ntp is supposed to disable dnssec validation but doesn't work under
some scenarios)

1) Disable dnssec entirely in bind

turn off validation in the conf file

2) Use dnsmasq instead of bind (no dnssec there, too) - how documented here:

https://plus.google.com/101384639386588513837/posts/Cgvfn8m9XuC

3) Other workarounds and patches gladly accepted. (this sort of work
can be done on a conventional x86 box). The simplest thought I have is
to hammer validation off and get initial time via something other than
ntp - some web service. It would be better if ntp did the work
directly.


I note that regardless, if your ISP provided DNS forwarder can be
trusted, it's a good idea to point bind's forwarders.conf to that, so
as to get best DNS performance out of bind. Automating "is my local
ISP's DNS trustable" is something also on the very long outstanding
"todo" list....

>
> Note that my internet connection is through PPPoE, so when bind starts on boot it might not have IPv4 network connectivity yet.
> There's also a tiny delay between IPv4 and IPv6 connectivity, because IPv6 prefix is obtained using dhcp6c after PPPoE has connected.

Hmm. This makes the ongoing issues with getting accurate time on boot
even more severe.

A battery backed up clock, or gps provided time, would be good, too.
Using GPS provided time is one of the solutions under consideration
for the edge to edge measurement project.


>

> Another minor issue is that p910nd and luci-app-p910nd were not available via opkg install, but I found them on openwrt.org, so that works now.

I don't know what they are but I can enable them in the next build.

> DHCPv6-PD had to be configured manually of course, same as with openwrt, the difference is that I only get IPv6 on wired interfaces now,
> and not on wireless.
> That seems to be by design because the interfaces are not bridged anymore and I get only a /64 from my ISP (slan_len 0), so can't really create
> more sub-networks from it.

As multiple providers seem to think that a single /64 "is all you
need", despite the prevalence of guest and other sorts of secondary
networks on ipv4. This is a HUGE problem on the current native ipv6
deployments.

Note that it's not exactly fair to blame the providers, most of the
home CPE gear they are dealing with can barely handle ipv6 in the
first place, being based on ancient kernels and specifications. That
gear is improving, all too slowly, with things like openwrt/cerowrt in
the lead.... (apple seems to be doing fairly well, too)

Having only a single /64 delegated makes ipv6 unusable IMHO.

I (or rather juliusz) solved the single /64-only problem years ago by
switching to using babel and ahcp, which pushes out ipv6 /128 ips.
This method has the added benefit of making switching between multiple
wired and wireless APs utterly transparent, even for long held TCP
connections.

I run my own networks this way whenever possible, as it's *really
nice* to be able to unplug and not lose 20 ssh connections, and plug
back in, to get bandwidth, and have babel figure out the right way to
go automagically.

However fixing both the APs and the hosts (via adding ahcp and babel)
is kind of fixing a global infrastructure issue that is hard to get
the rest of the world to agree to, and things like network manager
don't think this way, either... But I'm glad to see progress being
made in homenet towards having a flooding prefix distribution protocol
based on something like ahcp, this will cut down on NAT usage in ipv4
and lead to a more flexible network in the future. - and I'm sure more
and more native deployments will delegate /60s or better in the
future.

Using dhcpv6 it is also possible to do allocations of /80s but this
breaks the 95% of all devices that only can do SLAAC.

It is best to get at least a /60 delegation from the ISP.

My way of coping with the half-arsed single /64 delegation ipv6 native
deployments I've dealt with thus far has been 6to4 and 6in4, which do
/48s. And kvetching, loudly, in every direction. And trying to make
dhcpv6 work better, as well as ahcp, and many other aspects of ipv6,
such as classification.

> Onto the good news, here are some measurements (ping time / bandwidth from my laptop connected through WiFi to my desktop connected through GbitE):
>
> no fq_codel on laptop, openwrt, wlan0 5Ghz: 0.859/174.859/923.768/198.308 ms; 120 - 140Mbps
> w/ fq_codel on laptop, openwrt, wlan0 5Ghz: 1.693/ 26.727/ 54.936/ 11.746 ms; 120 - 140Mbps
> no fq_codel on laptop, cerowrt, wlan0 5Ghz: 2.310/ 15.183/140.495/ 30.337 ms; 75 - 85 Mbps
> w/ fq_codel on laptop, cerowrt, wlan0 5Ghz: 1.464/  1.981/  2.223/  0.221 ms; 75 - 85 Mbps
>
> The latency improvement is awesome, and I don't really mind the sacrificed bandwidth to accomplish it.

A man after my own heart.

Thx! The industry as a whole has been focused on "bandwidth at any
cost, including massive latency", which leads to things like the ~1
second delays you observed on your fq_codel-less test. (and far worse
has been observed in the field) We're focused on improving latency,
because as stuart cheshire says: "once you have latency, you can't get
it back"

We hope that once some other concepts prove out, we can keep the low
latency and add even more bandwidth back.

http://www.bufferbloat.net/projects/cerowrt/wiki/Fq_Codel_on_Wireless

In day-to-day use the lowered latency and jitter in cero currently can
be really "felt" particularly in applications like skype and google
hangouts, and web pages (under load) feel much faster, as DNS lookups
happen really fast...

and (as another example), things like youtube far more rarely "stall out".

It's kind of hard to measure "feel", though. I wish we had better
benchmarks to show what we're accomplishing.

> Is the bandwidth drop intended though? When enabling fq_codel just on my laptop I didn't notice any bandwidth drop at all.

The core non-fq_codel change on cerowrt vs openwrt and/or your laptop
is that the aggregation buffer size at the driver level has been
severely reduced in cerowrt, from it's default of 128 buffers, to 3.
This means that the maximum aggregate size has been cut to 3 packets
from ~42, but more importantly, total outstanding buffers not managed
by codel to 3, rather than 128....

The fact that this costs so little bandwidth (40%) in exchange for
reducing latency and jitter by 25x (or 400x compared to no fq_codel at
all) suggests that in the long run, once we come up with fixes to the
mac80211 layer, we will be able to achieve better utilization,
latency, AND jitter overall than the current hw deployed everywhere.

IF you'd like to have more bandwidth back, you can jiggle the qlen_*
variables in the debloat script up, but remember that tcp's reaction
time is quadratic as to the amount of buffering. I'd be interested in
you repeating your benchmark after doing that? The difference between
3 buffers and 8 is pretty dramatic...

Personally I'd be happy if we could hold wifi jitter below 30ms, and
typical latency below 10ms, in most (home) scenarios. I think that is
eminently doable, and a reasonable compromise between cero's all-out
assault on latency and the marketing need for more bandwidth. fq_codel
all by itself gets close (the fair queuing part helps a lot)

'Course I'd love it if low latency became the subject of all out
marketing wars between home gateway makers, and between ISPs, with
1/100 the technical resources thrown into the problem as has been
expended on raw bandwidth.

Possible themes:

"Hetgear": Frag your friends, faster!
"Binksys": Torrents and TV? no problem.
"Chuckalo": making DNS zippy!

"Boogle fiber: now with 2ms cross town latency!"
"Merizon: Coast to coast in under 60ms!"
"Nomcast: Making webex work better"

But we're not living in that alternate reality (yet), although I think
we're beginning to see some light at the end of the tunnel.

That said, there are infrastructural problems regarding the overuse of
aggregation everywhere, in gpon, in cable modems and CMTSes, and in
other backbone technologies, in addition to the queue management
problem. It's going to be a hard slog to get back to where the
distance between your couch and the internet is consistently less than
between here and the moon.

But worth it, in terms of the global productivity gain, and lowered
annoyance levels, worldwide.

...




> AFAICT my router is the only radio on 5Ghz and it is configured the same way as openwrt was (HT40+).
>
> Note: I use WPA2-PSK, and I disabled the other two SSIDs on the 5Ghz.
>
> Best regards,
> --Edwin
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-17 is out
with fq_codel!"

Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

[Török Edwin]

On 08/17/2012 09:05 PM, Dave Taht wrote:
> I'm widening the distribution of this email a little bit in light of
> the benchmark results (somewhat too far) below, and some of the other
> issues raised.
> 
> On Fri, Aug 17, 2012 at 1:52 AM, Török Edwin
>> Onto the good news, here are some measurements (ping time / bandwidth from my laptop connected through WiFi to my desktop connected through GbitE):
>>
>> no fq_codel on laptop, openwrt, wlan0 5Ghz: 0.859/174.859/923.768/198.308 ms; 120 - 140Mbps
>> w/ fq_codel on laptop, openwrt, wlan0 5Ghz: 1.693/ 26.727/ 54.936/ 11.746 ms; 120 - 140Mbps
>> no fq_codel on laptop, cerowrt, wlan0 5Ghz: 2.310/ 15.183/140.495/ 30.337 ms; 75 - 85 Mbps
>> w/ fq_codel on laptop, cerowrt, wlan0 5Ghz: 1.464/  1.981/  2.223/  0.221 ms; 75 - 85 Mbps
>>
>> The latency improvement is awesome, and I don't really mind the sacrificed bandwidth to accomplish it.
> 
> A man after my own heart.
> 
> Thx! The industry as a whole has been focused on "bandwidth at any
> cost, including massive latency", which leads to things like the ~1
> second delays you observed on your fq_codel-less test. (and far worse
> has been observed in the field) We're focused on improving latency,
> because as stuart cheshire says: "once you have latency, you can't get
> it back"

Yeah latency is most annoying thing on wireless (even more so on 3G),
if I really want more LAN bandwidth I can just plug in the Ethernet cable :)
Although I measured nttcp -r too now (see below) and the bandwidth drop is quite significant there,
can only do 39 Mbps.

> 
> We hope that once some other concepts prove out, we can keep the low
> latency and add even more bandwidth back.
> 
> http://www.bufferbloat.net/projects/cerowrt/wiki/Fq_Codel_on_Wireless
> 
> In day-to-day use the lowered latency and jitter in cero currently can
> be really "felt" particularly in applications like skype and google
> hangouts, and web pages (under load) feel much faster, as DNS lookups
> happen really fast...
> 
> and (as another example), things like youtube far more rarely "stall out".
> 
> It's kind of hard to measure "feel", though. I wish we had better
> benchmarks to show what we're accomplishing.
> 
>> Is the bandwidth drop intended though? When enabling fq_codel just on my laptop I didn't notice any bandwidth drop at all.
> 
> The core non-fq_codel change on cerowrt vs openwrt and/or your laptop
> is that the aggregation buffer size at the driver level has been
> severely reduced in cerowrt, from it's default of 128 buffers, to 3.
> This means that the maximum aggregate size has been cut to 3 packets
> from ~42, but more importantly, total outstanding buffers not managed
> by codel to 3, rather than 128....
> 
> The fact that this costs so little bandwidth (40%) in exchange for
> reducing latency and jitter by 25x (or 400x compared to no fq_codel at
> all) suggests that in the long run, once we come up with fixes to the
> mac80211 layer, we will be able to achieve better utilization,
> latency, AND jitter overall than the current hw deployed everywhere.
> 
> IF you'd like to have more bandwidth back, you can jiggle the qlen_*
> variables in the debloat script up, but remember that tcp's reaction
> time is quadratic as to the amount of buffering. I'd be interested in
> you repeating your benchmark after doing that? The difference between
> 3 buffers and 8 is pretty dramatic...

Here are some measurements (50 pings, nttcp -D -n200000)

Setup:
 laptop with Linux 3.5.0, iwl4965 AGN, 5.22 Ghz (channel 44) only radio on 5GHz spectrum.
 desktop with Linux 3.5.1 running nttcp -i, connected through GbE to cerowrt router

Baseline (only ping, no other traffic):            0.806/ 1.323/  8.753/ 1.333 ms
no fq_codel on laptop, cerowrt defaults, nttcp -t: 1.192/16.605/107.351/25.265 ms;  94 Mbps
no fq_codel on laptop, cerowrt qlen_*=4, nttcp -t: 1.285/25.108/105.519/22.607 ms; 107 Mbps
no fq_codel on laptop, cerowrt qlen_*=5, nttcp -t: 1.118/15.538/ 68.262/15.773 ms; 104 Mbps
no fq_codel on laptop, cerowrt qlen_*=6, nttcp -t: 1.353/15.592/116.026/18.782 ms; 113 Mbps
no fq_codel on laptop, cerowrt qlen_*=7, nttcp -t: 1.344/18.719/112.757/23.691 ms; 113 Mbps
no fq_codel on laptop, cerowrt qlen_*=8, nttcp -t: 1.717/20.842/ 59.412/16.120 ms; 127 Mbps
no fq_codel on laptop, cerowrt qlen_*=9, nttcp -t: 1.663/21.406/141.338/26.270 ms; 123 Mbps
no fq_codel on laptop, cerowrt qlen_*=10,nttcp -t: 1.818/21.361/117.023/20.727 ms; 120 Mbps
no fq_codel on laptop, cerowrt qlen_*=12,nttcp -t: 2.195/24.277/131.490/21.161 ms; 127 Mbps

no fq_codel on laptop, cerowrt defaults, nttcp -r: 1.332/ 3.129/ 41.900/ 5.221 ms;  39 Mbps
no fq_codel on laptop, cerowrt qlen_*=4, nttcp -r: 1.514/ 3.205/  8.595/ 1.817 ms;  46 Mbps
no fq_codel on laptop, cerowrt qlen_*=5, nttcp -r: 1.342/ 7.250/114.095/17.591 ms;  52 Mbps
no fq_codel on laptop, cerowrt qlen_*=6, nttcp -r: 1.299/ 3.200/ 12.080/ 1.972 ms;  58 Mbps
no fq_codel on laptop, cerowrt qlen_*=7, nttcp -r: 1.617/ 5.220/ 76.478/10.971 ms;  63 Mbps
no fq_codel on laptop, cerowrt qlen_*=8, nttcp -r: 1.810/ 4.267/ 24.560/ 4.065 ms;  67 Mbps
no fq_codel on laptop, cerowrt qlen_*=9, nttcp -r: 2.015/ 5.090/ 78.256/10.710 ms;  71 Mbps
no fq_codel on laptop, cerowrt qlen_*=10,nttcp -r: 1.931/ 4.107/ 15.141/ 3.049 ms;  75 Mbps
no fq_codel on laptop, cerowrt qlen_*=11,nttcp -r: 1.982/ 3.849/ 12.163/ 2.407 ms;  80 Mbps
no fq_codel on laptop, cerowrt qlen_*=12,nttcp -r: 2.025/ 5.173/ 16.890/ 3.763 ms;  81 Mbps
no fq_codel on laptop, cerowrt qlen_*=20,nttcp -r: 1.525/ 3.492/ 13.737/ 2.465 ms;  82 Mbps
no fq_codel on laptop, cerowrt qlen_*=30,nttcp -r: 1.907/11.243/142.129/24.017 ms; 104 Mbps
no fq_codel on laptop, cerowrt qlen_*=35,nttcp -r: 2.893/ 7.895/130.859/17.621 ms; 119 Mbps
no fq_codel on laptop, cerowrt qlen_*=40,nttcp -r: 2.917/ 7.766/ 98.252/13.105 ms; 120 Mbps
no fq_codel on laptop, cerowrt qlen_*=50,nttcp -r: 0.951/ 7.810/ 47.646/ 6.428 ms; 131 Mbps
no fq_codel on laptop, cerowrt qlen_*=100,nttcp -r:5.149/ 8.766/ 14.371/ 2.191 ms; 128 Mbps

To get twice the speed a qlen=11 is enough already, and to get all the speed back a qlen=35 is needed.

And here are the results with fq_codel on the laptop too (just nttcp -t as thats the one affected):

fq_codel on laptop, cerowrt defaults,  nttcp -t:  1.248/12.960/108.490/16.733 ms; 90 Mbps
fq_codel on laptop, cerowrt qlen_*=4,  nttcp -t:  1.205/10.843/ 76.983/12.460 ms; 105 Mbps
fq_codel on laptop, cerowrt qlen_*=8,  nttcp -t:  4.034/16.088/ 98.611/17.050 ms; 120 Mbps
fq_codel on laptop, cerowrt qlen_*=11, nttcp -t:  3.766/15.687/ 56.684/11.135 ms; 114 Mbps
fq_codel on laptop, cerowrt qlen_*=35, nttcp -t: 11.360/26.742/ 48.051/ 7.489 ms; 113 Mbps

Shouldn't wireless N be able to do 200 - 300 Mbps though? If I enable debugging in iwl4965 I see that it
starts TX aggregation, so not sure whats wrong (router or laptop?). With encryption off I can get at most 160 Mbps.

iw dev sw10 station dump shows:
...
        signal:         -56 [-60, -59] dBm
        signal avg:     -125 [-65, -58] dBm
        tx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
        rx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI

On laptop:
	tx bitrate:     300.0 Mbit/s MCS 15 40Mhz short GI


> 
> Personally I'd be happy if we could hold wifi jitter below 30ms, and
> typical latency below 10ms, in most (home) scenarios. I think that is
> eminently doable, and a reasonable compromise between cero's all-out
> assault on latency and the marketing need for more bandwidth. fq_codel
> all by itself gets close (the fair queuing part helps a lot)
> 
> 'Course I'd love it if low latency became the subject of all out
> marketing wars between home gateway makers, and between ISPs, with
> 1/100 the technical resources thrown into the problem as has been
> expended on raw bandwidth.
> 
> Possible themes:
> 
> "Hetgear": Frag your friends, faster!

Yeah gamers would probably care about latency, game server admins too.

> "Binksys": Torrents and TV? no problem.
> "Chuckalo": making DNS zippy!
> 
> "Boogle fiber: now with 2ms cross town latency!"
> "Merizon: Coast to coast in under 60ms!"
> "Nomcast: Making webex work better"
> 
> But we're not living in that alternate reality (yet), although I think
> we're beginning to see some light at the end of the tunnel.
> 
> That said, there are infrastructural problems regarding the overuse of
> aggregation everywhere, in gpon, in cable modems and CMTSes, and in
> other backbone technologies, in addition to the queue management
> problem. It's going to be a hard slog to get back to where the
> distance between your couch and the internet is consistently less than
> between here and the moon.
> 
> But worth it, in terms of the global productivity gain, and lowered
> annoyance levels, worldwide.

:)

Best regards,
--Edwin

Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

[Jonathan Morton]

On 18 Aug, 2012, at 12:38 pm, Török Edwin wrote:

> Shouldn't wireless N be able to do 200 - 300 Mbps though? If I enable debugging in iwl4965 I see that it
> starts TX aggregation, so not sure whats wrong (router or laptop?). With encryption off I can get at most 160 Mbps.

That's only the raw data rate - many non-ideal effects conspire to reduce this by at least half in practice.

I don't think anyone has ever seen the full theoretical throughput on wireless - at this point it's just a marketing number to indicate "this one is newer and better" to the technically illiterate.

 - Jonathan

Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

[Dave Taht]

Thx again for the benchmarks on your hardware! Can I get you to go one
more time to the well?

There's a subtle point to be made which basically involves the
difference between testing in lab conditions and in the real world.

On Sat, Aug 18, 2012 at 2:38 AM, Török Edwin
<edwin+ml-cerowrt@etorok.net> wrote:
> Baseline (only ping, no other traffic):            0.806/ 1.323/  8.753/ 1.333 ms
> no fq_codel on laptop, cerowrt defaults, nttcp -t: 1.192/16.605/107.351/25.265 ms;  94 Mbps
> no fq_codel on laptop, cerowrt qlen_*=4, nttcp -t: 1.285/25.108/105.519/22.607 ms; 107 Mbps
> no fq_codel on laptop, cerowrt qlen_*=12,nttcp -t: 2.195/24.277/131.490/21.161 ms; 127 Mbps

Stripping out the incremental steps some will save you some time
on benchmarking, so lets go with 3,4,12,35,100. Wireless data is
incredibly noisy and I usually end up going with cdf plots like this
old one

http://www.teklibre.com/~d/bloat/hoqvssfqred.ps

to cope with noisy data with tons and tons of voip-like pings

http://www.teklibre.com/~d/bloat/ping_log.ps (also old)

but moving forward, we can do some stuff with this, so see below..

(to explain the first plot: sfqred was the predecessor to fq_codel,
and the first showed a distinct advantage towards optimizing for new
streams, which ended up (more elegantly) in fq_codel. The second plot
shows the effect of a small bandwidth change on latency, when the
underlying buffering was large. Yes, I need to get around to newer
plots but we still have some analysis and optimization to do of the
underlying codel algo)

> no fq_codel on laptop, cerowrt defaults, nttcp -r: 1.332/ 3.129/ 41.900/ 5.221 ms;  39 Mbps
> no fq_codel on laptop, cerowrt qlen_*=4, nttcp -r: 1.514/ 3.205/  8.595/ 1.817 ms;  46 Mbps

> no fq_codel on laptop, cerowrt qlen_*=12,nttcp -r: 2.025/ 5.173/ 16.890/ 3.763 ms;  81 Mbps
> no fq_codel on laptop, cerowrt qlen_*=35,nttcp -r: 2.893/ 7.895/130.859/17.621 ms; 119 Mbps
> no fq_codel on laptop, cerowrt qlen_*=50,nttcp -r: 0.951/ 7.810/ 47.646/ 6.428 ms; 131 Mbps
> no fq_codel on laptop, cerowrt qlen_*=100,nttcp -r:5.149/ 8.766/ 14.371/ 2.191 ms; 128 Mbps
>
> To get twice the speed a qlen=11 is enough already, and to get all the speed back a qlen=35 is needed.

This is an incomplete conclusion. It is incomplete in that A) these
tests were done under laboratory conditions at the highest data rate
(MCS15), and B), it was with a single point to point link to an AP
which normally would be handling more than one client. C) it tests a
single full throttle TCP stream when typical websites and usage
involve 70+ dns lookups and 70 separate short streams.

I can live with B and C) for now, although I note that the chrome
benchmark while doing a full blown stream test as you are doing now in
the background and ping is quite useful for looking at C. Let's tackle
A...

>
> And here are the results with fq_codel on the laptop too (just nttcp -t as thats the one affected):
>
> fq_codel on laptop, cerowrt defaults,  nttcp -t:  1.248/12.960/108.490/16.733 ms; 90 Mbps
> fq_codel on laptop, cerowrt qlen_*=4,  nttcp -t:  1.205/10.843/ 76.983/12.460 ms; 105 Mbps
> fq_codel on laptop, cerowrt qlen_*=8,  nttcp -t:  4.034/16.088/ 98.611/17.050 ms; 120 Mbps
> fq_codel on laptop, cerowrt qlen_*=11, nttcp -t:  3.766/15.687/ 56.684/11.135 ms; 114 Mbps
> fq_codel on laptop, cerowrt qlen_*=35, nttcp -t: 11.360/26.742/ 48.051/ 7.489 ms; 113 Mbps

So, if you could move your laptop to where it gets MCS4 on a fairly
reliable basis, and repeat the tests? a wall or three will do it.

I will predict several things:

1) the bulk of the buffering problem is going to move to your laptop,
as it has weaker antennas than the wndrs. Most likely you will end up
with tx on the one side higher than rx on the other.

2) you will see much higher jitter and latency and much lower
throughput. Your results will also get wildly more variable run to
run. (I tend to run tests for 2 minutes or longer and toss out the
first few seconds)

3) The lower fixed buffering sizes on cero's qlens will start making a
lot more sense, but it may be hard to see due to 1 and 2.

The thing I don't honestly know is how well fq_codel reacts to sudden
bandwidth changes when the underlying device driver (the iwl in this
case) is overbuffered or how well codel's target idea really works in
the wifi case in general. It would be nice to have some data on it.
(hint, hint)

Some work was done on debloating the iwl last year, I don't know if
any of the work made it into mainline.

Lastly, I put a version of Linux 3.6-rc2 up here.

http://snapon.lab.bufferbloat.net/~cero1/deb/

It has a fix to codel in it that was needed (I think but have not
checked to see if it's in 3.5.1), and it also incorporates "TCP small
queues", which reduces tcp-related buffering in pfifo_fast enormously,
and helps on other qdiscs as well. Switching to it will invalidate the
testing you've done so far...

(another reason why I'm reluctant to post graphs on codel/fq_codel
right now is that good stuff keeps happening above/below it in Linux),

please don't change your kernel out before trying that test... (and I
make no warranties about the reliability/usefulness of a rc2!)


> Shouldn't wireless N be able to do 200 - 300 Mbps though? If I enable debugging in iwl4965 I see that it
> starts TX aggregation, so not sure whats wrong (router or laptop?). With encryption off I can get at most 160 Mbps.

A UDP test will get you in the 270Mbit range usually.

>
> iw dev sw10 station dump shows:
> ...
>         signal:         -56 [-60, -59] dBm
>         signal avg:     -125 [-65, -58] dBm
>         tx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
>         rx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
>
> On laptop:
>         tx bitrate:     300.0 Mbit/s MCS 15 40Mhz short GI

In non-lab conditions you generally don't lock into a rate. The
minstrel algorithm tries various strategies to get the packets
through, so you can
get a grip on what's really happening by looking at the rc_stats file
for your particular device.

example here:


http://www.bufferbloat.net/projects/cerowrt/wiki/Minstrel_Wireless_Rate_Selection

Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

[Steinar H. Gunderson]

On Sat, Aug 18, 2012 at 10:07:45AM -0700, Dave Taht wrote:
> Some work was done on debloating the iwl last year, I don't know if
> any of the work made it into mainline.

Based on personal experience with 3.6.0-rc1 and my iwl3945, I think iwlwifi
is pretty bloated still. But this is unscientific; it's just “it feels that
way”, no hard numbers.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

[Török Edwin]

On 08/18/2012 08:07 PM, Dave Taht wrote:
> Thx again for the benchmarks on your hardware! Can I get you to go one
> more time to the well?

Yes, but you have to wait until I have some time to do it.

> Stripping out the incremental steps some will save you some time
> on benchmarking, so lets go with 3,4,12,35,100. Wireless data is
> incredibly noisy and I usually end up going with cdf plots like this
> old one
>> To get twice the speed a qlen=11 is enough already, and to get all the speed back a qlen=35 is needed.
> 
> This is an incomplete conclusion. It is incomplete in that A) these
> tests were done under laboratory conditions at the highest data rate
> (MCS15), and B), it was with a single point to point link to an AP
> which normally would be handling more than one client. C) it tests a
> single full throttle TCP stream when typical websites and usage
> involve 70+ dns lookups and 70 separate short streams.
> 
> I can live with B and C) for now, although I note that the chrome
> benchmark while doing a full blown stream test as you are doing now in
> the background and ping is quite useful for looking at C. Let's tackle
> A...
> 
>>
>> And here are the results with fq_codel on the laptop too (just nttcp -t as thats the one affected):
>>
>> fq_codel on laptop, cerowrt defaults,  nttcp -t:  1.248/12.960/108.490/16.733 ms; 90 Mbps
>> fq_codel on laptop, cerowrt qlen_*=4,  nttcp -t:  1.205/10.843/ 76.983/12.460 ms; 105 Mbps
>> fq_codel on laptop, cerowrt qlen_*=8,  nttcp -t:  4.034/16.088/ 98.611/17.050 ms; 120 Mbps
>> fq_codel on laptop, cerowrt qlen_*=11, nttcp -t:  3.766/15.687/ 56.684/11.135 ms; 114 Mbps
>> fq_codel on laptop, cerowrt qlen_*=35, nttcp -t: 11.360/26.742/ 48.051/ 7.489 ms; 113 Mbps
> 
> So, if you could move your laptop to where it gets MCS4 on a fairly
> reliable basis, and repeat the tests? a wall or three will do it.

I've put my laptop in a place where I got MCS4 on TX most of the time.
RX is MCS4 most of the time too, but it is switching to MCS5, 7, 11, 12 and back to MCS4
quite a lot.

> please don't change your kernel out before trying that test... (and I
> make no warranties about the reliability/usefulness of a rc2!)

Here are the results with fq_codel on the laptop, and same 3.5.0 kernel:

qlen 100, nttcp -t:  5.966/57.104/192.017/26.674 ms; 52.2376 Mbps
qlen  35, nttcp -t: 15.636/54.823/108.921/19.762 ms; 52.4675 Mbps
qlen  12, nttcp -t:  4.768/29.439/132.924/27.159 ms; 51.2619 Mbps
qlen  4,  nttcp -t:  2.631/20.500/152.741/31.549 ms; 40.3949 Mbps
qlen def, ntccp -t:  2.010/21.851/317.085/49.323 ms; 35.8268 Mbps

qlen 100, nttcp -r: 23.225/44.101/142.835/21.181 ms; 36.6789 Mbps
qlen  35, nttcp -r:  3.755/23.413/ 83.530/15.329 ms; 35.4602 Mbps
qlen  12, nttcp -r:  4.318/10.251/ 96.773/12.008 ms; 31.1557 Mbps
qlen   4, nttcp -r:  2.733/ 4.507/ 16.353/ 1.917 ms; 24.6688 Mbps
qlen def, nttcp -r:  2.119/ 4.999/ 64.968/ 7.275 ms; 27.3645 Mbps

Note that the laptop was on battery this time, so that may add some jitter
(CPU freq switching, wifi power saving?), but shouldn't matter for >10ms quantities.

Looks like the iwl4965 is somewhat bloated, with those 100ms+ latencies.

I don't know what happened there, but with the default qlen (2,3,3,3) I get the 317 ms max latency,
whereas with qlen 4 I get 152 ms max latency on TX. The average is also better with qlen 4.
Same observation goes for the RX side.

> 
> I will predict several things:
> 
> 1) the bulk of the buffering problem is going to move to your laptop,
> as it has weaker antennas than the wndrs. Most likely you will end up
> with tx on the one side higher than rx on the other.

Yes the laptop TX latencies are worse.

> 
> 2) you will see much higher jitter and latency and much lower
> throughput. Your results will also get wildly more variable run to
> run. (I tend to run tests for 2 minutes or longer and toss out the
> first few seconds)

On TX it is quite consistently in MCS4 (according to watch iw wlan0 station dump),
but on RX its jumping quite a lot.

> 
> 3) The lower fixed buffering sizes on cero's qlens will start making a
> lot more sense, but it may be hard to see due to 1 and 2.

qlen 12 and 4 look good. The default looks worse though.

> 
> The thing I don't honestly know is how well fq_codel reacts to sudden
> bandwidth changes when the underlying device driver (the iwl in this
> case) is overbuffered or how well codel's target idea really works in
> the wifi case in general. It would be nice to have some data on it.
> (hint, hint)

The bandwidth varies quite a lot on RX even if both the laptop and router
are perfectly still. So the -r numbers above should be what you are looking for.
If you want some other data let me know.

> 
> Some work was done on debloating the iwl last year, I don't know if
> any of the work made it into mainline.
> 
> Lastly, I put a version of Linux 3.6-rc2 up here.
> 
> http://snapon.lab.bufferbloat.net/~cero1/deb/
> 
> It has a fix to codel in it that was needed (I think but have not
> checked to see if it's in 3.5.1), and it also incorporates "TCP small
> queues", which reduces tcp-related buffering in pfifo_fast enormously,
> and helps on other qdiscs as well. Switching to it will invalidate the
> testing you've done so far...

I assume these are in the upstream 3.6-rc3 too, right?

Here is just one measurement done with 3.6-rc3 on the laptop and fq_codel
(same location as above tests, approx MCS4):
qlen def, nttcp -t, 2.871/15.655/375.777/44.212 ms; 35.2776 Mbps
qlen def, nttcp -r, 1.406/ 3.434/ 12.763/ 1.649 ms; 24.3334 Mbps

It looks somewhat better.

> 
> (another reason why I'm reluctant to post graphs on codel/fq_codel
> right now is that good stuff keeps happening above/below it in Linux),
> 
> 
> 
>> Shouldn't wireless N be able to do 200 - 300 Mbps though? If I enable debugging in iwl4965 I see that it
>> starts TX aggregation, so not sure whats wrong (router or laptop?). With encryption off I can get at most 160 Mbps.
> 
> A UDP test will get you in the 270Mbit range usually.

nttcp -T -u -D -n2000 gives ~180 Mbps at most, and with -r I can't make sense of it (looks like most gets dropped):
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l    16384    0.08    0.00      1.6090  13107.2000       5     61.38  500000.0
1  8192000    0.08    0.04    845.8113   1820.6973    2003  25850.83   55646.6

> 
>>
>> iw dev sw10 station dump shows:
>> ...
>>         signal:         -56 [-60, -59] dBm
>>         signal avg:     -125 [-65, -58] dBm
>>         tx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
>>         rx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
>>
>> On laptop:
>>         tx bitrate:     300.0 Mbit/s MCS 15 40Mhz short GI
> 
> In non-lab conditions you generally don't lock into a rate. The
> minstrel algorithm tries various strategies to get the packets
> through, so you can
> get a grip on what's really happening by looking at the rc_stats file
> for your particular device.
> 
> example here:
> 
> 
> http://www.bufferbloat.net/projects/cerowrt/wiki/Minstrel_Wireless_Rate_Selection
> 

I looked at the rc_stats file by cd-ing into the stations dir on the router. After disabling/enabling the radio
the stations subdir was gone though:
root@OpenWrt:~# ls /sys/kernel/debug/ieee80211/phy1/netdev\:sw10/stations/ -al
drwxr-xr-x    2 root     root             0 Aug 25 10:28 .
drwxr-xr-x    3 root     root             0 Aug 25 10:28 ..

So unfortunately I'm without an rc_stats now (until I reboot the router probably?).

Best regards,
--Edwin

Re: [Bloat] [Cerowrt-devel] cerowrt 3.3.8-17: nice latency improvements, some issues with bind

[Dave Taht]

On Sat, Aug 25, 2012 at 6:56 AM, Török Edwin
<edwin+ml-cerowrt@etorok.net> wrote:
> On 08/18/2012 08:07 PM, Dave Taht wrote:
>> Thx again for the benchmarks on your hardware! Can I get you to go one
>> more time to the well?
>
> Yes, but you have to wait until I have some time to do it.

No worries. Doing good science takes time.

>
>> Stripping out the incremental steps some will save you some time
>> on benchmarking, so lets go with 3,4,12,35,100. Wireless data is
>> incredibly noisy and I usually end up going with cdf plots like this
>> old one
>>> To get twice the speed a qlen=11 is enough already, and to get all the speed back a qlen=35 is needed.
>>
>> This is an incomplete conclusion. It is incomplete in that A) these
>> tests were done under laboratory conditions at the highest data rate
>> (MCS15), and B), it was with a single point to point link to an AP
>> which normally would be handling more than one client. C) it tests a
>> single full throttle TCP stream when typical websites and usage
>> involve 70+ dns lookups and 70 separate short streams.
>>
>> I can live with B and C) for now, although I note that the chrome
>> benchmark while doing a full blown stream test as you are doing now in
>> the background and ping is quite useful for looking at C. Let's tackle
>> A...
>>
>>>
>>> And here are the results with fq_codel on the laptop too (just nttcp -t as thats the one affected):
>>>
>>> fq_codel on laptop, cerowrt defaults,  nttcp -t:  1.248/12.960/108.490/16.733 ms; 90 Mbps
>>> fq_codel on laptop, cerowrt qlen_*=4,  nttcp -t:  1.205/10.843/ 76.983/12.460 ms; 105 Mbps
>>> fq_codel on laptop, cerowrt qlen_*=8,  nttcp -t:  4.034/16.088/ 98.611/17.050 ms; 120 Mbps
>>> fq_codel on laptop, cerowrt qlen_*=11, nttcp -t:  3.766/15.687/ 56.684/11.135 ms; 114 Mbps
>>> fq_codel on laptop, cerowrt qlen_*=35, nttcp -t: 11.360/26.742/ 48.051/ 7.489 ms; 113 Mbps
>>
>> So, if you could move your laptop to where it gets MCS4 on a fairly
>> reliable basis, and repeat the tests? a wall or three will do it.
>
> I've put my laptop in a place where I got MCS4 on TX most of the time.
> RX is MCS4 most of the time too, but it is switching to MCS5, 7, 11, 12 and back to MCS4
> quite a lot.
>
>> please don't change your kernel out before trying that test... (and I
>> make no warranties about the reliability/usefulness of a rc2!)
>
> Here are the results with fq_codel on the laptop, and same 3.5.0 kernel:
>
> qlen 100, nttcp -t:  5.966/57.104/192.017/26.674 ms; 52.2376 Mbps
> qlen  35, nttcp -t: 15.636/54.823/108.921/19.762 ms; 52.4675 Mbps
> qlen  12, nttcp -t:  4.768/29.439/132.924/27.159 ms; 51.2619 Mbps
> qlen  4,  nttcp -t:  2.631/20.500/152.741/31.549 ms; 40.3949 Mbps
> qlen def, ntccp -t:  2.010/21.851/317.085/49.323 ms; 35.8268 Mbps
>
> qlen 100, nttcp -r: 23.225/44.101/142.835/21.181 ms; 36.6789 Mbps
> qlen  35, nttcp -r:  3.755/23.413/ 83.530/15.329 ms; 35.4602 Mbps
> qlen  12, nttcp -r:  4.318/10.251/ 96.773/12.008 ms; 31.1557 Mbps
> qlen   4, nttcp -r:  2.733/ 4.507/ 16.353/ 1.917 ms; 24.6688 Mbps
> qlen def, nttcp -r:  2.119/ 4.999/ 64.968/ 7.275 ms; 27.3645 Mbps
>
> Note that the laptop was on battery this time, so that may add some jitter
> (CPU freq switching, wifi power saving?), but shouldn't matter for >10ms quantities.

Thank you for so clearly showing the trendline and relationship between
overbuffering, bandwidth, latency, and jitter on linux wifi in this
combination of these two drivers and OSes!

(I am inclined to throw out the second qlen 4 result as anomalous however)

(Did I add enough qualifications to the above statement?)

It does look like qlen 12 (presently) fits within my overall goals.
However (to me) the next step is switching the ath9k driver's buffering itself
from a straight fifo to (a tree?) trying to inspect its queue(s)
for possible aggregate-able packets and fq-ing (again) the result.

a better method (probably) would be for it to tell the overlying qdisc
"I want up to x packets or y bytes for station z", and the overlying
qdisc to be doing that job, and thus the codel notion of "maxpacket"
could apply to each station.

"maxpacket" is kind of misnamed, what it means is the maximum number
of bytes that can be delivered in one go - so it is MTU for devices
that don't have TSO or GSO enabled, size of a TSO (something less than
64k) for TSO/GSO, and should (probably!!! we're not there yet) be
equal to
"proposed next aggregate-able size/bytes for this destination" in wifi.

> Looks like the iwl4965 is somewhat bloated, with those 100ms+ latencies.

Ya think? It turns out one of my laptops has the 5100AGN, which is similar.
Somewhere on this list last year was a long discussion and some
proposed patches for the iwl series... I think the guy working on it
got swamped by some phd work, though.

right now that box is used as a wired endpoint (and has a SR71 card in it).

A few x86 boxes were just donated that I can replace that with, and do
a bit more wireless testing next week than I presently do. (and the
x86 boxes will dramatically expand testing longer RTTs, which I care
about a lot)

(THANK YOU VYATTA!)

Regrettably, first I have to get those boxes here, then setup, a
working OS and kernel on them, and then running netem...

> I don't know what happened there, but with the default qlen (2,3,3,3) I get the 317 ms max latency,
> whereas with qlen 4 I get 152 ms max latency on TX. The average is also better with qlen 4.
> Same observation goes for the RX side.

We have a potential interaction with the default quantums I'm using on cero,
which are 256, rather than 1500, (which is the default). In that case,
we can end up with 3 timestamped ipv4 acks in a row, but not 4, so a
given stream can "leak over" into the next potential aggregate, which
might be arbitrarily shortened by incorporating another portion of a
stream for another destination.

So, I'm inclined to bump it up to 12 for the cerowrt userbase as the
cost in normal usage is low and the benefit high, (note the same
problem above will occur, just slightly less often, and on average the
aggregates will be larger, so it's a win)

but in the interest of science and continuing to analyze codel's
behavior I'm going to keep it at 3 for while longer. (feel free to use
values that make you happy, just clearly tell me when you do, please!)

fiddling with the qlen is a very blunt hammer for the real job that
needs to be happening in the qdisc and driver, regardless. I hope we
can get much smarter about it soon, but at least in my case that
requires more insight into the ath9k than I have currently. Felix is
probably pretty wrapped up in the openwrt freeze, Andrew has another
day job...

>>
>> I will predict several things:
>>
>> 1) the bulk of the buffering problem is going to move to your laptop,
>> as it has weaker antennas than the wndrs. Most likely you will end up
>> with tx on the one side higher than rx on the other.
>
> Yes the laptop TX latencies are worse.
>
>>
>> 2) you will see much higher jitter and latency and much lower
>> throughput. Your results will also get wildly more variable run to
>> run. (I tend to run tests for 2 minutes or longer and toss out the
>> first few seconds)
>
> On TX it is quite consistently in MCS4 (according to watch iw wlan0 station dump),
> but on RX its jumping quite a lot.

As good as the minstrel algorithm is, I've often felt it could be improved
with deeper analysis of what really happens in the wireless-n cases,
particularly in the case of retries and within-aggregate packet loss.

Tuning it for -g took a year of data collection and a ton of analysis
and cash... and the (excellent) paper on it is unpublishable because
it so far exceeds the MPU. I doubt there are 12 people in the world
that deeply understand how minstrel works, and I wish there were
thousands... there is a wealth of information in it that could be used
for other things, like improving the behavior of mesh routing
protocols.

>>
>> 3) The lower fixed buffering sizes on cero's qlens will start making a
>> lot more sense, but it may be hard to see due to 1 and 2.
>
> qlen 12 and 4 look good. The default looks worse though.
>
>>
>> The thing I don't honestly know is how well fq_codel reacts to sudden
>> bandwidth changes when the underlying device driver (the iwl in this
>> case) is overbuffered or how well codel's target idea really works in
>> the wifi case in general. It would be nice to have some data on it.
>> (hint, hint)
>
> The bandwidth varies quite a lot on RX even if both the laptop and router
> are perfectly still. So the -r numbers above should be what you are looking for.
> If you want some other data let me know.

I'll try not to abuse your time, but if I can convince you to be able to
duplicate your experiments exactly, when needed, it would be an enormous help.

>>
>> Some work was done on debloating the iwl last year, I don't know if
>> any of the work made it into mainline.
>>
>> Lastly, I put a version of Linux 3.6-rc2 up here.
>>
>> http://snapon.lab.bufferbloat.net/~cero1/deb/
>>
>> It has a fix to codel in it that was needed (I think but have not
>> checked to see if it's in 3.5.1), and it also incorporates "TCP small
>> queues", which reduces tcp-related buffering in pfifo_fast enormously,
>> and helps on other qdiscs as well. Switching to it will invalidate the
>> testing you've done so far...
>
> I assume these are in the upstream 3.6-rc3 too, right?

yes. The rc3 I just put up there has some subtle changes to codel in
it, however that differ from the mainline. I'll have to clearly
distinguish between that and mainline better in the future.

> Here is just one measurement done with 3.6-rc3 on the laptop and fq_codel
> (same location as above tests, approx MCS4):
> qlen def, nttcp -t, 2.871/15.655/375.777/44.212 ms; 35.2776 Mbps
> qlen def, nttcp -r, 1.406/ 3.434/ 12.763/ 1.649 ms; 24.3334 Mbps
>
> It looks somewhat better.

12 remains the sanest win right now. But too early to change it this month.

thx again!

>>
>> (another reason why I'm reluctant to post graphs on codel/fq_codel
>> right now is that good stuff keeps happening above/below it in Linux),
>>
>>
>>
>>> Shouldn't wireless N be able to do 200 - 300 Mbps though? If I enable debugging in iwl4965 I see that it
>>> starts TX aggregation, so not sure whats wrong (router or laptop?). With encryption off I can get at most 160 Mbps.
>>
>> A UDP test will get you in the 270Mbit range usually.
>
> nttcp -T -u -D -n2000 gives ~180 Mbps at most, and with -r I can't make sense of it (looks like most gets dropped):
>      Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
> l    16384    0.08    0.00      1.6090  13107.2000       5     61.38  500000.0
> 1  8192000    0.08    0.04    845.8113   1820.6973    2003  25850.83   55646.6

I'll think about this on another day. Feel free to do pfifo_fast on this test
a couple times in either direction to get a baseline.

Doing badly on this test right now doesn't bother me at all...

>
>>
>>>
>>> iw dev sw10 station dump shows:
>>> ...
>>>         signal:         -56 [-60, -59] dBm
>>>         signal avg:     -125 [-65, -58] dBm
>>>         tx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
>>>         rx bitrate:     300.0 MBit/s MCS 15 40Mhz short GI
>>>
>>> On laptop:
>>>         tx bitrate:     300.0 Mbit/s MCS 15 40Mhz short GI
>>
>> In non-lab conditions you generally don't lock into a rate. The
>> minstrel algorithm tries various strategies to get the packets
>> through, so you can
>> get a grip on what's really happening by looking at the rc_stats file
>> for your particular device.
>>
>> example here:
>>
>>
>> http://www.bufferbloat.net/projects/cerowrt/wiki/Minstrel_Wireless_Rate_Selection
>>
>
> I looked at the rc_stats file by cd-ing into the stations dir on the router. After disabling/enabling the radio
> the stations subdir was gone though:
> root@OpenWrt:~# ls /sys/kernel/debug/ieee80211/phy1/netdev\:sw10/stations/ -al
> drwxr-xr-x    2 root     root             0 Aug 25 10:28 .
> drwxr-xr-x    3 root     root             0 Aug 25 10:28 ..
>
> So unfortunately I'm without an rc_stats now (until I reboot the router probably?).
>
> Best regards,
> --Edwin



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-17 is out
with fq_codel!"