* [Bloat] capturing packets and applying qdiscs @ 2015-03-26 18:00 Isaac Konikoff 0 siblings, 0 replies; 7+ messages in thread From: Isaac Konikoff @ 2015-03-26 18:00 UTC (permalink / raw) To: Dave Taht; +Cc: codel, bloat Hi Dave, Can you please review my setup and let me know how to improve my application of the qdiscs? I've been applying manually, but I'm not sure that is the best method, or if the values really make sense. Sorry if this has been covered ad nauseum in codel or bloat threads over the past 4+ years... I've been capturing packets on a dedicated monitor box using the following method: tshark -i moni1 -w <file> where moni1 is ath9k on channel 149 (5745 MHz), width: 40 MHz, center1: 5755 MHz The system under test is a lanforge ath10k ap being driven by another lanforge system using ath9k clients to associate and run traffic tests. The two traffic tests I'm running are: 1. netperf-wrapper batch consisting of: tcp_download, tcp_upload, tcp_bidirectional, rrul, rrul_be and rtt_fair4be on 4 sta's. 2. lanforge wifi capacity test using tcp-download incrementing 4 sta's per minute up to 64 sta's with each iteration attempting 500Mbps download per x number of sta's. The qdiscs I am using are applied to the virtual ap interface which is the egress interface for download tests. I also applied the same qdisc to the ap's eth1 for the few upload tests. Is this sane? qdiscs used, deleting each before trying the next: 1. default pfifo_fast 2. tc qdisc add dev vap1 root fq_codel 3. tc qdisc add dev vap1 root fq_codel target 5ms interval 100ms noecn 4. tc qdisc add dev vap1 root fq_codel limit 2000 target 3ms interval 40ms noecn Any suggestions you have would be helpful. Thanks, Isaac ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Bloat] capturing packets and applying qdiscs @ 2015-03-26 21:39 Isaac Konikoff 2015-03-27 0:39 ` David Lang 2015-03-27 1:19 ` Dave Taht 0 siblings, 2 replies; 7+ messages in thread From: Isaac Konikoff @ 2015-03-26 21:39 UTC (permalink / raw) To: bloat; +Cc: codel, cerowrt-devel Hi All, Looking for some feedback in my test setup... Can you please review my setup and let me know how to improve my application of the qdiscs? I've been applying manually, but I'm not sure that is the best method, or if the values really make sense. Sorry if this has been covered ad nauseum in codel or bloat threads over the past 4+ years... I've been capturing packets on a dedicated monitor box using the following method: tshark -i moni1 -w <file> where moni1 is ath9k on channel 149 (5745 MHz), width: 40 MHz, center1: 5755 MHz The system under test is a lanforge ath10k ap being driven by another lanforge system using ath9k clients to associate and run traffic tests. The two traffic tests I'm running are: 1. netperf-wrapper batch consisting of: tcp_download, tcp_upload, tcp_bidirectional, rrul, rrul_be and rtt_fair4be on 4 sta's. 2. lanforge wifi capacity test using tcp-download incrementing 4 sta's per minute up to 64 sta's with each iteration attempting 500Mbps download per x number of sta's. The qdiscs I am using are applied to the virtual ap interface which is the egress interface for download tests. I also applied the same qdisc to the ap's eth1 for the few upload tests. Is this sane? qdiscs used, deleting each before trying the next: 1. default pfifo_fast 2. tc qdisc add dev vap1 root fq_codel 3. tc qdisc add dev vap1 root fq_codel target 5ms interval 100ms noecn 4. tc qdisc add dev vap1 root fq_codel limit 2000 target 3ms interval 40ms noecn Any suggestions you have would be helpful. Thanks, Isaac ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Bloat] capturing packets and applying qdiscs 2015-03-26 21:39 Isaac Konikoff @ 2015-03-27 0:39 ` David Lang 2015-03-27 16:38 ` Isaac Konikoff 2015-03-27 1:19 ` Dave Taht 1 sibling, 1 reply; 7+ messages in thread From: David Lang @ 2015-03-27 0:39 UTC (permalink / raw) To: Isaac Konikoff; +Cc: codel, cerowrt-devel, bloat On Thu, 26 Mar 2015, Isaac Konikoff wrote: > Hi All, > > Looking for some feedback in my test setup... > > Can you please review my setup and let me know how to improve my application > of the qdiscs? I've been applying manually, but I'm not sure that is the best > method, or if the values really make sense. Sorry if this has been covered ad > nauseum in codel or bloat threads over the past 4+ years... > > I've been capturing packets on a dedicated monitor box using the following > method: > > tshark -i moni1 -w <file> > > where moni1 is ath9k on channel 149 (5745 MHz), width: 40 MHz, center1: 5755 > MHz > > The system under test is a lanforge ath10k ap being driven by another > lanforge system using ath9k clients to associate and run traffic tests. > > The two traffic tests I'm running are: > > 1. netperf-wrapper batch consisting of: tcp_download, tcp_upload, > tcp_bidirectional, rrul, rrul_be and rtt_fair4be on 4 sta's. > > 2. lanforge wifi capacity test using tcp-download incrementing 4 sta's per > minute up to 64 sta's with each iteration attempting 500Mbps download per x > number of sta's. what results are you getting? and what results are you hoping to get to? David Lang > The qdiscs I am using are applied to the virtual ap interface which is the > egress interface for download tests. I also applied the same qdisc to the > ap's eth1 for the few upload tests. Is this sane? > > qdiscs used, deleting each before trying the next: > 1. default pfifo_fast > 2. tc qdisc add dev vap1 root fq_codel > 3. tc qdisc add dev vap1 root fq_codel target 5ms interval 100ms noecn > 4. tc qdisc add dev vap1 root fq_codel limit 2000 target 3ms interval 40ms > noecn > > Any suggestions you have would be helpful. > > Thanks, > Isaac > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Bloat] capturing packets and applying qdiscs 2015-03-27 0:39 ` David Lang @ 2015-03-27 16:38 ` Isaac Konikoff 2015-03-27 17:15 ` Aaron Wood 0 siblings, 1 reply; 7+ messages in thread From: Isaac Konikoff @ 2015-03-27 16:38 UTC (permalink / raw) To: David Lang; +Cc: codel, cerowrt-devel, bloat On 03/26/2015 05:39 PM, David Lang wrote: > On Thu, 26 Mar 2015, Isaac Konikoff wrote: > >> Hi All, >> >> Looking for some feedback in my test setup... >> >> Can you please review my setup and let me know how to improve my >> application of the qdiscs? I've been applying manually, but I'm not >> sure that is the best method, or if the values really make sense. >> Sorry if this has been covered ad nauseum in codel or bloat threads >> over the past 4+ years... >> >> I've been capturing packets on a dedicated monitor box using the >> following method: >> >> tshark -i moni1 -w <file> >> >> where moni1 is ath9k on channel 149 (5745 MHz), width: 40 MHz, >> center1: 5755 MHz >> >> The system under test is a lanforge ath10k ap being driven by another >> lanforge system using ath9k clients to associate and run traffic tests. >> >> The two traffic tests I'm running are: >> >> 1. netperf-wrapper batch consisting of: tcp_download, tcp_upload, >> tcp_bidirectional, rrul, rrul_be and rtt_fair4be on 4 sta's. >> >> 2. lanforge wifi capacity test using tcp-download incrementing 4 >> sta's per minute up to 64 sta's with each iteration attempting >> 500Mbps download per x number of sta's. > > what results are you getting? and what results are you hoping to get to? > > David Lang I'll share my results shortly, but the main idea is that I'm doing the captures as part of our effort to improve the ath10k driver. Just one comparison is that with many clients the ath10k ap throughput tails off whereas some non-ath10k ap's are able to sustain high throughput for many clients, but even that depends on manufacturer and firmware combos. I'll be able to point this behaviour out better once I get the files uploaded... ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Bloat] capturing packets and applying qdiscs 2015-03-27 16:38 ` Isaac Konikoff @ 2015-03-27 17:15 ` Aaron Wood 0 siblings, 0 replies; 7+ messages in thread From: Aaron Wood @ 2015-03-27 17:15 UTC (permalink / raw) To: Isaac Konikoff; +Cc: codel, cerowrt-devel, bloat [-- Attachment #1: Type: text/plain, Size: 2751 bytes --] I do this often at work, using a separate machine to capture traffic using wireshark. Wireshark makes a lot of the analysis fairly straightforward (especially with it's excellent packet dissectors). By capturing in radiotap mode, you get RSSI/noise levels on the 802.11n packet, the rates involved, everything. It's very nice for digging into issues. Unfortunately, the next problem is a "can't see the forest for the trees", as there are no good high-level analysis tools for captured traffic that I've found. Most of the commercial packages seem to offer summary stats, but not much more (nothing like airtime utilization over time, negotiated rates over time, aggregate/per-station throughput over time, etc.) -Aaron On Fri, Mar 27, 2015 at 9:38 AM, Isaac Konikoff <konikofi@candelatech.com> wrote: > > > On 03/26/2015 05:39 PM, David Lang wrote: > >> On Thu, 26 Mar 2015, Isaac Konikoff wrote: >> >> Hi All, >>> >>> Looking for some feedback in my test setup... >>> >>> Can you please review my setup and let me know how to improve my >>> application of the qdiscs? I've been applying manually, but I'm not sure >>> that is the best method, or if the values really make sense. Sorry if this >>> has been covered ad nauseum in codel or bloat threads over the past 4+ >>> years... >>> >>> I've been capturing packets on a dedicated monitor box using the >>> following method: >>> >>> tshark -i moni1 -w <file> >>> >>> where moni1 is ath9k on channel 149 (5745 MHz), width: 40 MHz, center1: >>> 5755 MHz >>> >>> The system under test is a lanforge ath10k ap being driven by another >>> lanforge system using ath9k clients to associate and run traffic tests. >>> >>> The two traffic tests I'm running are: >>> >>> 1. netperf-wrapper batch consisting of: tcp_download, tcp_upload, >>> tcp_bidirectional, rrul, rrul_be and rtt_fair4be on 4 sta's. >>> >>> 2. lanforge wifi capacity test using tcp-download incrementing 4 sta's >>> per minute up to 64 sta's with each iteration attempting 500Mbps download >>> per x number of sta's. >>> >> >> what results are you getting? and what results are you hoping to get to? >> >> David Lang >> > I'll share my results shortly, but the main idea is that I'm doing the > captures as part of our effort to improve the ath10k driver. Just one > comparison is that with many clients the ath10k ap throughput tails off > whereas some non-ath10k ap's are able to sustain high throughput for many > clients, but even that depends on manufacturer and firmware combos. > > I'll be able to point this behaviour out better once I get the files > uploaded... > > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > [-- Attachment #2: Type: text/html, Size: 3680 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Bloat] capturing packets and applying qdiscs 2015-03-26 21:39 Isaac Konikoff 2015-03-27 0:39 ` David Lang @ 2015-03-27 1:19 ` Dave Taht 2015-03-27 1:37 ` Dave Taht 1 sibling, 1 reply; 7+ messages in thread From: Dave Taht @ 2015-03-27 1:19 UTC (permalink / raw) To: Isaac Konikoff; +Cc: codel, cerowrt-devel, bloat On Thu, Mar 26, 2015 at 2:39 PM, Isaac Konikoff <konikofi@candelatech.com> wrote: > Hi All, > > Looking for some feedback in my test setup... > > Can you please review my setup and let me know how to improve my application > of the qdiscs? I've been applying manually, but I'm not sure that is the > best method, or if the values really make sense. Sorry if this has been > covered ad nauseum in codel or bloat threads over the past 4+ years... > > I've been capturing packets on a dedicated monitor box using the following > method: > > tshark -i moni1 -w <file> > > where moni1 is ath9k on channel 149 (5745 MHz), width: 40 MHz, center1: 5755 > MHz For those of you that don't know how to do aircaps, it is pretty easy. We are going to be doing a lot more of this as make-wifi-fast goes along, so... install aircrack-ng via whatever means you have available (works best on ath9k, seems to work on iwl, don't know about other devices) run: airmon-ng start your_wifi_device your_channel This will create a monX device of some sort, which you can then capture with tshark or wireshark. There are all sorts of other cool features here where - for example - you can post-hoc decrypt a wpa session, etc. Note that usually you will have trouble using the device for other things, so I tend to just run it with the ethernet also connected. We are in dire need of tools that can analyze aircap'd stuff at different rates, look at beacons, interpacket gaps, wireless g fallbacks, etc. If anyone knows f anything good, please post to the list. > The system under test is a lanforge ath10k ap being driven by another > lanforge system using ath9k clients to associate and run traffic tests. > > The two traffic tests I'm running are: > > 1. netperf-wrapper batch consisting of: tcp_download, tcp_upload, > tcp_bidirectional, rrul, rrul_be and rtt_fair4be on 4 sta's. Cool. > 2. lanforge wifi capacity test using tcp-download incrementing 4 sta's per > minute up to 64 sta's with each iteration attempting 500Mbps download per x > number of sta's. > > The qdiscs I am using are applied to the virtual ap interface which is the > egress interface for download tests. I also applied the same qdisc to the > ap's eth1 for the few upload tests. Is this sane? > > qdiscs used, deleting each before trying the next: > 1. default pfifo_fast > 2. tc qdisc add dev vap1 root fq_codel > 3. tc qdisc add dev vap1 root fq_codel target 5ms interval 100ms noecn 1) Test 2 and test 3 are essentially the same, unless you have also enabled ecn on both sides of the tcp connection with sysctl -w net.ipv4.tcp_ecn=1 #or the equivalent in sysctl.conf The ecn vs non-ecn results tend to smoother results for tcp and mildly higher packet loss on the measurement flows. 2) I do not have an ath10k in front of me. The ath9k presents 4 queues controlled by mq (and then some sub-qdisc) when it is in operation, as does the iwl. Does the ath10k only present one queue? On the two chipsets mentioned first, the queues are mapped to the 802.11e VO,VI,BE, and BK queues - very inefficiently. I have long maintained the VO queue should be obsoleted in favor of the VI queue, and in general I find wireless-n works better if these queues are entirely disabled on the AP. This extremely old piece of code does more of the right thing for the mq'd style of wifi interface, although it is pretty wrong for everything else (notably, we typically only use a reduced quantum of 300 on some low speed devices, we never got around to making the tg3 work right, and the tc filter is not the right thing for wifi, either) https://github.com/dtaht/deBloat/blob/master/src/debloat.sh > 4. tc qdisc add dev vap1 root fq_codel limit 2000 target 3ms interval 40ms > noecn Here there are about 3 assumptions wrong. 1) 1000 packets is still quite enough for even 802.11ac wifi (or so I think). 2) although fiddling with the target and interval is done here, there is so much underlying buffering that these numbers are not going to help much in the face of them on wifi. I typically actually run with a much larger target (30ms) to cope with wifi's mac access jitter - with the default interval when trying to improve per-station performance along with... 3) The real parameters that will help wifi on an AP somewhat is to use a tc dst filter (rather than the default 5 tuple filter) on fq_codel to sort stuff into per station queues, and to use a quantum in the 4500 range, which accounts for either the max number of packets that can be put in a txop (42 on wireless-n), and/or 3 big packets - neither solution being a good one when wifi can handle 64k in a single burst, and ac, more. Even then, the results are far less than pleasing. What is needed, and what we are going to do, is add real per-station queuing at the lowest layer and then put something fq_codel like on top of each... and that work hasn't started yet. The tc filter method I just described will not work on station ids and thus will treat ipv4 and ipv6 traffic for the same destination differently. Now I do have wifi results for this stuff - somewhere - and the right tc filter for dst filtering on a per mq basis, but it turns out I think I left all that behind a natted box that I can't get back to til thursday next week. and as always I appreciate every scrap of data, every experiment, every result obtained via every method, in order to more fully bracket the real problems and demonstrate progress against wifi's problems, if and when we start making it. a tarball of what you got would be nice to have around. You will see absolutely terrible per-sta download performance on the rrul and rrul_be tests in particular with any of the qdiscs. > > Any suggestions you have would be helpful. > > Thanks, > Isaac > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat -- Dave Täht Let's make wifi fast, less jittery and reliable again! https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Bloat] capturing packets and applying qdiscs 2015-03-27 1:19 ` Dave Taht @ 2015-03-27 1:37 ` Dave Taht 0 siblings, 0 replies; 7+ messages in thread From: Dave Taht @ 2015-03-27 1:37 UTC (permalink / raw) To: Isaac Konikoff; +Cc: codel, cerowrt-devel, bloat On Thu, Mar 26, 2015 at 6:19 PM, Dave Taht <dave.taht@gmail.com> wrote: > On Thu, Mar 26, 2015 at 2:39 PM, Isaac Konikoff > <konikofi@candelatech.com> wrote: >> Hi All, >> >> Looking for some feedback in my test setup... >> >> Can you please review my setup and let me know how to improve my application >> of the qdiscs? I've been applying manually, but I'm not sure that is the >> best method, or if the values really make sense. Sorry if this has been >> covered ad nauseum in codel or bloat threads over the past 4+ years... >> >> I've been capturing packets on a dedicated monitor box using the following >> method: >> >> tshark -i moni1 -w <file> >> >> where moni1 is ath9k on channel 149 (5745 MHz), width: 40 MHz, center1: 5755 >> MHz > > For those of you that don't know how to do aircaps, it is pretty easy. > We are going to be doing a lot more of this as make-wifi-fast goes > along, so... > > install aircrack-ng via whatever means you have available (works best > on ath9k, seems to work on iwl, don't know about other devices) > > run: > > airmon-ng start your_wifi_device your_channel > > This will create a monX device of some sort, which you can then > capture with tshark or wireshark. There are all sorts of other cool > features here where - for example - you can post-hoc decrypt a wpa > session, etc. > > Note that usually you will have trouble using the device for other > things, so I tend to just run it with the ethernet also connected. > > We are in dire need of tools that can analyze aircap'd stuff at > different rates, look at beacons, interpacket gaps, wireless g > fallbacks, etc. If anyone knows f anything good, please post to the > list. > >> The system under test is a lanforge ath10k ap being driven by another >> lanforge system using ath9k clients to associate and run traffic tests. >> >> The two traffic tests I'm running are: >> >> 1. netperf-wrapper batch consisting of: tcp_download, tcp_upload, >> tcp_bidirectional, rrul, rrul_be and rtt_fair4be on 4 sta's. > > Cool. > >> 2. lanforge wifi capacity test using tcp-download incrementing 4 sta's per >> minute up to 64 sta's with each iteration attempting 500Mbps download per x >> number of sta's. >> >> The qdiscs I am using are applied to the virtual ap interface which is the >> egress interface for download tests. I also applied the same qdisc to the >> ap's eth1 for the few upload tests. Is this sane? by all means apply fq_codel on the ethernet devices. At these speeds you will see the fq part engage at least occasionally, which you can see by tc -s qdisc show dev eth1 showing a few overlimits or new flows. If it does not engage at these speeds, you will see maxpacket not crack 256 as that statistic is not collected unless stuff engages. if maxpacket is greater than 1514 you have offloads turned on somewhere. >> qdiscs used, deleting each before trying the next: >> 1. default pfifo_fast >> 2. tc qdisc add dev vap1 root fq_codel >> 3. tc qdisc add dev vap1 root fq_codel target 5ms interval 100ms noecn > > 1) Test 2 and test 3 are essentially the same, unless you have also > enabled ecn on both sides of the tcp connection with > > sysctl -w net.ipv4.tcp_ecn=1 #or the equivalent in sysctl.conf > > The ecn vs non-ecn results tend to smoother results for tcp and mildly > higher packet loss on the measurement flows. > > 2) I do not have an ath10k in front of me. The ath9k presents 4 queues > controlled by > mq (and then some sub-qdisc) when it is in operation, as does the iwl. > Does the ath10k only present one queue? > > On the two chipsets mentioned first, the queues are mapped to the 802.11e > VO,VI,BE, and BK queues - very inefficiently. I have long maintained > the VO queue should be obsoleted in favor of the VI queue, and in > general I find wireless-n works better if these queues are entirely > disabled on the AP. > > This extremely old piece of code does more of the right thing for the > mq'd style of > wifi interface, although it is pretty wrong for everything else > (notably, we typically only use a reduced quantum of 300 on some low > speed devices, we never got around to making the tg3 work right, and > the tc filter is not the right thing for wifi, either) > > https://github.com/dtaht/deBloat/blob/master/src/debloat.sh > >> 4. tc qdisc add dev vap1 root fq_codel limit 2000 target 3ms interval 40ms >> noecn > > Here there are about 3 assumptions wrong. > > 1) 1000 packets is still quite enough for even 802.11ac wifi (or so I think). > 2) although fiddling with the target and interval is done here, there > is so much underlying buffering that these numbers are not going to > help much in the face of them on wifi. I typically actually run with a > much larger target (30ms) to cope with wifi's mac access jitter - with > the default interval when trying to improve per-station performance > along with... > > 3) The real parameters that will help wifi on an AP somewhat is to use > a tc dst filter (rather than the default 5 tuple filter) on fq_codel > to sort stuff into per station queues, and to use a quantum in the > 4500 range, which accounts for either the max number of packets that > can be put in a txop (42 on wireless-n), and/or 3 big packets - > neither solution being a good one when wifi can handle 64k in a single > burst, and ac, more. > > > Even then, the results are far less than pleasing. What is needed, and > what we are going to do, is add real per-station queuing at the lowest > layer and then put something fq_codel like on top of each... and that > work hasn't started yet. The tc filter method I just described will > not work on station ids and thus will treat ipv4 and ipv6 traffic for > the same destination differently. > > Now I do have wifi results for this stuff - somewhere - and the right > tc filter for dst filtering on a per mq basis, but it turns out I > think I left all that behind a natted box that I can't get back to til > thursday next week. > > and as always I appreciate every scrap of data, every experiment, > every result obtained via every method, in order to more fully bracket > the real problems and demonstrate progress against wifi's problems, if > and when we start making it. a tarball of what you got would be nice > to have around. > > You will see absolutely terrible per-sta download performance on the > rrul and rrul_be tests in particular with any of the qdiscs. >> >> Any suggestions you have would be helpful. >> >> Thanks, >> Isaac >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat > > > > -- > Dave Täht > Let's make wifi fast, less jittery and reliable again! > > https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb -- Dave Täht Let's make wifi fast, less jittery and reliable again! https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-03-27 17:15 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-03-26 18:00 [Bloat] capturing packets and applying qdiscs Isaac Konikoff 2015-03-26 21:39 Isaac Konikoff 2015-03-27 0:39 ` David Lang 2015-03-27 16:38 ` Isaac Konikoff 2015-03-27 17:15 ` Aaron Wood 2015-03-27 1:19 ` Dave Taht 2015-03-27 1:37 ` Dave Taht
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox