[Cake] overhead for double nat VDSL2 connection

Cake - FQ_codel the next generation
 help / color / mirror / Atom feed

* [Cake] overhead for double nat VDSL2 connection
@ 2017-12-17 10:45 Mark Captur
  2017-12-17 15:23 ` Sebastian Moeller
  0 siblings, 1 reply; 7+ messages in thread
From: Mark Captur @ 2017-12-17 10:45 UTC (permalink / raw)
  To: cake

[-- Attachment #1: Type: text/plain, Size: 7855 bytes --]

My setup is as follows

vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2
(wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x

Here is my current SQM config

config queue 'eth1'
        option debug_logging '0'
        option verbosity '5'
        option qdisc 'cake'
        option qdisc_advanced '1'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option qdisc_really_really_advanced '1'
        option script 'layer_cake.qos'
        option interface 'eth0.2'
        option enabled '1'
        option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost
diffserv4'
        option upload '2400'
        option linklayer 'ethernet'
        option overhead '8'
        option squash_dscp '1'
        option squash_ingress '1'
        option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
        option download '0'

config queue
        option debug_logging '0'
        option verbosity '5'
        option download '0'
        option qdisc 'cake'
        option script 'layer_cake.qos'
        option qdisc_advanced '1'
        option squash_dscp '0'
        option squash_ingress '0'
        option ingress_ecn 'ECN'
        option qdisc_really_really_advanced '1'
        option egress_ecn 'ECN'
        option interface 'eth0.1'
        option enabled '1'
        option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost
diffserv4'
        option upload '30000'
        option linklayer 'ethernet'
        option overhead '8'


Is the overhead correct? should i use the bridged-ptm keyword (or should i
use pppoe-ptm). I tried using ATM_overhead_detector and get the following
output

869001 lines parsed...
Found 144921 ping packets in
/home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202.txt
Elapsed time is 972.739 seconds.
Minimum size of ping payload used: 16 bytes.
warning: division by zero
warning: called from
    ATM_overhead_detector at line 201 column 15
warning: legend: ignoring extra labels
Unknown or ambiguous terminal name 'wxt'
Unknown or ambiguous terminal name 'wxt'
Saved figure (5) to:
/home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_data.png
lower bound estimate for one ATM cell RTT based of specified up and
downlink is 0.052419 ms.
estimate for one ATM cell RTT based on linear fit of the ping sweep data is
0.052419 ms.
Starting brute-force search for optimal stair fit, might take a while...
Unknown or ambiguous terminal name 'wxt'
Unknown or ambiguous terminal name 'wxt'
Best staircase fit cumulative difference is: 2.6952
Best linear fit cumulative difference is: 2.7314
Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952 linear
fit 2.7314
remaining ATM cell length after ICMP header is 5 bytes.
ICMP RTT of a single ATM cell is 0.05871 ms.

Estimated overhead preceding the IP header: 48 bytes
Saved figure (6) to:
/home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_results.png

According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/
48 bytes overhead indicate
Overhead of 0 bytes is not possible so assume 1 full packet (48 bytes)
overhead...
Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag terminated at
modem
Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD [8] (0),
Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR
(8) : Total 36

Add the following to both the egress root qdisc:
A) Assuming the router connects over ethernet to the DSL-modem:
stab mtu 2048 tsize 128 overhead 48 linklayer atm

Add the following to both the ingress root qdisc:

A) Assuming the router connects over ethernet to the DSL-modem:
stab mtu 2048 tsize 128 overhead 48 linklayer atm

Elapsed time is 1042.06 seconds.
Done...
ans = [](0x0)



Could my VDSL2 be using ATM? so i have the wrong overhead set?

here is my tc -s qdisc output

root@OpenWrt-Mi3G:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0)
  maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4
dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22
hard_header_len 14
 Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 36684b of 4Mb
 capacity estimate: 30Mbit
                   Bulk  Best Effort        Video        Voice
  thresh       1875Kbit       30Mbit       15Mbit     7500Kbit
  target          9.7ms        2.5ms        2.5ms        2.5ms
  interval       57.2ms       50.0ms       50.0ms       50.0ms
  pk_delay          0us        5.6ms        855us         15us
  av_delay          0us        436us         28us         10us
  sp_delay          0us         27us          8us          6us
  pkts                0        33679          457         1346
  bytes               0     19869595        80517        69940
  way_inds            0          249            0            0
  way_miss            0         3099           19            7
  way_cols            0            0            0            0
  drops               0           10            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            1            1            0
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0         7064         2982          350

qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit diffserv4
dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22
hard_header_len 14
 Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 94752b of 4Mb
 capacity estimate: 2400Kbit
                   Bulk  Best Effort        Video        Voice
  thresh        150Kbit     2400Kbit     1200Kbit      600Kbit
  target        121.1ms        7.6ms       15.1ms       30.3ms
  interval      242.2ms       55.1ms       62.6ms       77.8ms
  pk_delay          0us       29.5ms         19us        4.4ms
  av_delay          0us        3.8ms          1us        2.3ms
  sp_delay          0us         50us          1us          9us
  pkts                0        34979           31         1099
  bytes               0     23342354         2930       162457
  way_inds            0          177            0            0
  way_miss            0         2003           27           16
  way_cols            0            0            0            0
  drops               0          205            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            1            0            0
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0         8342           98          598

qdisc noqueue 0: dev wlan1 root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc noqueue 0: dev wlan0 root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum
1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0


Thanks,
Mark

[-- Attachment #2: Type: text/html, Size: 12164 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [Cake] overhead for double nat VDSL2 connection
  2017-12-17 10:45 [Cake] overhead for double nat VDSL2 connection Mark Captur
@ 2017-12-17 15:23 ` Sebastian Moeller
  2017-12-17 16:00   ` Mark Captur
  2017-12-17 19:23   ` Kevin Darbyshire-Bryant
  0 siblings, 2 replies; 7+ messages in thread
From: Sebastian Moeller @ 2017-12-17 15:23 UTC (permalink / raw)
  To: Mark Captur; +Cc: cake

Hi Mark,

> On Dec 17, 2017, at 11:45, Mark Captur <mark.captur@gmail.com> wrote:
> 
> My setup is as follows
> 
> vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
> 
> Here is my current SQM config
> config queue 'eth1'
>         option debug_logging '0'
>         option verbosity '5'
>         option qdisc 'cake'
>         option qdisc_advanced '1'
>         option ingress_ecn 'ECN'
>         option egress_ecn 'NOECN'
>         option qdisc_really_really_advanced '1'
>         option script 'layer_cake.qos'
>         option interface 'eth0.2'
>         option enabled '1'
>         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost diffserv4'
>         option upload '2400'
>         option linklayer 'ethernet'
>         option overhead '8'
>         option squash_dscp '1'
>         option squash_ingress '1'
>         option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
>         option download '0'
> 
> config queue
>         option debug_logging '0'
>         option verbosity '5'
>         option download '0'
>         option qdisc 'cake'
>         option script 'layer_cake.qos'
>         option qdisc_advanced '1'
>         option squash_dscp '0'
>         option squash_ingress '0'
>         option ingress_ecn 'ECN'
>         option qdisc_really_really_advanced '1'
>         option egress_ecn 'ECN'
>         option interface 'eth0.1'
>         option enabled '1'
>         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost diffserv4'
>         option upload '30000'
>         option linklayer 'ethernet'
>         option overhead '8'
> 
> Is the overhead correct? should i use the bridged-ptm keyword (or should i use pppoe-ptm).

	The overhead certainly seems confusing. Personally, I dislike the overhead related compound keywords like *-ptm and would recommend the following:
1) remove the bridged-ptm from the eqdisc/iqdisc fields
2) add "mpu 64" to the eqdisc/iqdisc fields
3) set overhad to 8+18+4 = 30 bytes (you will need to account for everything added on the bottleneck, so your packets will be MTU 1492, to leave room for the PPPoE header that the modem adds).
4) DO not set the ptm keyword at all, instead make sure to set the shaper bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring needless operations per packet).

4) tell us about your ISP and plan ;)




> I tried using ATM_overhead_detector and get the following output

	Well, yes sorry, this will not work at all for links using PTM; in theory VDSL links could be using ATM and still be in compliance with the ITU specs (at lest that is my reading) but fortunately nobody seems to be doing that. (Also more unfortunately no ISP sseems to be using PTM encoding on ADSL links, but I digress).

> 
> 869001 lines parsed...
> Found 144921 ping packets in /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202.txt
> Elapsed time is 972.739 seconds.
> Minimum size of ping payload used: 16 bytes.
> warning: division by zero
> warning: called from
>     ATM_overhead_detector at line 201 column 15
> warning: legend: ignoring extra labels
> Unknown or ambiguous terminal name 'wxt'
> Unknown or ambiguous terminal name 'wxt'
> Saved figure (5) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_data.png
> lower bound estimate for one ATM cell RTT based of specified up and downlink is 0.052419 ms.
> estimate for one ATM cell RTT based on linear fit of the ping sweep data is 0.052419 ms.
> Starting brute-force search for optimal stair fit, might take a while...
> Unknown or ambiguous terminal name 'wxt'
> Unknown or ambiguous terminal name 'wxt'
> Best staircase fit cumulative difference is: 2.6952
> Best linear fit cumulative difference is: 2.7314
> Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952 linear fit 2.7314

	Yes, that difference is simply too small to be meaningful. I am currently collecting reference values for these for different technologies and it seems on real ATM/AAL5 links linear residuals are >= 2 * stair residuals, but curently the code simply reports the smaller value as more likely...


> remaining ATM cell length after ICMP header is 5 bytes.
> ICMP RTT of a single ATM cell is 0.05871 ms.
> 
> Estimated overhead preceding the IP header: 48 bytes
> Saved figure (6) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_results.png
> 
> According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/
> 48 bytes overhead indicate
> Overhead of 0 bytes is not possible so assume 1 full packet (48 bytes) overhead...
> Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag terminated at modem
> Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD [8] (0), Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 36
> 
> Add the following to both the egress root qdisc:
> A) Assuming the router connects over ethernet to the DSL-modem:
> stab mtu 2048 tsize 128 overhead 48 linklayer atm
> 
> Add the following to both the ingress root qdisc:
> 
> A) Assuming the router connects over ethernet to the DSL-modem:
> stab mtu 2048 tsize 128 overhead 48 linklayer atm
> 
> Elapsed time is 1042.06 seconds.
> Done...
> ans = [](0x0)
> 
> 
> Could my VDSL2 be using ATM? so i have the wrong overhead set?

	Yes in theory it could, but I am almost 100% certain it is not doing that (AFAICT ISPs try to get rid of all ATM equipment ASAP). But I also believe you have the wrong overhead specified... (if you insist on the keywords try pppoe-ptm). Now, please keep in mind that you properly need to "model" the encapsulation on the bottleneck link, if your ISP uses a traffic shaper on the otherside of your link you need to actually model that shapers settings. As far as I can see, ISPs will model the per packet overhead roughly correctly, but will reduce the bandwidth from the sync values reported by the modem, this requires a few iterations of trial and error to get right. Also some ISPs also use VLAN tags on the bottleneck link...

Hope this helps...

Best Regards
	Sebastian


> 
> here is my tc -s qdisc output
> 
> root@OpenWrt-Mi3G:~# tc -s qdisc
> qdisc noqueue 0: dev lo root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
>  Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0)
>   maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> qdisc noqueue 0: dev br-lan root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4 dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
>  Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 36684b of 4Mb
>  capacity estimate: 30Mbit
>                    Bulk  Best Effort        Video        Voice
>   thresh       1875Kbit       30Mbit       15Mbit     7500Kbit
>   target          9.7ms        2.5ms        2.5ms        2.5ms
>   interval       57.2ms       50.0ms       50.0ms       50.0ms
>   pk_delay          0us        5.6ms        855us         15us
>   av_delay          0us        436us         28us         10us
>   sp_delay          0us         27us          8us          6us
>   pkts                0        33679          457         1346
>   bytes               0     19869595        80517        69940
>   way_inds            0          249            0            0
>   way_miss            0         3099           19            7
>   way_cols            0            0            0            0
>   drops               0           10            0            0
>   marks               0            0            0            0
>   ack_drop            0            0            0            0
>   sp_flows            0            1            1            0
>   bk_flows            0            1            0            0
>   un_flows            0            0            0            0
>   max_len             0         7064         2982          350
> 
> qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit diffserv4 dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
>  Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 94752b of 4Mb
>  capacity estimate: 2400Kbit
>                    Bulk  Best Effort        Video        Voice
>   thresh        150Kbit     2400Kbit     1200Kbit      600Kbit
>   target        121.1ms        7.6ms       15.1ms       30.3ms
>   interval      242.2ms       55.1ms       62.6ms       77.8ms
>   pk_delay          0us       29.5ms         19us        4.4ms
>   av_delay          0us        3.8ms          1us        2.3ms
>   sp_delay          0us         50us          1us          9us
>   pkts                0        34979           31         1099
>   bytes               0     23342354         2930       162457
>   way_inds            0          177            0            0
>   way_miss            0         2003           27           16
>   way_cols            0            0            0            0
>   drops               0          205            0            0
>   marks               0            0            0            0
>   ack_drop            0            0            0            0
>   sp_flows            0            1            0            0
>   bk_flows            0            1            0            0
>   un_flows            0            0            0            0
>   max_len             0         8342           98          598
> 
> qdisc noqueue 0: dev wlan1 root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc noqueue 0: dev wlan0 root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
>  Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> 
> Thanks,
> Mark
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [Cake] overhead for double nat VDSL2 connection
  2017-12-17 15:23 ` Sebastian Moeller
@ 2017-12-17 16:00   ` Mark Captur
  2017-12-17 22:30     ` Sebastian Moeller
  2017-12-17 19:23   ` Kevin Darbyshire-Bryant
  1 sibling, 1 reply; 7+ messages in thread
From: Mark Captur @ 2017-12-17 16:00 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: cake

[-- Attachment #1: Type: text/plain, Size: 16771 bytes --]

Thanks for the comprehensive and quick reply :) below are some further
answers

On 17 December 2017 at 16:23, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Mark,
>
> > On Dec 17, 2017, at 11:45, Mark Captur <mark.captur@gmail.com> wrote:
> >
> > My setup is as follows
> >
> > vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2
> (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
> >
> > Here is my current SQM config
> > config queue 'eth1'
> >         option debug_logging '0'
> >         option verbosity '5'
> >         option qdisc 'cake'
> >         option qdisc_advanced '1'
> >         option ingress_ecn 'ECN'
> >         option egress_ecn 'NOECN'
> >         option qdisc_really_really_advanced '1'
> >         option script 'layer_cake.qos'
> >         option interface 'eth0.2'
> >         option enabled '1'
> >         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost
> diffserv4'
> >         option upload '2400'
> >         option linklayer 'ethernet'
> >         option overhead '8'
> >         option squash_dscp '1'
> >         option squash_ingress '1'
> >         option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
> >         option download '0'
> >
> > config queue
> >         option debug_logging '0'
> >         option verbosity '5'
> >         option download '0'
> >         option qdisc 'cake'
> >         option script 'layer_cake.qos'
> >         option qdisc_advanced '1'
> >         option squash_dscp '0'
> >         option squash_ingress '0'
> >         option ingress_ecn 'ECN'
> >         option qdisc_really_really_advanced '1'
> >         option egress_ecn 'ECN'
> >         option interface 'eth0.1'
> >         option enabled '1'
> >         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost
> diffserv4'
> >         option upload '30000'
> >         option linklayer 'ethernet'
> >         option overhead '8'
> >
> > Is the overhead correct? should i use the bridged-ptm keyword (or should
> i use pppoe-ptm).
>
>         The overhead certainly seems confusing. Personally, I dislike the
> overhead related compound keywords like *-ptm and would recommend the
> following:
> 1) remove the bridged-ptm from the eqdisc/iqdisc fields
>
Done

> 2) add "mpu 64" to the eqdisc/iqdisc fields
>
Done

> 3) set overhad to 8+18+4 = 30 bytes (you will need to account for
> everything added on the bottleneck, so your packets will be MTU 1492, to
> leave room for the PPPoE header that the modem adds).
>
Dne

> 4) DO not set the ptm keyword at all, instead make sure to set the shaper
> bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615
> (to account for ptm's 64/65 encoding _without_ incurring needless
> operations per packet).
>
Done  0.98 (2440 is .98 of 2480) for upstream but had to reduce to .95 for
downstream as other wise ping go very high when line saturated


root@OpenWrt-Mi3G:~# cat /etc/config/sqm

config queue 'eth1'
        option debug_logging '0'
        option verbosity '5'
        option qdisc 'cake'
        option qdisc_advanced '1'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option qdisc_really_really_advanced '1'
        option script 'layer_cake.qos'
        option interface 'eth0.2'
        option enabled '1'
        option linklayer 'ethernet'
        option squash_dscp '1'
        option squash_ingress '1'
        option download '0'
        option upload '2440'
        option overhead '30'
        option iqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost'
        option eqdisc_opts 'nat rtt 50000 mpu 64 dual-srchost diffserv4'

config queue
        option debug_logging '0'
        option verbosity '5'
        option download '0'
        option qdisc 'cake'
        option script 'layer_cake.qos'
        option qdisc_advanced '1'
        option squash_dscp '0'
        option squash_ingress '0'
        option ingress_ecn 'ECN'
        option qdisc_really_really_advanced '1'
        option egress_ecn 'ECN'
        option interface 'eth0.1'
        option enabled '1'
        option linklayer 'ethernet'
        option overhead '30'
        option eqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost diffserv4'
        option upload '30500'


4) tell us about your ISP and plan ;)

ISP is Go Malta VDSL2 70/10, however i'm very far from cabinet so i sync at
32/2.48

>
>
>
> > I tried using ATM_overhead_detector and get the following output
>
>         Well, yes sorry, this will not work at all for links using PTM; in
> theory VDSL links could be using ATM and still be in compliance with the
> ITU specs (at lest that is my reading) but fortunately nobody seems to be
> doing that. (Also more unfortunately no ISP sseems to be using PTM encoding
> on ADSL links, but I digress).
>
> >
> > 869001 lines parsed...
> > Found 144921 ping packets in /home/mark/src/ATM_overhead_
> detector/ping_sweep__20171217_075202.txt
> > Elapsed time is 972.739 seconds.
> > Minimum size of ping payload used: 16 bytes.
> > warning: division by zero
> > warning: called from
> >     ATM_overhead_detector at line 201 column 15
> > warning: legend: ignoring extra labels
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Saved figure (5) to: /home/mark/src/ATM_overhead_
> detector/ping_sweep__20171217_075202_data.png
> > lower bound estimate for one ATM cell RTT based of specified up and
> downlink is 0.052419 ms.
> > estimate for one ATM cell RTT based on linear fit of the ping sweep data
> is 0.052419 ms.
> > Starting brute-force search for optimal stair fit, might take a while...
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Best staircase fit cumulative difference is: 2.6952
> > Best linear fit cumulative difference is: 2.7314
> > Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952
> linear fit 2.7314
>
>         Yes, that difference is simply too small to be meaningful. I am
> currently collecting reference values for these for different technologies
> and it seems on real ATM/AAL5 links linear residuals are >= 2 * stair
> residuals, but curently the code simply reports the smaller value as more
> likely...
>
>
> > remaining ATM cell length after ICMP header is 5 bytes.
> > ICMP RTT of a single ATM cell is 0.05871 ms.
> >
> > Estimated overhead preceding the IP header: 48 bytes
> > Saved figure (6) to: /home/mark/src/ATM_overhead_detector/ping_sweep__
> 20171217_075202_results.png
> >
> > According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/
> > 48 bytes overhead indicate
> > Overhead of 0 bytes is not possible so assume 1 full packet (48 bytes)
> overhead...
> > Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag terminated at
> modem
> > Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD [8]
> (0), Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM
> AAL5 SAR (8) : Total 36
> >
> > Add the following to both the egress root qdisc:
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Add the following to both the ingress root qdisc:
> >
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Elapsed time is 1042.06 seconds.
> > Done...
> > ans = [](0x0)
> >
> >
> > Could my VDSL2 be using ATM? so i have the wrong overhead set?
>
>         Yes in theory it could, but I am almost 100% certain it is not
> doing that (AFAICT ISPs try to get rid of all ATM equipment ASAP). But I
> also believe you have the wrong overhead specified... (if you insist on the
> keywords try pppoe-ptm). Now, please keep in mind that you properly need to
> "model" the encapsulation on the bottleneck link, if your ISP uses a
> traffic shaper on the otherside of your link you need to actually model
> that shapers settings. As far as I can see, ISPs will model the per packet
> overhead roughly correctly, but will reduce the bandwidth from the sync
> values reported by the modem, this requires a few iterations of trial and
> error to get right. Also some ISPs also use VLAN tags on the bottleneck
> link...
>
> Hope this helps...
>
> Best Regards
>         Sebastian
>
> Here is tc output
>

root@OpenWrt-Mi3G:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 13974072314 bytes 14402910 pkts (dropped 0, overlimits 0)
  maxpacket 17032 drop_overlimit 0 new_flow_count 132368 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc cake 8069: dev eth0.1 root refcnt 2 bandwidth 31500Kbit diffserv4
dual-dsthost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
 Sent 89833020 bytes 59632 pkt (dropped 5, overlimits 44617 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 14112b of 4Mb
 capacity estimate: 31500Kbit
                   Bulk  Best Effort        Video        Voice
  thresh       1968Kbit    31500Kbit    15750Kbit     7875Kbit
  target          9.2ms        2.5ms        2.5ms        2.5ms
  interval       56.7ms       50.0ms       50.0ms       50.0ms
  pk_delay          7us        4.3ms        3.5ms        3.2ms
  av_delay          0us        183us        173us        271us
  sp_delay          0us          7us          8us          7us
  pkts                4          408        20384        38841
  bytes             717        64851     30954992     58820110
  way_inds            0            0            0            0
  way_miss            4           64            4           11
  way_cols            0            0            0            0
  drops               0            0            5            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            0            0            0
  bk_flows            0            0            1            0
  un_flows            0            0            0            0
  max_len           189         1536         2988         2988

qdisc cake 8067: dev eth0.2 root refcnt 2 bandwidth 2440Kbit diffserv4
dual-srchost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
 Sent 3780967 bytes 33806 pkt (dropped 130, overlimits 12207 requeues 0)
 backlog 10860b 9p requeues 0
 memory used: 116928b of 4Mb
 capacity estimate: 2440Kbit
                   Bulk  Best Effort        Video        Voice
  thresh      152496bit     2440Kbit     1220Kbit      610Kbit
  target        119.1ms        7.4ms       14.9ms       29.8ms
  interval      238.3ms       54.9ms       62.4ms       77.3ms
  pk_delay          0us      101.5ms        8.2ms          7us
  av_delay          0us       49.7ms        2.6ms          0us
  sp_delay          0us        6.8ms        245us          0us
  pkts                0        22757        11178           10
  bytes               0      3029896       951692         1096
  way_inds            0            0            0            0
  way_miss            0           61            3            3
  way_cols            0            0            0            0
  drops               0          130            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            1            1            0
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0        11700          120          179

qdisc noqueue 0: dev wlan1 root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc noqueue 0: dev wlan0 root refcnt 2
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum
1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0

Best regards,
Mark

> >
> > here is my tc -s qdisc output
> >
> > root@OpenWrt-Mi3G:~# tc -s qdisc
> > qdisc noqueue 0: dev lo root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum
> 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> >  Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0)
> >   maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0
> >   new_flows_len 0 old_flows_len 0
> > qdisc noqueue 0: dev br-lan root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4
> dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22
> hard_header_len 14
> >  Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues 0)
> >  backlog 0b 0p requeues 0
> >  memory used: 36684b of 4Mb
> >  capacity estimate: 30Mbit
> >                    Bulk  Best Effort        Video        Voice
> >   thresh       1875Kbit       30Mbit       15Mbit     7500Kbit
> >   target          9.7ms        2.5ms        2.5ms        2.5ms
> >   interval       57.2ms       50.0ms       50.0ms       50.0ms
> >   pk_delay          0us        5.6ms        855us         15us
> >   av_delay          0us        436us         28us         10us
> >   sp_delay          0us         27us          8us          6us
> >   pkts                0        33679          457         1346
> >   bytes               0     19869595        80517        69940
> >   way_inds            0          249            0            0
> >   way_miss            0         3099           19            7
> >   way_cols            0            0            0            0
> >   drops               0           10            0            0
> >   marks               0            0            0            0
> >   ack_drop            0            0            0            0
> >   sp_flows            0            1            1            0
> >   bk_flows            0            1            0            0
> >   un_flows            0            0            0            0
> >   max_len             0         7064         2982          350
> >
> > qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit diffserv4
> dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22
> hard_header_len 14
> >  Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 requeues 0)
> >  backlog 0b 0p requeues 0
> >  memory used: 94752b of 4Mb
> >  capacity estimate: 2400Kbit
> >                    Bulk  Best Effort        Video        Voice
> >   thresh        150Kbit     2400Kbit     1200Kbit      600Kbit
> >   target        121.1ms        7.6ms       15.1ms       30.3ms
> >   interval      242.2ms       55.1ms       62.6ms       77.8ms
> >   pk_delay          0us       29.5ms         19us        4.4ms
> >   av_delay          0us        3.8ms          1us        2.3ms
> >   sp_delay          0us         50us          1us          9us
> >   pkts                0        34979           31         1099
> >   bytes               0     23342354         2930       162457
> >   way_inds            0          177            0            0
> >   way_miss            0         2003           27           16
> >   way_cols            0            0            0            0
> >   drops               0          205            0            0
> >   marks               0            0            0            0
> >   ack_drop            0            0            0            0
> >   sp_flows            0            1            0            0
> >   bk_flows            0            1            0            0
> >   un_flows            0            0            0            0
> >   max_len             0         8342           98          598
> >
> > qdisc noqueue 0: dev wlan1 root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc noqueue 0: dev wlan0 root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum
> 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> >  Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
> >   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
> >   new_flows_len 0 old_flows_len 0
> >
> > Thanks,
> > Mark
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 23529 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [Cake] overhead for double nat VDSL2 connection
  2017-12-17 16:00   ` Mark Captur
@ 2017-12-17 22:30     ` Sebastian Moeller
  0 siblings, 0 replies; 7+ messages in thread
From: Sebastian Moeller @ 2017-12-17 22:30 UTC (permalink / raw)
  To: Mark Captur; +Cc: cake

HI Mark,


> On Dec 17, 2017, at 17:00, Mark Captur <mark.captur@gmail.com> wrote:
> 
> Thanks for the comprehensive and quick reply :) below are some further answers

	Always happy to help.

> 
> On 17 December 2017 at 16:23, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Mark,
> 
> > On Dec 17, 2017, at 11:45, Mark Captur <mark.captur@gmail.com> wrote:
> >
> > My setup is as follows
> >
> > vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
> >
> > Here is my current SQM config
> > config queue 'eth1'
> >         option debug_logging '0'
> >         option verbosity '5'
> >         option qdisc 'cake'
> >         option qdisc_advanced '1'
> >         option ingress_ecn 'ECN'
> >         option egress_ecn 'NOECN'
> >         option qdisc_really_really_advanced '1'
> >         option script 'layer_cake.qos'
> >         option interface 'eth0.2'
> >         option enabled '1'
> >         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost diffserv4'
> >         option upload '2400'
> >         option linklayer 'ethernet'
> >         option overhead '8'
> >         option squash_dscp '1'
> >         option squash_ingress '1'
> >         option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
> >         option download '0'
> >
> > config queue
> >         option debug_logging '0'
> >         option verbosity '5'
> >         option download '0'
> >         option qdisc 'cake'
> >         option script 'layer_cake.qos'
> >         option qdisc_advanced '1'
> >         option squash_dscp '0'
> >         option squash_ingress '0'
> >         option ingress_ecn 'ECN'
> >         option qdisc_really_really_advanced '1'
> >         option egress_ecn 'ECN'
> >         option interface 'eth0.1'
> >         option enabled '1'
> >         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost diffserv4'
> >         option upload '30000'
> >         option linklayer 'ethernet'
> >         option overhead '8'
> >
> > Is the overhead correct? should i use the bridged-ptm keyword (or should i use pppoe-ptm).
> 
>         The overhead certainly seems confusing. Personally, I dislike the overhead related compound keywords like *-ptm and would recommend the following:
> 1) remove the bridged-ptm from the eqdisc/iqdisc fields
> Done 
> 2) add "mpu 64" to the eqdisc/iqdisc fields
> Done 
> 3) set overhad to 8+18+4 = 30 bytes (you will need to account for everything added on the bottleneck, so your packets will be MTU 1492, to leave room for the PPPoE header that the modem adds).
> Dne 
> 4) DO not set the ptm keyword at all, instead make sure to set the shaper bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring needless operations per packet).
> Done  0.98 (2440 is .98 of 2480) for upstream but had to reduce to .95 for downstream as other wise ping go very high when line saturated

	Sure, the 98.46% of sync is a theoretical upper limit and not a really reachable for ingress (but you might want tto try the new "ingress" keyword on eth0.1)

>  
> 
> root@OpenWrt-Mi3G:~# cat /etc/config/sqm
> 
> config queue 'eth1'
>         option debug_logging '0'
>         option verbosity '5'
>         option qdisc 'cake'
>         option qdisc_advanced '1'
>         option ingress_ecn 'ECN'
>         option egress_ecn 'NOECN'
>         option qdisc_really_really_advanced '1'
>         option script 'layer_cake.qos'
>         option interface 'eth0.2'
>         option enabled '1'
>         option linklayer 'ethernet'
>         option squash_dscp '1'
>         option squash_ingress '1'
>         option download '0'
>         option upload '2440'
>         option overhead '30'
>         option iqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost'
>         option eqdisc_opts 'nat rtt 50000 mpu 64 dual-srchost diffserv4'
> 
> config queue
>         option debug_logging '0'
>         option verbosity '5'
>         option download '0'
>         option qdisc 'cake'
>         option script 'layer_cake.qos'
>         option qdisc_advanced '1'
>         option squash_dscp '0'
>         option squash_ingress '0'
>         option ingress_ecn 'ECN'
>         option qdisc_really_really_advanced '1'
>         option egress_ecn 'ECN'
>         option interface 'eth0.1'
>         option enabled '1'
>         option linklayer 'ethernet'
>         option overhead '30'
>         option eqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost diffserv4'
>         option upload '30500'
> 
> 
> 4) tell us about your ISP and plan ;)
> ISP is Go Malta VDSL2 70/10, however i'm very far from cabinet so i sync at 32/2.48 

	Thanks, I had a look, but i could not easily whether they use a VLAN on their vdsl2 link.

Best Regards
	Sebastian
	
> 
> 
> 
> > I tried using ATM_overhead_detector and get the following output
> 
>         Well, yes sorry, this will not work at all for links using PTM; in theory VDSL links could be using ATM and still be in compliance with the ITU specs (at lest that is my reading) but fortunately nobody seems to be doing that. (Also more unfortunately no ISP sseems to be using PTM encoding on ADSL links, but I digress).
> 
> >
> > 869001 lines parsed...
> > Found 144921 ping packets in /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202.txt
> > Elapsed time is 972.739 seconds.
> > Minimum size of ping payload used: 16 bytes.
> > warning: division by zero
> > warning: called from
> >     ATM_overhead_detector at line 201 column 15
> > warning: legend: ignoring extra labels
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Saved figure (5) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_data.png
> > lower bound estimate for one ATM cell RTT based of specified up and downlink is 0.052419 ms.
> > estimate for one ATM cell RTT based on linear fit of the ping sweep data is 0.052419 ms.
> > Starting brute-force search for optimal stair fit, might take a while...
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Best staircase fit cumulative difference is: 2.6952
> > Best linear fit cumulative difference is: 2.7314
> > Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952 linear fit 2.7314
> 
>         Yes, that difference is simply too small to be meaningful. I am currently collecting reference values for these for different technologies and it seems on real ATM/AAL5 links linear residuals are >= 2 * stair residuals, but curently the code simply reports the smaller value as more likely...
> 
> 
> > remaining ATM cell length after ICMP header is 5 bytes.
> > ICMP RTT of a single ATM cell is 0.05871 ms.
> >
> > Estimated overhead preceding the IP header: 48 bytes
> > Saved figure (6) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_results.png
> >
> > According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/
> > 48 bytes overhead indicate
> > Overhead of 0 bytes is not possible so assume 1 full packet (48 bytes) overhead...
> > Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag terminated at modem
> > Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD [8] (0), Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 36
> >
> > Add the following to both the egress root qdisc:
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Add the following to both the ingress root qdisc:
> >
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Elapsed time is 1042.06 seconds.
> > Done...
> > ans = [](0x0)
> >
> >
> > Could my VDSL2 be using ATM? so i have the wrong overhead set?
> 
>         Yes in theory it could, but I am almost 100% certain it is not doing that (AFAICT ISPs try to get rid of all ATM equipment ASAP). But I also believe you have the wrong overhead specified... (if you insist on the keywords try pppoe-ptm). Now, please keep in mind that you properly need to "model" the encapsulation on the bottleneck link, if your ISP uses a traffic shaper on the otherside of your link you need to actually model that shapers settings. As far as I can see, ISPs will model the per packet overhead roughly correctly, but will reduce the bandwidth from the sync values reported by the modem, this requires a few iterations of trial and error to get right. Also some ISPs also use VLAN tags on the bottleneck link...
> 
> Hope this helps...
> 
> Best Regards
>         Sebastian
> 
> Here is tc output
> 
> root@OpenWrt-Mi3G:~# tc -s qdisc
> qdisc noqueue 0: dev lo root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
>  Sent 13974072314 bytes 14402910 pkts (dropped 0, overlimits 0)
>   maxpacket 17032 drop_overlimit 0 new_flow_count 132368 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> qdisc noqueue 0: dev br-lan root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc cake 8069: dev eth0.1 root refcnt 2 bandwidth 31500Kbit diffserv4 dual-dsthost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
>  Sent 89833020 bytes 59632 pkt (dropped 5, overlimits 44617 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 14112b of 4Mb
>  capacity estimate: 31500Kbit
>                    Bulk  Best Effort        Video        Voice
>   thresh       1968Kbit    31500Kbit    15750Kbit     7875Kbit
>   target          9.2ms        2.5ms        2.5ms        2.5ms
>   interval       56.7ms       50.0ms       50.0ms       50.0ms
>   pk_delay          7us        4.3ms        3.5ms        3.2ms
>   av_delay          0us        183us        173us        271us
>   sp_delay          0us          7us          8us          7us
>   pkts                4          408        20384        38841
>   bytes             717        64851     30954992     58820110
>   way_inds            0            0            0            0
>   way_miss            4           64            4           11
>   way_cols            0            0            0            0
>   drops               0            0            5            0
>   marks               0            0            0            0
>   ack_drop            0            0            0            0
>   sp_flows            0            0            0            0
>   bk_flows            0            0            1            0
>   un_flows            0            0            0            0
>   max_len           189         1536         2988         2988
> 
> qdisc cake 8067: dev eth0.2 root refcnt 2 bandwidth 2440Kbit diffserv4 dual-srchost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
>  Sent 3780967 bytes 33806 pkt (dropped 130, overlimits 12207 requeues 0)
>  backlog 10860b 9p requeues 0
>  memory used: 116928b of 4Mb
>  capacity estimate: 2440Kbit
>                    Bulk  Best Effort        Video        Voice
>   thresh      152496bit     2440Kbit     1220Kbit      610Kbit
>   target        119.1ms        7.4ms       14.9ms       29.8ms
>   interval      238.3ms       54.9ms       62.4ms       77.3ms
>   pk_delay          0us      101.5ms        8.2ms          7us
>   av_delay          0us       49.7ms        2.6ms          0us
>   sp_delay          0us        6.8ms        245us          0us
>   pkts                0        22757        11178           10
>   bytes               0      3029896       951692         1096
>   way_inds            0            0            0            0
>   way_miss            0           61            3            3
>   way_cols            0            0            0            0
>   drops               0          130            0            0
>   marks               0            0            0            0
>   ack_drop            0            0            0            0
>   sp_flows            0            1            1            0
>   bk_flows            0            1            0            0
>   un_flows            0            0            0            0
>   max_len             0        11700          120          179
> 
> qdisc noqueue 0: dev wlan1 root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc noqueue 0: dev wlan0 root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
>  Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> 
> Best regards,
> Mark 
> >
> > here is my tc -s qdisc output
> >
> > root@OpenWrt-Mi3G:~# tc -s qdisc
> > qdisc noqueue 0: dev lo root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> >  Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0)
> >   maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0
> >   new_flows_len 0 old_flows_len 0
> > qdisc noqueue 0: dev br-lan root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4 dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
> >  Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues 0)
> >  backlog 0b 0p requeues 0
> >  memory used: 36684b of 4Mb
> >  capacity estimate: 30Mbit
> >                    Bulk  Best Effort        Video        Voice
> >   thresh       1875Kbit       30Mbit       15Mbit     7500Kbit
> >   target          9.7ms        2.5ms        2.5ms        2.5ms
> >   interval       57.2ms       50.0ms       50.0ms       50.0ms
> >   pk_delay          0us        5.6ms        855us         15us
> >   av_delay          0us        436us         28us         10us
> >   sp_delay          0us         27us          8us          6us
> >   pkts                0        33679          457         1346
> >   bytes               0     19869595        80517        69940
> >   way_inds            0          249            0            0
> >   way_miss            0         3099           19            7
> >   way_cols            0            0            0            0
> >   drops               0           10            0            0
> >   marks               0            0            0            0
> >   ack_drop            0            0            0            0
> >   sp_flows            0            1            1            0
> >   bk_flows            0            1            0            0
> >   un_flows            0            0            0            0
> >   max_len             0         7064         2982          350
> >
> > qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit diffserv4 dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
> >  Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 requeues 0)
> >  backlog 0b 0p requeues 0
> >  memory used: 94752b of 4Mb
> >  capacity estimate: 2400Kbit
> >                    Bulk  Best Effort        Video        Voice
> >   thresh        150Kbit     2400Kbit     1200Kbit      600Kbit
> >   target        121.1ms        7.6ms       15.1ms       30.3ms
> >   interval      242.2ms       55.1ms       62.6ms       77.8ms
> >   pk_delay          0us       29.5ms         19us        4.4ms
> >   av_delay          0us        3.8ms          1us        2.3ms
> >   sp_delay          0us         50us          1us          9us
> >   pkts                0        34979           31         1099
> >   bytes               0     23342354         2930       162457
> >   way_inds            0          177            0            0
> >   way_miss            0         2003           27           16
> >   way_cols            0            0            0            0
> >   drops               0          205            0            0
> >   marks               0            0            0            0
> >   ack_drop            0            0            0            0
> >   sp_flows            0            1            0            0
> >   bk_flows            0            1            0            0
> >   un_flows            0            0            0            0
> >   max_len             0         8342           98          598
> >
> > qdisc noqueue 0: dev wlan1 root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc noqueue 0: dev wlan0 root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> >  Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
> >   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
> >   new_flows_len 0 old_flows_len 0
> >
> > Thanks,
> > Mark
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [Cake] overhead for double nat VDSL2 connection
  2017-12-17 15:23 ` Sebastian Moeller
  2017-12-17 16:00   ` Mark Captur
@ 2017-12-17 19:23   ` Kevin Darbyshire-Bryant
  2017-12-17 22:35     ` Sebastian Moeller
  1 sibling, 1 reply; 7+ messages in thread
From: Kevin Darbyshire-Bryant @ 2017-12-17 19:23 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: Mark Captur, cake

[-- Attachment #1: Type: text/plain, Size: 3701 bytes --]



> On 17 Dec 2017, at 15:23, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi Mark,
> 
>> On Dec 17, 2017, at 11:45, Mark Captur <mark.captur@gmail.com> wrote:
>> 
>> My setup is as follows
>> 
>> vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
>> 
>> Here is my current SQM config
>> config queue 'eth1'
>>        option debug_logging '0'
>>        option verbosity '5'
>>        option qdisc 'cake'
>>        option qdisc_advanced '1'
>>        option ingress_ecn 'ECN'
>>        option egress_ecn 'NOECN'
>>        option qdisc_really_really_advanced '1'
>>        option script 'layer_cake.qos'
>>        option interface 'eth0.2'
>>        option enabled '1'
>>        option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost diffserv4'
>>        option upload '2400'
>>        option linklayer 'ethernet'
>>        option overhead '8'
>>        option squash_dscp '1'
>>        option squash_ingress '1'
>>        option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
>>        option download '0'
>> 
>> config queue
>>        option debug_logging '0'
>>        option verbosity '5'
>>        option download '0'
>>        option qdisc 'cake'
>>        option script 'layer_cake.qos'
>>        option qdisc_advanced '1'
>>        option squash_dscp '0'
>>        option squash_ingress '0'
>>        option ingress_ecn 'ECN'
>>        option qdisc_really_really_advanced '1'
>>        option egress_ecn 'ECN'
>>        option interface 'eth0.1'
>>        option enabled '1'
>>        option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost diffserv4'
>>        option upload '30000'
>>        option linklayer 'ethernet'
>>        option overhead '8'
>> 
>> Is the overhead correct? should i use the bridged-ptm keyword (or should i use pppoe-ptm).


Beware of using  option linklayer ‘ethernet’ without option linklayer_advanced ‘1’ & option linklayer_adaptation_mechanism ‘default’.  Or use linklayer ‘none’.  Failure to do so will make sqm-scripts use STAB for the link accounting (in essence it lies to cake about the size of packets being passed through it).  Better to use cake’s built-in compensation - fewer modules, less code.

I was bitten by this myself very recently, lost 4 hours of my life & many recompiles before I realised an innocent looking setting (linklayer_advanced) was messing with the packet size (seen by looking at max_len from tc)

> 
> 	The overhead certainly seems confusing. Personally, I dislike the overhead related compound keywords like *-ptm and would recommend the following:
> 1) remove the bridged-ptm from the eqdisc/iqdisc fields
> 2) add "mpu 64" to the eqdisc/iqdisc fields
> 3) set overhad to 8+18+4 = 30 bytes (you will need to account for everything added on the bottleneck, so your packets will be MTU 1492, to leave room for the PPPoE header that the modem adds).
MTU 1492 ain’t necessarily so - some vdsl modems in bridge mode support RFC 4638 with baby jumbo frames, thus supporting the normal ethernet MTU of 1500 (and obviating the need to TCP MSS clamping) - ie. if you can do it, you should ;-).  You still need to account for the 8 byte PPPOE overhead of course.
> 4) DO not set the ptm keyword at all, instead make sure to set the shaper bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring needless operations per packet).
> 
> 4) tell us about your ISP and plan ;)
> 
Cheers,

Kevin D-B

GPG fingerprint: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [Cake] overhead for double nat VDSL2 connection
  2017-12-17 19:23   ` Kevin Darbyshire-Bryant
@ 2017-12-17 22:35     ` Sebastian Moeller
  2017-12-18  3:56       ` Mark Captur
  0 siblings, 1 reply; 7+ messages in thread
From: Sebastian Moeller @ 2017-12-17 22:35 UTC (permalink / raw)
  To: Kevin Darbyshire-Bryant; +Cc: Mark Captur, cake

Hi Kevin, hi Mark,

Kevin is 100% right!

> On Dec 17, 2017, at 20:23, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> 
> 
>> On 17 Dec 2017, at 15:23, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Mark,
>> 
>>> On Dec 17, 2017, at 11:45, Mark Captur <mark.captur@gmail.com> wrote:
>>> 
>>> My setup is as follows
>>> 
>>> vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
>>> 
>>> Here is my current SQM config
>>> config queue 'eth1'
>>>       option debug_logging '0'
>>>       option verbosity '5'
>>>       option qdisc 'cake'
>>>       option qdisc_advanced '1'
>>>       option ingress_ecn 'ECN'
>>>       option egress_ecn 'NOECN'
>>>       option qdisc_really_really_advanced '1'
>>>       option script 'layer_cake.qos'
>>>       option interface 'eth0.2'
>>>       option enabled '1'
>>>       option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost diffserv4'
>>>       option upload '2400'
>>>       option linklayer 'ethernet'
>>>       option overhead '8'
>>>       option squash_dscp '1'
>>>       option squash_ingress '1'
>>>       option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
>>>       option download '0'
>>> 
>>> config queue
>>>       option debug_logging '0'
>>>       option verbosity '5'
>>>       option download '0'
>>>       option qdisc 'cake'
>>>       option script 'layer_cake.qos'
>>>       option qdisc_advanced '1'
>>>       option squash_dscp '0'
>>>       option squash_ingress '0'
>>>       option ingress_ecn 'ECN'
>>>       option qdisc_really_really_advanced '1'
>>>       option egress_ecn 'ECN'
>>>       option interface 'eth0.1'
>>>       option enabled '1'
>>>       option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost diffserv4'
>>>       option upload '30000'
>>>       option linklayer 'ethernet'
>>>       option overhead '8'
>>> 
>>> Is the overhead correct? should i use the bridged-ptm keyword (or should i use pppoe-ptm).
> 
> 
> Beware of using  option linklayer ‘ethernet’ without option linklayer_advanced ‘1’ & option linklayer_adaptation_mechanism ‘default’.  Or use linklayer ‘none’.  

I guess "linklayer cake" would be better than none...

> Failure to do so will make sqm-scripts use STAB for the link accounting (in essence it lies to cake about the size of packets being passed through it).  Better to use cake’s built-in compensation - fewer modules, less code.

	I agree, even though for ptm this is less dire than for ATM; still use linklayer default or linklayer cake.

> 
> I was bitten by this myself very recently, lost 4 hours of my life & many recompiles before I realised an innocent looking setting (linklayer_advanced) was messing with the packet size (seen by looking at max_len from tc)

	I am beginning to wonder whether the attempt to hide complexity behind linklayer_advanced is not simply misguided and should be jettisoned...



> 
>> 
>> 	The overhead certainly seems confusing. Personally, I dislike the overhead related compound keywords like *-ptm and would recommend the following:
>> 1) remove the bridged-ptm from the eqdisc/iqdisc fields
>> 2) add "mpu 64" to the eqdisc/iqdisc fields
>> 3) set overhad to 8+18+4 = 30 bytes (you will need to account for everything added on the bottleneck, so your packets will be MTU 1492, to leave room for the PPPoE header that the modem adds).
> MTU 1492 ain’t necessarily so - some vdsl modems in bridge mode support RFC 4638 with baby jumbo frames, thus supporting the normal ethernet MTU of 1500 (and obviating the need to TCP MSS clamping) - ie. if you can do it, you should ;-).  You still need to account for the 8 byte PPPOE overhead of course.

	And again, Kevin is correct; my rationale might have been faulty but the recommendation still was correct though ;)...


>> 4) DO not set the ptm keyword at all, instead make sure to set the shaper bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring needless operations per packet).
>> 
>> 4) tell us about your ISP and plan ;)
>> 
> Cheers,
> 
> Kevin D-B

Thanks for your input, as always very much appreciated!

Best Regards
	Sebastian


> 
> GPG fingerprint: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [Cake] overhead for double nat VDSL2 connection
  2017-12-17 22:35     ` Sebastian Moeller
@ 2017-12-18  3:56       ` Mark Captur
  0 siblings, 0 replies; 7+ messages in thread
From: Mark Captur @ 2017-12-18  3:56 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: Kevin Darbyshire-Bryant, cake

[-- Attachment #1: Type: text/plain, Size: 7824 bytes --]

Thank to both of you!

On 17 December 2017 at 23:35, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Kevin, hi Mark,
>
> Kevin is 100% right!
>
> > On Dec 17, 2017, at 20:23, Kevin Darbyshire-Bryant <
> kevin@darbyshire-bryant.me.uk> wrote:
> >
> >
> >
> >> On 17 Dec 2017, at 15:23, Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> Hi Mark,
> >>
> >>> On Dec 17, 2017, at 11:45, Mark Captur <mark.captur@gmail.com> wrote:
> >>>
> >>> My setup is as follows
> >>>
> >>> vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master
> eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
> >>>
> >>> Here is my current SQM config
> >>> config queue 'eth1'
> >>>       option debug_logging '0'
> >>>       option verbosity '5'
> >>>       option qdisc 'cake'
> >>>       option qdisc_advanced '1'
> >>>       option ingress_ecn 'ECN'
> >>>       option egress_ecn 'NOECN'
> >>>       option qdisc_really_really_advanced '1'
> >>>       option script 'layer_cake.qos'
> >>>       option interface 'eth0.2'
> >>>       option enabled '1'
> >>>       option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost
> diffserv4'
> >>>       option upload '2400'
> >>>       option linklayer 'ethernet'
> >>>       option overhead '8'
> >>>       option squash_dscp '1'
> >>>       option squash_ingress '1'
> >>>       option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
> >>>       option download '0'
> >>>
> >>> config queue
> >>>       option debug_logging '0'
> >>>       option verbosity '5'
> >>>       option download '0'
> >>>       option qdisc 'cake'
> >>>       option script 'layer_cake.qos'
> >>>       option qdisc_advanced '1'
> >>>       option squash_dscp '0'
> >>>       option squash_ingress '0'
> >>>       option ingress_ecn 'ECN'
> >>>       option qdisc_really_really_advanced '1'
> >>>       option egress_ecn 'ECN'
> >>>       option interface 'eth0.1'
> >>>       option enabled '1'
> >>>       option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost
> diffserv4'
> >>>       option upload '30000'
> >>>       option linklayer 'ethernet'
> >>>       option overhead '8'
> >>>
> >>> Is the overhead correct? should i use the bridged-ptm keyword (or
> should i use pppoe-ptm).
> >
> >
> > Beware of using  option linklayer ‘ethernet’ without option
> linklayer_advanced ‘1’ & option linklayer_adaptation_mechanism ‘default’.
> Or use linklayer ‘none’.
>
> I guess "linklayer cake" would be better than none...
>
Yes if i use link layer cake explicitly (option linklayer_advanced ‘1’ &
option linklayer_adaptation_mechanism ‘cake’) it works better i.e. higher
bandwidth with no effect on pings

>
> > Failure to do so will make sqm-scripts use STAB for the link accounting
> (in essence it lies to cake about the size of packets being passed through
> it).  Better to use cake’s built-in compensation - fewer modules, less code.
>
>         I agree, even though for ptm this is less dire than for ATM; still
> use linklayer default or linklayer cake.
>
> >
> > I was bitten by this myself very recently, lost 4 hours of my life &
> many recompiles before I realised an innocent looking setting
> (linklayer_advanced) was messing with the packet size (seen by looking at
> max_len from tc)
>
>         I am beginning to wonder whether the attempt to hide complexity
> behind linklayer_advanced is not simply misguided and should be
> jettisoned...
>
I agree i though that by leaving  linklayer_advanced off i was allowing it
to choose default i.e. cake! what about max_len below? is that ok?

qdisc cake 802b: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv3
dual-dsthost nat rtt 50.0ms noatm overhead 30 via-ethernet total_overhead
30 hard_header_len 14 mpu 64
 Sent 169756771 bytes 121528 pkt (dropped 4732, overlimits 118063 requeues
0)
 backlog 8352b 6p requeues 0
 memory used: 80640b of 4Mb
 capacity estimate: 30Mbit
                   Bulk  Best Effort        Voice
  thresh       1875Kbit       30Mbit     7500Kbit
  target          9.7ms        2.5ms        2.5ms
  interval       57.2ms       50.0ms       50.0ms
  pk_delay          0us        3.2ms        1.6ms
  av_delay          0us        2.3ms        283us
  sp_delay          0us        324us         18us
  pkts                0       125657          609
  bytes               0    176142105       205486
  way_inds            0            0            0
  way_miss            0           61            2
  way_cols            0            0            0
  drops               0         4732            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         3012         2032

qdisc cake 8029: dev eth0.2 root refcnt 2 bandwidth 2440Kbit diffserv3
dual-srchost nat rtt 50.0ms noatm overhead 30 via-ethernet total_overhead
30 hard_header_len 14 mpu 64
 Sent 7184293 bytes 74928 pkt (dropped 99, overlimits 46203 requeues 0)
 backlog 54b 1p requeues 0
 memory used: 155232b of 4Mb
 capacity estimate: 2440Kbit
                   Bulk  Best Effort        Voice
  thresh      152496bit     2440Kbit      610Kbit
  target        119.1ms        7.4ms       29.8ms
  interval      238.3ms       54.9ms       77.3ms
  pk_delay          0us        1.3ms        526us
  av_delay          0us        272us         10us
  sp_delay          0us          9us         10us
  pkts                0        75018           10
  bytes               0      7200943         1290
  way_inds            0            1            0
  way_miss            0           73            8
  way_cols            0            0            0
  drops               0           99            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         3008          288


>
>
> >
> >>
> >>      The overhead certainly seems confusing. Personally, I dislike the
> overhead related compound keywords like *-ptm and would recommend the
> following:
> >> 1) remove the bridged-ptm from the eqdisc/iqdisc fields
> >> 2) add "mpu 64" to the eqdisc/iqdisc fields
> >> 3) set overhad to 8+18+4 = 30 bytes (you will need to account for
> everything added on the bottleneck, so your packets will be MTU 1492, to
> leave room for the PPPoE header that the modem adds).
> > MTU 1492 ain’t necessarily so - some vdsl modems in bridge mode support
> RFC 4638 with baby jumbo frames, thus supporting the normal ethernet MTU of
> 1500 (and obviating the need to TCP MSS clamping) - ie. if you can do it,
> you should ;-).  You still need to account for the 8 byte PPPOE overhead of
> course.
>
>         And again, Kevin is correct; my rationale might have been faulty
> but the recommendation still was correct though ;)...
>
>
> >> 4) DO not set the ptm keyword at all, instead make sure to set the
> shaper bandwidth to <= sync bandwidth * 64/65 = sync bandwidth *
> 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring
> needless operations per packet).
> >>
> >> 4) tell us about your ISP and plan ;)
> >>
> > Cheers,
> >
> > Kevin D-B
>
> Thanks for your input, as always very much appreciated!
>
> Best Regards
>         Sebastian


Thanks & regards
Mark

>
> >
> > GPG fingerprint: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A
>
>

[-- Attachment #2: Type: text/html, Size: 11324 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2017-12-18  3:56 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-12-17 10:45 [Cake] overhead for double nat VDSL2 connection Mark Captur
2017-12-17 15:23 ` Sebastian Moeller
2017-12-17 16:00   ` Mark Captur
2017-12-17 22:30     ` Sebastian Moeller
2017-12-17 19:23   ` Kevin Darbyshire-Bryant
2017-12-17 22:35     ` Sebastian Moeller
2017-12-18  3:56       ` Mark Captur

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox