[Cerowrt-devel] odhcp6c went crazy flooding Comcast with DHCPv6 SOLICITs

Dave Taht dave.taht at gmail.com
Sat Feb 1 11:43:48 EST 2014


I am painfully aware that a change in comcast's deployment starting in
late december started messing up older versions of cerowrt, openwrt,
etc.

(short version, they started announcing ras every three seconds, which
triggers a reload/reconfiguration of openwrt that
takes longer than 3 seconds...)

The openwrt folk fixed it upstream a few weeks back and the last
couple development releases of cero have the fixes.
(ghu help those that are merely ipv6 certified and not paying attention)

https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-January/002093.html

I have since produced several comcast specific releases. This one is
pretty stable

http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/comcast/3.10.28-4/

I recomend all comcast users on cerowrt upgrade ASAP as comcast is aggressively
turning on ipv6 everywhere. (and it's pretty wonderful when it works)

Note that you should make a backup,
reflash from scratch, and re-apply your mods via the gui as much as
changed since 3.8.

I usually take a backup via scp root at gw.home.lan:/overlay and then
look hard at the modified files
in /etc and /etc/config


As this is still a dev release it has some problems, notably HT40+
mode seems to be borked
on 5ghz. Also I am discovering that providers deploying ipv6 (AT&T,
now possibly comcast) seem
to be (inadvertently) disabling various forms of tunnelling. I still
haven't got he to work again
simultaneously with native ipv6 on comcast. (but unlike the first
comcast-specific release,
trying to enable one doesn't blow up the router)

Another option for you is to disable the dhcp-pd request on your
current version so you don't get native ipv6...

I hope to resume marching towards a final stable release over the
coming weeks. In the interim,
on comcast, upgrade ASAP.


On Sat, Feb 1, 2014 at 5:29 AM, Chuck Anderson <cra at wpi.edu> wrote:
> This morning my Linux PC which has a direct connection to my Comcast
> cable modem (no router in between) lost its IPv4 address.  During
> troubleshooting, I noticed that the dhclient was unable to get an IPv4
> address from Comcast.  I ran tcpdump and discovered that the CeroWRT
> router, also connected to the same cable modem via a switch, was
> flooding the WAN with DHCPv6 SOLICIT packets with about 4700
> packets/sec, 6.6 megabits/sec of traffic!  I immediately unplugged
> CeroWRT from the WAN and then my PC was able to get an IPv4 address
> from Comcast.
>
> I know CeroWrt 3.7.5-2 is old at this point, but I'm wondering if
> something else changed to cause this behavior.  Maybe Comcast
> IPv6-enabled my CMTS finally?  I've been using HE tunnels for IPv6,
> one on a Linksys OpenWRT for my "production" network and a separate
> tunnel on this CeroWRT for "testing".
>
> There is one other change that was made to my Linux PC--I booted into
> a new kernel yesterday morning and had a similar problem with the
> inability to get an IPv4 address via DHCP from Comcast for about 30
> minutes, then it just started working on its own.  (I hadn't noticed
> initially since I was using IPv6 to get where I needed to go.)  I
> didn't have time to troubleshoot it at the time, but I assumed it was
> due to this IPv6 change in the Fedora kernel:
>
> * Wed Jan 29 2014 Justin M. Forbes <jforbes at fedoraproject.org> - 3.12.9-201
> - ipv6 addrconf: revert /proc/net/if_inet6 ifa_flag format (rhbz 1056711)
>
> * Tue Jan 28 2014 Josh Boyer <jwboyer at fedoraproject.org>
> - Add patch from Stanislaw Gruszka to fix ath9k BUG (rhbz 990955)
>
> * Mon Jan 27 2014 Justin M. Forbes <jforbes at fedoraproject.org> - 3.12.9-200
> - Backport new IPv6 address flag IFA_F_NOPREFIXROUTE and IFA_F_MANAGETEMPADDR (rhbz 1056711)
> - Linux v3.12.9
> - i915: remove pm_qos request on error (rhbz 1057533)
>
> See https://bugzilla.redhat.com/show_bug.cgi?id=1056711 for details
> about that.
>
> Each time this loss of IPv4 happened, I noticed the NIC link went down
> right before it started.  Maybe the flooding was happening yesterday
> morning too, and the flooding caused my poor 5-port Netgear switch to
> flake out and flap the NIC links?  Alternatively, maybe the link flap
> itself was what caused odhcp6c to wig out and start flooding in the
> first place?  Unfortunately I don't have a tcpdump from yesterday
> morning to confirm this.
>
> CeroWRT status:
>
> Router Name  cerowrt
> Router Model NETGEAR WNDR3700v2
> Firmware Version     CeroWrt Modena 3.7.5-2 / LuCI Trunk (trunk+svn)
> Kernel Version       3.7.5
> Local Time           Sat Feb 1 07:54:43 2014
> Uptime               58d 6h 56m 51s
>
> The DHCPv6 client is odhcp6c:
>
> root at cerowrt:~# ps |grep dhc
>   980 root      1720 S    udhcpc -p /var/run/udhcpc-ge00.pid -s /lib/netifd/dh
>  1335 root       804 R    odhcp6c -s /lib/netifd/dhcpv6.script -Ntry -P60 ge00
>  3725 root      1704 S    grep dhc
>
> Here is an example packet from the DHCPv6 flood:
>
> No.     Time        Source                Destination           Protocol Length Info
>       1 0.000000    fe80::c63d:c7ff:feb0:8f41 ff02::1:2             DHCPv6   179    Solicit XID: 0x45eb91 CID: 00030001c43dc7b08f41
>
> Frame 1: 179 bytes on wire (1432 bits), 179 bytes captured (1432 bits)
>     Encapsulation type: Ethernet (1)
>     Arrival Time: Feb  1, 2014 07:20:27.723633000 EST
>     [Time shift for this packet: 0.000000000 seconds]
>     Epoch Time: 1391257227.723633000 seconds
>     [Time delta from previous captured frame: 0.000000000 seconds]
>     [Time delta from previous displayed frame: 0.000000000 seconds]
>     [Time since reference or first frame: 0.000000000 seconds]
>     Frame Number: 1
>     Frame Length: 179 bytes (1432 bits)
>     Capture Length: 179 bytes (1432 bits)
>     [Frame is marked: False]
>     [Frame is ignored: False]
>     [Protocols in frame: eth:ipv6:udp:dhcpv6]
>     [Coloring Rule Name: UDP]
>     [Coloring Rule String: udp]
> Ethernet II, Src: Netgear_b0:8f:41 (c4:3d:c7:b0:8f:41), Dst: IPv6mcast_00:01:00:02 (33:33:00:01:00:02)
>     Destination: IPv6mcast_00:01:00:02 (33:33:00:01:00:02)
>         Address: IPv6mcast_00:01:00:02 (33:33:00:01:00:02)
>         .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
>         .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
>     Source: Netgear_b0:8f:41 (c4:3d:c7:b0:8f:41)
>         Address: Netgear_b0:8f:41 (c4:3d:c7:b0:8f:41)
>         .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
>         .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
>     Type: IPv6 (0x86dd)
> Internet Protocol Version 6, Src: fe80::c63d:c7ff:feb0:8f41 (fe80::c63d:c7ff:feb0:8f41), Dst: ff02::1:2 (ff02::1:2)
>     0110 .... = Version: 6
>         [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
>     .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
>         .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
>         .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
>         .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
>     .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
>     Payload length: 125
>     Next header: UDP (17)
>     Hop limit: 1
>     Source: fe80::c63d:c7ff:feb0:8f41 (fe80::c63d:c7ff:feb0:8f41)
>     [Source SA MAC: Netgear_b0:8f:41 (c4:3d:c7:b0:8f:41)]
>     Destination: ff02::1:2 (ff02::1:2)
>     [Source GeoIP: Unknown]
>     [Destination GeoIP: Unknown]
> User Datagram Protocol, Src Port: dhcpv6-client (546), Dst Port: dhcpv6-server (547)
>     Source port: dhcpv6-client (546)
>     Destination port: dhcpv6-server (547)
>     Length: 125
>     Checksum: 0xda1c [validation disabled]
>         [Good Checksum: False]
>         [Bad Checksum: False]
> DHCPv6
>     Message type: Solicit (1)
>     Transaction ID: 0x45eb91
>     Elapsed time
>         Option: Elapsed time (8)
>         Length: 2
>         Value: ffff
>         Elapsed-time: 655350 ms
>     Option Request
>         Option: Option Request (6)
>         Length: 10
>         Value: 00170018003800160015
>         Requested Option code: DNS recursive name server (23)
>         Requested Option code: Domain Search List (24)
>         Requested Option code: Unknown (56)
>         Requested Option code: SIP Servers IPv6 Address List (22)
>         Requested Option code: SIP Server Domain Name List (21)
>     Client Identifier
>         Option: Client Identifier (1)
>         Length: 10
>         Value: 00030001c43dc7b08f41
>         DUID: 00030001c43dc7b08f41
>         DUID Type: link-layer address (3)
>         Hardware type: Ethernet (1)
>         Link-layer address: c4:3d:c7:b0:8f:41
>     Reconfigure Accept
>         Option: Reconfigure Accept (20)
>         Length: 0
>     Fully Qualified Domain Name
>         Option: Fully Qualified Domain Name (39)
>         Length: 10
>         Value: 00076365726f77727400
>         0000 0... = Reserved: 0x00
>         .... .0.. = N bit: Server should perform DNS updates
>         .... ..0. = O bit: Server has not overridden client's S bit preference
>         .... ...0 = S bit: Server should not perform forward DNS updates
>         Domain: cerowrt
>     Identity Association for Non-temporary Address
>         Option: Identity Association for Non-temporary Address (3)
>         Length: 12
>         Value: 000000010000000000000000
>         IAID: 00000001
>         T1: 0
>         T2: 0
>     Identity Association for Prefix Delegation
>         Option: Identity Association for Prefix Delegation (25)
>         Length: 41
>         Value: 000000010000000000000000001a00190000000000000000...
>         IAID: 00000001
>         T1: 0
>         T2: 0
>         IA Prefix
>             Option: IA Prefix (26)
>             Length: 25
>             Value: 00000000000000003c000000000000000000000000000000...
>             Preferred lifetime: 0
>             Valid lifetime: 0
>             Prefix length: 60
>             Prefix address: :: (::)
>
> 0000  33 33 00 01 00 02 c4 3d c7 b0 8f 41 86 dd 60 00   33.....=...A..`.
> 0010  00 00 00 7d 11 01 fe 80 00 00 00 00 00 00 c6 3d   ...}...........=
> 0020  c7 ff fe b0 8f 41 ff 02 00 00 00 00 00 00 00 00   .....A..........
> 0030  00 00 00 01 00 02 02 22 02 23 00 7d da 1c 01 45   .......".#.}...E
> 0040  eb 91 00 08 00 02 ff ff 00 06 00 0a 00 17 00 18   ................
> 0050  00 38 00 16 00 15 00 01 00 0a 00 03 00 01 c4 3d   .8.............=
> 0060  c7 b0 8f 41 00 14 00 00 00 27 00 0a 00 07 63 65   ...A.....'....ce
> 0070  72 6f 77 72 74 00 00 03 00 0c 00 00 00 01 00 00   rowrt...........
> 0080  00 00 00 00 00 00 00 19 00 29 00 00 00 01 00 00   .........)......
> 0090  00 00 00 00 00 00 00 1a 00 19 00 00 00 00 00 00   ................
> 00a0  00 00 3c 00 00 00 00 00 00 00 00 00 00 00 00 00   ..<.............
> 00b0  00 00 00                                          ...
>
> The CeroWRT system log is attached.  Nothing looks strange except the
> loss of ge00 link around 6:24 this morning, which is right around when
> I lost IPv4 connectivity to my Linux PC (I have a system monitoring
> this IP and it SMS's me if it goes down).  My PC's NIC link went down
> at exactly the same time.  At 7:24 is when I unplugged CeroWRT.
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html



More information about the Cerowrt-devel mailing list