From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.toke.dk (mail.toke.dk [52.28.52.200]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 20CA63B2A4 for ; Sat, 31 Mar 2018 15:50:48 -0400 (EDT) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1522525847; bh=4PI8ANZ6Z5GHsQap8IrcW64oItSFtVjGhpwWkveQUvs=; h=From:To:Subject:In-Reply-To:References:Date:From; b=Rs6jtKGw2lNkr7+Iq551SsxGMxcyx5UcVodvx18Qz9Q73V8u18pJvlmk9zCtCt6x+ 7EOASW3MfDr73BkUvJnZ4TFeEBWWOEr7/hRIw4sOpEsjipOFjV5FNTf8BD/Qr994SQ 0q7f3UEr7DuTqeFj6zmjoHMg59gnLQPU438WWFkwnQy7csiR3/wZljsR7EjeG8k8g+ MZnZC4HZs+8BYkokJ4Z4zCzGeoHU5ewHjZfUoe6ZK6DYr4E4wbu9clwUxYz72VhOqx vXyvyG+QLLDBWz8lp7ZRHwGYF3jKGisUAGfcETRc4hpS9PDcRP91Jm+ckaOC+oeh7S rbvuhhs/jfcBQ== To: Rich Brown , bloat In-Reply-To: References: Date: Sat, 31 Mar 2018 21:50:45 +0200 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87y3i8hvmy.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain Subject: Re: [Bloat] bufferbloat.net server having troubles? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2018 19:50:48 -0000 Rich Brown writes: > I just went to bufferbloat.net and > https://www.bufferbloat.net/projects/bloat/wiki/RRUL_Chart_Explanation/ > and am receiving intermittent 502 & 522 errors. Is anyone else seeing > this? Let me know if you need more details. Thanks. Yeah, the box running the web server is having some issues with NULL pointer dereferences in tcp_push() in the kernel crashing processes running TCP. Haven't been able to figure out why :/ [332756.817052] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [332756.817072] IP: tcp_push+0x40/0x120 [332756.817075] PGD 0 P4D 0 [332756.817082] Oops: 0002 [#11] SMP PTI [332756.817085] Modules linked in: fuse md4 nls_utf8 cifs ccm dns_resolver fscache wireguard(O) ip6_udp_tunnel udp_tunnel ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ ipv6 ip6table_filter ip6table_mangle ip6_tables ipt_REJECT nf_reject_ipv4 xt_policy xt_set xt_hashlimit xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable _nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c crc32c_generic xt_TCPMSS xt_tcpudp iptable_mangle ip_set_hash_ip tun ip_set nfnetlink sunrpc snd_ hda_codec_hdmi nvidia_drm(PO) nvidia_modeset(PO) nls_iso8859_1 nls_cp437 intel_rapl vfat nvidia(PO) fat uvcvideo videobuf2_vmalloc x86_pkg_temp_thermal snd_usb_audio intel_power clamp videobuf2_memops videobuf2_v4l2 videobuf2_core coretemp joydev dcdbas videodev kvm_intel mousedev [332756.817155] snd_usbmidi_lib iTCO_wdt input_leds snd_hda_codec_realtek snd_rawmidi evdev mei_wdt iTCO_vendor_support media snd_seq_device drm_kms_helper mac_hid dell_smm_hwm on snd_hda_codec_generic kvm drm irqbypass snd_hda_intel intel_cstate intel_rapl_perf snd_hda_codec pcspkr snd_hda_core agpgart ipmi_devintf snd_hwdep ipmi_msghandler snd_pcm sy scopyarea sysfillrect sysimgblt fb_sys_fops snd_timer i2c_i801 snd mei_me soundcore mei lpc_ich shpchp wmi button vmmon(O) vmw_vmci vboxnetflt(O) vboxnetadp(O) pci_stub vboxpci( O) vboxdrv(O) tcp_bbr sit tunnel4 ip_tunnel sg ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto algif_skcipher af_alg hid_generic usbhid hid arc4 rt2800usb rt2x00usb rt2800li b rt2x00lib led_class mac80211 cfg80211 rfkill dm_crypt dm_mod sd_mod crct10dif_pclmul crc32_pclmul crc32c_intel [332756.817239] ghash_clmulni_intel pcbc xhci_pci xhci_hcd ehci_pci ahci ehci_hcd libahci aesni_intel aes_x86_64 crypto_simd glue_helper libata cryptd tg3 libphy scsi_mod e1000 e usbcore ptp usb_common pps_core sch_fq [332756.817267] CPU: 0 PID: 22603 Comm: nginx Tainted: P D O 4.14.29-1-lts #1 [332756.817270] Hardware name: Dell Inc. Precision T3610/09M8Y8, BIOS A15 01/04/2018 [332756.817273] task: ffff97ccd2914880 task.stack: ffffbe1e04cc4000 [332756.817279] RIP: 0010:tcp_push+0x40/0x120 [332756.817282] RSP: 0018:ffffbe1e04cc7d00 EFLAGS: 00010246 [332756.817286] RAX: 0000000000000000 RBX: ffff97cb98b43b80 RCX: 0000000000000001 [332756.817289] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff97cb98b43b80 [332756.817292] RBP: 00000000000021f0 R08: 00000000000005a8 R09: 00000000000005a8 [332756.817295] R10: ffff97cb98b43cd8 R11: 0000000000000000 R12: 00000000000005a8 [332756.817298] R13: 0000000000000040 R14: ffff97cb98b43cd8 R15: 00000000ffffffe0 [332756.817302] FS: 00007f909a778b80(0000) GS:ffff97ceafc00000(0000) knlGS:0000000000000000 [332756.817305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [332756.817308] CR2: 0000000000000038 CR3: 000000031172a001 CR4: 00000000001606f0 [332756.817311] Call Trace: [332756.817320] tcp_sendmsg_locked+0xb10/0xe50 [332756.817328] ? sock_poll+0x70/0x90 [332756.817334] tcp_sendmsg+0x27/0x40 [332756.817339] sock_write_iter+0xa3/0x110 [332756.817347] __vfs_write+0x102/0x180 [332756.817353] vfs_write+0xad/0x1a0 [332756.817358] SyS_write+0x52/0xc0 [332756.817366] do_syscall_64+0x67/0x120 [332756.817379] RIP: 0033:0x7f909a1958b4 [332756.817382] RSP: 002b:00007ffff5e51c48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [332756.817386] RAX: ffffffffffffffda RBX: 0000000000002f4c RCX: 00007f909a1958b4 [332756.817389] RDX: 0000000000002f4c RSI: 000055be7827c36c RDI: 0000000000000016 [332756.817392] RBP: 000055be78050340 R08: 0000000000000000 R09: 0000000000000000 [332756.817395] R10: 000055be7804eab0 R11: 0000000000000246 R12: 000055be7827c36c [332756.817398] R13: 0000000000002f4c R14: 00007f909a778b00 R15: 000055be78272150 [332756.817401] Code: 00 48 8b 87 60 01 00 00 4c 8d 97 58 01 00 00 ba 00 00 00 00 41 89 f3 49 39 c2 48 0f 44 c2 41 81 e3 00 80 00 00 0f 85 9d 00 00 00 <80> 48 38 08 8b 97 74 06 00 00 89 97 7c 06 00 00 83 e6 01 74 0c [332756.817461] RIP: tcp_push+0x40/0x120 RSP: ffffbe1e04cc7d00 [332756.817463] CR2: 0000000000000038 [332756.817469] ---[ end trace 797c2d8c9eead6f1 ]--- -Toke