From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-1" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 5196F21F64A for ; Sat, 26 Jul 2014 14:48:51 -0700 (PDT) Received: from hms-beagle.home.lan ([217.231.210.84]) by mail.gmx.com (mrgmx101) with ESMTPSA (Nemesis) id 0LiHc7-1WhIul04At-00nPZf; Sat, 26 Jul 2014 23:48:44 +0200 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Sebastian Moeller In-Reply-To: Date: Sat, 26 Jul 2014 23:48:42 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <598BFF8C-37B6-4F5B-9F0D-A0D69A1F32B7@gmx.de> References: To: David Lang X-Mailer: Apple Mail (2.1878.6) X-Provags-ID: V03:K0:I7HxtNf34vVk7ulIB6XEMhFRsEosH3QY0+nndjdRoPShSN1CGl7 hZz+aLPpeQwH7xh9CWcqw6B9BBupRP75kPVek+te3ntVygHv754GWW3rlSH5JmC+/FxmQCB jrkkRkg44IciVoivLfowt2lRT7BWJjcYIML58Nizhf2NuOLneOaGi363/EAJ6WKeA2LXHfW 4I82MrOSgpED2k39XKY5w== Cc: Wes Felter , cerowrt-devel@lists.bufferbloat.net Subject: Re: [Cerowrt-devel] Ideas on how to simplify and popularize bufferbloat control for consideration. X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 21:48:51 -0000 Hi David, On Jul 26, 2014, at 23:14 , David Lang wrote: > On Sat, 26 Jul 2014, Sebastian Moeller wrote: >=20 >> On Jul 26, 2014, at 22:21 , David Lang wrote: >>=20 >>> On Sat, 26 Jul 2014, Sebastian Moeller wrote: >>>=20 >>>>=20 >>>> On Jul 25, 2014, at 22:57 , David Lang wrote: >>>>=20 >>>>> The trouble is that to measure bandwidth, you have to be able to = send and receive a lot of traffic. >>>>=20 >>>> Well that is what you typically do, but you can get away with = less measurement traffic: in an ideal quiescent network sending two = packets back to back should give you the bandwidth (packet size / = incoming time difference of both packets), or send two packets of = different size (needs synchronized clocks, then difference of packet = sizes / difference of transfer times). >>>=20 >>> Except that your ideal network doesn't exist in the real world. You = are never going to have the entire network quiescent, the router you are = going to be talking to is always going to have other things going on, = which can affect it's timing. >>=20 >> Sure, the two packets a required per measurement, guess I would = calculate the average and confidence interval over several of these = (potentially by a moving window) to get a handle on the variability. I = have done some RTT measurements on a ADSL link and can say that = realistically one needs in the hundreds data points per packet size. = This sounds awe full, but at least it does not require to saturate the = link and hence works without dedicated receivers on the other end... >>=20 >>>=20 >>>>> unless the router you are connecting to is running some sort of = service to support that, >>>>=20 >>>> But this still requires some service on the other side. You = could try to use ICMP packets, but these will only allow to measure RTT = not one-way delays (if you do this on ADSL you will find the RTT = dominated by the typically much slower uplink path). If network = equipment would be guaranteed to use NTP for decent clock = synchronization and would respond to timestamp ICMP messages with = timestamp reply measuring bandwidth might be =93cheap=94 enough to keep = running in the background, though. >>>> Since this looks too simple there must be a simple reason why = this would fail. (It would be nice if ping packets with timestamps would = have required the echo server top also store its incoming timestamp in = the echo, but I digress) >>>> I note that gargoyle uses a sparse stream of ping packets to a = close host and uses increases in RTT as proxy for congestion and signal = to throttle down stream link=85 >>>=20 >>> As you say, anything that requires symmetrical traffic (like ICMP = isn't going to work, and routers do not currently offer any service that = will. >>=20 >> Well I think the gargoyle idea is feasible given that there is a = reference implementation out in the wild ;). >=20 > I'm not worried about an implementation existing as much as the = question of if it's on the routers/switches by default, and if it isn't, = is the service simple enough to be able to avoid causing load on these = devices and to avoid having any security vulnerabilities (or DDos = potential) But with gargoyle the idea is to monitor a sparse ping stream to = the closest host responding and interpreting a sudden increase in RTT as = a sign the the upstreams buffers are filling up and using this as signal = to throttle on the home router. My limited experience shows that quite = often close hosts will respond to pings... >=20 >>>>> you can't just test that link, you have to connect to something = beyond that. >>>>=20 >>>> So it would be sweet if we could use services that are running = on the machines anyway, like ping. That way the =93load=94 of all the = leaf nodes of the internet continuously measuring their bandwidth could = be handled in a distributed fashion avoiding melt-downs by synchronized = measurement streams=85 >>>=20 >>> Well, let's talk about what we would like to have on the router >>>=20 >>> As I see it, we want to have two services >>>=20 >>> 1. a service you send a small amount of data to and it responds by = sending you a large amount of data (preferrably with the most accurate = timestamps it has and the TTL of the packets it received) >>>=20 >>> 2. a service you send a large amount of data to and it responds by = sending you small responses, telling you how much data it has received = (with a timestamp and what the TTL of the packets it received were) >>>=20 >>> questions: >>>=20 >>> A. Protocol: should these be UDP/TCP/SCTP/raw IP packets/??? >>>=20 >>> TCP has the problem of slow start so it would need substantially = more traffic to flow to reach steady-state. >>>=20 >>> anything else has the possibility of taking a different path through = the router/switch software and so the performance may not be the same. >>=20 >> You thing UDP would not work out? >=20 > I don't trust that UDP would go through the same codepaths and delays = as TCP Why should a router care=20 >=20 > even fw_codel handles TCP differently Does it? I thought UDP typically reacts differently to fq_codels = dropping strategy but fq_codel does not differentiate between protocols = (last time I looked at the code I came to that conclusion, but I am not = very fluent in C so I might be simply wrong here) >=20 > so if we measure with UDP, does it really reflect the 'real world' of = TCP? But we care for UDP as well, no? >=20 >>> B. How much data is needed to be statistically accurate? >>>=20 >>> Too many things can happen for 1-2 packets to tell you the answer. = The systems on both ends are multi-tasking, and at high speeds, = scheduling jitter will throw off your calculations with too few packets. >>=20 >> Yeah, but you can (to steal an I idea from Rick Jones netperf) = just keep measuring until the confidence interval around the mean of the = data falls below a set magnitude. But for the purpose of traffic shaping = you do not need the exact link bandwidth anyway just a close enough = proxy to start the search for a decent set point from a reasonable = position. I think that the actual shaping rates need to be iteratively = optimized. >>=20 >>>=20 >>> C. How can this be prevented from being used for DoS attacks, either = against the thing running the service or against someone else via a = reflected attack if it's a forgable protocol (i.e. UDP) >>=20 >> Well, if it only requires a sparse packet stream it is not going = to be to useful for DOS attacks, >=20 > unless it can be requested a lot Well yes, hence sparse stream, if we can make sure to always = just send sparse streams we will stay on the backwaters of services = useful for DOS I would guess, we just need not to be the low hanging = fruit :) . >=20 >>> One thought I have is to require a high TTL on the packets for the = services to respond to them. That way any abuse of the service would = have to take place from very close on the network. >>>=20 >>> Ideally these services would only respond to senders that are = directly connected, but until these services are deployed and enabled by = default, there is going to be a need to be the ability to 'jump over' = old equipment. This need will probably never go away completely. >>=20 >> But if we need to modify DSLAMs and CMTSs it would be much nicer = if we could just ask nicely what the current negotiated bandwidths are = ;) >=20 > negotiated bandwith and effective bandwidth are not the same >=20 > what if you can't talk to the devices directly connected to the DSL = line, but only to a router one hop on either side? In my limited experience the typical bottleneck is the DSL line, = so if we shape for that we are fine=85 Assume for a moment the DSLAM = uplink is so congested because of oversubscription of the DSLAM, that = now this constitutes the bottleneck. Now the available bandwidth for = each user depends on the combined traffic of all users, not a situation = we can reasonable shape for anyway (I would hope that ISPs monitor this = situation and would remedy it by adding uplink capacity, so this = hopefully is just a transient event).=20 >=20 > for example, I can't buy (at least not for anything close to a = reasonable price) a router to run at home that has a DSL port on it, so = I will always have some device between me and the DSL. http://wiki.openwrt.org/toh/tp-link/td-w8970 or = http://www.traverse.com.au/products ? If you had the DSL modem in the = router under cerowrts control you would not need to use a traffic shaper = for your uplink, as you could apply the BQL ideas to the ADSL driver. >=20 > If you have a shared media (cable, wireless, etc), the negotiated = speed is meaningless. Not exactly meaningless, if gives you an upper bound... >=20 > In my other location, I have a wireless link that is ethernet to the = dish on the roof, I expect the other end is a similar setup, so I can = never see the link speed directly (not to mention the fact that rain can = degrade the effective link speed) One more case for measuring the link speed continuously! >=20 >>> Other requirements or restrictions? >>=20 >> I think the measurement should be fast and continuous=85 >=20 > Fast yes, because we want to impact the network as little as possible >=20 > continuous?? I'm not so sure. Do conditions really change that much? You just gave an example above for changing link conditions, by = shared media... > And as I ask in the other thread, how much does it hurt if your = estimates are wrong? I think I sent a plot to that regard. >=20 > for wireless links the conditions are much more variable, but we don't = really know what is going to work well there. Wireless as in point 2 point links or in wifi? Best Regards Sebastian >=20 > David Lang