<html><head></head><body style="zoom: 0%;"><div dir="auto">Thanks, this is very interesting. I wrote code to DMA packets in an early Cisco switch and the hardware ASIC that did the movement across the fabric would provide a simple status of success or not. Unfortunately, the ASIC would at times indicate success and never move the packet across the fabric and then went to a state of using the wrong egress for all subsequent packets. It wasn't possible to change the ASIC as that was locked down years earlier. Luckily, we had the ability to query the ASIC to get more information on what it actually did so the code could see when it needed a fix. We did the FDIR, lost a bunch of packets, and assumed TCP would handle it. Of course, TCP designers assumed the loss was due to congestion so those state machines were incorrect but would ultimately recover.<br><br></div>
<div dir="auto">I started my career working on a NASA FDDI network. SW had gotten so complex that all the states could not be inspected by humans like done on the Shuttle, nor even tested by conmputers. The strategy became commercial off the shelf (COTS) because, through "market magic", it was assumed fully tested.<br><br></div>
<div dir="auto">I think the same naivety is now applied to open source code. There is no magic here either. Testing is way beyond simple scenarios repeated over and over again as the only test that matters.<br><br></div>
<div dir="auto">Networks and distributed systems have bugs. I think a current Linux kernel is 30M lines of code and 1100 config options. Good luck in testing that.<br><br></div>
<div dir="auto">This is beyond complex and not easy. FDIR has to be designed in from the get go.<br><br></div>
<div dir="auto">Bob</div>
<div class="gmail_quote" >On Oct 23, 2023, at 4:22 PM, Karl Auerbach <<a href="mailto:karl@cavebear.com" target="_blank">karl@cavebear.com</a>> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="blue">On 10/23/23 2:54 PM, rjmcmahon wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> Home networks today are embarrassing to me. Our industry is woefully <br> behind here.</blockquote><br>I would be more expansive.<br><br>(Bringing this back to network neutrality - my argument, not clearly <br>suggested below, is that "neutrality" is more than bandwidth or <br>connectivity but ought also ought to include other aspects including <br>robust and repairable service in the face of reasonably foreseeable <br>events. By-the-way, when I was involved in the early days of the net, I <br>worked for groups [such as the US Joint Chiefs] who thought that routers <br>being vaporized by nuclear explosions were "reasonable foreseeable".)<br><br>The lawyer half of me lives in fear of the harm that can come from bad <br>code in network devices. I've seen the growth of strict product <br>liability laws in the consumer space (sometimes resulting in those silly <br>"do not eat" labels on silica gel packets, but also resulting in <br>important steps, like pressure-release closures on cleaning products <br>that contain dry sodium hydroxide, or dual braking systems in automobiles.)<br><br>And the railroad nut in me remembers that Murphy's law is as strong as <br>ever. (Just ask "Why are highway and railroad traffic control signals <br>red and green [actually a quite bluish green]?" [Hint, they originally <br>were red and white, and sometimes the red colored lens would fall out.])<br><br>When I was working with the DARPA Robotics Challenge my job was to <br>introduce network problems - the kinds of things that can happen in real <br>life when a robot operates in a disaster zone. I could introduce a <br>simple change - like increasing the level of lost Ethernet frames when a <br>robot went through a door into a (simulated) concrete reactor building - <br>and the robot would simply stop or fall over.<br><br>I've seen videos of animal surgery performed by remote control over a <br>long distance (50km) network link where the doctors presumed that the <br>net was endlessly flawless. (I have this mental image of a robotic <br>scalpel overshooting its cut due to a non-idempotent command contained <br>in a packet that was replicated on the net.)<br><br>And I've seen users of satellites fail to remember that every now and <br>then, from the point of view of a ground station, a satellite may <br>transit across the face of the sun (a highly predictable event) and be <br>temporarily blinded and unable to receive data.<br><br>Many of our implementations today are hanging on only because modern <br>machines have gobs upon gobs of memory and nobody notices if a couple of <br>gigabytes leak or are uselessly allocated for a few minutes.<br><br>(For instance, one way to stop a Linux stack is to send it patterns of <br>tiny IPv4 fragments that overlap or have gaps so that reassembly is not <br>possible (or difficult) and buffers just sit there waiting for a rather <br>long timeout before being reclaimed.)<br><br>It seems that everybody and her brothers think they can write code. And <br>they do. And in our open source world the code they write is often <br>protocol code. Often it is badly written protocol code containing <br>monumental flaws, such as use of "integer" types in C (when "unsigned <br>uint16" or similar is needed), failure to recognize that number spaces <br>wrap, assumptions that "everything is in ASCII" or that character <br>sequences do not contain null bytes. (Last time I looked some major <br>libraries went down in flames when string data in packets happened to <br>contain nulls - the code was using ancient Unix/C string routines.) I <br>once sent several SIP phones into the weeds when I sent length fields <br>(in text form) with leading zero characters (e.g. 050 rather than 50) - <br>some code treated that as octal!)<br><br>It would certainly be nice if we had a body of network implementation <br>design/implementation rules - similar in concept to engineering design <br>rules used in bridges, aircraft, electrical networks, etc - for use when <br>writing code. Any one who wanted to do something outside of those rules <br>could do so, but would be strongly "encouraged" to seek the advice and <br>oversight of others.<br><br>Once the Interop show net was brought to a stop (by infinitely looping <br>packets) when two brands of routers had different notions how to expand <br>IPv4 multicast addresses into MAC addresses. (I can't remember the <br>details, but when every light in the NOC turned red everybody in the <br>Interop NOC turned to look at me, guessing [incorrectly in this <br>instance] that I was the cause.])<br><br>It would be nice if we built our network devices so that they each had a <br>little introspective daemon that frequently asked "am I healthy, am I <br>still connected, are packets still moving through me?" (For consumer <br>devices an answer of "no" could trigger a full device reboot or reset.)<br><br>For larger devices, such as routers, we could have some machinery, <br>internal or external, that did a bit of modelling and informed the <br>routing machinery of anticipated queue lengths and similar metrics. <br>Then the router could monitor itself to check if it was wobbling outside <br>of those anticipated ranges and take appropriate action to signal the <br>issue. (I was once quite surprised to learn on at least one large type <br>of router that it was difficult-to-impossible to obtain queue length <br>data because so much function had been pushed into hardware that had few <br>test or measurement points.)<br><br>My grandfather and father were radio and TV repair guys. I learned from <br>an early age the value of good tools and of looking outside the basic <br>operation of a device for symptoms. (You could often hear a failing <br>capacitor or inductor; or you could smell a slowly burning resistor.) <br>Our modern networks and code usually lack that kind of observational <br>(and active testing) plane.<br><br>I can see a big net neutrality differentiator between providers being <br>"time to detect" and "time to repair".<br><br> --karl--<br><br></pre></blockquote></div></body></html>