From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6F2C53B2A4 for ; Sun, 23 Oct 2022 10:08:19 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1666534095; bh=eGtRPo6AkMVNWi1JmWIdBJIAFZj+PNwDnDy+dS2u6Mc=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=XHvImpsfklORmxvNGl3yMTm/Wj5giBfvIpwYBQVx9HQHYiLsnGgJsFGBupe7KMsbm 7qOp1tPs1a2bZA98Wi9GkAq//FpVgsRFHfhzbtV4/Y2IXK7d7rfmsEyJdQmj6wEkIt oDpR25sPb2zwyLNRwI6oxJDgjwJTyvrwe9FKoW78= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([95.116.66.209]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MrhUK-1pPISZ1lSu-00newl; Sun, 23 Oct 2022 16:08:15 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: Date: Sun, 23 Oct 2022 16:08:14 +0200 Cc: starlink@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: <961E84CB-390F-4BF8-94AA-C9000459DE46@gmx.de> References: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> <0e811787-cc55-8db9-2a4b-7706813769da@indexexchange.com> <850ACED5-3345-441E-9A92-BF27B21EABF5@gmx.de> <78ae1fce-b23a-ef67-e486-72c26d2bbe30@indexexchange.com> <93A92B00-AB22-414A-9A05-B887262E5A8E@gmx.de> To: Dave Collier-Brown X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:9OZTcYQdWy6O8PhOhBSqIkrN4vVxKiaxamqj4V3QjwKh14dgRhe KmCGnfB6X4xgZ/tlj9G3YZLZgjZ5c6y5vrg5LqwbCMjvzS/yQNhYQuvKKp/b6RKZxVGKhWb 6g0ux+zWzft4kNDWUXhZzLdrdClg/PyXExnF/glU64Levv3FNOLqSXLs0cPZS+jRY5Zw9el vXQa0Nw1ehb5roA0l2+pw== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:c+7PpNY0/jw=:2cFdP1BS2gVBNmTiwHZ9qW 3d20E3kwMyu3gT28bRhQjRLUSaG03V2ER2SrgTcHxJ1K224AJTvJ0d69RDSRuRs3NF2wFy1vG ZPC0gg0f4/BfzHfS8ftnKRfjGGldPmyz+EtOM7RDv5b54u+SDb6X/clJWKOQA82IotlZGMbQZ 3USV2SrJ55k+OQwvkCcuDFb9Z4zMIZNrhxN5rYawNHh0vdwAKVjdhjVafuFysRMeONdFz+jIv m9YBrIvQwnXJ/adKySSI2F9a8Q2pp0y2TEe9CiQ9nZJwMrsLsaZ5/7BCNekmV3Va6k6eOPuiM 37SpCrC9vcDqO63RP9v4oiL6gPIhvv/NSi9I3p6jMnV0VgORtiwI/Yff3YIGQj2Il8atPcWar DmZkwoEa0ri0SRJ6i8hf9qAwIU31dsT7SSH2mUcZAxOLPg0Y1kvSsU3BqJKJUMRarHAFtYghm Ew/dJs8zfXHySiYs2Q3SPqxq70Kx3aT4NfWFvFtazuLSKA4egvdSzK8PDFebcem1vavw9rVqE OJ2VMvxGUSuCOpMkDvlYkpTmn68b+mjUF1JhlC9SRTJSOd4sZXLLPKafPpmtLbBxlNzxL3jIV brDgPecWckm7aEAoL7eVs+WYNjzoKVXZnf+a7ODjLC93eIEk9aibbXDMV/PjSSsGXx+3i5j8+ GARC6VnZ0D/p+l2oO0jYNrqWc6Mv+pQqsOEH3sfTjABkRdq/UFZTJkm1rzZFnLWqZONA53+6S cKyLBO4clX+1EbXCfJEKNftzK2lC6OVeDSydy/GOmABePZ6xKYncsMh6go8zgxG2xWEgjxoIN IxP3NXVu3ELpElB3e4CTffEajH6PIeo9FoZLzJNle/tl5D6sTtZZwH/hEcUWE6PYHxF9+v5DF dFgRfJsm9likQis3mWzZUMdyTbfifC0YJhz2MxRFWh+fZW7pPO9GQXAc9KCOPMfbSmktT46Lm YrSVPjbcANrEfXWBDgor3r4I8ouNhB136OIfKZ3aqr+aOgUenZDIjpKWrgsjJ2GeDszkPqOiR tD66lUznVjcmlxk3vwn3ihdBqZMzwtr1MohwTjSgshzTLjlG4mDMcHK0inpUKmGTk1KDAD4ey Z+al8XK6ENlbNfHQagdUhrvRyF08PwEHUgUi/Yocqk8FObsQSOzEZd5uL7QH5h2eYPBQ361CK AUvjuPdHOWA2p9TqmgowIaLzf61uYOO/RQUsIJpLSW3YuHE53cKWfKkA3ldgQBppLH4//+81l mh7V6ewQo+gj9cRnFjGt4+Q0Bg0BNGu6ookg0irMT/NngWExnSYgr77cGv4xT3leKVSaLqXzc DiLOYpqxjfgTw8FAZB/Vaf4AUD5zYjCqbJtuZl0iDFKQaIzqTRyvmzvV9/JIF9iUxsgy+zhBl PNbFdnvMPWpqlescp9MxaoDtT6HNkufDeK3OT/OiDzF2YrxADXa203ZevQyQRmMaA+qwhX6iI Jh2UiqbXNcEV5KMuG2trurO2T06mFn15CWn2xafAIqhHRm5rCLdyryma9YQQrfvdLQHPhLIiS gAGxEDNjuAdMkdk0GWWoPYUpOq6oiPBYAQTMCaa7lLlrMyYnrNEoRX6k3wMkOQuUJH7CYvxP2 BLx+1m4P61M2pjJcbeaQmJApHwSJAE+DujCNMg22eeVzBQoi9wrxXW5XleIy4ZRjGyOExubr2 afyAr27W6nq0xVoKMDGRyJzAI4QcO2mgDK1Y/XnHhxcO17WIw== Subject: Re: [Starlink] [Rpm] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2022 14:08:19 -0000 Hi David, > On Oct 23, 2022, at 16:00, Dave Collier-Brown = wrote: >=20 >=20 > On 10/23/22 09:52, Sebastian Moeller wrote: >> [SM] Okay. In the context of cake-autorate = (https://can01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithu= b.com%2Flynxthecat%2FCAKE-autorate%2Fblob%2Fmain%2FREADME.md&data=3D05= %7C01%7C%7C0de1c0ddf51f41aef70c08dab4fde5ad%7Cb07c069022b843668d8d7b845d08= 8e18%7C1%7C0%7C638021299817872156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw= MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sda= ta=3DLwP%2F45%2FIdIJgdFGITFEgYa2egg4xyQEqkE21HYKb0nM%3D&reserved=3D0) = we implemented a flight recorder type logging that continuously logs the = last X (configurable) epoch and stores bot shaper and achieved rates as = well as the results from the latency probes. This script can be used = with rate setting disabled to record relevant data and the user just = needs to remember to export the data after experiencing = interesting/abnormal events. Sure this does not have per application = resolution, but should give some idea about current latency as well as = current traffic. I will admit though that this logging is not exactly = cheap CPU-wise and lacks the precision of packet captures... but it can = be operated as flight recorder where relevant events can be = exported/stored post-hoc... >>=20 > Ah, now that sounds interesting. I especially like the use of "flight = recorder", as one could express using it as "whenever your MS Teams = session crashes ..." [SM] Yepp, there is a cost though such a log file easily gets = into the multi-dozend MB size range if the logging interval is set high, = but the last 10 minutes should not be that costly (then again many = routers are both storage and RAM limited, so this might be a problem). >>>> Protocols with less fields readable like QUIC would require special = care to evaluate the spin-bit if that exists. Or just resort to active = polling and ping* each CPE once per second or so (for a course = resolution, you could increase the polling rate on detecting anomalies = thereby risking to make congestion slightly worse). None of this will = allow to measure within home network congestion though, but it might = still be a wortwhile diagnostic to know that the access link is OK, = while the user reports latency issues. >>> If one has a good way to capture RTT and data rate for one = problematic app, say zoom, then one could see that network problems were = happening at the same time as lags and dropouts. >> [SM] As above logging all traffic is relatively easy, per = application or per flow will require different tools or packet = captures... >>=20 >>> ISPs would positively hate that, of course. >> [SM] Assuming they come out of this looking bad, if the = outcome is to imply the local WiFi being the root cause ISPs might = actually appreciate it ;) >=20 > Ah, I hadn't thought of that! [SM] I admit that currently ay such localization would require = an additional trace showing congestion/latency increase from internal = hosts, while the autorate logs show no such issue on the wan link. This = is where iperf2's bounceback could be helpful. Regards Sebastian >=20 > --dave >=20 >=20 >>=20 >> Regards >> Sebastian >>=20 >>> --dave >>>=20 >>>=20 >>>=20 >>>=20 >>>> Regards >>>> Sebastian >>>>=20 >>>> *) I think there are dedicated devices available that allow to ping = large numbers of IPs in a periodic fashion. >>>>=20 >>>>=20 >>>>=20 >>>>> --dave >>>>>=20 >>>>>=20 >>>>>=20 >>>>> On 10/23/22 07:57, Sebastian Moeller via Starlink wrote: >>>>>=20 >>>>>> Hi Glenn, >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm = >>>>>>>=20 >>>>>>> wrote: >>>>>>>=20 >>>>>>> As a classic died in the wool empiricist, granted that you can = identify "misery" factors, given a population of 1,000 users, how do you = propose deriving a misery index for that population? >>>>>>>=20 >>>>>>> We can measure download, upload, ping, jitter pretty much = without user intervention. For the measurements you hypothesize, how = you you automatically extract those indecies without subjective user = contamination. >>>>>>>=20 >>>>>>> I.e. my download speed sucks. Measure the download speed. >>>>>>>=20 >>>>>>> My isp doesn't fix my problem. Measure what? How? >>>>>>>=20 >>>>>>> Human survey technology is 70+ years old and it still has = problems figuring out how to correlate opinion with fact. >>>>>>>=20 >>>>>>> Without an objective measurement scheme that doesn't require = human interaction, the misery index is a cool hypothesis with no way to = link to actual data. What objective measurements can be made? Answer = that and the index becomes useful. Otherwise it's just consumer whining. >>>>>>>=20 >>>>>>> Not trying to be combative here, in fact I like the concept you = support, but I'm hard pressed to see how the concept can lead to data, = and the data lead to policy proposals. >>>>>>>=20 >>>>>>>=20 >>>>>> [SM] So it seems that outside of seemingly simple to test = throughput numbers*, the next most important quality number (or the most = important depending on subjective ranking) is how does latency change = under "load". Absolute latency is also important albeit static high = latency can be worked around within limits so the change under load = seems more relevant. >>>>>> All of flent's RRUL test, apple's networkQuality/RPM, and = iperf2's bounceback test offer methods to asses latency change under = load**, as do waveforms bufferbloat tests and even to a degree Ookla's = speedtest.net. IMHO something like latency increase under load or = apple's responsiveness measure RPM (basically the inverse of the latency = under load calculated on a per minute basis, so it scales in the typical = higher numbers are better way, unlike raw latency under load numbers = where smaller is better). >>>>>> IMHO what networkQuality is missing ATM is to measure and = report the unloaded RPM as well as the loaded the first gives a measure = over the static latency the second over how well things keep working if = capacity gets tight. They report the base RTT which can be converted to = RPM. As an example: >>>>>>=20 >>>>>> macbook:~ user$ networkQuality -v >>>>>> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D >>>>>> Upload capacity: 24.341 Mbps >>>>>> Download capacity: 91.951 Mbps >>>>>> Upload flows: 20 >>>>>> Download flows: 16 >>>>>> Responsiveness: High (2123 RPM) >>>>>> Base RTT: 16 >>>>>> Start: 10/23/22, 13:44:39 >>>>>> End: 10/23/22, 13:44:53 >>>>>> OS Version: Version 12.6 (Build 21G115) >>>>>>=20 >>>>>> Here RPM 2133 corresponds to 60000/2123 =3D 28.26 ms latency = under load, while the Base RTT of 16ms corresponds to 60000/16 =3D 3750 = RPM, son on this link load reduces the responsiveness by 3750-2123 =3D = 1627 RPM a reduction by 100-100*2123/3750 =3D 43.4%, and that is with = competent AQM and scheduling on the router. >>>>>>=20 >>>>>> Without competent AQM/shaping I get: >>>>>> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D >>>>>> Upload capacity: 15.101 Mbps >>>>>> Download capacity: 97.664 Mbps >>>>>> Upload flows: 20 >>>>>> Download flows: 12 >>>>>> Responsiveness: Medium (427 RPM) >>>>>> Base RTT: 16 >>>>>> Start: 10/23/22, 13:51:50 >>>>>> End: 10/23/22, 13:52:06 >>>>>> OS Version: Version 12.6 (Build 21G115) >>>>>> latency under load: 60000/427 =3D 140.52 ms >>>>>> base RPM: 60000/16 =3D 3750 RPM >>>>>> reduction RPM: 100-100*427/3750 =3D 88.6% >>>>>>=20 >>>>>>=20 >>>>>> I understand apple's desire to have a single reported number with = a single qualifier medium/high/... because in the end a link is only = reliably usable if responsiveness under load stays acceptable, but with = two numbers it is easier to see what one's ISP could do to help. (I = guess some ISPs might already be unhappy with the single number, so this = needs some diplomacy/tact) >>>>>>=20 >>>>>> Regards >>>>>> Sebastian >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> *) Seemingly as quite some ISPs operate their own speedtest = servers in their network and ignore customers not reaching the = contracted rates into speedtest-servers located in different ASs. As the = product is called internet access I a inclined to expect that my ISP = maintains sufficient peering/transit capacity to reach the next tier of = AS at my contracted rate (the EU legislative seems to agree, see EU = directive 2015/2120). >>>>>>=20 >>>>>> **) Most do by creating load themselves and measuring throughput = at the same time, bounceback IIUC will focus on the latency measurement = and leave the load generation optional (so offers a mode to measure = responsiveness of a live network with minimal measurement traffic). = @Bob, please correct me if this is wrong. >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Fri, Oct 21, 2022, 5:20 PM Dave Taht >>>>>>>=20 >>>>>>> >>>>>>>=20 >>>>>>> wrote: >>>>>>> One of the best talks I've ever seen on how to measure customer >>>>>>> satisfaction properly just went up after the P99 Conference. >>>>>>>=20 >>>>>>> It's called Misery Metrics. >>>>>>>=20 >>>>>>> After going through a deep dive as to why and how we think and = act on >>>>>>> percentiles, bins, and other statistical methods as to how we = use the >>>>>>> web and internet are *so wrong* (well worth watching and = thinking >>>>>>> about if you are relying on or creating network metrics today), = it >>>>>>> then points to the real metrics that matter to users and the = ultimate >>>>>>> success of an internet business: Timeouts, retries, misses, = failed >>>>>>> queries, angry phone calls, abandoned shopping carts and loss of >>>>>>> engagement. >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> = https://can01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fwww.p9= 9conf.io%2Fsession%2Fmisery-metrics-consequences%2F&data=3D05%7C01%7C%= 7C0de1c0ddf51f41aef70c08dab4fde5ad%7Cb07c069022b843668d8d7b845d088e18%7C1%= 7C0%7C638021299817872156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI= joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3D%2FO= e2eo9f7JQ8bnQRB23HEaeXq6G9QxSQ%2FZkNb%2F6ctyU%3D&reserved=3D0 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> The ending advice was - don't aim to make a specific percentile >>>>>>> acceptable, aim for an acceptable % of misery. >>>>>>>=20 >>>>>>> I enjoyed the p99 conference more than any conference I've = attended in years. >>>>>>>=20 >>>>>>> -- >>>>>>> This song goes out to all the folk that thought Stadia would = work: >>>>>>>=20 >>>>>>>=20 >>>>>>> = https://can01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fwww.li= nkedin.com%2Fposts%2Fdtaht_the-mushroom-song-activity-6981366665607352320-= FXtz&data=3D05%7C01%7C%7C0de1c0ddf51f41aef70c08dab4fde5ad%7Cb07c069022= b843668d8d7b845d088e18%7C1%7C0%7C638021299817872156%7CUnknown%7CTWFpbGZsb3= d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C300= 0%7C%7C%7C&sdata=3DALKX4qknTgJBAiBET9j2yfdyhuEmM5rs2Ng3%2B09rat4%3D&am= p;reserved=3D0 >>>>>>>=20 >>>>>>>=20 >>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>=20 >>>>>>> -- >>>>>>> You received this message because you are subscribed to the = Google Groups "discuss" group. >>>>>>> To unsubscribe from this group and stop receiving emails from = it, send an email to >>>>>>>=20 >>>>>>> discuss+unsubscribe@measurementlab.net >>>>>>>=20 >>>>>>> . >>>>>>> To view this discussion on the web visit >>>>>>>=20 >>>>>>> = https://can01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgroups= .google.com%2Fa%2Fmeasurementlab.net%2Fd%2Fmsgid%2Fdiscuss%2FCAA93jw4w27a1= EO_QQG7NNkih%252BC3QQde5%253D_7OqGeS9xy9nB6wkg%2540mail.gmail.com&data= =3D05%7C01%7C%7C0de1c0ddf51f41aef70c08dab4fde5ad%7Cb07c069022b843668d8d7b8= 45d088e18%7C1%7C0%7C638021299817872156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4= wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&am= p;sdata=3DHVk9tgu97ElRdvdHiiE3PSuEzT6PM731Ag4XMIVDJIU%3D&reserved=3D0 >>>>>>>=20 >>>>>>> . >>>>>>> _______________________________________________ >>>>>>> Rpm mailing list >>>>>>>=20 >>>>>>>=20 >>>>>>> Rpm@lists.bufferbloat.net >>>>>>> = https://can01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.= bufferbloat.net%2Flistinfo%2Frpm&data=3D05%7C01%7C%7C0de1c0ddf51f41aef= 70c08dab4fde5ad%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C6380212998178= 72156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6= Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3D9Qd2WIP0ONe2zt%2FX3r0ws= 3QQMkRNjfmeY7dl9LH6T9k%3D&reserved=3D0 >>>>>> _______________________________________________ >>>>>> Starlink mailing list >>>>>>=20 >>>>>>=20 >>>>>> Starlink@lists.bufferbloat.net >>>>>> = https://can01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.= bufferbloat.net%2Flistinfo%2Fstarlink&data=3D05%7C01%7C%7C0de1c0ddf51f= 41aef70c08dab4fde5ad%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C63802129= 9817872156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ= BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DhEuwM2IalFt67cx%2F= kqQuHNR%2FL%2B8pwH0PKtMCiFMb6yU%3D&reserved=3D0 >>>>> -- >>>>> David Collier-Brown, | Always do right. This will gratify >>>>> System Programmer and Author | some people and astonish the rest >>>>>=20 >>>>>=20 >>>>> dave.collier-brown@indexexchange.com >>>>> | -- Mark Twain >>>>>=20 >>>>> CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, = including any and all attachments, contains confidential information = intended only for the person(s) to whom it is addressed. Any = dissemination, distribution, copying or disclosure is strictly = prohibited and is not a waiver of confidentiality. If you have received = this telecommunication in error, please notify the sender immediately by = return electronic mail and delete the message from your inbox and = deleted items folders. This telecommunication does not constitute an = express or implied agreement to conduct transactions by electronic = means, nor does it constitute a contract offer, a contract amendment or = an acceptance of a contract offer. Contract terms contained in this = telecommunication are subject to legal review and the completion of = formal documentation and are not binding until same is confirmed in = writing and has been signed by an authorized signatory. >>>>>=20 >>>>> _______________________________________________ >>>>> Starlink mailing list >>>>>=20 >>>>> Starlink@lists.bufferbloat.net >>>>> = https://can01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flists.= bufferbloat.net%2Flistinfo%2Fstarlink&data=3D05%7C01%7C%7C0de1c0ddf51f= 41aef70c08dab4fde5ad%7Cb07c069022b843668d8d7b845d088e18%7C1%7C0%7C63802129= 9817872156%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ= BTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3DhEuwM2IalFt67cx%2F= kqQuHNR%2FL%2B8pwH0PKtMCiFMb6yU%3D&reserved=3D0 >>> -- >>> David Collier-Brown, | Always do right. This will gratify >>> System Programmer and Author | some people and astonish the rest >>>=20 >>> dave.collier-brown@indexexchange.com | -- Mark Twain >=20 > --=20 > David Collier-Brown, | Always do right. This will gratify > System Programmer and Author | some people and astonish the rest > dave.collier-brown@indexexchange.com | -- Mark Twain >=20