From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 24A8E3CB41 for ; Mon, 12 Jul 2021 16:50:51 -0400 (EDT) Received: by mail-ed1-x52b.google.com with SMTP id ec55so5133391edb.1 for ; Mon, 12 Jul 2021 13:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=eJbks3bivJS4cPTq9eGVcjYlcWGIlz292IJv825Mr3Y=; b=f9KylBlPtWhoVUYgqBQu2M+gGsluz6NuKIHgKkZ+Rag6vXmFM4gPNO3DXWmje3L2uk FGcbTnR9YW+B8nwRswslkfFGHZxHGONYrMdD4Qx40OuVbNLJ6yM8CKJSI30trdPEbpHy 3oGYsRyLKTYC/qIuNFg00oYl87WQpO3B3jMkg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=eJbks3bivJS4cPTq9eGVcjYlcWGIlz292IJv825Mr3Y=; b=PStkF2+vIrJrqQselpt5e2Lv5Xg2Boov3CRwDheuapn37AJedlWSffyha8K+1cws7n Ym2tX4RWFi16S3t5IYWFZ09AxSZSjyI4xAp0gtCGZuEFoZr9jkpQKh6iwjwNqyezMIj4 cvA66i7yOmF9+XPFS7YrL530XStrcdh6tx1jyYzhY+r+UfshBE2xAA1sYxqJKmcJ3JXp 6fMf/VXz/1Gqusarhzk06KL/o8AtZhEaC/suIDfRu4choBXwUJ4qemQL22QE23kLIz87 ty/pyEksBbWJUr0OkhYZ3px0Godwfbz68jBR6sjZdVOn5057vLKNa/RFxs7A5FPxzhus sUWw== X-Gm-Message-State: AOAM532emktR5UY3u+FgejZdAegZlEsi59XYMnJ9CUJ0G8RH/SAkH4CZ q1/6q5D2W3sQdfUz6p+LBm0doyCZXMqA8t+H1tU00Mfg0QHv03zi1OOMuNwXR7a6qkmG7ObDSxT d9EVlKfRY9lUWls6feTA9tKZ3zezHqYMiTg== X-Google-Smtp-Source: ABdhPJwzJ08+2SMD310h8mypdnCHS2xvrrZ86DMWE/Gzw1wEkQlonJJnVb3tP9O8THwY7vt4Fw7azDFV9kuWZh6eg3o= X-Received: by 2002:a05:6402:14c7:: with SMTP id f7mr947605edx.84.1626123049913; Mon, 12 Jul 2021 13:50:49 -0700 (PDT) MIME-Version: 1.0 References: <1625188609.32718319@apps.rackspace.com> <989de0c1-e06c-cda9-ebe6-1f33df8a4c24@candelatech.com> <1625773080.94974089@apps.rackspace.com> <1625859083.09751240@apps.rackspace.com> <1626111630.69692379@apps.rackspace.com> <9c3d61c1-7013-414e-964d-9e83f596e69d@candelatech.com> <1e8bdf58-2a21-f543-a248-be58bcbddbcf@candelatech.com> In-Reply-To: From: Bob McMahon Date: Mon, 12 Jul 2021 13:50:39 -0700 Message-ID: To: David Lang Cc: Ben Greear , "starlink@lists.bufferbloat.net" , Make-Wifi-fast , Leonard Kleinrock , Cake List , "Livingood, Jason" , "codel@lists.bufferbloat.net" , cerowrt-devel , bloat Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="000000000000c5c72c05c6f3462c" X-Mailman-Approved-At: Tue, 13 Jul 2021 09:31:40 -0400 Subject: Re: [Starlink] [Cake] [Bloat] Little's Law mea culpa, but not invalidating my main point X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 20:50:51 -0000 --000000000000c5c72c05c6f3462c Content-Type: multipart/alternative; boundary="000000000000c0955e05c6f3468f" --000000000000c0955e05c6f3468f Content-Type: text/plain; charset="UTF-8" Agreed that UDP is important but it's also much easier to test and debug for WiFi coders. We find it's the connect() and TCP control loop that challenges WiFi logic systems on end hosts. APs are a whole different story per things like OFDMA. Nothing is simple anymore it seems. Reminds me of the standard model developed over time at CERN. It ain't E=MC**2 Bob On Mon, Jul 12, 2021 at 1:36 PM David Lang wrote: > I have seen some performance tests that do explicit DNS timing tests > separate > from other throughput/latency tests. > > Since DNS uses UDP (even if it then falls back to TCP in some cases), UDP > performance (and especially probability of loss at congested links) is > very > important. > > David Lang > > On Mon, 12 Jul 2021, Ben Greear wrote: > > > UDP is better for getting actual packet latency, for sure. TCP is > > typical-user-experience-latency though, > > so it is also useful. > > > > I'm interested in the test and visualization side of this. If there > were a > > way to give engineers > > a good real-time look at a complex real-world network, then they have > > something to go on while trying > > to tune various knobs in their network to improve it. > > > > I'll let others try to figure out how build and tune the knobs, but the > data > > acquisition and > > visualization is something we might try to accomplish. I have a feeling > I'm > > not the > > first person to think of this, however....probably someone already has > done > > such > > a thing. > > > > Thanks, > > Ben > > > > On 7/12/21 1:04 PM, Bob McMahon wrote: > >> I believe end host's TCP stats are insufficient as seen per the > "failed" > > congested control mechanisms over the last decades. I think Jaffe > pointed > > this out in > >> 1979 though he was using what's been deemed on this thread as > "spherical > > cow queueing theory." > >> > >> "Flow control in store-and-forward computer networks is appropriate for > > decentralized execution. A formal description of a class of > "decentralized > > flow control > >> algorithms" is given. The feasibility of maximizing power with such > > algorithms is investigated. On the assumption that communication links > behave > > like M/M/1 > >> servers it is shown that no "decentralized flow control algorithm" can > > maximize network power. Power has been suggested in the literature as a > > network > >> performance objective. It is also shown that no objective based only on > the > > users' throughputs and average delay is decentralizable. Finally, a > > restricted class > >> of algorithms cannot even approximate power." > >> > >> https://ieeexplore.ieee.org/document/1095152 > >> > >> Did Jaffe make a mistake? > >> > >> Also, it's been observed that latency is non-parametric in it's > > distributions and computing gaussians per the central limit theorem for > OWD > > feedback loops > >> aren't effective. How does one design a control loop around things that > are > > non-parametric? It also begs the question, what are the feed forward > knobs > > that can > >> actually help? > >> > >> Bob > >> > >> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear > > wrote: > >> > >> Measuring one or a few links provides a bit of data, but seems like > if > > someone is trying to understand > >> a large and real network, then the OWD between point A and B needs > to > > just be input into something much > >> more grand. Assuming real-time OWD data exists between 100 to 1000 > > endpoint pairs, has anyone found a way > >> to visualize this in a useful manner? > >> > >> Also, considering something better than ntp may not really scale to > > 1000+ endpoints, maybe round-trip > >> time is only viable way to get this type of data. In that case, > maybe > > clever logic could use things > >> like trace-route to get some idea of how long it takes to get > 'onto' > > the internet proper, and so estimate > >> the last-mile latency. My assumption is that the last-mile latency > is > > where most of the pervasive > >> assymetric network latencies would exist (or just ping 8.8.8.8 > which is > > 20ms from everywhere due to > >> $magic). > >> > >> Endpoints could also triangulate a bit if needed, using some anchor > > points in the network > >> under test. > >> > >> Thanks, > >> Ben > >> > >> On 7/12/21 11:21 AM, Bob McMahon wrote: > >> > iperf 2 supports OWD and gives full histograms for TCP write to > > read, TCP connect times, latency of packets (with UDP), latency of > "frames" > > with > >> > simulated video traffic (TCP and UDP), xfer times of bursts with > low > > duty cycle traffic, and TCP RTT (sampling based.) It also has support > for > > sampling (per > >> > interval reports) down to 100 usecs if configured with > > --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've > released > > all this as open source. > >> > > >> > OWD only works if the end realtime clocks are synchronized using > a > > "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data > centers > > don't > >> provide > >> > sufficient level of clock accuracy and the GPS pulse per second > * to > > colo and vm customers. > >> > > >> > https://iperf2.sourceforge.io/iperf-manpage.html > >> > > >> > Bob > >> > > >> > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed < > dpreed@deepplum.com > > >> >> wrote: > >> > > >> > > >> > On Monday, July 12, 2021 9:46am, "Livingood, Jason" > > > >> > >> said: > >> > > >> > > I think latency/delay is becoming seen to be as important > > certainly, if not a more direct proxy for end user QoE. This is all > still > > evolving and I > >> have > >> > to say is a super interesting & fun thing to work on. :-) > >> > > >> > If I could manage to sell one idea to the management > hierarchy > > of communications industry CEOs (operators, vendors, ...) it is this one: > >> > > >> > "It's the end-to-end latency, stupid!" > >> > > >> > And I mean, by end-to-end, latency to complete a task at a > > relevant layer of abstraction. > >> > > >> > At the link level, it's packet send to packet receive > > completion. > >> > > >> > But at the transport level including retransmission buffers, > > it's datagram (or message) origination until the acknowledgement arrives > for > > that > >> message being > >> > delivered after whatever number of retransmissions, freeing > the > > retransmission buffer. > >> > > >> > At the WWW level, it's mouse click to display update > > corresponding to completion of the request. > >> > > >> > What should be noted is that lower level latencies don't > > directly predict the magnitude of higher-level latencies. But longer > lower > > level latencies > >> almost > >> > always amplfify higher level latencies. Often non-linearly. > >> > > >> > Throughput is very, very weakly related to these latencies, > in > > contrast. > >> > > >> > The amplification process has to do with the presence of > > queueing. Queueing is ALWAYS bad for latency, and throughput only helps > if it > > is in exactly the > >> > right place (the so-called input queue of the bottleneck > > process, which is often a link, but not always). > >> > > >> > Can we get that slogan into Harvard Business Review? Can we > get > > it taught in Managerial Accounting at HBS? (which does address > > logistics/supply chain > >> queueing). > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > This electronic communication and the information and any files > > transmitted with it, or attached to it, are confidential and are > intended > > solely for the > >> use of > >> > the individual or entity to whom it is addressed and may contain > > information that is confidential, legally privileged, protected by > privacy > > laws, or > >> otherwise > >> > restricted from disclosure to anyone else. If you are not the > > intended recipient or the person responsible for delivering the e-mail > to the > > intended > >> recipient, > >> > you are hereby notified that any use, copying, distributing, > > dissemination, forwarding, printing, or copying of this e-mail is > strictly > > prohibited. If you > >> > received this e-mail in error, please return the e-mail to the > > sender, delete it from your computer, and destroy any printed copy of it. > >> > >> > >> -- > >> Ben Greear >> > >> Candela Technologies Inc http://www.candelatech.com > >> > >> > >> This electronic communication and the information and any files > transmitted > > with it, or attached to it, are confidential and are intended solely for > the > > use of > >> the individual or entity to whom it is addressed and may contain > > information that is confidential, legally privileged, protected by > privacy > > laws, or otherwise > >> restricted from disclosure to anyone else. If you are not the intended > > recipient or the person responsible for delivering the e-mail to the > intended > > recipient, > >> you are hereby notified that any use, copying, distributing, > dissemination, > > forwarding, printing, or copying of this e-mail is strictly prohibited. > If > > you > >> received this e-mail in error, please return the e-mail to the sender, > > delete it from your computer, and destroy any printed copy of it. > > > > > > -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it. --000000000000c0955e05c6f3468f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Agreed that UDP is important but it's also much easier= to test and debug for WiFi coders. We find it's the connect() and TCP = control loop that challenges WiFi logic systems on end hosts. APs=C2=A0are= =C2=A0a whole different story per things like OFDMA.

Nothing is simp= le anymore it seems. Reminds me of the standard model developed over time a= t CERN. It ain't E=3DMC**2

Bob

On Mon, Jul 12, 2021 at 1:36 PM Da= vid Lang <david@lang.hm> wrote:<= br>
I have seen some= performance tests that do explicit DNS timing tests separate
from other throughput/latency tests.

Since DNS uses UDP (even if it then falls back to TCP in some cases), UDP <= br> performance (and especially probability of loss at congested links) is very=
important.

David Lang

On Mon, 12 Jul 2021, Ben Greear wrote:

> UDP is better for getting actual packet latency, for sure.=C2=A0 TCP i= s
> typical-user-experience-latency though,
> so it is also useful.
>
> I'm interested in the test and visualization side of this.=C2=A0 I= f there were a
> way to give engineers
> a good real-time look at a complex real-world network, then they have =
> something to go on while trying
> to tune various knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, bu= t the data
> acquisition and
> visualization is something we might try to accomplish.=C2=A0 I have a = feeling I'm
> not the
> first person to think of this, however....probably someone already has= done
> such
> a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
>> I believe end host's TCP stats are insufficient as seen per th= e "failed"
> congested control mechanisms over the last decades. I think Jaffe poin= ted
> this out in
>> 1979 though he was using what's been deemed on this thread as = "spherical
> cow queueing theory."
>>
>> "Flow control in store-and-forward computer networks is appro= priate for
> decentralized execution. A formal description of a class of "dece= ntralized
> flow control
>> algorithms" is given. The feasibility of maximizing power wit= h such
> algorithms is investigated. On the assumption that communication links= behave
> like M/M/1
>> servers it is shown that no "decentralized flow control algor= ithm" can
> maximize network power. Power has been suggested in the literature as = a
> network
>> performance objective. It is also shown that no objective based on= ly on the
> users' throughputs and average delay is decentralizable. Finally, = a
> restricted class
>> of algorithms cannot even approximate power."
>>
>> https://ieeexplore.ieee.org/document/1095152
>>
>> Did Jaffe make a mistake?
>>
>> Also, it's been observed that latency=C2=A0is non-parametric i= n it's
> distributions and computing gaussians=C2=A0per the central limit theor= em for OWD
> feedback loops
>> aren't effective. How does one design a control loop around th= ings that are
> non-parametric? It also begs the question, what are the feed forward k= nobs
> that can
>> actually help?
>>
>> Bob
>>
>> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear <greearb@candelatech.com
> <mailto:greearb@candelatech.com>> wrote:
>>
>>=C2=A0 =C2=A0 =C2=A0Measuring one or a few links provides a bit of = data, but seems like if
> someone is trying to understand
>>=C2=A0 =C2=A0 =C2=A0a large and real network, then the OWD between = point A and B needs to
> just be input into something much
>>=C2=A0 =C2=A0 =C2=A0more grand.=C2=A0 Assuming real-time OWD data e= xists between 100 to 1000
> endpoint pairs, has anyone found a way
>>=C2=A0 =C2=A0 =C2=A0to visualize this in a useful manner?
>>
>>=C2=A0 =C2=A0 =C2=A0Also, considering something better than ntp may= not really scale to
> 1000+ endpoints, maybe round-trip
>>=C2=A0 =C2=A0 =C2=A0time is only viable way to get this type of dat= a.=C2=A0 In that case, maybe
> clever logic could use things
>>=C2=A0 =C2=A0 =C2=A0like trace-route to get some idea of how long i= t takes to get 'onto'
> the internet proper, and so estimate
>>=C2=A0 =C2=A0 =C2=A0the last-mile latency.=C2=A0 My assumption is t= hat the last-mile latency is
> where most of the pervasive
>>=C2=A0 =C2=A0 =C2=A0assymetric network latencies would exist (or ju= st ping 8.8.8.8 which is
> 20ms from everywhere due to
>>=C2=A0 =C2=A0 =C2=A0$magic).
>>
>>=C2=A0 =C2=A0 =C2=A0Endpoints could also triangulate a bit if neede= d, using some anchor
> points in the network
>>=C2=A0 =C2=A0 =C2=A0under test.
>>
>>=C2=A0 =C2=A0 =C2=A0Thanks,
>>=C2=A0 =C2=A0 =C2=A0Ben
>>
>>=C2=A0 =C2=A0 =C2=A0On 7/12/21 11:21 AM, Bob McMahon wrote:
>>=C2=A0 =C2=A0 =C2=A0 > iperf 2 supports OWD and gives full histo= grams for TCP write to
> read, TCP connect times, latency of packets (with UDP), latency of &qu= ot;frames"
> with
>>=C2=A0 =C2=A0 =C2=A0 > simulated=C2=A0video=C2=A0traffic (TCP an= d UDP), xfer times of bursts with low
> duty cycle traffic, and TCP RTT (sampling based.) It also has support = for
> sampling (per
>>=C2=A0 =C2=A0 =C2=A0 > interval reports) down to 100 usecs if co= nfigured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We'= ve released
> all this as open source.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 > OWD only works if the end realtime clocks= are synchronized using a
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *m= ost data centers
> don't
>>=C2=A0 =C2=A0 =C2=A0provide
>>=C2=A0 =C2=A0 =C2=A0 > sufficient level of clock accuracy and th= e GPS pulse per second * to
> colo and vm customers.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 > https://iperf2.sou= rceforge.io/iperf-manpage.html
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 > Bob
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 > On Mon, Jul 12, 2021 at 10:40 AM David P.= Reed <dpreed@d= eepplum.com
> <mailto:dp= reed@deepplum.com> <mailto:dpreed@deepplum.com
>>=C2=A0 =C2=A0 =C2=A0<mailto:dpreed@deepplum.com>>> wrote:
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0On Monday, July 12, 20= 21 9:46am, "Livingood, Jason"
> <J= ason_Livingood@comcast.com <mailto:Jason_Livingood@comcast.com>
>>=C2=A0 =C2=A0 =C2=A0<mailto:Jason_Livingood@comcast.com
> <mailto:Jason_Livingood@comcast.com>>> said:
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0 > I think latency/= delay is becoming seen to be as important
> certainly, if not a more direct proxy for end user QoE. This is all st= ill
> evolving and I
>>=C2=A0 =C2=A0 =C2=A0have
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0to say is a super inte= resting & fun thing to work on. :-)
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0If I could manage to s= ell one idea to the management hierarchy
> of communications industry CEOs (operators, vendors, ...) it is this o= ne:
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0"It's the end= -to-end latency, stupid!"
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0And I mean, by end-to-= end, latency to complete a task at a
> relevant layer of abstraction.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0At the link level, it&= #39;s packet send to packet receive
> completion.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0But at the transport l= evel including retransmission buffers,
> it's datagram (or message) origination until the acknowledgement a= rrives for
> that
>>=C2=A0 =C2=A0 =C2=A0message being
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0delivered after whatev= er number of retransmissions, freeing the
> retransmission buffer.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0At the WWW level, it&#= 39;s mouse click to display update
> corresponding to completion of the request.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0What should be noted i= s that lower level latencies don't
> directly predict the magnitude of higher-level latencies. But longer l= ower
> level latencies
>>=C2=A0 =C2=A0 =C2=A0almost
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0always amplfify higher= level latencies. Often non-linearly.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0Throughput is very, ve= ry weakly related to these latencies, in
> contrast.
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0The amplification proc= ess has to do with the presence of
> queueing. Queueing is ALWAYS bad for latency, and throughput only help= s if it
> is in exactly the
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0right place (the so-ca= lled input queue of the bottleneck
> process, which is often a link, but not always).
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0Can we get that slogan= into Harvard Business Review? Can we get
> it taught in Managerial Accounting at HBS? (which does address
> logistics/supply chain
>>=C2=A0 =C2=A0 =C2=A0queueing).
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 >
>>=C2=A0 =C2=A0 =C2=A0 > This electronic communication and the inf= ormation and any files
> transmitted with it, or attached to it, are confidential and are inten= ded
> solely for the
>>=C2=A0 =C2=A0 =C2=A0use of
>>=C2=A0 =C2=A0 =C2=A0 > the individual or entity to whom it is ad= dressed and may contain
> information that is confidential, legally privileged, protected by pri= vacy
> laws, or
>>=C2=A0 =C2=A0 =C2=A0otherwise
>>=C2=A0 =C2=A0 =C2=A0 > restricted from disclosure to anyone else= . If you are not the
> intended recipient or the person responsible for delivering the e-mail= to the
> intended
>>=C2=A0 =C2=A0 =C2=A0recipient,
>>=C2=A0 =C2=A0 =C2=A0 > you are hereby notified that any use, cop= ying, distributing,
> dissemination, forwarding, printing, or copying of this e-mail is stri= ctly
> prohibited. If you
>>=C2=A0 =C2=A0 =C2=A0 > received this e-mail in error, please ret= urn the e-mail to the
> sender, delete it from your computer, and destroy any printed copy of = it.
>>
>>
>>=C2=A0 =C2=A0 =C2=A0--
>>=C2=A0 =C2=A0 =C2=A0Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.c= om>>
>>=C2=A0 =C2=A0 =C2=A0Candela Technologies Inc http://www.candelatec= h.com
>>
>>
>> This electronic communication and the information and any files tr= ansmitted
> with it, or attached to it, are confidential and are intended solely f= or the
> use of
>> the individual or entity to whom it is addressed and may contain <= br> > information that is confidential, legally privileged, protected by pri= vacy
> laws, or otherwise
>> restricted from disclosure to anyone else. If you are not the inte= nded
> recipient or the person responsible for delivering the e-mail to the i= ntended
> recipient,
>> you are hereby notified that any use, copying, distributing, disse= mination,
> forwarding, printing, or copying of this e-mail is strictly prohibited= . If
> you
>> received this e-mail in error, please return the e-mail to the sen= der,
> delete it from your computer, and destroy any printed copy of it.
>
>
>

This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it. --000000000000c0955e05c6f3468f-- --000000000000c5c72c05c6f3462c Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIQagYJKoZIhvcNAQcCoIIQWzCCEFcCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg gg3BMIIFDTCCA/WgAwIBAgIQeEqpED+lv77edQixNJMdADANBgkqhkiG9w0BAQsFADBMMSAwHgYD VQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE AxMKR2xvYmFsU2lnbjAeFw0yMDA5MTYwMDAwMDBaFw0yODA5MTYwMDAwMDBaMFsxCzAJBgNVBAYT AkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYDVQQDEyhHbG9iYWxTaWduIEdDQyBS MyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA vbCmXCcsbZ/a0fRIQMBxp4gJnnyeneFYpEtNydrZZ+GeKSMdHiDgXD1UnRSIudKo+moQ6YlCOu4t rVWO/EiXfYnK7zeop26ry1RpKtogB7/O115zultAz64ydQYLe+a1e/czkALg3sgTcOOcFZTXk38e aqsXsipoX1vsNurqPtnC27TWsA7pk4uKXscFjkeUE8JZu9BDKaswZygxBOPBQBwrA5+20Wxlk6k1 e6EKaaNaNZUy30q3ArEf30ZDpXyfCtiXnupjSK8WU2cK4qsEtj09JS4+mhi0CTCrCnXAzum3tgcH cHRg0prcSzzEUDQWoFxyuqwiwhHu3sPQNmFOMwIDAQABo4IB2jCCAdYwDgYDVR0PAQH/BAQDAgGG MGAGA1UdJQRZMFcGCCsGAQUFBwMCBggrBgEFBQcDBAYKKwYBBAGCNxQCAgYKKwYBBAGCNwoDBAYJ KwYBBAGCNxUGBgorBgEEAYI3CgMMBggrBgEFBQcDBwYIKwYBBQUHAxEwEgYDVR0TAQH/BAgwBgEB /wIBADAdBgNVHQ4EFgQUljPR5lgXWzR1ioFWZNW+SN6hj88wHwYDVR0jBBgwFoAUj/BLf6guRSSu TVD6Y5qL3uLdG7wwegYIKwYBBQUHAQEEbjBsMC0GCCsGAQUFBzABhiFodHRwOi8vb2NzcC5nbG9i YWxzaWduLmNvbS9yb290cjMwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5j b20vY2FjZXJ0L3Jvb3QtcjMuY3J0MDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFs c2lnbi5jb20vcm9vdC1yMy5jcmwwWgYDVR0gBFMwUTALBgkrBgEEAaAyASgwQgYKKwYBBAGgMgEo CjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAN BgkqhkiG9w0BAQsFAAOCAQEAdAXk/XCnDeAOd9nNEUvWPxblOQ/5o/q6OIeTYvoEvUUi2qHUOtbf jBGdTptFsXXe4RgjVF9b6DuizgYfy+cILmvi5hfk3Iq8MAZsgtW+A/otQsJvK2wRatLE61RbzkX8 9/OXEZ1zT7t/q2RiJqzpvV8NChxIj+P7WTtepPm9AIj0Keue+gS2qvzAZAY34ZZeRHgA7g5O4TPJ /oTd+4rgiU++wLDlcZYd/slFkaT3xg4qWDepEMjT4T1qFOQIL+ijUArYS4owpPg9NISTKa1qqKWJ jFoyms0d0GwOniIIbBvhI2MJ7BSY9MYtWVT5jJO3tsVHwj4cp92CSFuGwunFMzCCA18wggJHoAMC AQICCwQAAAAAASFYUwiiMA0GCSqGSIb3DQEBCwUAMEwxIDAeBgNVBAsTF0dsb2JhbFNpZ24gUm9v dCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTA5 MDMxODEwMDAwMFoXDTI5MDMxODEwMDAwMFowTDEgMB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENB IC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDMJXaQeQZ4Ihb1wIO2hMoonv0FdhHFrYhy/EYCQ8eyip0E XyTLLkvhYIJG4VKrDIFHcGzdZNHr9SyjD4I9DCuul9e2FIYQebs7E4B3jAjhSdJqYi8fXvqWaN+J J5U4nwbXPsnLJlkNc96wyOkmDoMVxu9bi9IEYMpJpij2aTv2y8gokeWdimFXN6x0FNx04Druci8u nPvQu7/1PQDhBjPogiuuU6Y6FnOM3UEOIDrAtKeh6bJPkC4yYOlXy7kEkmho5TgmYHWyn3f/kRTv riBJ/K1AFUjRAjFhGV64l++td7dkmnq/X8ET75ti+w1s4FRpFqkD2m7pg5NxdsZphYIXAgMBAAGj QjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSP8Et/qC5FJK5N UPpjmove4t0bvDANBgkqhkiG9w0BAQsFAAOCAQEAS0DbwFCq/sgM7/eWVEVJu5YACUGssxOGhigH M8pr5nS5ugAtrqQK0/Xx8Q+Kv3NnSoPHRHt44K9ubG8DKY4zOUXDjuS5V2yq/BKW7FPGLeQkbLmU Y/vcU2hnVj6DuM81IcPJaP7O2sJTqsyQiunwXUaMld16WCgaLx3ezQA3QY/tRG3XUyiXfvNnBB4V 14qWtNPeTCekTBtzc3b0F5nCH3oO4y0IrQocLP88q1UOD5F+NuvDV0m+4S4tfGCLw0FREyOdzvcy a5QBqJnnLDMfOjsl0oZAzjsshnjJYS8Uuu7bVW/fhO4FCU29KNhyztNiUGUe65KXgzHZs7XKR1g/ XzCCBUkwggQxoAMCAQICDBhL7k9eiTHfluW70TANBgkqhkiG9w0BAQsFADBbMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTExMC8GA1UEAxMoR2xvYmFsU2lnbiBHQ0MgUjMg UGVyc29uYWxTaWduIDIgQ0EgMjAyMDAeFw0yMTAyMjIwNDQyMDRaFw0yMjA5MDEwODA5NDlaMIGM MQswCQYDVQQGEwJJTjESMBAGA1UECBMJS2FybmF0YWthMRIwEAYDVQQHEwlCYW5nYWxvcmUxFjAU BgNVBAoTDUJyb2FkY29tIEluYy4xFDASBgNVBAMTC0JvYiBNY01haG9uMScwJQYJKoZIhvcNAQkB Fhhib2IubWNtYWhvbkBicm9hZGNvbS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB AQDyY95HWFm48WhKUyFbAS9JxiDqBHBdAbgjx4iF46lkqZdVkIJ8pGfcXoGd10Vp9yL5VQevDAt/ A/Jh22uhSgKR9Almeux9xWGhG8cyZwcCwYrsMt84FqCgEQidT+7YGNdd9oKrjU7mFC7pAnnw+cGI d3NFryurgnNPwfEK0X7HwRsga5pM+Zelr/ZM8MkphE1hCvTuPGakNylOFhP+wKL8Bmhsq5tNIInw DrPV5EPUikwiGMDmkX8o6roGiUwyqAp8dMZKJZ/vS/aWEELV+gm21Btr7eqdAWyqm09McVpkM4th v/FOYcj8DeJr8MXmHW53gN2fv0BzQjqAdrdCBPNRAgMBAAGjggHZMIIB1TAOBgNVHQ8BAf8EBAMC BaAwgaMGCCsGAQUFBwEBBIGWMIGTME4GCCsGAQUFBzAChkJodHRwOi8vc2VjdXJlLmdsb2JhbHNp Z24uY29tL2NhY2VydC9nc2djY3IzcGVyc29uYWxzaWduMmNhMjAyMC5jcnQwQQYIKwYBBQUHMAGG NWh0dHA6Ly9vY3NwLmdsb2JhbHNpZ24uY29tL2dzZ2NjcjNwZXJzb25hbHNpZ24yY2EyMDIwME0G A1UdIARGMEQwQgYKKwYBBAGgMgEoCjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxz aWduLmNvbS9yZXBvc2l0b3J5LzAJBgNVHRMEAjAAMEkGA1UdHwRCMEAwPqA8oDqGOGh0dHA6Ly9j cmwuZ2xvYmFsc2lnbi5jb20vZ3NnY2NyM3BlcnNvbmFsc2lnbjJjYTIwMjAuY3JsMCMGA1UdEQQc MBqBGGJvYi5tY21haG9uQGJyb2FkY29tLmNvbTATBgNVHSUEDDAKBggrBgEFBQcDBDAfBgNVHSME GDAWgBSWM9HmWBdbNHWKgVZk1b5I3qGPzzAdBgNVHQ4EFgQUpyXYr5rh8cZzkns+zXmMG1YkBk4w DQYJKoZIhvcNAQELBQADggEBACfauRPak93nzbpn8UXqRZqg6iUZch/UfGj9flerMl4TlK5jWulz Y+rRg+iWkjiLk3O+kKu6GI8TLXB2rsoTnrHYij96Uad5/Ut3Q5F4S0ILgOWVU38l0VZIGGG0CzG1 eLUgN2zjLg++xJuzqijuKQCJb/3+il2MTJ8dcDaXuYcjg7Vt6+EtCBS1SGMVhOTH4Fp50yGWj8ZA bPF1uuJM+dGLJLheUizCr5J/OBEdENg+DSmrqoZ+kZd76iRaF2CkhboR2394Ft8lFlKQiU0q8lnR 9/kdZ0F0iCcUfhaLaGYWujW7N0LZ+rQuTfuPGLx9zZNeNMWSZi/Pc8vdCO7EnlIxggJtMIICaQIB ATBrMFsxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYDVQQDEyhH bG9iYWxTaWduIEdDQyBSMyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwAgwYS+5PXokx35blu9EwDQYJ YIZIAWUDBAIBBQCggdQwLwYJKoZIhvcNAQkEMSIEIOozTVTNsqK796VW+EcLGKn+o8WMZMWKSIcd mj3BJqjCMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIxMDcxMjIw NTA1MFowaQYJKoZIhvcNAQkPMVwwWjALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAsGCWCGSAFl AwQBAjAKBggqhkiG9w0DBzALBgkqhkiG9w0BAQowCwYJKoZIhvcNAQEHMAsGCWCGSAFlAwQCATAN BgkqhkiG9w0BAQEFAASCAQAZiYZo05hH9v2bDsUOYH9JoETr39aKXY7LGET5yoAB/Mnmr8szUDhK eyA4tk7sCF0bD+G+uUh9zZKdKd/fvTRWi8e6vtqoDAf3cX4dGjc3bguZ+aXpEUupBFNFQfVZ+K7W 2h5Cp7WbLqdKx+ima4lwnN4wPaPiOw56ewSfLbSFE4YK0gh2HiKWSSrGm9POab1B3vsWqNfkRrrw sxtM18sU2RXKKXvzOvJ5FVU/ShoyZQhiqAyncP5SxZAaQWg9/0eqXs94qgQQ9lRi+uF8ny9vpdfg Lf3HoYT2ZHcJleTiu58Lrrr7jJbxN+5nZ50gOv+wTS2NWJjmzYAk+MGkCq55 --000000000000c5c72c05c6f3462c--