[Bloat] tiny monsters: multicast packets

[Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2534 bytes --]

So after my experiments [1] yesterday with the wndr3700v2 hardware[2], I
came away
even more convinced that the wireless world and the wired worlds should
not be bridged together.

All the AQMs out there assume that it takes the same period of time to
deliver a packet consisting of X bytes to the next hop. Wired more or less
does that.

Wireless breaks that assumption. It wasn't so bad, back in the early days of
802.11b - 802.11b ran as fast as 11Mbit, and multicast, at 2... for a ratio
of 5.5x1.

11g came around, and runs at 54Mbit, and - (if you don't run in mixed mode,
still supporting B), you can multicast at 6Mbit. But nearly everyone does
still run mixed mode, so multicast is stuck at 2... for a ratio of 26x1....
instead of a mere 9x1.

Soo... 11n has come around, and I shudder to think of what packet rate ratio
of "normal" vs "multicast packets" do to the assumptions of the rest of the
stack.

So with just a little multicast going through your wireless network, any
assumptions the higher level portions of the stack might make are invalid.
HTB? hah, uses fixed buckets... RED? a single multicast packet is a monster
packet, how's it supposed to find it in the swamp?

Worse, most multicast packets are statistically rare and needed for the
network to actually continue to function.

In my last 2 months of travel, I have seen multicast packets, such as ARP,
DHCP, MDNS, and now babel, all failing far, far, far more often than is
desirable. I have seen DHCP fail completely for hours at a time, I've seen
ARP take dozens of queries to resolve.

Nextly, It is trivial to trigger the symptoms of bufferbloat, with a
multicast stream.

Perhaps eBDP can handle multicast well, but certainly AQMs are going to have
headaches that are difficult to solve at ratios between normal and multicast
packets this poor.

Lastly, most home router vendors bridge wired and wireless together, sort of
like jamming together "jet engines and a vw bugs",  and I finally broke them
apart [3], to try and look at them separately - as even the switch is
displaying 100+ms of buffering when I slam it with 4 simultaneous iperf
streams [4]

1: https://lists.bufferbloat.net/pipermail/bloat-devel/2011-May/000156.html
2: http://www.bufferbloat.net/projects/bismark/wiki/Capetown
3: http://www.bufferbloat.net/issues/186
4: http://www.bufferbloat.net/projects/bismark-testbed/wiki/Experiment_-_QoS

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 3016 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Jonathan Morton]

On 29 May, 2011, at 4:23 pm, Dave Taht wrote:

> In my last 2 months of travel, I have seen multicast packets, such as ARP, DHCP, MDNS, and now babel, all failing far, far, far more often than is desirable. I have seen DHCP fail completely for hours at a time, I've seen ARP take dozens of queries to resolve. 

And the irony is that the lower speed is specifically chosen for multicast in order to make sure all clients in range can hear them reliably.  Broadcast packets are not supposed to be large ones, but wireless framing must add a lot of fixed overhead.

Given that the AP surely knows which hosts are attached to it at any given time, and what link rate they are currently sustaining, surely a saner design would have been either:

1) Broadcast the packet at the lowest link rate for all known attached hosts.

2) Unicast the packet to each attached host in turn, at that host's current link rate.

The latter sounds wasteful, but would still be a win on 802.11g in compatibility mode.  It also turns the AP into a star-topology hub, so hosts would send their broadcast packets by unicast to the AP, which would repeat them.

But presumably the brokenness is now baked firmly into the standard, and is therefore inescapable.  So the workaround is to isolate the broadcast domains of wired networks and wireless networks by making the home router into...  a router.  Wireless on one subnet, wired on another, and so ARP between the two turns into ARP to the router alone - much more scalable.

I should check whether my Airport Base Station already supports that.

 - Jonathan

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 3070 bytes --]

On Sun, May 29, 2011 at 8:10 AM, Jonathan Morton <chromatix99@gmail.com>wrote:

>
> On 29 May, 2011, at 4:23 pm, Dave Taht wrote:
>
> > In my last 2 months of travel, I have seen multicast packets, such as
> ARP, DHCP, MDNS, and now babel, all failing far, far, far more often than is
> desirable. I have seen DHCP fail completely for hours at a time, I've seen
> ARP take dozens of queries to resolve.
>
> And the irony is that the lower speed is specifically chosen for multicast
> in order to make sure all clients in range can hear them reliably.
>  Broadcast packets are not supposed to be large ones, but wireless framing
> must add a lot of fixed overhead.
>
> Given that the AP surely knows which hosts are attached to it at any given
> time, and what link rate they are currently sustaining, surely a saner
> design would have been either:
>
> 1) Broadcast the packet at the lowest link rate for all known attached
> hosts.
>
> 2) Unicast the packet to each attached host in turn, at that host's current
> link rate.
>
> The latter sounds wasteful, but would still be a win on 802.11g in
> compatibility mode.  It also turns the AP into a star-topology hub, so hosts
> would send their broadcast packets by unicast to the AP, which would repeat
> them.
>
> But presumably the brokenness is now baked firmly into the standard, and is
> therefore inescapable.  So the workaround is to isolate the broadcast
> domains of wired networks and wireless networks by making the home router
> into...  a router.  Wireless on one subnet, wired on another, and so ARP
> between the two turns into ARP to the router alone - much more scalable.
>
> I should check whether my Airport Base Station already supports that.
>

I just managed (after fighting with the switch in the wndr3700, and having
to disable vlan support) to treat the wireless and wired lans separately
[1].

Result - 130+Mbit performance on iperf on the lan (up from 60Mbit), which is
still pretty low - [2]

AND multicast stopped failing, in the limited testing I'd done so far.
Yea... [3]

Yes, I think home routers should route, not bridge between widely disparate
network types.

Bridging between a gige interface, a wireless N 5.x gigE interface, and a
2.4 ghz interface in compatability mode is just begging for trouble.
Especially with a switch on the gigE interface that is also doing buffering
itself.

Switching to routing throughout does induce more complexity in the network
(3 or more subnets, rather than 1), and for the end-user...

... and all the multicast-isms that have arisen in the last decade such as
mdns and daap would need to be looked at, to see if they can actually be
made to work... (IGMPv2 ?)

but I can no longer think of a better solution to the nearly intolerable
performance of today's wireless networks than starting to route them, and
dealing with the consequences above.



>  - Jonathan
>
>

1: http://www.bufferbloat.net/issues/186
2: I'm still seeing enormous delays in the switch, however
3: http://www.bufferbloat.net/projects/bismark-testbed/wiki/Experiment_-_QoS

[-- Attachment #2: Type: text/html, Size: 3941 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Juliusz Chroboczek]

> Result - 130+Mbit performance on iperf on the lan (up from 60Mbit),
> which is still pretty low

Are you seeing high CPU load in interrupt context?  (Run top.)

-- Juliusz

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 587 bytes --]

On Sun, May 29, 2011 at 9:33 AM, Juliusz Chroboczek <jch@pps.jussieu.fr>wrote:

> > Result - 130+Mbit performance on iperf on the lan (up from 60Mbit),
> > which is still pretty low
>
> Are you seeing high CPU load in interrupt context?  (Run top.)
>
>
Yes. 99% sirq.

Screenshot:

 http://www.bufferbloat.net/attachments/download/33/routertest.png

The one in green is the one 'routing', the others (3 out of the 4) are the
ones generating traffic with iperf.

-- Juliusz
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 1291 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Juliusz Chroboczek]

>> Are you seeing high CPU load in interrupt context?  (Run top.)

> Yes. 99% sirq.

Could be due to a simplistic Ethernet driver.  If you have the time and
energy, you may want to ask on dev.openwrt.org.

-- Juliusz

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

On Sun, May 29, 2011 at 9:51 AM, Juliusz Chroboczek <jch@pps.jussieu.fr>wrote:

> >> Are you seeing high CPU load in interrupt context?  (Run top.)
>
> > Yes. 99% sirq.
>
> Could be due to a simplistic Ethernet driver.  If you have the time and
> energy, you may want to ask on dev.openwrt.org.
>
> I will have some energy and time, shortly.

That said, several great openwrt people are on this list, and may be able to
weigh in.

I'm glad that the limit of about 130Mbit on the ethernet side for gigE could
be mitigated with a better driver. (and that said, 130Mbit is "good enough"
for most of the world)

On the other hand, making the switch that lays underneath this driver, work
well, looks hard.

Does anybody here speak enough Taiwanese to get enough detail on the
rtl8366s to see what it would take to enable fair queueing and port
mirroring?

or have a relationship with realtek they could use to get this info?

The datasheet has insufficient detail, and yet the switch seems enormously
capable, at least in theory. The kind of numbers under load I've seen thus
far (ranging from .9ms to 170ms) suggest port starvation.

http://realtek.info/pdf/rtl8366s_8366sr_datasheet_vpre-1.4_20071022.pdf



> -- Juliusz
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 2171 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]

sorry, I meant to reply all.

Thanks for so quickly seeing the real cause of the upper limit.

On Sun, May 29, 2011 at 10:07 AM, Dave Taht <dave.taht@gmail.com> wrote:

>
>
> On Sun, May 29, 2011 at 9:51 AM, Juliusz Chroboczek <jch@pps.jussieu.fr>wrote:
>
>> >> Are you seeing high CPU load in interrupt context?  (Run top.)
>>
>> > Yes. 99% sirq.
>>
>> Could be due to a simplistic Ethernet driver.  If you have the time and
>> energy, you may want to ask on dev.openwrt.org.
>>
>> I will have some energy and time, shortly.
>
> That said, several great openwrt people are on this list, and may be able
> to weigh in.
>
> I'm glad that the limit of about 130Mbit on the ethernet side for gigE
> could be mitigated with a better driver. (and that said, 130Mbit is "good
> enough" for most of the world)
>
> On the other hand, making the switch that lays underneath this driver, work
> well, looks hard.
>
> Does anybody here speak enough Taiwanese to get enough detail on the
> rtl8366s to see what it would take to enable fair queueing and port
> mirroring?
>
> or have a relationship with realtek they could use to get this info?
>
> The datasheet has insufficient detail, and yet the switch seems enormously
> capable, at least in theory. The kind of numbers under load I've seen thus
> far (ranging from .9ms to 170ms) suggest port starvation.
>
> http://realtek.info/pdf/rtl8366s_8366sr_datasheet_vpre-1.4_20071022.pdf
>
>
>
>>  -- Juliusz
>>
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://the-edge.blogspot.com
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 2975 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

The datasheet has insufficient detail, and yet the switch seems enormously
>> capable, at least in theory. The kind of numbers under load I've seen thus
>> far (ranging from .9ms to 170ms) suggest port starvation.
>>
>> http://realtek.info/pdf/rtl8366s_8366sr_datasheet_vpre-1.4_20071022.pdf
>>
>
In looking over an earlier datasheet, for an earlier revision of this switch
chip, the details for possibly enabling port mirroring (so as to be able to
look at streams using an external monitoring box), as well as twiddling with
fair queuing (to address the port starvation issue) are there, in sections
8.10, and 8.13.

http://realtek.info/pdf/rtl8366_8369_datasheet_1-1.pdf

They are not in the later datasheet, but the features are at least,
mentioned there.

I'm happy. Nothing left to do but code it up and see what happens.
That's the fun part.




>
>>
>>
>>>  -- Juliusz
>>>
>>
>>
>>
>> --
>> Dave Täht
>> SKYPE: davetaht
>> US Tel: 1-239-829-5608
>> http://the-edge.blogspot.com
>>
>
>
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://the-edge.blogspot.com
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 2864 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Eric Dumazet]

Le dimanche 29 mai 2011 à 10:07 -0600, Dave Taht a écrit :
> 
> 
> On Sun, May 29, 2011 at 9:51 AM, Juliusz Chroboczek
> <jch@pps.jussieu.fr> wrote:
>         >> Are you seeing high CPU load in interrupt context?  (Run
>         top.)
>         
>         > Yes. 99% sirq.
>         
>         
>         Could be due to a simplistic Ethernet driver.  If you have the
>         time and
>         energy, you may want to ask on dev.openwrt.org.
>         
> I will have some energy and time, shortly. 
> 
> That said, several great openwrt people are on this list, and may be
> able to weigh in.
> 
> I'm glad that the limit of about 130Mbit on the ethernet side for gigE
> could be mitigated with a better driver. (and that said, 130Mbit is
> "good enough" for most of the world)
> 

Hmm, 130 Mbits sound very low. What linux version is it, and what the
driver is exactly, so that I take a look ?

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

On Sun, May 29, 2011 at 10:33 AM, Eric Dumazet <eric.dumazet@gmail.com>wrote:

> Le dimanche 29 mai 2011 à 10:07 -0600, Dave Taht a écrit :
> >
> >
> > On Sun, May 29, 2011 at 9:51 AM, Juliusz Chroboczek
> > <jch@pps.jussieu.fr> wrote:
> >         >> Are you seeing high CPU load in interrupt context?  (Run
> >         top.)
> >
> >         > Yes. 99% sirq.
> >
> >
> >         Could be due to a simplistic Ethernet driver.  If you have the
> >         time and
> >         energy, you may want to ask on dev.openwrt.org.
> >
> > I will have some energy and time, shortly.
> >
> > That said, several great openwrt people are on this list, and may be
> > able to weigh in.
> >
> > I'm glad that the limit of about 130Mbit on the ethernet side for gigE
> > could be mitigated with a better driver. (and that said, 130Mbit is
> > "good enough" for most of the world)
> >
>
> Hmm, 130 Mbits sound very low. What linux version is it, and what the
> driver is exactly, so that I take a look ?
>
>
>
The ethernet driver is the ag71xx driver as present in the wndr3700v2 (and
mucho related atheros hardware).  Regrettably so far as I can tell, this one
is out of tree, and is incorporated in the openwrt build via a string of
patches.

https://dev.openwrt.org/browser/trunk/target/linux/ar71xx/files/drivers/net/ag71xx

I think.

Gabor sent patches for it to me for ethtool that I have not yet tried.

The kernel being used in capetown[1] is 2.6.37.6. - patched forward from
2.6.39 for the pfifo ecn bug, the ipv6 ecn bug, and several other
bufferbloat related things like sfb...


1: http://www.bufferbloat.net/projects/bismark/wiki/Capetown
-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 2592 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Eric Dumazet]

Le dimanche 29 mai 2011 à 11:02 -0600, Dave Taht a écrit :

>         
> 
> The ethernet driver is the ag71xx driver as present in the wndr3700v2
> (and mucho related atheros hardware).  Regrettably so far as I can
> tell, this one is out of tree, and is incorporated in the openwrt
> build via a string of patches. 
> 
> https://dev.openwrt.org/browser/trunk/target/linux/ar71xx/files/drivers/net/ag71xx

I'll take a look, thanks.

> Gabor sent patches for it to me for ethtool that I have not yet tried.
> 
> The kernel being used in capetown[1] is 2.6.37.6. - patched forward
> from 2.6.39 for the pfifo ecn bug, the ipv6 ecn bug, and several other
> bufferbloat related things like sfb...
> 

Hmm, maybe you could try "perf top" if available to make sure cpu time
is used in this driver, not in another layer ?

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Sun, May 29, 2011 at 11:17 AM, Eric Dumazet <eric.dumazet@gmail.com>wrote:

> Le dimanche 29 mai 2011 à 11:02 -0600, Dave Taht a écrit :
>
> > The kernel being used in capetown[1] is 2.6.37.6. - patched forward
> > from 2.6.39 for the pfifo ecn bug, the ipv6 ecn bug, and several other
> > bufferbloat related things like sfb...
> >
>
> Hmm, maybe you could try "perf top" if available to make sure cpu time
> is used in this driver, not in another layer ?
>
>
>
>
perf-top does not appear to have been ported to openwrt, and as cool as it
is, I do not know if the related performance counters exist for mips.

it would be darn useful though.

All I know at the moment is that sirq is at 99% at the given workload,
according busybox's top.

http://www.bufferbloat.net/attachments/download/33/routertest.png

It is mildly early to point at the driver as being the issue - it could be
the switch or something else entirely. Would iptables or qos rules show up
in sirq?


-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 1612 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Jonathan Morton]

On 29 May, 2011, at 8:40 pm, Dave Taht wrote:

> It is mildly early to point at the driver as being the issue - it could be the switch or something else entirely. Would iptables or qos rules show up in sirq?

I doubt it.  I run some pretty old hardware sometimes, and I've never seen sirq so high except for PIO IDE adapters.

The Pentium-MMX with it's VIA Rhine adapter keeps up with 100Mbps transfers without exhibiting those symptoms.

 - Jonathan

Re: [Bloat] tiny monsters: multicast packets

[Eric Dumazet]

Le dimanche 29 mai 2011 à 11:40 -0600, Dave Taht a écrit :

>         
> 
> perf-top does not appear to have been ported to openwrt, and as cool
> as it is, I do not know if the related performance counters exist for
> mips.
> 
> it would be darn useful though.
> 
> All I know at the moment is that sirq is at 99% at the given
> workload, 
> according busybox's top.
> 
> http://www.bufferbloat.net/attachments/download/33/routertest.png
> 
> It is mildly early to point at the driver as being the issue - it
> could be the switch or something else entirely. Would iptables or qos
> rules show up in sirq?
> 

Yes, about all this stuff is run in softirq. QOS can be a problem if
high resolution kernel time services are expensive on this platform.

You could try a raw routing stuff, with no QOS / iptables, just to see
how much packets can be received/sent by this unit.

And if it stills sucks, just do an input test only, dropping packets
immediately with "iptables -I INPUT -i name -j DROP" (to check how many
packets/second can be received by the driver)

Re: [Bloat] tiny monsters: multicast packets

[Juliusz Chroboczek]

> And the irony is that the lower speed is specifically chosen for
> multicast in order to make sure all clients in range can hear them
> reliably.

It was my understanding that it was done for compatibility with older
devices, since 2 Mbit/s is the fastest rate supported by pre-B
spread-spectrum hardware.

> 2) Unicast the packet to each attached host in turn,

Both DVMR and the multicast part of BATMAN-adv do that for router-router
links.

A better link-layer solution, IMHO, would be to multicast (with ARQ) the
packet at a reasonably high rate (say, the median of the STAs subscribed
to the multicast group), and then unicast it to all STAs that failed to
return an ACK.  Interestingly, if the new multicast frame format is
incompatible with the normal 802.11 format, then this scheme is
compatible with legacy devices, which won't ever see the multicast frame
and hence won't return an ACK.

> So the workaround is to isolate the broadcast domains of wired
> networks and wireless networks by making the home router into...
> a router.  Wireless on one subnet, wired on another, and so ARP
> between the two turns into ARP to the router alone - much more
> scalable.

OpenWRT is your friend.

-- Juliusz

Re: [Bloat] tiny monsters: multicast packets

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2600 bytes --]

On Sun, May 29, 2011 at 9:21 AM, Juliusz Chroboczek <jch@pps.jussieu.fr>wrote:

> > And the irony is that the lower speed is specifically chosen for
> > multicast in order to make sure all clients in range can hear them
> > reliably.
>
> It was my understanding that it was done for compatibility with older
> devices, since 2 Mbit/s is the fastest rate supported by pre-B
> spread-spectrum hardware.
>
>
And thus, everybody loses. I doubt there is much 802.11b gear still active
in the field.


> > 2) Unicast the packet to each attached host in turn,
>
> Both DVMR and the multicast part of BATMAN-adv do that for router-router
> links.
>
> A better link-layer solution, IMHO, would be to multicast (with ARQ) the
> packet at a reasonably high rate (say, the median of the STAs subscribed
> to the multicast group),


I'm not deeply familiar with many of the  multicast protocols in use today,
such as MDNS. I AM under the impression that IGMP has not been well used or
tested recently.

I did start building uftp which I think I understand, and lets me have very
tunable rates, and definately uses igmp.


> and then unicast it to all STAs that failed to
> return an ACK.  Interestingly, if the new multicast frame format is
> incompatible with the normal 802.11 format, then this scheme is
> compatible with legacy devices, which won't ever see the multicast frame
> and hence won't return an ACK.
>
>
While I can see this helping for many protocols ARP seems a problem, in that
you'd like to stop broadcasting after you get a good response, and that's at
a higher layer.

> So the workaround is to isolate the broadcast domains of wired
> > networks and wireless networks by making the home router into...
> > a router.  Wireless on one subnet, wired on another, and so ARP
> > between the two turns into ARP to the router alone - much more
> > scalable.
>
> OpenWRT is your friend.
>
>
It certainly is! I would never have got this much insight into these
problems, so fast, had I not poked into the depths of the linux kernel,
patched it up (with the help of many here on this list) and tried to build a
version of openwrt based on everything we've learned about bufferbloat so
far, and setup a lab to stress it out.

Future directions are becoming apparent, rapidly.


> -- Juliusz
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blogspot.com

[-- Attachment #2: Type: text/html, Size: 3936 bytes --]

Re: [Bloat] tiny monsters: multicast packets

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On 05/29/2011 11:57 AM, Dave Taht wrote:
>
>
> On Sun, May 29, 2011 at 9:21 AM, Juliusz Chroboczek 
> <jch@pps.jussieu.fr <mailto:jch@pps.jussieu.fr>> wrote:
>
>     > And the irony is that the lower speed is specifically chosen for
>     > multicast in order to make sure all clients in range can hear them
>     > reliably.
>
>     It was my understanding that it was done for compatibility with older
>     devices, since 2 Mbit/s is the fastest rate supported by pre-B
>     spread-spectrum hardware.
>
>
> And thus, everybody loses. I doubt there is much 802.11b gear still 
> active in the field.
I think Juliusz is referring to old stuff pre-B, like the Digital 
RoamAbout.  Certainly that is long gone, and I think pretty much all b 
hardware is gone, though having a "screw your performance to make 
something antique run" option someplace might become necessary for someone.
                 - Jim


[-- Attachment #2: Type: text/html, Size: 1931 bytes --]