[Thumbgps-devel] Jitter and latency in a linux system might be a problem. Minimal BoM crosspost.

[Thumbgps-devel] Jitter and latency in a linux system might be a problem. Minimal BoM crosspost.

[tz]

The PPS change detect thread in gpsd might have latency problems if
the kernel is busy and/or it isn't a high priority - a plain userland
program shows about 1mS jitter - a loop with the ioctl to block until
a pin-change followed by gettimeofday will vary over about a 1000 uS
band on my dual-core netbook.  If you really need 100nS, you might
need to use the kernel PPS, preferably with a dual core or better CPU
so the interrupt can capture the nanosecond timer reliably.

I'm not sure how much of the jitter might be due to  the USB latency
to detect an interrupt condition at the far end, but USB 2.0 ought to
be fairly fast.  However it might debounce or otherwise delay the
detection of a serial line change to eliminate noise, so might be
introducing a latency, e.g. CTS might need to be in the new state for
tens or hundreds of microseconds before an event happens to avoid
noise spikes.

If it is a USB latency you  might need to feed the 1PPS into a GPIO
pin (maybe the WPS if it isn't being utilized by the router if it is
on an interrupt).

But to state the problem clearly, you want the clock to have 100nS
accuracy, but there is no way in userland and/or USB to get better
than 1mS jitter even if the edges going into a serial device are
precise.  There is a variable latency - error - introduced.  If we
need to do this via Linux-PPS we should decide now - it should be
adaptable to the open router software as it just grabs the nanosecond
timestamp upon an interrupt.  But it would require updating the
kernel.

If this is doing more than the timing measurement, e.g. routing, the
jitter might end up being much greater.  Using an embedded linux
device only running the timing stuff would likely be much more well
behaved, but there are some kernel latencies no matter what - most
routers aren't multi-core.  If you don't grab the timestamp in the
interrupt like Linux-PPS, you won't come near the target accuracy or
precision.

Many routers have a "serial console", but few have anything more than
tx and rx broken out.  They would be direct serial hardware so have
less device-to-kernel latency.  Many are on standard 0.1" spaced
headers so it might be possible to do an adapter board that would
plug-in to the header and connect to the GPS - you would have to open
the router and install the adapter/gps and run the antenna wire out
(maybe drilling a hole for it).  And adjust the serial console kernel
setting so it didn't think it would be a login.

There are off the shelf $30 USB GPS units which say they are using a
SkyTraq chipset but without (visible indication of support for) PPS -
I assume one or more is actually using SkyTraq - the descriptions say
SkyTraq but also some large number of tracking channels which SkyTraq
doesn't do (it has under 20, 2 or 4 for search), but the description
says 50+.

Any SkyTraq GPS (I have the Venus 624 and 628, and their GLONASS/GPS
modules), the "$" character in the first sentence can be synced to the
UTC second and comes out at a fixed delay from the PPS edge, just over
1mS to stop bit with no visible jitter so technically it would be as
accurate as the PPS itself, and the latency could be compensated for.
At 115kBaud there will be over 500mS of serial silence before the $,
so it can be detected fairly easily, perhaps even in a Linux-PPS
interrupt timestamp (if $, grab temp timestamp, if temptimestamp is
over 500mS from last timestamp, update - treat as PPS).  If the
CTS/DCD line switch ioctl jitter is acceptable, this is about the same
and requires no modifications.

Also using just the NMEA mode in an unmodified GPSD, the latency error
for the time was very consistent - about 17 mS with the standard
sentence set,  there will be slightly more jitter since it invokes the
PPS routine after the final sentence in the NMEA set
(GPGGA/GSA/GSVx3/GPRMC), the sentences vary in length.  If you just
counted the characters (they are a continual burst), you could divide
by the baud rate to get the position of the start bit of the '$', but
it would likely have the same userland or USB jitter.

Another problem is even if you get the clock at 100nS accuracy, how do
you send packets out or record their arrival at an exact time or
calculate the jitter there - the network stack and drivers?  You might
need to instrument part of the network stack to record the nanosecond
timestamps, or will have to live with the error from the network stack
variable latency.

Perhaps a possibility would be an external clock that you could
trigger a capture-time event when the packet arrived and read out
slower.  The external timer could be synced to UTC and very precise
(I've played with syncing signals from an Atmel chip to PPS), so the
internal ntp synced clock might just be approximate.  It doesn't fix
the kernel jitter but would act like an external precise and accurate
stopwatch.

Does this have to be an actual router, or could it sit on the network
right at the gateway?  If you just need to measure "router-to-router",
it might be easier to use something like an arduino with ethernet
shield and emit a UDP or ICMP packet EXACTLY at the UTC second
(assuming no ethernet collisions, but a good switch should be able to
avoid them).   There are embedded servers inside what appear to be
large ethernet connectors which are fairly inexpensive.  Perhaps a UTC
synchronized pingbot - ping it and it will do the echo exactly at the
UTC second.  Two pingbots could measure to well under 1mS precision
and accuracy.  Basically an embedded system that would only know UDP,
but you could tell it to send a packet to a destination, but it would
do so within microseconds of the PPS. and tag anything in and/or
return an echo with the incoming timestamp, e.g. over a command line
interface:
uutcping 1.2.3.4
(arrives at 67.890mS after UTC)
(remote returns in a packet, some ID header, 67.890, and the GPRMC
sentence from the previous second)
(output is RTT, remote time, remote GPRMC)
115.234 - 67.890 - $GPRMC...

I suggest a pure embedded timing unit with really limited function
because it would be easy to get it down to 100uS precision.  Otherwise
you are back to Linux-PPS if you want it.

-----

Cross post from "Calling all Openhardware...", edited.  This is the
best I can come up with, but others might know of some other devices
which might work better.

Assuming you compile GPSD with PPS on CTS it might be done with the
following three main parts:
http://www.sparkfun.com/products/464 (Magnetic Mount active antenna
with long cable)
http://www.sparkfun.com/products/11058 (SkyTraq breakout)
http://www.sparkfun.com/products/9873 (FTDI 3.3v Basic)
$13, 50, and 15 respectively for $78, Quantity 1.  100+ drops 20%.
You would need a simple single-sided board with some headers to attach
- all at 0.1" centering, very easy to layout and build but would need
headers, e.g. http://www.sparkfun.com/products/553 and
http://www.sparkfun.com/products/116 - the PPS goes to CTS, the FTDI
provides 3.3v @ 50mA - just enough for the GPS, and then just do TX -
RXI and RX to TXO.  Then some kind of packaging.  You might have to
configure the SkyTraq the first time, and AGPS helps (I have code for
that - ftp download, then serial packetized upload with acks).
SkyTraq has a fast search mode that uses more power so until it locks
might pull over 50mA, that is one of the configurations that have to
be changed.

I have an unpackaged version running now using jumper wires, but
haven't recompiled GPSD to use PPS_CTS yet.  I have tested the
consistency of the CTS edge.

The SkyTraq PPS is accurate to 60nS mainly due to the 16Mhz internal clock rate.

Even if you get a $30 GPS unit you would have to break it open and do
some soldering under a stereo microscope, assuming you can attach the
PPS to DCD or CTS or another pin easily.

Re: [Thumbgps-devel] Jitter and latency in a linux system might be a problem. Minimal BoM crosspost.

[Patrick Maupin]

On Mon, Mar 12, 2012 at 2:09 PM, tz <thomas@mich.com> wrote:
> The PPS change detect thread in gpsd might have latency problems if
> the kernel is busy and/or it isn't a high priority - a plain userland
> program shows about 1mS jitter - a loop with the ioctl to block until
> a pin-change followed by gettimeofday will vary over about a 1000 uS
> band on my dual-core netbook.

I thought sub-ms wasn't really necessary.  Sub-us would be useful in a
testbench for characterizing devices to make sure they meet ms...

> I'm not sure how much of the jitter might be due to  the USB latency
> to detect an interrupt condition at the far end, but USB 2.0 ought to
> be fairly fast.

That really depends on how the scheduler is set up.  A driver can
request 'x' interrupts per second.  But then there will be latency
inside the USB part as well.

>  However it might debounce or otherwise delay the
> detection of a serial line change to eliminate noise

That latency should be fairly fixed.  But we can measure it with a test setup.

> But to state the problem clearly, you want the clock to have 100nS
> accuracy,

Where was that listed as a requirement?  I think there's some
misunderstanding here.


> but there is no way in userland and/or USB to get better
> than 1mS jitter even if the edges going into a serial device are
> precise.

Which is probably good enough.

> Any SkyTraq GPS (I have the Venus 624 and 628, and their GLONASS/GPS
> modules), the "$" character in the first sentence can be synced to the
> UTC second and comes out at a fixed delay from the PPS edge, just over
> 1mS to stop bit with no visible jitter so technically it would be as
> accurate as the PPS itself, and the latency could be compensated for.

That's useful.  With a testbench we can test this, and with a small
CPLD in conjunction with such a module and a serial to USB converter,
we might even be able to make this work over USB by wiggling CTS or
DSR.  In other words, if we find a module without PPS, but *with*
accurate first-character message timing, we can wiggle a modem control
line when the first character starts to force an immediate USB
interrupt response, rather than waiting on the entire message plus
some extra latency, which is what the USB chip would normally do.

> At 115kBaud there will be over 500mS of serial silence before the $,
> so it can be detected fairly easily, perhaps even in a Linux-PPS
> interrupt timestamp (if $, grab temp timestamp, if temptimestamp is
> over 500mS from last timestamp, update - treat as PPS).  If the
> CTS/DCD line switch ioctl jitter is acceptable, this is about the same
> and requires no modifications.

Right.

> Also using just the NMEA mode in an unmodified GPSD, the latency error
> for the time was very consistent - about 17 mS with the standard
> sentence set,  there will be slightly more jitter since it invokes the
> PPS routine after the final sentence in the NMEA set
> (GPGGA/GSA/GSVx3/GPRMC), the sentences vary in length.  If you just
> counted the characters (they are a continual burst), you could divide
> by the baud rate to get the position of the start bit of the '$', but
> it would likely have the same userland or USB jitter.

It would be nice if you could backtrack like this.  Wouldn't need any
extr hardware.

> Perhaps a possibility would be an external clock that you could
> trigger a capture-time event when the packet arrived and read out
> slower.  The external timer could be synced to UTC and very precise
> (I've played with syncing signals from an Atmel chip to PPS), so the
> internal ntp synced clock might just be approximate.  It doesn't fix
> the kernel jitter but would act like an external precise and accurate
> stopwatch.

I think there are a lot of really good possibilities, but the first
thing to do is a hardware-assisted testbench to take accurate
measurements, so we know the source and size of all the latencies in
the system.  I haven't heard back from esr about what he thinks of
this approach, though, so I am reluctant to start work on anything
until we have some consensus on the idea that we should grab several
devices and USB modules and USB chips and see how well they really
operate.

> Does this have to be an actual router, or could it sit on the network
> right at the gateway?  If you just need to measure "router-to-router",
> it might be easier to use something like an arduino with ethernet
> shield and emit a UDP or ICMP packet EXACTLY at the UTC second
> (assuming no ethernet collisions, but a good switch should be able to
> avoid them).   There are embedded servers inside what appear to be
> large ethernet connectors which are fairly inexpensive.  Perhaps a UTC
> synchronized pingbot - ping it and it will do the echo exactly at the
> UTC second.  Two pingbots could measure to well under 1mS precision
> and accuracy.  Basically an embedded system that would only know UDP,
> but you could tell it to send a packet to a destination, but it would
> do so within microseconds of the PPS. and tag anything in and/or
> return an echo with the incoming timestamp, e.g. over a command line
> interface:

This is an interesting idea.  I had wondered about ethernet, but the
idea of setting up anything NTPish on a tiny micro seemed daunting.  I
like the idea of such a UDP request/response mechanism.

> Assuming you compile GPSD with PPS on CTS it might be done with the
> following three main parts:

I think we should compile a list of ways to build acceptable units.
But I really believe that to do so, we ought to test all the
components...

> The SkyTraq PPS is accurate to 60nS mainly due to the 16Mhz internal clock rate.

Have you checked that over days?  I'm intrigued by the graph that I
just saw of things going haywire.

> Even if you get a $30 GPS unit you would have to break it open and do
> some soldering under a stereo microscope, assuming you can attach the
> PPS to DCD or CTS or another pin easily.

Absolutely.  But esr's stated end product (which I think should be
possibly part of our range of solutions) is a tiny USB key.  I found
that one for $23, and it's particularly noteworthy because I've seen
pictures of it disassembled on the internet and it appears to have
everything we need except one blue wire.  I've also seen others on
Amazon for $19.  There's no way you could buy all the parts that
cheaply.

Re: [Thumbgps-devel] Jitter and latency in a linux system might be a problem. Minimal BoM crosspost.

[tz]

On Mon, Mar 12, 2012 at 4:44 PM, Patrick Maupin <pmaupin@gmail.com> wrote:
> On Mon, Mar 12, 2012 at 2:09 PM, tz <thomas@mich.com> wrote:

> I thought sub-ms wasn't really necessary.  Sub-us would be useful in a
> testbench for characterizing devices to make sure they meet ms...

>> But to state the problem clearly, you want the clock to have 100nS
>> accuracy,
>
> Where was that listed as a requirement?  I think there's some
> misunderstanding here.

I would think so to, but I think ESRs original post or one of the
requirements comments said 100nS.

It would help if we knew how hard the number was.

>> Any SkyTraq GPS (I have the Venus 624 and 628, and their GLONASS/GPS
>> modules), the "$" character in the first sentence can be synced to the
>> UTC second and comes out at a fixed delay from the PPS edge, just over
>> 1mS to stop bit with no visible jitter so technically it would be as
>> accurate as the PPS itself, and the latency could be compensated for.
>
> That's useful.  With a testbench we can test this, and with a small
> CPLD in conjunction with such a module and a serial to USB converter,
> we might even be able to make this work over USB by wiggling CTS or
> DSR.  In other words, if we find a module without PPS, but *with*
> accurate first-character message timing, we can wiggle a modem control
> line when the first character starts to force an immediate USB
> interrupt response, rather than waiting on the entire message plus
> some extra latency, which is what the USB chip would normally do.

I've been using uS gettimeofday on Linux and with an oscilloscope
connected to the PPS and data lines.

> I think there are a lot of really good possibilities, but the first
> thing to do is a hardware-assisted testbench to take accurate
> measurements, so we know the source and size of all the latencies in
> the system.  I haven't heard back from esr about what he thinks of
> this approach, though, so I am reluctant to start work on anything
> until we have some consensus on the idea that we should grab several
> devices and USB modules and USB chips and see how well they really
> operate.

Garmin has the 18x-LVM (oem) which has a PPS and from what I recall (I
haven't hooked it up in a while and I don't know where it is) the NMEA
sentences didn't have a lot of jitter, but I'd need to put it back on
my scope.

The Sirf3 apparently doesn't work.  I've seen MTK and Ublox, but I
don't know about them.

> This is an interesting idea.  I had wondered about ethernet, but the
> idea of setting up anything NTPish on a tiny micro seemed daunting.  I
> like the idea of such a UDP request/response mechanism.

You don't need ntpd.  You simply need precise PPS since you can then
measure the actual crystal frequency, e.g. if it "10 MHz" ends up
being 10.123456 MHz, you can divide the counts into this to find the
time with high precision.  You can play some PLL games, but I was
close to syncing an SMPTE stream to PPS for my ION-GNSS entry last
october where the stream edges would be synced even though the micro
had a ceramic resonator.

>> The SkyTraq PPS is accurate to 60nS mainly due to the 16Mhz internal clock rate.
>
> Have you checked that over days?  I'm intrigued by the graph that I
> just saw of things going haywire.

The only problem I have seen was at 20Hz with AGPS on my motorcycle
under nonrepeatable conditions - sometimes it would not get a lock
though it had 10 satellites in view with strong signals, I had to
reset it.

If it loses lock, the PPS disappears, but I should probably try my 628
and glonass and leave both on over a weekend.

>> Even if you get a $30 GPS unit you would have to break it open and do
>> some soldering under a stereo microscope, assuming you can attach the
>> PPS to DCD or CTS or another pin easily.
>
> Absolutely.  But esr's stated end product (which I think should be
> possibly part of our range of solutions) is a tiny USB key. ...

I've seen this one with the same description from a few searches:

http://www.chinazrh.com/wholesale-gps-receiver-usb-dongle-for-computers-netbook-laptop-umpc-p-728.html

Re: [Thumbgps-devel] Jitter and latency in a linux system might be a problem. Minimal BoM crosspost.

[Eric S. Raymond]

tz <thomas@mich.com>:
> The PPS change detect thread in gpsd might have latency problems if
> the kernel is busy and/or it isn't a high priority - a plain userland
> program shows about 1mS jitter - a loop with the ioctl to block until
> a pin-change followed by gettimeofday will vary over about a 1000 uS
> band on my dual-core netbook.  If you really need 100nS, you might
> need to use the kernel PPS, preferably with a dual core or better CPU
> so the interrupt can capture the nanosecond timer reliably.

We don't need 100ns.  Our target accuracy is 1ms.  This simplifies
things a lot.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: [Thumbgps-devel] Jitter and latency in a linux system might be a problem. Minimal BoM crosspost.

[Eric S. Raymond]

tz <thomas@mich.com>:
> I would think so to, but I think ESRs original post or one of the
> requirements comments said 100nS.
> 
> It would help if we knew how hard the number was.

I've specified 1ms becauser that's a convenient order of magnitude below the
expected 10ms accuracty of NTP time, and the first application of this defice 
will be to check the accuracy of NTP time.

There's evidence from the ZTI Z050 spec sheet that this is an
achievable target - they quote 1ms timing accuracy.

> I've seen this one with the same description from a few searches:
> 
> http://www.chinazrh.com/wholesale-gps-receiver-usb-dongle-for-computers-netbook-laptop-umpc-p-728.html

I've tracked down their support address and asked these questions:

1. The SkyTraq 6 emits 1PPS. Is this signal brought out to the USB bus
where it can be seen as an emulated DCD or CTS change?

2. What USB-to-serial converter does it use?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>