J. George and R. J. Wilkes, 4/1/97
Components of the SuperK UTC Clock
The GPS System
The SK UTC Clock Hardware
SK UTC Clock Online Software
How To Determine Event Time From Data Supplied In The Antiheader:
Offline software to retrieve UTC time
CORRECTIONS, CAUTIONS AND PROBLEM HANDLING:
How to Correct LTCTRG
LTCGPS Recording Problem
LTCGPS Reset at Data Buffer Start
Run Start difficulties
Missing LTCTRG or LTCGPS data
Missing GPS signal
Missing Past Year Offsets
Components of the SuperK UTC Clock
The GPS clock system provides UTC (Universal Time, Coordinated) timestamps for SuperK data. The system consists of three major components: the TrueTime GPS Receiver (Model XLDC-602), the VME Receiver (TrueTime Model VME-SG) with fiber-optic interface, and the Local Time Clock board. The GPS Receiver is located on the mezzanine of the Radon Hut, with its antenna mounted on the hut’s roof. It is connected to the Central Hut by a 2 km long cable containing two optical fibers (one a spare). The fiber cable is connected to the VME Fiber-Optic Interface located in the OD-DAQ VME crate. The Local Time Clock (LTC) board is mounted in the same VME crate, and connected to the fiber interface module by an auxiliary cable on the back side of the modules. Both modules are controlled and read by sukant via the VME bus. All equipment except the LTC is commercially made; the LTC was designed and built by Hans Berns in Seattle.
The GPS System
The GPS system consists of about 20 satellites maintained by the US Department of Defense, each transmitting the coordinated "GPS Time" according to its onboard atomic clock. GPS Time differs from UTC only in the absence of the leap seconds which are periodically inserted in UTC. Most GPS receivers (including ours) automatically take the shift into account using data downloaded from the satellites, so the time reported is UTC. The satellites’ onboard clocks are regularly conditioned to match GPS time according to a ground-based reference clock system (actually about 20 individual high-precision atomic time standards). The satellites also broadcast their ephemerides, so their position in space can be accurately calculated as a function of time. The ephemerides also are regularly recalculated and updated. Since the GPS satellites are constantly referenced to a national standards laboratory time base, the GPS system provides a simple and inexpensive way to obtain high precision absolute time, synchronized to UTC, without purchasing and constantly recalibrating an atomic clock. The GPS system is designed to give standard errors of about 150m per single position fix and 150 nsec per time fix.
The GPS receiver attempts to lock in 4~6 GPS satellites simultaneously. Using standard hardware and algorithms, it can then simultaneously fit for its own geographic position (latitude, longitude, altitude above sea level) and the time. Once the clock has run for about 24 hours, it has determined its position to maximum statistical precision and thereafter needs only one satellite in view to calculate accurate time (as long as its antenna is not moved). If no satellites are visible, the GPS clock uses its own internal stable oscillator to provide accurate time estimates until it can be recalibrated by a new satellite fix. Such situations are extremely rare, given the number of satellites in operation; we checked a 24 hr period in 9/96 and found no time when there were fewer than 3 satellites locked.
The SK UTC Clock Hardware
The output produced by the GPS receiver (in addition to its own display screen) is in two forms: a standard serial time code format known as IRIG-B, and a 1 Hz square wave whose leading edges are synchronized to seconds rollovers in UTC. The IRIG-B code is in turn supplied in two forms: AM (amplitude modulated 10 kHz), or DC-shifted pulse-width modulation. The DC-shifted version has rectangular pulses whose width carries the information. We use the DCPM mode since that is more suitable for fiber-optic links.
The fiber optic interface module (next to the VME receiver in the central hut) transforms optical signals into DC-shifted IRIG-B signals delivered to the back panel of the VME receiver. The fiber-optic module is familiar to everyone as the module with the flashing green light just under Hans Berns’s "OD-DAQ Status Box" on the TV monitor showing the Central Hut VME crate; if the green light stops flashing, it means the fiber has been disconnected (or the GPS receiver is down for some reason). The VME receiver module has the AM mode as default, so upon restart, the anticollector program must set a bit to tell the board to use the DC-shifted signal; this can lead to startup failures if the restart procedure is not followed. Once that is done, the DCPM signal is detected, the board phase locks to the IRIG-Btime code, and UTC accurate to 1m sec can be read via the VME bus.
The VME receiver can be programmed to put UTC time on the VME bus (month, day, hour, min, sec, microsec) in several different ways: at regular intervals, or when commanded via the VME bus, or when triggered by an "external event" input on the front panel. We use the latter mode. A pulse input to one of the external event connectors causes the VME-SG module to freeze the current time values in a set of internal registers, and generate an interrupt on the VME bus, which is passed to sukant via the Bit3 interface, and handled there.
The external event signal we use is taken from a selected bit on the LTC counter. A set of jumpers on the LTC board allow one to select one of the upper bits as the rollover bit, carried through a driver chip and onto an external line. Choice of bit determines the time (in LTC ticks) between GPS clock readings, modulo powers of 2. We selected bit 29, counting from 0 to 31, ie, the 3rd most significant bit. The LTC clock pulse interval is 20ns, so (2^29)*20ns is a little over 10 seconds. This is the width of a half cycle (1 or 0) for that bit, so the clock trigger occurs every ~21 seconds. This signal is taken from the back of the LTC board on a twisted pair to the external event input on the back of the VME-SG module (there is another EXT EVT trigger input on the front panel, but the cabling was simpler via the back of the VME crate).
The sequence of events is as follows:
Every 21.474 seconds, the GPS module gets a rising edge on its EXT EVT input. This freezes the current time in a set of internal registers.
If the GPS module has been properly initialized (interrupt on external event set, and master interrupt enable set: see setup_gps() in anti-collector), it will generate an interrupt on the VME bus.
The Bit3 interface has its jumpers set to pass IRQ1-4 to sukant on its cable. Thus the VME interrupt causes an Sbus interrupt in sukant.
An interrupt handler on sukant (see the GPS routines in /usr/local/bit3/944/v3.2/src/bts_stub.c) recognizes the interrupt, tells the Bit3 adapter to start a VME interrupt-acknowledge cycle on the VME side, and sends a UNIX signal (SIGUSR2) to any process which has registered for it.
Anticollector registered itself (in setup_gps), so it receives the signal SIGUSR2 as notification that the GPS module has received an external event.
At the next convenient opportunity (defined as when no data is waiting in the V533 FIFO modules), anticollector does the following:
Reads and releases the latched VME-SG time registers.
Copies this GPS time into the event header template, which will be used for all subsequent events.
Copies the value of the LTC count from last recorded event trigger (LTCTRG) in the header template, as the LTC value roughly corresponding to the GPS update (LTCGPS).
Since the LTC count is not separately latched by a GPS trigger, LTCGPS must be corrected from this slightly later value by the offline software, using the definition of a GPS trigger as rollover of bit 29 of the LTC. Hence bit 29 of the corrected LTCGPS is 1 and all lower bits are 0. The upper two bits can be taken from LTCGPS.
SK UTC Clock Online Software
When the collector reads out the frozen time registers, it converts the time into a Unix timeval structure containing seconds and microseconds. Note that the IRIG-B time code delivered by the VME-SG module does not include the year. So at that point, the time is given as seconds since 00:00 Jan 1 of the current year. (The microseconds part of the time value is stored separately.)
To simplify handling of time data by offline users, we add in the offset from the start of UNIX time (UNIX's "birthday", Jan 1,1970) to the beginning of the current year. Thus, the seconds part of the GPS time becomes the number of seconds since 00:00 1 Jan 1970. When this offset is calculated, it goes to the start of the current year as determined by the system clock on sukant. (Note that the offset calculation does not depend on sukant's time being precisely correct, only that the year be set correctly, a reasonably safe assumption.) UNIX automatically handles leap years, etc., for all years prior to the current year in determining the offset. The UTC from the GPS receiver is only from the start of the current year, but leap years, leap seconds, etc., are also handled by the satellite system, so all calendar details for the current year and years past become transparent to the user.
Thus the time structure (containing year, month, day, hour, minutes, seconds) can be handled conveniently with standard UNIX time utilities. In this way, users can retrieve month, day, hour, min, sec data without worrying about leap years, length of months, etc - it is all done automatically by Unix. However, the offset introduces a minor complication which must be taken into account by users of the time data.
When anticollector calculates the offset for past years to add to the GPS time, it uses the mktime routine in C. We did not realize initially that mktime is a partner function of localtime, which interprets times relative to the local time zone. This means that the offset is actually calculated from Jan 1, 1970 in local time, or JST. When decoding time words, routines typically use the gmtime routine, which interprets the result as UTC. This is correct: use of localtime to decode the result would make the event time a function of the time zone in which the analysis was done! However, the two together mean that there are 9 hours subtracted in the decoding which were not restored in the encoding, namely, the 9 hours between 00:00 Jan 1, 1970 UTC and 00:00 Jan 1, 1970 JST. This problem is easily corrected by adding the 9 hours back into total_sec before using gmtime to interpret the result as UTC, ie, add.
time_zone_correction = 32400 seconds.
How To Determine Event Time From Data Supplied In The Antiheader:
The antiheader contains the following 32-bit (unsigned int) words to be used for time deterimination. The names given are those used in online and onsite analysis software. The names in parentheses are those used in the offsite analysis library, which are not always accurately mnemonic.
LTCGPS = local clock count recorded at last trigger before the latest GPS reading (not at exact time of last GPS interrupt).(local_to_gps)
NOTE: LTCGPS was used in earlier methods we suggested for UTC extraction, but is actually not needed and should no longer be used. See note below on "LTCGPS Recording Problem".
LTCTRG = local clock count at current event’s trigger time. (local_trg)
NSGPS = GPS time in seconds since beginning of current year at the last GPS reading. (gps_sec)
NUSGPS = GPS microseconds at the last GPS reading. (gps_usec)
Time is calculated in the following manner (using online/onsite nomenclature):
LTCTRG must be first be corrected by the prescription below, to give the actual LTC reading when the GPS update was made. The result is defined as LTCGPS_corrected.
The time offset to the event from the GPS update in seconds is given by:
dt_evt = (LTCTRG - LTCGPS_corrected)*(20.0e-9 seconds/LTC count)
total_sec = NSGPS + integer portion of dt_evt + time_zone_correction (explained above).
total_usec = NUSGPS + (fractional portion of dt_evt)*1.e6
UTC time is given by gmtime(total_sec) to seconds precision, where gmtime is the Unix time utility that takes the time in seconds since start of UNIX time, and returns year, month, day, hour, minute and seconds in a structure (see man gmtime for your favorite C or FORTRAN usage).
Higher precision is gained by adding total_usec (in microseconds)
Offline software to retrieve UTC time
The above calculations are contained in function UTCtime.c, available in the US code repository (in /atmpd/ska/). This should be added to the onsite code library soon (we must translate to F77).
The package consists of UTCtime.c, UTCtime.h, and UTCtimeT.c (a test program to exercise the function), written by P. Mass in 4/97. Usage (in c) is:
void UTCtime(struct ahead_bank *ahead, struct tm *utc, unsigned int *useconds)
The structure utc is a standard Unix time structure, defined in /usr/include/time.h as follows:
struct tm { /* see ctime(3) */
int tm_sec; /* seconds after the minute [0-60] */
int tm_min; /* minutes after the hour [0-59] */
int tm_hour; /* hours since midnight [0-23] */
int tm_mday; /* day of the month [1-31] */
int tm_mon; /* months since January [0-11] */
int tm_year; /* years since 1900 */
Following is a patch from UTCtimeT.c showing how to use the UTCtime function:
struct tm *utczone;
void UTCtime(struct ahead_bank *, struct tm *, unsigned int *);
if(*ahead_link)
{
ahead = (struct ahead_bank *)skbank(ahead_link);
test_sec=0;
utczone = gmtime((time_t *)&test_sec);
UTCtime(ahead,utczone,&usec);
printf(" UTC corrected time: %u %2u %2u %02u:%02u:%02u.%06u\n"
,utczone->tm_year, (utczone->tm_mon + 1), utczone->tm_mday,
utczone->tm_hour, utczone->tm_min, utczone->tm_sec,usec);
}
Corrections, Cautions and Problem Handling
How to Correct LTCTRG
The correction of LTCGPS is slightly complicated by the choice of the rising edge of bit 29 to trigger the GPS. This means that halfway through the 21 sec GPS update period, bit 29 carries to bit 30, and the upper two bits of the LTC count change in the middle of the cycle. Note also that LTCGPS is the LTC count at the last event before the GPS is read out, not at GPS readout. Normally, the V533 FIFO is empty when the GPS trigger occurs, so the previous trigger was recorded before the GPS trigger. However, if the GPS trigger comes in the midst of a large burst of events, it is possible that the last trigger will be later than the GPS trigger by the time the V533 is read out. This has never been seen in practice, but in principle is quite possible.
In spite of these caveats, the system is actually robust. With the following prescription either case can be handled equally well. The only requirement is that LTCGPS be anywhere within +/- 10.7 seconds of the GPS trigger. This is a very loose requirement.
Usually, LTCGPS will contain the LTC count shortly before the GPS trigger. Bit 29 will be 0 (the rising edge has not yet come). In this case, the upper 2 bits are valid since they will not change when bit 29 becomes 1. The corrected LTCGPS is just:
LTCGPS_corrected = ((LTCGPS & 0xC0000000) | 0x20000000)
where & is a bitwise AND, and | is a bitwise OR. Clearly this preserves bits 30 and 31, sets bit 29, and leaves the rest 0.
If LTCGPS is taken from a trigger just after the GPS trigger, then bit 29 will already be set to 1. If this is the case, the upper bits will not yet have carried since this happens when bit 29 drops to 0 again. The same formula can be used for this case.
LTCGPS Recording Problem
In late March 1997, a bug was discovered in the way LTCGPS is determined. The result is that reported UTC may be 21.474 sec earlier than the true time. Shifts of 2*21.474 sec occur rarely. Larger multiples are possible in principle, but require an improbable conspiracy of circumstances and have not been seen.
This is due to the following bug in versions of anticollector used prior to run 3750. When the GPS was read out, the value of LTCTRG for last event trigger was retrieved from an array containing event header information and stored as LTCGPS for the current GPS update cycle. Unfortunately, an array index was improperly handled, and the value of LTCTRG taken was an old value from a previous buffer of data. It is almost certainly from the next previous buffer, but it is possible for the value to have been several GPS updates old. The result of this is that LTCGPS must be considered unreliable for all data prior to run 3750 (late March 1997) when the correction was made. The possibility of error is a shift of 21 seconds (down) from the actual UTC. A shift of 42 seconds is very occasionally possible if the phases of the GPS update cycle and the collector buffer readout cycle unluckily coincide so that there are two GPS updates contained in one collector data buffer of 256 events. At current rates the two cycles have similar frequencies to this can happen, though not often. A longer shift is remotely possible in theory but would require such a specially tuned set of coincidences involving the burst structure of the event trigger so as not to be believable. Such a case would affect only 256 events if it were possible at all.
This does not preclude any use of the GPS data. All of the information contained in LTCGPS is also available in LTCTRG.
There is only one new rule which must be added. In the previous prescription, LTCTRG for any trigger within +/- 10.7 seconds could be used in place of LTCGPS with no adverse consequences. What if the trigger is more than 10.7 seconds later than the GPS update? In this case, bit 29 goes into the second half of the cycle and drops to 0 with a carry to bit 30. The same prescription holds, but the upper bits are now one ahead of what they would be had I actually recorded the GPS update. The count of the upper bits must be reduced by 1. Note there is now no concern about whether the trigger was latched before or after the GPS update. The LTCTRG itself makes it clear which GPS update cycle the event belongs to.
LTCGPS Reset at Data Buffer Start
After discovery of the LTCGPS recording problem, a new error was found. Anticollector handles data in buffers of roughly 256 events. At the start of each buffer, LTCGPS was being re-initialized to be equal to the current LTCTRG. This was fine for half of the time because of the loose conditions on the original prescription for correcting LTCGPS. When LTCTRG was in the latter half of the GPS update cycle, however, the original prescription failed and provided a +21.4 second shift as described in the LTCGPS recording problem.
The correction given for the LTCGPS recording problem completely handles this case as well. No new rules are required. This effect is not yet corrected in the anticollector. It is hoped that a test run with an updated version will reveal whether there are as yet unknown problems which could be fixed in a single software upgrade rather than many small ones.
Run Start difficulties
Since there are no previous triggers to give information at the start of a run, there have been +21.4 second shifts for the first few events of a run, until the first GPS update with triggers flowing. It is believed that these problems are further manifestations of the same mistake in correcting LTCGPS when bit 29 has carried. A test run of the fixed anticollector should determine if there is some remaining error.
Missing LTCTRG or LTCGPS data
Even at present running, there are occasional (few per thousand triggers) events which are passed to the event builder with LTCTRG or LTCGPS = 0. It is believed that this is related to the overlapping global and end-of-bip triggers recorded by the V533 modules. The exact difficulty is as yet unknown.
For now, users should simply consider any data with LTCTRG=0 to be invalid for the purpose of determining GPS time. This is fairly safe, less data will be discarded if we simply ignore real triggers with LTCTRG=0.
Missing GPS signal
Especially in early data it was possible for the link to the satellite receiver to go down. This happened during run starts in early data when the gps module was taking some time to lock to the input signal. This was corrected by initializing the gps module sooner, allowing more time to lock on. The optical fiber also caused some trouble with a break near a connector. Bumping the connector could cause the gps module to lose lock with the input signal. Even today a loss of signal can happen if power goes down in the radon hut.
The input time code conditions an internal oscillator in the VME-SG module. Loss of the input signal means the module continues to use the internal clock, but it is no longer disciplined. For short breaks in service this is ok as the internal clock has been kept close to the actual time. When starting a run, however, the internal clock is somewhat random and has no actual knowledge of the correct time.
The status of the input signal is coded into the data stream as bits 16 and 17 of the status word (the last word of the event header). If those two bits are 10 (bit 17 is MSB), this is sync-gen mode, phaselocked to the input signal. If they are 01, this is an input reference error, there is no input signal whatsoever. Anything else means the status is undetermined. This means the signal is present and detected, but the phase error of the internal clock is still greater than 100 nanoseconds but (I think?) less than 1 microsecond. If this condition persists for more than about 10 seconds it probably indicates a problem with the input signal such as excessive noise.
Starting up with no input signal will make the GPS output jump to Jan 1 of the current year followed by normal incrementing as the internal clock keeps time to its own accuracy. Any GPS data without a valid sync-gen flag should be discarded, although it could be used for relative timing within a run.
Missing Past Year Offsets
Occasionally in the very early data it was possible for LTCGPS to become 0. In some cases events got by without being updated properly. In these cases the GPS output will decode to Jan 1, 1970. As a general rule of thumb, any GPS data which does not decode to a decade in which the detector existed should be discarded.
With the mode of operation described, the LTC clock, which uses a simple stable oscillator chip, is not "disciplined". The LTC clock is used only to interpolate between GPS clock readings. The intervals between GPS reads are short enough that the errors due to LTC drift are on the order of a few hundred nsec, more than sufficient for SuperK, where microsec accuracy is adequate for the most demanding application (supernova burst timing).
One improvement is to run the LTC from the ID's 50 MHz oscillator signal (for the 48-bit ID clock) instead of our own on-board oscillator. Thus the ID and OD clock ticks will be synchronized. This will be done by the end of September, 1997.
However, with some additional effort (which will be required for K2K) we could use the same hardware to provide increased accuracy. We could use the 1 pps squarewave output of the GPS clock to condition the LTC counter by using these edges (which are synchronized with seconds transitions in GPS time to GPS precision) to reset the counter. In other words, instead of using the 29th bit and interpolating LTC readings over a ~20 sec interval, we could in effect roll over the LTC counter every second, and thus the precision of the time values produced would be limited only by the LTC oscillator drift over 1 second. The LTC could also be upgraded by using an oven-stabilized oscillator.