K2K GPS Time Clock System Shift Manual

R. J. Wilkes, H. G. Berns, last update: 23 Sep. 2004

QUICK REFERENCE

• Shift Duties
• GPS System Online Monitor (updated half-hourly)
• Watch For Problems
  • How To Recover/Restart After Power Outage
• Experts

INTRODUCTION

HARDWARE

The GPS receivers are located in the North Hall Control room. Details of the hardware installation are given here.

There are two GPS receivers: TrueTime XL-DC (primary clock) and Motorola UT (backup clock, mounted as a daughtercard on a board in the VME crate).

Each receiver delivers two kinds of data: a character string with date and conventional time-of-day, and a 1-PPS (pulse-per-second) signal. The leading edges of the 1 PPS pulses are accurately synchronized with seconds rollovers in GPS time.

We determine event (spill trigger) times in 3 semi-independent ways, which are then compared as a data quality check.

1. The TrueTime (primary) clock has a built-in event time output which delivers the spill trigger time with 30 nsec precision. The absolute accuracy for all GPS data is approximately 100 nsec.
2. We independently determine the fractional-second part of the time using a 50 MHz counter (ie, clock with 20 nsec ticks) on the LTC (Local Time Clock) board in the VME crate. Spill triggers latch the TrueTime clock date/time data, and also latch the LTC count. 1-PPS triggers are separately latched and logged. The difference between a spill count and the most recent 1 PPS count gives fractional-second part of the trigger time in units of 20-nsec ticks. This is added to the date and time (to integer seconds) supplied by the clock to obtain the exact GPS time of the spill.
3. The same procedure (with an independent counter, but using the same 50 MHz source oscillator) is applied to the Motorola (secondary) receiver 1-PPS signal.

A 120-second running average of LTC counts per second (defined by consecutive 1-PPS edges) is used to determine the oscillator frequency and hence the time scale. Numerous checks are imposed to correct for routine minor glitches.

If the sytem is operating properly, the TrueTime "event time" (supplied directly by the commercial module) should be within a few 20-nsec ticks of the "1-PPS" times delivered by the LTC board from the TrueTime and Motorola 1-PPS signals. Also, the two clocks' 1-PPS times should also be within a few ticks of one another. The online display window on nubl01 shows such a comparison.

Note: The 50 MHz LTC base frequency was improved in the summer 2003 with an external 10 MHz rubidium frequency standard and a 1:5 PLL frequency multiplier to provide a highly stable 50 MHz source to the LTC module with 10^-11 stability.
The rubidium clock's output frequency is phase-locked to the 1PPS signal of the Motorola GPS and the resulting 50 MHz reference signal is therefore expected to stay stable at 50,000,000.00 Hz with a small error of only +/- 0.01 Hz or better.
The rubidium clock also provides its own 1PPS output, also phase-locked to the incoming 1PPS of the Motorola GPS, and with an accuracy of better than +/- 10 ns. This 1PPS signal is now used for computing the GPS2 (backup) time. The absolute error between GPS1 and GPS2 is now typically 20 ns rms and +/- 100 ns at about 99.99% of the time.

REALTIME SOFTWARE

The VME crate and TrueTime clock data are acquired by a process gpstrg running on nubl01. This process continually checks data quality, and also writes data to files for use by K2K run control, and for transmission to SuperK. See documentation by Hans Berns for a description of the algorithm. See below for how to restart the process if necessary.

Normally, gpstrg should be running at all times, and should not be terminated by killing its window or ctrl-c. If desired, data logging can be paused or the process can be stopped by creating flag files resident in nubl01:/home/online/flags/:

• init.flag causes a software reset of the VME modules. The flag file is deleted upon successful completion of the reset.
• wait.flag halts data logging (delete file to resume)
• stop.flag terminates gpstrg; the GPS_TRIGGER window will ask for a return to close. After successful termination, stop.flag is deleted.

DATA STRUCTURES

NEVER edit or delete GPS code, data or log files!

See nubl01:/home/online/log/summary.current for a data quality summary (mainly for experts, will not be explained in detail here) since the last restart of gpstrg. An archive of all such summaries is in summaries.gpstrg.

The data used for the quality control histograms are stored in nubl01:/home/online/log/histograms/ltchist.yymmdd.

In addition, there are 3 log files created daily and updated event-by-event or 1PPS-by-1PPS if the debug.flag file exists in the flags directory. Plots and a summary of the data of the last 7 days can be viewed here:

1. ~log/data/gpstrg.data.YYMMDD contains one line for each spill with CPU time, spill number, VMETRG 48-bit time, and the 3 final GPS times in UNIX format (TrueTime EventTime, TrueTime TRG-1PPS time, Motorola TRG-1PPS time).
2. ~log/data/ttet.data.YYMMDD contains raw TrueTime EventTime as received via the serial port, with year value added.
3. ~log/performance/ltcperf.YYMMDD contains one line for each GPS second rollover (1PPS) with CPU time, data quality flags, raw LTC times, momentary and averaged LTC frequency (only if the debug.log file exists).

Daily (C shift) should check the histogram of GPS quality control data. Click here to view it. See below for explanation of quantities plotted. The histograms should all be narrower than about 100 nsec half-width at half max. Contact an expert if not. THAT'S ALL for routine duties.

The GPS system should not require routine attention. Periodically, you can check the grey GPS_TRIGGER window on nubl01. It should update with every spill:

############## event 00001 ######## TRG_time 1.680944620 ##############
TrueTime EventTime:  266-2004 21:07:06.739152225 = 1095887226.739152225
TrueTime TRG-1PPS: 09-22-2004 21:07:06.739152240 = 1095887226.739152240   -15ns
Motorola TRG-1PPS: 09-22-2004 21:07:06.739152222 = 1095887226.739152222     3ns   18ns
#@ LTC6 event 00001 TrueTime: 21:07:06.739152220 = 1095887226.739152220   -20ns
#@                  Motorola: 21:07:06.739152240 = 1095887226.739152240    18ns
#@                  Rubidium: 21:07:06.739152240 = 1095887226.739152240    15ns

############## event 00002 ######## TRG_time 3.680436320 ##############
TrueTime EventTime:  266-2004 21:07:08.738643946 = 1095887228.738643946
TrueTime TRG-1PPS: 09-22-2004 21:07:08.738643940 = 1095887228.738643940     6ns
Motorola TRG-1PPS: 09-22-2004 21:07:08.738643838 = 1095887228.738643838   108ns  102ns
#@ LTC6 event 00002 TrueTime: 21:07:08.738643920 = 1095887228.738643920   -20ns
#@                  Motorola: 21:07:08.738643860 = 1095887228.738643860    22ns
#@                  Rubidium: 21:07:08.738643940 = 1095887228.738643940    -6ns

############## event 00003 ######## TRG_time 5.679927280 ##############
TrueTime EventTime:  266-2004 21:07:10.738134874 = 1095887230.738134874
TrueTime TRG-1PPS: 09-22-2004 21:07:10.738134900 = 1095887230.738134900   -26ns
Motorola TRG-1PPS: 09-22-2004 21:07:10.738134819 = 1095887230.738134819    55ns   81ns
#@ LTC6 event 00003 TrueTime: 21:07:10.738134860 = 1095887230.738134860   -40ns
#@                  Motorola: 21:07:10.738134820 = 1095887230.738134820     1ns
#@                  Rubidium: 21:07:10.738134880 = 1095887230.738134880     6ns

The display gives 7 lines per spill event with the following information (as shown above)

• header line: Spill (event) number and raw time code (seconds since last 48-bit clock rollover)
• line 1: Primary clock (TrueTime) event time = GPS1 data: day-of-year, hr:min:sec.nanosec; unix timecode. This is the time delivered directly by the TrueTime clock.
• line 2: Primary clock (TrueTime) 1-PPS time: date, hr:min:sec.nanosec; unix timecode; dt1 (see definitions below)
• line 3: Secondary clock (TrueTime) 1-PPS time: date, hr:min:sec.nanosec; unix timecode; dt2 dt3
• line 4: Primary clock (TrueTime) 1-PPS time as recorded by the newly installed LTC (version 6) module: hr:min:sec.ns; unix timecode; dt4.
  [Note: The LTC6 module is expected to be more stable and accurate than the original LTC (version 5) module. It has its own Fifo buffer for all trigger data and UART interface for the Motorola GPS, i.e. it combines 3 modules of the original setup. It also provides additional trigger inputs for Rubidium clock 1PPS and the spill trigger signal. Currently the LTC6 module is running in parallel with the original setup, so we will have direct performance comparison between the two setups and can keep the original data format as is but with additional quality checks.]
• line 5: Secondary clock (Motorola) 1-PPS time as recorded by the LTC6 module: hr:min:sec.nanosec; unix time; dt5.
• line 6: Rubidium clock 1-PPS time as recorded by the LTC6 module = GPS2 data (since October 2003): hr:min:sec.nanosec; unix time; dt6.

• dt1 = difference between line 2 and line 1 - a measure of LTC counter consistency and TrueTime receiver functionality
• dt2 = difference between line 3 and line 1 - a measure of LTC counter consistency and Motorola receiver functionality
• dt3 = difference between Motorola time and TrueTime time - a measure of agreement between the two independent GPS receivers.
• dt4 = difference between line 4 and line 2 - a comparison of TrueTime 1-PPS data recorded by the new LTC (version 6) and old LTC (version 5).
• dt5 = difference between line 5 and line 3 - a comparison of Motorola 1-PPS data recorded by LTC6 and LTC5.
• dt6 = difference between line 6 and line 1 - a direct comparison between GPS1 data (TrueTime EventTime) and GPS2 data (Rubidium 1-PPS).

• relative 1PPS pulse jitter of TrueTime receiver versus LTC oscillator counts
• relative 1PPS pulse jitter of Motorola receiver versus LTC oscillator counts
• absolute 1PPS pulse difference between TrueTime and Motorola receiver (a good agreement check between the two GPS receivers)
• drift between TrueTime "Event Time" versus TrueTime "TRG-1PPS" time of spill trigger events (= dt1 above; measure of consistency of TrueTime receiver data)
• drift between TrueTime "Event Time" versus Motorola "TRG-1PPS" time of spill trigger events (= dt2 above; alternative agreement check between TrueTime and Motorola data)
• drift between TrueTime "TRG-1PPS" time versus Motorola "TRG-1PPS" time of spill trigger events (= dt3 above; second alternative agreement check of the two GPS receivers)

• Error/Warning messages in GPS_TRIGGER window.
• Stopped display: clocks down? Nubl01 down?
• TrueTime and Motorola 1-PPS times differ by >150 nsec: one clock has lost its satellite fixes? If not, wait 5 minutes: the software will initiate auto-recovery procedures.
• On the LTC6 board (in VME crate, see diagram posted on side of equipment rack), LEDs are not blinking - contact an expert if this persists more than 5 min.

Normally gpstrg should self-correct within 5 minutes. For example, gpstrg will automatically restart if nubl01 loses power or reboots for any reason. The GPS receivers should regain synchronization promptly within 1-5 minutes if there is no hardware problem. If we lose an antenna or one of the receivers fails, there is nothing the shift workers can do: CONTACT AN EXPERT.

Note that as long as either of the 2 clocks shows good data, we are OK, and it is not necessary to make an emergency call to experts, just send email. The software is designed to provide usable GPS times for K2K/Super-K event synchronization as long as one of the two receivers is delivering valid data, since both clock times are recorded at every spill.

If you need to phone the experts, please note time difference when calling: Seattle Time = Japan Time - 17 hours (16 hours in summer).

HOW TO RECOVER/RESTART AFTER POWER FAILURE

• Check that the TrueTime receiver and VME crate have restarted, if not try recycling power switches.
• Usually, nubl01 will not be affected by a short power outage. It is hooked up to a UPS. All GPS system electronics (VME crate, TrueTime receiver and Rubidium frequency standard module) are also protected from short power outages by a separate UPS. If the VME crate power was shortly off after a longer power outage than the UPS battery could provide backup power, then gpstrg needs to re-initialize. Open a kterm or xterm and type the following command:

  /home/online/bin/restart_gpstrg

  This script will stop gpstrg properly and close the GPS_TRIGGER window. Then 10 seconds later, gpstrg will restart and a new grey GPS_TRIGGER appears. The window will show one or two errors in the TrueTime setup - these are normal. Once normal DAQ is established there should be no further errors.

• If nubl01 turned off during the power outage it should restart (reboot) automatically and run gpstrg from its startup script - once power is back. However, if the grey GPS_TRIGGER window does not appear, then run the /home/online/bin/restart_gpstrg script as mentioned above.

SPARES

A spare antenna is kept in the N-Hall control room cabinet (across from beam TV monitors). Replacement of GPS hardware should only be performed after contacting an expert.
NOTE: Tools and spare parts in this cabinet are the personal property of H. Berns and J. Wilkes. They may be borrowed in an emergency but must be promptly replaced.

name	email	phone(s)
Hans Berns	berns@phys.washington.edu	+1-206-685-4725 (UW office, Seattle) +1-206-713-5919 (mobile, while in US) 0578-5-9616 (while at SuperK)
Rik Gran	gran@phys.washington.edu	+1-206-543-9584
Jeff Wilkes	wilkes@phys.washington.edu	+1-206-543-4232 (UW office, Seattle) +1-206-282-5715 (home, Seattle)
UW group cell phone in Japan (if a UW member is at Super-K or KEK)		0901-636-4087