Alternative Clock Management for WSJT-X
Matt Roberts - matt-at-kk5jy-dot-net
Published: 2020-04-12
Updated: 2020-04-21
A unique requirement for running most
WSJT-X modes is
that the computers on either end of the link have to run with pre-synchronized local
clocks. It is a rather bizarre requirement, that has to do with how the DSP
algorithms search for bits in the incoming audio. While home users of the
software can use NTP
to synchronize their clocks to existing high-precision Internet time sources, mobile
and portable users can find it a challenge to access to quality time sources while
operating from the bush.
One popular solution is to use an inexpensive
GPS dongle with
one's portable computer. The GPS system of satellites can provide a time source
competitive with the best NTP sources on the Internet, and inexpensive USB receivers
make this a cost-effective option.
However, the GPS option has some disadvantages:
- Finding a quality GPS receiver dongle that offers reliable
PPS is
increasingly difficult. Most receivers can get to within +/- one second,
but to get high precision, PPS is required. Since WSJT-X relies on the
fidelity of the clock to optimize search for weak signals, more precision is
definitely better.
- Using GPS requires a receiving antenna that can be placed in the clear. A
USB receiver with a built-in antenna will have serious challenges operating from
within an Airstream, for example.
- Using what is essentially a high-power broadcast service to recover an FT8 or
FT4 clock on the air just isn't faithful to the self-contained nature of
amateur radio. ;-)
One activity that I enjoy is operating HF from remote and rural locations. To
make use of FT8 and FT4 while out in the rough, I needed a good time source, but I
didn't want to rely on GPS or my cell phone (which is so old, it barely makes
calls). So I set out to find a good clock source that can be used with
WSJT-X from anywhere, not just where GPS or network services are available.
As it turns out, the best solution was staring me right in the face... why not just
adjust the system clock from the dozens of incoming FT8 signals on the radio?
Much like NTP, the numerous signals arriving from the HF radio provide plentiful time
data, from which one could synchronize the clock, and fortunately, WSJT-X calculates
and reports the offset between each decoded packet and the local system clock, as shown in
Figure 1.
The column circled in red, taken from a screenshot of the WSJT-X running FT8, is the
per-decode clock error estimate calculated by the software. This is part of the
data reported by the application via the UDP and ALL.TXT interfaces. This
information can be used to estimate the mean clock offset between the PC and all the
people transmitting on a given channel, and then used to correct the local clock to
match theirs.
Figure 1: WSJT-X Clock Error
In fact, this approach turned out to be better than NTP or GPS, since the
real goal is to align the local clock with the remote clocks of other users.
The optimal solution isn't a high-precision match with UTC—the optimal solution
is to provide a high-precision match to the majority of other stations on the channel, so
as to optimize the DSP search for weak signals.
The Tools
Unfortunately, the required conversion—from clock error reports to clock
correction actions— is not a feature of the core WSJT-X software. However,
since the software reports the needed data via UDP for third-party applications, I was
able to write such an application to do what I wanted. The solution has two
parts:
- A small utility, tweaktime, for adjusting the system clock by a
small number of milliseconds.
- A second utility, ctweaktime, providing a simple user interface
for the first utility.
The second item provides a text-mode interface to the first item, allowing the user to
quickly adjust the time with a few simple key clicks. One pair of keys changes
the clock increment to be applied, and a second pair of keys applies that increment,
either up or down, to the system clock. With this simple set-up, aligning the
local clock to the incoming FT8 signals can be done in just a few time slots.
A third optional item was developed to automate the adjustment process, so that the
time is automatically kept in alignment with the incoming decodes. While this
is definitely handy, the manual process is sufficient on its own, and so the
automatic version will be discussed in its own section, below.
The software builds in just a few seconds, even on Raspbian. It consists of two
small C++ programs.
Prerequisite Packages
The build process only requires the build-essential package to be installed
on systems derived from Debian (like Mint, Ubuntu, Raspbian, etc.). This
package is installed by default on most installations. To make sure the package
is available, simply start a terminal prompt, and run the following commands:
sudo apt-get update
sudo apt-get install build-essential
Building the Software
The software comes as a tarball. Decompress and build it with this sequence:
tar -Jxf tweaktime-20200412a.tar.xz
cd tweaktime
make
sudo make install
Adjusting the Clock
Figure 2: ctweaktime Output Example
The basic process for hand-adjusting the clock is as follows:
- Start WSJT-X.
- Start a terminal window, and in that window run sudo ctweaktime.
- Arrange the windows so that you can see them all at the same time.
- Use a coarse step size (e.g., 100ms or 200ms) to adjust the clock time over
several slot times, until the horizontal lines in the Wide Graph align
with the incoming signal edges.
- At this point, any incoming signals
should decode successfully.
- Using a finer step value (e.g., 20ms), fine-tune the clock adjustment as you
operate, to bring the DT value close to zero.
The step size to be applied is changed using the + and - keys.
The clock is adjusted by that step size by using the < and >
keys. That's it!
Figure 2 shows an example of the program while making adjustments.
Modern PCs have very accurate clocks, so once you are close to an average DT value
of zero, you should be able to operate for several hours before having to adjust
again.
While it probably sounds obvious, it is worth pointing out that if you are running
NTP on your system, and if that system is connected to the Internet, the NTP daemon
will quickly undo any changes you make using this utility. That's its job,
after all.
Automating the Process
The utilities described above are enough to quickly bring the PC clock to heel when
NTP and GPS are unavailable. It's a visual clock synchronization, since
the human operator is manually aligning the signals to the local clock, via the
Wide Graph and the DT measurements. While quite effective, the
WSJT-X interfaces allow another computer program to read the DT values, and
automatically adjust the local clock to achieve a best-fit to the incoming
signal decode edges.
Work smarter, not harder, right?
In some ways, this is very similar to NTP, just without the Internet. NTP also
uses measurements from several anonymous sources, does math on the measured errors,
and then selects the best adjustment to achieve a best-fit to the available data.
Figure 3: Automation Output Example
As an experiment, I wrote a simple
.Net program to
automatically adjust the local PC clock to the WSJT-X traffic on the air. It
uses the UDP interface to watch the DT values, performs some statistical math
on those values, and periodically makes small clock adjustments to synchronize the
clock time to the radio channel.
The first step is to count up how many decode samples are available. Adjustments
are only done once enough data is built up to make a good estimate, even if that takes
several time slots. Older data is discarded, and only a fixed number of samples
are allowed from any one station. Then the utility computes the
mean and
variance to place some bounds
on how much dispersion
is present in the decode error estimates from WSJT-X. This is used to exclude
samples that are outliers,
improving the estimate of the time offset of the samples that remain. Only
those samples within a certain number of standard deviations from the mean
are used.
If enough data is present, the resulting error estimate is used to call the
tweaktime utility, adjusting the clock. The response is smoothed
by applying only a fraction of the needed correction per estimate, spreading
the correction out over time, which helps control jitter. After each
adjustment, the existing data is cleared and accumulation starts again.
An example of the utility running is shown in Figure 3, as a .Net
console application.
The utility has a very simple user interface, and is configured using an XML file.
A better user interface (allowing easy adjustment of those settings) would definitely
be useful. That said, the utility in its current form appears to run very well
against FT8 and FT4 data on both lightly-loaded and very crowded channels.
The automation utility isn't quite ready for release, but if anyone is interested in
helping me test this on other systems, drop me an email, and we can talk about what
is needed for a release.
Some Conclusions
It is certainly possible, and relatively easy, to operate WSJT-X with near-optimal
clock synchronization, using only the incoming FT* signals from an HF receiver.
When using an automated process, it is also possible to keep the clock in a
near-optimal state for decoding, even better than with NTP or GPS.
Since the DSP within WSJT-X is doing all of the math to calculate time offset errors
of incoming signals, the extra overhead needed to keep the local system clock in
synchronization with those signals is almost zero.
Observant users will notice that we aren't recoving the actual value of the
current time using this method. We are only able to recover the leading edges
of the transmissions of other users. The granularity—or
modulus—of
the recovered clock is the time slot length of the mode being used. E.g., fifteen
seconds for FT8, or 7.5 seconds for FT4. Fortunately, as long as we align the
local clock's leading edges to the other stations' local clocks to that level of
granularity, the actual value of the clocks is irrelevant. Clock recovery
doesn't care about the wall-clock time, only the alignment of the relative bits being
transmitted and expected by the receiver.
In order for this approach to work, most participants—particularly the stronger
stations—on a channel should still synchronize to
UTC, or some
other common time standard. Fortunately, that's the situation we have
today. Most people operate from home or some other fixed location, and are able to synchronize
via NTP or GPS. As long as this is the case, having a few mobile or portable
stations synchronizing to "the pack" should be fine. The only time
this might become an issue is during a portable event, like
Field Day.
This leads me to a critique of the WSJT-X software, which is that it doesn't provide
its own clock recovery. Every other amateur radio digital mode designed for
computer demodulation (including the DV modes) uses clock recovery on the individual
signals from the receiver, in order to decode each of those signals. By omitting
this, WSJT-X relies on external recovery (via presynchronized PC clocks), which moves
half (or 6dB as measured at the sound card) of the signal content (the clock) out of
the HF channel, and into other media, such as the Internet or GPS. As a result,
the versitility is limited, the required SNR estimates are at least 6dB understated,
and the modes are dependent on communication services outside of amateur radio to
operate properly.
On the other hand, WSJT-X modes are very popular, and until WSJT-X adds its own clock
recovery (after all, the information is there in the software—we used it
above to synchronize the PC clock), we can rely on other means to recover it.
The method described above effectively piggy-backs the local PC clock on the NTP/GPS
synchronization used by other stations. In effect, we are leaning on "herd
synchronization" of everybody else in lieu of synchronizing to external
time services directly.
I have made thousands of FT8 and FT4 contacts, so I'm not trying to disparage the
authors of WSJT-X. They are obviously gifted engineers. But when we have
to work around a limitation like this, it's important to understand the limitation
we are trying to circumvent, and how that impacts operation generally.
Have a safe mobile HF operating season!
Software Downloads
The tweaktime and ctweaktime utilities are available for download in
source form.
The software was developed and tested on Linux, but it can also compile on Windows
using Visual Studio. Windows uses different system calls to read and set the
local clock, but the basic idea is the same. If anyone is willing to help me
do some extended testing on Windows, please drop me an email. For now, I only
operate from Linux, for a number of reasons, so the Windows support needs more
testing before I provide binaries for download.
2020-04-13 - Add tweaktime support for Win32 time calls, and update
the ctweaktime application to run under Windows Command Prompt.
2020-04-12 - Initial release.
Copyright (C) 2020 by Matt Roberts, KK5JY.
All Rights Reserved.
