About FT8

Matt Roberts - matt-at-kk5jy-dot-net

Published: 2023-01-31

Updated: 2023-06-28

FT8 is unquestionably popular in the amateur radio world. Among some, zealously so. My experiences with these modes are not typical of most operators, so I want to share some thoughts on the mode—some technical, and some not.

When I speak of FT8, this also relates to the majority of modes in the WSJT-X collection of modes — including FT4 and JT65 — as well as the reference software for these modes, WSJT-X.

An honest evaluation of FT8 as a mode doesn’t reveal any significant technical innovation. Narrow DSP digital modes existed long before FT8, and its popular predecessor, JT65. This is also true of other design features, such as wave-shaped MFSK, forward error correction, in-stream synchronization patterns, data whitening, interleaving, deep-searching decoders, a priori decoding assistance, presynchronized data clocks, etc.

Further, the Shannon Limit wasn't changed with the release of FT8. Modes with performance approaching that limit have existed for many years. So FT8's modulation design is not intrinsically more capable of digging signals out of noise than many other encoding schemes that preceded it.

So what makes this mode so popular among hobbyists?

The one "innovation" that seems to be key to the success of FT8 is the restriction of message format and content. That is, the mode is explicitly not intended for normal conversational text.

Most earlier digital modes in the hobby world used asynchronous messages of arbitrary length, while FT8 uses fixed time slots of fixed-length messages that are too short for typical conversation. When that is combined with the reference software’s strong emphasis on macro messages, and message sequence automation, the mode effectively limits typical operations to brief contest-like exchanges.

Such contest-like operation has been around for decades with other modes, as well, but what makes FT8 different is that it uniquely limited to such operation. Thus, FT8 operation is a 24/7/365 contest, and it operates with very similar on-air "rules."

This appeals to operators who want to make contacts and chase DX, but aren’t interested in "rag chewing" with strangers. In fact, the rise in popularity of these modes roughly coincides with the growth of usage of short-message services such as Twitter and SMS. So it is no surprise that a cultural shift encouraged WSJT-X style operation.

And that’s not a criticism.

After all, radio contesting is also very popular. What the WSJT-X team brought to the air was a family of modes — and associated reference software — that are useful for little else besides contest-style operations: Call CQ, trade reports, roger and 73.

As with any contest, removing any expectation of conversation allows one to focus exclusively on making more contacts to more locations in the time available.

The restriction of messages to fixed formats is even leveraged by the software's a priori decoding assistance, to help deduce the content of messages too weak to copy completely. This gives the illusion of SNR performance beyond the Shannon Limit. By sticking to a few short messages of fixed format, the decoder can get extra hints on the content, even if the error correction can’t dig out all the bits.

Man Meets Machine

My own experiences with FT8 started long after it was already popular. At first, I found it disinteresting, and some of the technical details mildly distasteful.

In particular, I dislike having to pre-synchronize the computer clock in order for the demodulator to function. I don't object to synchronizing clocks, but I do object to the fact that the demodulator can't function without it. The WSJT-X modes uniquely have this restriction. Existing modes like RTTY and Olivia extract the clock information from the data stream. This means that all of the information needed to decode them comes from their own transmission. The FT8 family lacks this ability.

It may seem like a small thing, but requiring use of a separate communication channel to operate, whether it is NTP, WWV, GPS, or anything else outside the radio channel, doesn't seem very "amateur radio."

However, the restricted message content that makes FT8 so popular means that it is easy to automate contacts, making large numbers of contacts with very little operating effort. So I decided to pursue FT8 as a challenge for my automation and software skills:

   convert(lemons) -> lemonade;

...or something like that.

Automated Amateur

The release of FT8 came with a new feature to WSJT-X, called "auto-sequencing," where the software chooses the next message to send, based on what the other station sends. Because FT8 used tighter timing than its predecessor (and FT4 used tighter timing, still), the team decided that auto-sequencing was necessary, since humans would struggle to operate the UI in the roughly one second per slot available to read the prior message and select the next one to send. Further, when calling CQ, the software even decides which station to answer when multiple replies are received.

This feature makes the stock software a "QSO robot," of sorts. Since the operator only decides to start or search for a contact, and the software makes all the subsequent decisions, the human is mostly removed from the decision logic.

Thus, the WSJT-X team has implemented most of the automation in their software that is needed to make contacts without significant human intervention. While they strongly discourage unattended robots, they have implemented most of the needed automation logic to do so.

Even without auto-sequencing, the nature of fixed messages of predictable content is ready-made for automation. The WSJT-X software isn’t a completely automated system on its own, nor is its API particularly well-suited for attaching additional automation. However, there are some improvements that can be made to increase efficiency and QSO rate, while reducing operator workload.

It’s also important to define some terms here. When I speak of "automation" in this context, I’m referring to the automation of the user interface (UI) task sequence. Whether and how that relates to "automatic" operation — which is a specific technical term used in some regulatory rules — is a question that I will touch on below. However, it is important to note that these are distinct terms used in distinct contexts.

Improving the Stock Automation

The stock WSJT-X software already implements automation of the message sequence of each contact, but by default, the automation stops after each contact. So the main improvement that can be made is automating the sequence of making multiple contacts.

For that task, there are two two basic automation schemes that can be implemented: "pouncing" and "running." These are also the two basic modes of any contest operation.

I experimented with "run" and "pounce" implemented as distinct modes within a single software project. It is also possible to run a combined mode where the software calls CQ, but opportunistically pounces stations in the absence of CQ replies. This combined mode is only a very small change to the run mode, so I will still treat the two as distinct modes.

First is the "run" automation. This enables the software to call CQ over some period of time, making contacts as other stations reply. When a contact is completed, the software resumes the run. The main task here is clicking "Enable TX" whenever it is disabled by WSJT-X. Otherwise, the auto-sequencing logic within WSJT-X is a reasonably complete implementation of the automation rules needed.

The second scheme is "pounce" automation. This enables the software to quietly watch decodes, the selectively attempt to reply to stations calling CQ. This was a somewhat more interesting mode to implement, as there are some custom rules that can be used for pouncing that don’t apply to running. Such rules might include only calling stations that are unique in the log, or located outside a certain geographic radius, or only calling DX stations. Nonetheless, most of the logic for running also applies here.

Again, both modes of automation will be familiar to contest operators, as these are essentially identical to how contesting is done in any other mode.

Nuts and Bolts

There are two ways to implement additional automation — either within WSJT-X’s C++ UI code, or "outboard" as a separate application that interacts with WSJT-X. For simplicity and licensing considerations, I chose the latter.

I chose C# as the language for the outboard automation module, mostly because I have C# libraries for geography calculations and ADIF files. In retrospect, another language like Ruby or Python might have provided better library support overall, and eliminated a lot of compile steps during debugging. Also, Mono support for the CLR on Linux can be a little buggy for lesser-used features. But after 9,000 lines of code, I'm not going to change now. CLI languages do have the advantage of being JIT-able, which should make them run faster, even on modest hardware like a RasPi.

WSJT-X provides decoded message text via two interfaces: the UDP API, and the ALL.TXT file. This allows the outboard application to watch both transmitted and received messages in real time. In practice, I found that the UDP reporting of decodes and encodes was unreliable, occasionally dropping a message. So I decided to tail the ALL.TXT file for message updates, which works quite well.

The UDP API also provides some control functions, but unfortunately, it does not allow complete control of the UI. There are still a few commands that can only be sent to WSJT-X via GUI elements such as buttons and text boxes. As a Linux user, I chose to use the XDO interface to accomplish this. Windows has a similar but distinct interface for UI interaction.

The UDP API has good support for automatic pouncing of stations by user software. I was pleasantly surprised by this, since the WSJT-X team has in the past shyed away from features that could be used to increase the automation of operation.

The stream of FT8 messages is the main input to the outboard automation logic. As the automation sees messages come and go, it makes decisions on how to steer the operation of WSJT-X. When actions are needed, they are sent to WSJT-X via UDP or XDO. UDP is preferred for speed where it works properly, but XDO is used where needed.

The automation module mostly alows WSJT-X to "just do its thing," intervening only when needed. Think of it like a spacecraft... WSJT-X is the main engine doing the heavy lifting, with the outboard automation module providing small thruster or gimbal inputs, to keep the main engine pointed in the best direction.

Well-Behaved Automation

Getting the basic logic for running and pouncing was the easy part. Again, the stock WSJT-X does most of the work. However, there’s a well-known military adage that no plan survives contact with the enemy. In this case, even well-planned code stumbles when making real contacts on the air.

While a naive implementation of either automation variant is almost trivial to code, not all FT8 participants follow a predicable or helpful recipe. There are also malicious and/or careless operators who send messages that easily confuse simple automation rules.

One common interference technique is where the attacker sends a grid square repeatedly to another station, in order to prevent that station from working others. This approach completely fouls WSJT-X’s auto-sequencing.

If grid square frames are interleaved with signal report frames, the watchdog timer can also be rendered useless as a countermeasure.

Either of these scenarios are essentially a denial-of-service attack. Such situations can be caused by ignorant or careless users, or by poorly written "robots." There are plenty of each on the air.

Because of such things, the first task for writing well-behaved automation is to implement a list of blocked calls. These are call signs that the automation will never try to call, and their frames are always ignored. The list should be easily updated with new calls as the automation operates.

The automation itself needs to watch for errant behavior, and block calls of misbehaving stations. This allows it to quickly deal with problems by just ignoring them. When the operator sees behavior that the automation can’t cope with, manual blocking of calls is also useful to help the automation to "just move on."

A related task for resilient automation is proper timeouts. We want to detect when the station we are working unexpectedly disappears (or gets stuck). The stock software opts for a single "watchdog" timer, which shuts down the automation if too many identical frames are sent. Just because a station vanishes in the fade, we don’t want to send and send until the transmitter times out. We want to work others, and hopefully the lost station will come back later.

The cousin of proper timeouts is state caching. We want to keep all the information we receive from other stations, such as grid squares and reports, in case a contact takes place piecemeal. It’s not unusual for a station to fade out before sending a final "roger" frame. If he comes back later, after we have worked someone else, and send only a "roger" frame to complete the contact, we want to log the entire contact with all the information without relying on unnecessary repeats. It’s also not unusual for a station to call CQ with their grid square, but when he calls us later, he may send only a signal report. If we keep the grid square from his CQ call, we can use that when logging his later abbreviated call. All of this should be done automatically.

Speaking of logging, there is no reason for a human to interact with logging. The FT8 messages are the same with every contact, including at most a call sign, grid square, and signal reports. Once the stations trade "roger" frames, the DX should be logged, without duplication. Although distinct from automating the contact sequence itself, this task is another good fit for automation.

Frequency Discipline

It’s important for us to avoid interfering with other stations, but one feature that isn’t really supported by the WSJT-X software is automatic frequency selection for "run" operation. It is an exercise for the operator to find a clear frequency to call. It is also one of the reasons FT8 can’t be completely automated; more on this later.

The WSJT-X software does provide help when selecting transmit frequency for pounce operations. When requesting the software to reply to a CQ, the default behavior is to move the transmit frequency to that of the CQ station. This helps avoid clobbering someone else when pouncing, and makes "pounce" automation easier to tame with respect to frequency discipline.

One task that outboard automation can perform is detection of when another station is clobbering our CQ frequency. At a minimum, this should be reported as a warning for the operator to deal with.

Some operators advocate calling CQ where somebody else is calling CQ, but on the opposite time slot. They claim that this improves channel efficiency. Unfortunately, such claims ignore the fact that others will be replying in that other slot, so calling CQ on someone else’s frequency almost guarantees interference.

The fixed timing of messages adds some complexity to the question of deconflicting stations. If two stations are calling CQ in the same time slot, they will not hear each other. If they are half-overlapping in frequency, they are always clobbering each other. Since they always start and stop transmitting at the same time, there is no good way for them to deconflict with each other. If you then reply to one, the other may think you are clobbering "his" frequency, when the conflict is between the runners, not with the caller.

Experiences

Use of extended automation over ~5800 hours of operation yielded 50,000 contacts between FT8 and FT4, on the HF and 6m bands, giving a QSO rate of about 8.6 per hour. Roughly 48% of contacts were unique calls, and 93% were unique to the combination of call, band, and mode (i.e., "unique" in the contesting sense). That means that each call was worked twice, on average. In general, I don't do FT8 contests, but those numbers do include a couple of Field Day events, which accounts for some of the duplicated calls.

Rather than spending time trying to quickly scan decodes and pick stations to contact, I write the logic, and let the PC do the work. Writing and maintaining the automation rules was a far more interesting hobby activity than mindlessly double-clicking on decodes. Many times, I can write or adjust the automation while the automation is busy making contacts. It's almost like having a second operator in the room.

It is strangely rewarding to watch software rake in one DX call after another without my help. It is a much more relaxing way to operate.

When running FT4, automation becomes even more powerful, since there is very little time to make decisions between decodes. Since FT4 was intended as a contest mode, having maximum automation helps maximize QSO rate, which is the point of contesting.

My preferred operation mode is "pounce." This is particularly helpful on a very busy frequency where calling stations are wall-to-wall. Rather than adding to the clutter by calling CQ, the software calls stations already running. On a busy channel, this can easily outperform running.

A combined operating mode, where the automation is running, but occasionally pouncing along the way, is a good way to keep the QSO rate up. Some operators only run, while some only pounce, so allowing the software to do both is handy for catching stations that would otherwise be missed. The complement of this mode is also useful — where the automation focuses on S&P, but occasionally sends a single CQ when the Q rate starts to fall.

Automated logging is the only way to fly. Once both stations have traded "roger" frames, the QSO is logged automatically. Some people prefer to only log after trading "73" but to each his own. The really helpful thing about automating the decision to log a call is that many times the waterfall is cluttered with multiple people calling, and often a blocked call or two." The auto-logging sorts it all out perfectly, every time.

Once one's FT8 logfile accumulates a few hundred contacts, the data can be used for some interesting things. We can look at the mean differential signal reports, to get a (rough) sense of how antenna systems are performing. When I say "mean differential," I'm talking about computing the difference in SNR estimates for each contact — the received report less the sent report — then averaging those differences across many conatacts.

The signal reports traded are SNR estimates, computed from within the necessary bandwidth of each received signal, and as such, they are subject to a lot of variation due to congestion, IMD, local noise sources, etc. Further, beacuse of the way the SNR values are calculated, and then "adjusted" to reference a 2500Hz noise bandwidth, the estimate values are unrealistically optimisitc. However, even with these issues, they are still far more consistent and uniform than the older RST system of reporting. This makes them more useful for spotting trends in the data.

Distance and bearing information from grid squares also provides some interesting statistics. When combined with the differential signal reports, it allows you to compute stats within (or outside) specific distances. Since nearly all QSOs include grid squares, the data is fairly complete, even for relatively small samples.

Most people on FT8 use "contest rules," where each station is worked only once per band and mode. A scan of my log shows that fully 91% of contacts were unique across call, band, and mode, while 52% of contacts had a unique call. So each call in the log was present twice, on average; but fewer than one in ten was a duplicate contact.

On the downside, I find that there is a lot of poor behavior on FT8 and FT4. There is frequent deliberate interference that can be easily verified by looking at PSKreporter. People will report my signal with a very high SNR, but then start a long run right on top of me.

Broken robots are common, and some of their transmitted message sequences are downright comical. After sending 52 grid squares, even the dumbest robot should know to give up. My favorites are the ones who send "RR73" frames for several minutes. How many times does somebody really need to say "goodbye," anyway?

I'm amazed how often somebody will reply to my CQ with a high SNR report, then repeat that report over and over. So did you really copy me at +15dB?

Then there's that guy who occasionally pirates a P5 call...

"Tail-callers" are frequent, where someone will work a running station, then start calling CQ or other callers without changing frequency; this blocks other people from working the runner indefinitely. Fortunately, this is one behavior that seems to be slowly declining. Automation can also help here, by quickly moving the frequency before doing the tail-call, so that the original runner is undisturbed.

QSO-busting is more widespread than bronco-busting. One expects a certain number of busted QSOs just from fading, weak signals, etc. But I'm amazed how many stations there are who make dozens or hundreds of contacts, but never send more than a single SNR frame, and never send an "R" frame. I used a script to scan my decode log, and there are literally hundreds of stations who never complete a QSO — one SNR report is all you get from them. I drop these guys in the blocked-call list, too.

The number of calls in the blocked list drifts between 8% and 12% of the total number of unique calls in my FT8 logbook. It seems unfortunate that one in ten licensed operators either can't figure out how to run a mode that is already mostly automated, or just don't care to do so in a minimally friendly way. I suppose it does help explain why online social media is such a mess.

In fairness, my operation isn't perfect, and both I and the software do make mistakes. At the very least, I do try to operate kindly...

On the bright side, the outboard automation is very helpful when it comes to dealing with misbehaving stations. Instead of me having to handle each and every malfunctioning caller, the automation does it for me, usually by blocking the call. Writing automation rules to detect such things is simple, and it really does help keep the QSO rate up. My "robot" earns his money.

I would not have done FT8 without the extra automation. To be honest, with all the "liddy" behavior, I don't see how people operate without extra automation. Some have described FT8 as "watching paint dry," and I tend to agree. Automating the WSJT-X user interface is definitely "working smarter, not harder."

Now let's talk about more mundane subjects...

Aren’t "Robots" Against the Rules?

As with many things, permissible operation depends on the regulator issuing your license, and how you operate. So there is no single nor simple answer to this.

In the US, FCC rules only address "automatic" operation as used to ensure rules-compliance without a control operator present at the control point (see 47 CFR 97.3(a)(6)). Operation without a control operator is restricted to specific frequencies, and, in the case of digital modes, to specific bandwidths and related technical considerations (see 47 CFR 97.221).

However, the US rules appear to be silent regarding the level of automation employed while a control operator is present. And this makes sense, even in terms of older modes, such as RTTY or CW. If you visit any serious contest station, you will find very high levels of automation of transmitted signals. Software macros are an obvious example, controlling not only content, but the timing of T/R switching.

Other modes, such as ALE, actually require such high levels of automation, and these modes can even do automated frequency selection, in addition to timing and content decisions.

WSJT-X itself takes this a step further, automating the content and timing of entire sequences of messages, which it calls "auto-sequence."

Nobody seriously questions whether such operation is permissible in the presence of a control operator. In fact, ARRL itself has repeatedly endorsed the use of software, including WSJT-X, with these features. They even endorse the use of the mode and software for logbook stuffing.

As long as a control operator is present at the control point during transmission, the discussion of whether high levels of UI automation are permissible under US rules appears to be moot.

If you live or operate outside of US FCC jurisdiction, you should consult the regulator(s) there, of course.

Things get a bit different when referring to contesting rules...

E.g., the ARRL contest rules make a half-hearted attempt to disqualify contest contacts where the control operator didn’t explicitly initiate each contact. Even here, the rule text is poorly written. Here's an example from the RTTY Roundup rules, fetched February 2023:

Each contact must include contemporaneous direct initiation by both operators making a contact...

This language doesn't do anything to prevent robots — even by operators who intend to follow the rules. If a robot calls CQ for four minutes, and makes four contacts, how is the human starting that CQ run any less "contemporaneous" than a human operator initiating a CQ run that lasts four four minutes and makes one contact? In both cases, the human started an automation sequence that did the rest of the work.

More importantly, distinguishing a well-written FT8 "robot" from someone using only stock WSJT-X is impossible, since the message timing and content could easily be identical between the two cases.

What ARRL has missed is that robotic communications are part of the design of FT8. When they endorsed such modes for use in their contests, the horse has left the gate — there will be robots in all FT8 contests, because WSJT-X is already a robot.

Aren’t "Robots" Still a Bad Idea?

A frequent objection to "FT8 robots" is centered around how much automation is being used. WSJT-X by itself is already highly automated — I would estimate that WSJT-X contains 95% of the automation needed to make multiple contacts without operator intervention. It does not provide enough interface for 100% automation of operations, since tasks like band and frequency selection still have to be done by hand.

In normal FT8 operation, the difference between the "stock" automation and the popular idea of a "robot" is, at most, a double-click action per contact. A double-click from the mouse initiates a "pounce" operation, while a single click initiates a CQ run. As a result, objections to automation of FT8 operation are nothing more than a matter of degree, and a small degree at that. People can object to 90%, 92%, or 99% automation, but such distinctions are essentially subjective.

Without complete automation of all needed tasks, unattended operation is not advisable, even where regulations permit. In the US, the allowed frequencies for unatteneded operation overlap with typical FT8 "watering holes" on the VHF bands only. But increasing the level of automation can make attended operation more smooth and efficient.

Another common objection to "FT8 robots" is that the typical 3kHz-wide "watering hole" frequencies used by FT8 may become congested if people run automation over long periods of time. The assumption is that requiring a human to click something for each contact will somehow introduce enough delay between contact attempts to prevent congestion.

The problem with the "congestion" argument is that it just doesn’t scale. What’s the difference between ten robots and twelve humans? At some point, popularity will cause the same issues, even if no "robots" are running amok on the bands.

The solution to congestion is the same as for all other modes — hams need to learn to spread out in the band, and not rely on a single 3kHz voice channel on each band. Does this require hams with older radios to "spin the dial" now and then? Of course it does, but blaming "robots" for congestion caused by popularity does a disservice to everybody on the air.

Newer receivers will likely solve the "congestion" quesiton for us. A $30 RTL-SDR can gulp down dozens of kiloherz and even present them to the decoder software; so as long as WSJT-X can decode more than 3kHz at a time, the "watering hole" can be as large as needed.

Final Thoughts

The exercise of improving the automation of WSJT-X was unexpectedly rewarding. It also helped me understand the popularity of the mode a bit better.

I still prefer modes that allow proper conversation. There are some some very capable modes that perform very well in poor SNR conditions, such as Olivia and PSK-R, among many others.

Then again, in the age of Twitter, I’m going to have to make the most of short-message modes, because they are certainly en vogue.

But I’m going to make it easy on myself, and on my mouse hand, to do so.

— Copyright (C) 2023 by Matt Roberts, KK5JY. All Rights Reserved.