FreeDV SNR Performance
Matt Roberts - matt-at-kk5jy-dot-net
Published: 2018-05-29
Updated: 2018-05-30
The FreeDV software, by David Rowe and his team, is
one of the first completely open-source and open-standard digital voice (DV) products
available for general amateur radio use. It is build around an open codec and
modem, and allows for a high-quality voice transmission to be done in around 1.1kHz
of spectrum. Compared to other DV modes, it is incredibly spectrally efficient.
David didn't stop there, however, and he has been working on a whole family of DV modes
that use even lower bit rates, making them more robust on our noisy HF bands.
What follows here is a brief study of the performance of FreeDV's modem and codec, when
used on a channel containing
AWGN, to
evaluate the effects of steady-state noise floor. This paper isn't intended as
a complete study of the effects of other noise types, including pulse or impulse
noise. The study also doesn't attempt to simulate the various kinds of path
distoration that can occur, such as Doppler, multipath, or fading. However, for
those wanting to compare the performance of David's DV mode to that of analog modes,
the tests below should give a good feel for how each degrades with decreasing
SNR, and by showing
how each modem compares to analog voice in identical noise conditions.
The Tests
For each test, a voice test file is first encoded into a noiseless data stream.
The data stream then has a specific noise level mixed with it, before the noisy data
stream is decoded back to voice audio. The noise is generated by a
PRNG with
a roughly Gaussian distribution.
The level of noise and signal are adjusted before being mixed, so as to avoid
clipping the result of the mix.
Separately for each test, the original voice test file has the same amount of noise
applied to it as the digital stream, to generate a second test output for each SNR
level, to allow for direct comparison between the DV and analog outputs. This
allows for the listener to hear what an analog
(e.g., SSB)
channel would sound like at that SNR, and compare it to what the DV modem would be
able to recover on that same noisy channel. The sampling frequency used for
all of the test files is 8kHz, so the passband is only slightly larger than what
would be encountered on a full-width SSB channel.
The only analog steps involved in the testing process were the original microphone
recording of the voice sample, and the playback of the resulting audio files in the
results table, below. For best results during playback, it would be
advantageous to listen to the result files using headphones, rather than speakers.
The intermediate test steps involved no analog processing, as the noise generaiton
and mixing process was done digitally and automatically under control of a script.
The modem encoding and decoding were done using the command line tools provided as part
of the codec2 package, and the intermediate results captured as raw digital
audio files.
The Results
The original test file can be found here,
as a WAV file download. The recording uses 8000Hz sample rate, which yields a
noise bandwidth of just under 4kHz.
The first modem tested was the 1600 version, which is the default for
the FreeDV application. The results at selected SNR values are collected
in Table 1. The table lists several columns:
- SNR
- This is the S/N ratio, measured in dB, of the environment in which the
codec and modem were tested.
- DV Output
- This is decoded audio, recovered by the DV modem from the noisy channel
at the specified SNR.
- DV Plot
- This is a plot of the DV Output audio.
- Analog Output
- This is original voice recording, with the same level of noise added to
it as was used in the DV modem test for that row.
- Analog Plot
- This is a plot of the Analog Output audio.
Each of the DV and Analog audio results can be downloaded as a WAV file. Each of the
plots can be enlarged by clicking on the thumbnail.
Table 1: Test Results
FreeDV 1600 vs. Analog
As you can see in the plots, the DV demodulator tries to recover something
from the data stream well into negative SNR values. The intelligible,
usable limit is somewhere around +2dB. Under non-gaussian noise conditions,
the lowest intelligible SNR will probably be a bit higher for both DV and analog
modes.
Nonetheless, the 1600 DV modem has a nice edge over its analog
counterpart, so it should be able to provide an improvement in received
quality and intelligibility over analog modes for most uses.
The 1600 mode isn't the only one supported by FreeDV, however. The
results for the original 700 mode can be seen in Table 2,
below.
Table 2: Test Results
FreeDV 700 vs. Analog
The 700 mode allows intelligible operation down to around 0dB SNR.
That's a nice improvement over the 1600 mode. The sound is a bit
tinny, but it is still quite usable, and the advantage over analog voice is quite
clear.
A newer version of the 700 mode, called 700C, was tested
next. The results are collected in Table 3, below.
Table 3: Test Results
FreeDV 700C vs. Analog
The SNR performance is similar to the 700 mode, but the 700C mode
has better fidelity in the restored waveform. However, the modem's ability
to synchronize to the received signal fell off earlier, with the modem missing
synchronization completely below -4dB.
The codec2 library used in FreeDV supports an even newer mode as of version
0.8. It is a refinement of the 700 mode called 700D. This
mode is supported in the FreeDV software as of version 1.3. Its performance
is shown in Table 4, below.
Table 4: Test Results
FreeDV 700D vs. Analog
The 700D mode allows intelligible operation down to somewhere between
-2dB and -3dB SNR, depending on one's tolerance for artifacts. That is an
impressive improvement on both the 700 and 1600 modes, and far better
than even an experienced analog voice operator could achieve on SSB. If you
listen to the corresponding analog audio, the signal is so completely buried
in noise, that many listeners would be hard-pressed to tell you if there was even a
voice signal present in the noise, at all. The fact that the modem can
decode a noise-free and completely understandable signal at that level is amazing.
Future Work
As explained above, this is a simple simulation of voice transmissions in AWGN.
HF band noise isn't anything close to Gaussian or white, and there are all kinds of
bizarre types of noise and distortion that can interfere with transmission. Each
of these could be modeled and simulated, which would produce new results for each
modem type.
Further, the SNR calculated assumes that the signals and noise have predictable and
steady RMS power levels, which is certainly not true for the signals — neither
the FreeDV digital signals, nor the analog voice signals. That makes computing
SNR an approximation at best. The tests performed above assume that the signals
and noise have RMS values that are a fixed fraction of their peak values, which is a
reasonable first-order estimate. That said, on-air SNR performance comparisons
should keep in mind that analog voice transmission is highly dependent upon
the real RMS power delivered to the RF signal.
There are probably some other improvements that I haven't considered, even to the
simplified tests whose results are reported above. I welcome input on how to
improve the tests and/or its assumptions to improve the correspondence between the
tests and real-world operation.
In the mean time, I encourage people to experiment with the new FreeDV modes, as
digital voice is the future of HF voice operation.
Copyright (C) 2018 by Matt Roberts, KK5JY.
All Rights Reserved.
