Skip to content

Commit 4aa1e5b

Browse files
authored
Keep missing values missing in resample_sig (#615)
Follow-up to @tompollard's note on #604. `resample_sig` fills NaN values before resampling, because the FFT would otherwise spread them over the whole signal. But the filled values were then returned as if they were real data, so the output had no trace of where the input was missing. Now output samples near a NaN input sample are set back to NaN. An output sample at position `t` (in input samples) is NaN if any input sample within half an output spacing of `t` is NaN (and within at least one input sample, so neighbours count when upsampling). The filling and resampling are unchanged, so samples away from the gaps keep the same values. For example, record 100 at 360 Hz with its first 50 samples set to NaN, resampled to 250 Hz, gives NaN for the first 35 output samples (positions below 50) and finite values after. Signals without NaN give the same output as before. This changes the output for signals with NaN, which is why it's a separate PR. The two tests from #604 now expect NaN at the filled positions. Tests: new tests check which output samples are NaN when upsampling, and when downsampling with a single NaN and with a gap between two output samples; they fail without the change. `pytest -m "not network"` passes (109 tests), and the docs build passes.
2 parents 68e458f + 79ceedf commit 4aa1e5b

3 files changed

Lines changed: 46 additions & 5 deletions

File tree

‎docs/changes.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,9 @@ Unreleased
6565
**Fix resample_sig for signals starting with NaN**
6666
``resample_sig`` (and the functions that use it, such as ``resample_singlechan`` and ``resample_multichan``) returned only NaN values if the signal started with NaN. Leading NaN values now take the first valid value before resampling, as trailing ones already took the last.
6767

68+
**Keep missing values missing in resample_sig**
69+
``resample_sig`` (and ``resample_singlechan`` and ``resample_multichan``) now return NaN for output samples with a NaN input value within half an output sample spacing (or within one input sample, when upsampling). NaN values are still filled before resampling, so the rest of the signal is unaffected, but the filled values are no longer returned as if they were real data.
70+
6871
**Reject aux notes that can't be written in wrann**
6972
``wrann`` now raises a ``ValueError`` if an ``aux_note`` is longer than 255 characters or contains characters outside Latin-1, instead of writing a corrupt annotation file. A longer note used to be read back as many extra annotations, and other characters were silently replaced.
7073

‎src/wfdb/processing/basic.py‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ def resample_sig(x, fs, fs_target):
4242
Returns
4343
-------
4444
resampled_x : ndarray
45-
Array of the resampled signal values.
45+
Array of the resampled signal values. Samples with a NaN value in
46+
`x` within half an output sample spacing (or within one input
47+
sample, when upsampling) are NaN.
4648
resampled_t : ndarray
4749
Array of the resampled signal locations.
4850
@@ -55,12 +57,23 @@ def resample_sig(x, fs, fs_target):
5557
new_length = int(x.shape[0] * fs_target / fs)
5658
# Fill NaN values before resampling. Leading and trailing NaN values take
5759
# the nearest valid value, since interpolate() leaves leading ones as NaN.
58-
if np.isnan(x).any():
60+
nan_mask = np.isnan(x)
61+
if nan_mask.any():
5962
x = pd.Series(x.reshape((-1,))).interpolate(limit_direction="both").values
6063
resampled_x, resampled_t = signal.resample(x, num=new_length, t=t)
6164
assert resampled_x.shape == resampled_t.shape and resampled_x.shape[0] == new_length
6265
assert np.all(np.diff(resampled_t) > 0)
6366

67+
# The filled values aren't real data, so mark output samples with a NaN
68+
# input sample within half an output spacing (and at least one input
69+
# sample, so neighbours count when upsampling) as NaN again.
70+
if nan_mask.any():
71+
half = max(fs / fs_target / 2, 1)
72+
n_nan = np.concatenate([[0], np.cumsum(nan_mask.reshape((-1,)))])
73+
lo = np.clip(np.floor(resampled_t - half).astype(int) + 1, 0, len(t))
74+
hi = np.clip(np.ceil(resampled_t + half).astype(int), 0, len(t))
75+
resampled_x[n_nan[hi] - n_nan[lo] > 0] = np.nan
76+
6477
return resampled_x, resampled_t
6578

6679

‎tests/test_processing.py‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,26 +38,51 @@ def test_resample_multi(self):
3838
assert new_sig.shape[1] == sig.shape[1]
3939

4040
def test_resample_leading_nan(self):
41-
"""Leading NaN values take the first valid value before resampling."""
41+
"""Leading NaN values are filled before resampling, then marked as NaN."""
4242
x = np.array([np.nan, np.nan, 1.0, 2.0, 3.0, 4.0])
4343
filled = np.array([1.0, 1.0, 1.0, 2.0, 3.0, 4.0])
4444

4545
new_x, _ = processing.resample_sig(x, fs=2, fs_target=4)
4646
expected, _ = processing.resample_sig(filled, fs=2, fs_target=4)
4747

48-
assert np.isfinite(new_x).all()
48+
# Output positions 0 to 1.5 are next to a NaN input sample
49+
expected[:4] = np.nan
4950
np.testing.assert_allclose(new_x, expected)
5051

5152
def test_resample_trailing_and_inner_nan(self):
52-
"""Inner NaN values are interpolated, and trailing ones take the last."""
53+
"""Inner and trailing NaN values are filled, then marked as NaN."""
5354
x = np.array([1.0, np.nan, 3.0, 4.0, np.nan, np.nan])
5455
filled = np.array([1.0, 2.0, 3.0, 4.0, 4.0, 4.0])
5556

5657
new_x, _ = processing.resample_sig(x, fs=2, fs_target=4)
5758
expected, _ = processing.resample_sig(filled, fs=2, fs_target=4)
5859

60+
# Positions 0.5 to 1.5 and 3.5 onwards are next to a NaN input sample
61+
expected[[1, 2, 3]] = np.nan
62+
expected[7:] = np.nan
5963
np.testing.assert_allclose(new_x, expected)
6064

65+
def test_resample_marks_nan(self):
66+
"""Output samples at or next to NaN input samples are NaN."""
67+
x = np.arange(12.0)
68+
x[6] = np.nan
69+
70+
new_x, new_t = processing.resample_sig(x, fs=4, fs_target=8)
71+
72+
assert np.isnan(new_x).tolist() == ((new_t > 5) & (new_t < 7)).tolist()
73+
74+
@pytest.mark.parametrize(
75+
"gap, expected", [([500], [50]), (list(range(501, 509)), [50, 51])]
76+
)
77+
def test_resample_marks_nan_downsampling(self, gap, expected):
78+
"""Gaps between two output samples are still marked when downsampling."""
79+
x = np.sin(np.arange(1000) / 50)
80+
x[gap] = np.nan
81+
82+
new_x, _ = processing.resample_sig(x, fs=1000, fs_target=100)
83+
84+
assert np.flatnonzero(np.isnan(new_x)).tolist() == expected
85+
6186
def test_normalize_bound(self):
6287
sig, _ = wfdb.rdsamp("sample-data/100")
6388
lb = -5

0 commit comments

Comments
 (0)