Skip to content

Commit 9336f76

Browse files
authored
Support reading records from fsspec's memory:// (#629)
> [!NOTE] > This code was implemented by AI. Adds support for `memory://` to `rdrecord`, `rdsamp` and `rdheader`, addressing #491 (comment). Supersedes #491. Adding this was already almost done via `fsspec`. What was needed was Windows support for path-joining POSIX paths and adding it to the supported protocols list. Besides that the PR adds documentation and tests.
2 parents 263624c + 5574959 commit 9336f76

6 files changed

Lines changed: 222 additions & 39 deletions

File tree

‎changelog.d/memory-uris.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
**Read records from memory**
2+
``rdrecord``, ``rdsamp`` and ``rdheader`` now accept ``memory://`` URIs, which read from the in-memory filesystem of ``fsspec``, as ``rdann`` already did. This also makes cloud URIs work for signal files and multi-segment records on Windows, where the file paths used to be built with backslashes.

‎docs/io.rst‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,38 @@ handles credential discovery automatically.
136136
against the configured PhysioNet index URL.
137137

138138

139+
Reading From Memory
140+
-------------------
141+
142+
Records and annotations can also be read from the in-memory filesystem of
143+
``fsspec``, for example when the files come from an upload form instead of
144+
from disk. Store all files of the record in the same directory, and pass a
145+
``memory://`` URI as the ``record_name`` argument:
146+
147+
.. code-block:: python
148+
149+
import fsspec
150+
import wfdb
151+
152+
fs = fsspec.filesystem("memory")
153+
fs.pipe("/upload/100.hea", hea_bytes)
154+
fs.pipe("/upload/100.dat", dat_bytes)
155+
fs.pipe("/upload/100.atr", atr_bytes)
156+
157+
record = wfdb.rdrecord("memory://upload/100")
158+
ann = wfdb.rdann("memory://upload/100", "atr")
159+
160+
The files stay in memory until they are removed, for example with
161+
``fs.rm("/upload", recursive=True)``.
162+
163+
.. warning::
164+
165+
The in-memory filesystem is shared by the whole process, and everything
166+
that opens a file shares a single file position. Use a unique directory
167+
for each upload, and do not read the same record from several threads at
168+
once, as this can return wrong data without raising an error.
169+
170+
139171
Downloading
140172
-----------
141173

‎src/wfdb/io/_coreio.py‎

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
import posixpath
23

34
import fsspec
@@ -6,7 +7,57 @@
67
from wfdb.io.download import config
78

89
# Cloud protocols
9-
CLOUD_PROTOCOLS = ["az://", "azureml://", "s3://", "gs://"]
10+
CLOUD_PROTOCOLS = ["az://", "azureml://", "s3://", "gs://", "memory://"]
11+
12+
13+
def _split_path(record_name):
14+
"""Split a local path or cloud URI into its directory and base name.
15+
16+
Unlike `os.path.split`, this keeps the protocol of a URI intact if the URI
17+
has no directory, so that "memory://100" gives "memory://" and "100".
18+
19+
Parameters
20+
----------
21+
record_name : str or os.PathLike
22+
The local path or cloud URI.
23+
24+
Returns
25+
-------
26+
dir_name : str
27+
The local directory or cloud URI of the directory.
28+
base_name : str
29+
The last component of the path.
30+
31+
"""
32+
path = os.fspath(record_name)
33+
if any(path.startswith(proto) for proto in CLOUD_PROTOCOLS):
34+
protocol, separator, location = path.partition("://")
35+
dir_name, base_name = posixpath.split(location)
36+
return protocol + separator + dir_name, base_name
37+
return os.path.split(path)
38+
39+
40+
def _join_path(dir_name, file_name):
41+
"""Join a local directory or cloud URI with a file name.
42+
43+
URIs always use forward slashes, regardless of the operating system.
44+
45+
Parameters
46+
----------
47+
dir_name : str
48+
The local directory or cloud URI.
49+
file_name : str
50+
The name of the file.
51+
52+
Returns
53+
-------
54+
str
55+
The path or URI of the file.
56+
57+
"""
58+
if any(dir_name.startswith(proto) for proto in CLOUD_PROTOCOLS):
59+
return posixpath.join(dir_name, file_name)
60+
return os.path.join(dir_name, file_name)
1061

1162

1263
def _open_file(

‎src/wfdb/io/_signal.py‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
import io
22
import math
33
import os
4-
import posixpath
54
import sys
65

76
import fsspec
@@ -1711,7 +1710,7 @@ def _rd_dat_file(
17111710

17121711
# Local or cloud dat file
17131712
elif pn_dir is None:
1714-
with fsspec.open(os.path.join(dir_name, file_name), "rb") as fp:
1713+
with fsspec.open(_coreio._join_path(dir_name, file_name), "rb") as fp:
17151714
fp.seek(start_byte)
17161715
sig_data = util.fromfile(
17171716
fp, dtype=np.dtype(DATA_LOAD_TYPES[fmt]), count=element_count
@@ -1916,7 +1915,7 @@ def _rd_compressed_file(
19161915
open_file = wfdb_archive.open(file_name)
19171916
else:
19181917
if pn_dir is None:
1919-
file_name = os.path.join(dir_name, file_name)
1918+
file_name = _coreio._join_path(dir_name, file_name)
19201919
open_file = _coreio._open_file(pn_dir, file_name, "rb")
19211920

19221921
with open_file as fp:
@@ -2600,10 +2599,9 @@ def _infer_sig_len(
26002599
if wfdb_archive is not None:
26012600
file_size = wfdb_archive.file_size(file_name)
26022601

2603-
# If this is a cloud path, use posixpath to construct the path and fsspec to open
2604-
# file
2602+
# If this is a cloud path, use fsspec to open the file
26052603
elif any(dir_name.startswith(proto) for proto in CLOUD_PROTOCOLS):
2606-
with fsspec.open(posixpath.join(dir_name, file_name), mode="rb") as f:
2604+
with fsspec.open(_coreio._join_path(dir_name, file_name), mode="rb") as f:
26072605
file_size = f.seek(0, os.SEEK_END)
26082606

26092607
# If the PhysioNet database path is provided, construct the download path using the

‎src/wfdb/io/record.py‎

Lines changed: 31 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
import pandas as pd
1111

1212
from wfdb.io import _header, _signal, _url, download, header, util
13-
from wfdb.io._coreio import CLOUD_PROTOCOLS
13+
from wfdb.io._coreio import CLOUD_PROTOCOLS, _join_path, _split_path
1414
from wfdb.io.archive import WFDBArchive, archive_for_writing, is_archive_path
1515

1616
# -------------- WFDB Signal Calibration and Classification ---------- #
@@ -169,8 +169,8 @@ class BaseRecord:
169169
files are searched for locally, or when `record_name` is a
170170
supported cloud URI, they are read directly from cloud storage
171171
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
172-
`azureml://`. Install the appropriate `fsspec` backend
173-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
172+
`azureml://`, `memory://`. Other than for `memory://`, install the
173+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
174174
n_sig : int, optional
175175
Total number of signals.
176176
fs : int, float, optional
@@ -700,8 +700,8 @@ class Record(BaseRecord, _header.HeaderMixin, _signal.SignalMixin):
700700
files are searched for locally, or when `record_name` is a
701701
supported cloud URI, they are read directly from cloud storage
702702
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
703-
`azureml://`. Install the appropriate `fsspec` backend
704-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
703+
`azureml://`, `memory://`. Other than for `memory://`, install the
704+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
705705
n_sig : int, optional
706706
Total number of signals.
707707
fs : float, optional
@@ -1080,8 +1080,8 @@ class MultiRecord(BaseRecord, _header.MultiHeaderMixin):
10801080
files are searched for locally, or when `record_name` is a
10811081
supported cloud URI, they are read directly from cloud storage
10821082
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
1083-
`azureml://`. Install the appropriate `fsspec` backend
1084-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
1083+
`azureml://`, `memory://`. Other than for `memory://`, install the
1084+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
10851085
n_sig : int, optional
10861086
Total number of signals.
10871087
fs : int, float, optional
@@ -1837,10 +1837,10 @@ def rdheader(record_name, pn_dir=None, rd_segments=False):
18371837
files are searched for locally, or when `record_name` is a
18381838
supported cloud URI, they are read directly from cloud storage
18391839
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
1840-
`azureml://`. Install the appropriate `fsspec` backend
1841-
(e.g., `s3fs`, `gcsfs`, `adlfs`). To read the header from a
1842-
local .wfdb archive, pass the path of the archive including the
1843-
.wfdb extension.
1840+
`azureml://`, `memory://`. Other than for `memory://`, install the
1841+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
1842+
To read the header from a local .wfdb archive, pass the path of
1843+
the archive including the .wfdb extension.
18441844
pn_dir : str, optional
18451845
Option used to stream data from Physionet. The Physionet
18461846
database directory from which to find the required record files.
@@ -1861,7 +1861,7 @@ def rdheader(record_name, pn_dir=None, rd_segments=False):
18611861
>>> ecg_record = wfdb.rdheader('100', pn_dir='mitdb')
18621862
18631863
"""
1864-
dir_name, base_record_name = os.path.split(record_name)
1864+
dir_name, base_record_name = _split_path(record_name)
18651865
file_name = f"{base_record_name}.hea"
18661866

18671867
# Read the header, and any segment headers, from a .wfdb archive
@@ -1873,10 +1873,9 @@ def rdheader(record_name, pn_dir=None, rd_segments=False):
18731873
_read_segment_headers(record, dir_name, pn_dir, archive)
18741874
return record
18751875

1876-
# If this is a cloud path, use posixpath to construct the path and fsspec to open
1877-
# file
1876+
# If this is a cloud path, use fsspec to open the file
18781877
if any(dir_name.startswith(proto) for proto in CLOUD_PROTOCOLS):
1879-
with fsspec.open(posixpath.join(dir_name, file_name), mode="r") as f:
1878+
with fsspec.open(_join_path(dir_name, file_name), mode="r") as f:
18801879
header_content = f.read()
18811880

18821881
# If the PhysioNet database path is provided, construct the download path using the
@@ -1942,7 +1941,7 @@ def _read_record_header(record_name, dir_name, pn_dir, wfdb_archive):
19421941
"""
19431942
if wfdb_archive is not None:
19441943
return _parse_header_content(wfdb_archive.read_header(record_name))
1945-
return rdheader(os.path.join(dir_name, record_name), pn_dir=pn_dir)
1944+
return rdheader(_join_path(dir_name, record_name), pn_dir=pn_dir)
19461945

19471946

19481947
def _read_segment_headers(record, dir_name, pn_dir, wfdb_archive):
@@ -2069,10 +2068,10 @@ def rdrecord(
20692068
files are searched for locally, or when `record_name` is a
20702069
supported cloud URI, they are read directly from cloud storage
20712070
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
2072-
`azureml://`. Install the appropriate `fsspec` backend
2073-
(e.g., `s3fs`, `gcsfs`, `adlfs`). To read a record from a local
2074-
.wfdb archive, pass the path of the archive including the .wfdb
2075-
extension.
2071+
`azureml://`, `memory://`. Other than for `memory://`, install the
2072+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
2073+
To read a record from a local .wfdb archive, pass the path of
2074+
the archive including the .wfdb extension.
20762075
sampfrom : int, optional
20772076
The starting sample number to read for all channels.
20782077
sampto : int, 'end', optional
@@ -2201,7 +2200,7 @@ def _rdrecord(
22012200
archive.
22022201
22032202
"""
2204-
dir_name, base_record_name = os.path.split(record_name)
2203+
dir_name, base_record_name = _split_path(record_name)
22052204
# Update the dir_name using abspath unless it is a cloud path
22062205
if not any(dir_name.startswith(proto) for proto in CLOUD_PROTOCOLS):
22072206
dir_name = os.path.abspath(dir_name)
@@ -2409,7 +2408,7 @@ def _rdrecord(
24092408
)
24102409
else:
24112410
record.segments[seg_num] = rdrecord(
2412-
os.path.join(dir_name, record.seg_name[seg_num]),
2411+
_join_path(dir_name, record.seg_name[seg_num]),
24132412
sampfrom=seg_ranges[i][0],
24142413
sampto=seg_ranges[i][1],
24152414
channels=seg_channels[i],
@@ -2470,8 +2469,8 @@ def rdsamp(
24702469
files are searched for locally, or when `record_name` is a
24712470
supported cloud URI, they are read directly from cloud storage
24722471
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
2473-
`azureml://`. Install the appropriate `fsspec` backend
2474-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
2472+
`azureml://`, `memory://`. Other than for `memory://`, install the
2473+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
24752474
sampfrom : int, optional
24762475
The starting sample number to read for all channels.
24772476
sampto : int, 'end', optional
@@ -2590,8 +2589,8 @@ def sampfreq(record_name, pn_dir=None):
25902589
files are searched for locally, or when `record_name` is a
25912590
supported cloud URI, they are read directly from cloud storage
25922591
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
2593-
`azureml://`. Install the appropriate `fsspec` backend
2594-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
2592+
`azureml://`, `memory://`. Other than for `memory://`, install the
2593+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
25952594
pn_dir : str, optional
25962595
Option used to stream data from Physionet. The Physionet
25972596
database directory from which to find the required record files.
@@ -2640,8 +2639,8 @@ def signame(record_name, pn_dir=None, sig_nums=[]):
26402639
files are searched for locally, or when `record_name` is a
26412640
supported cloud URI, they are read directly from cloud storage
26422641
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
2643-
`azureml://`. Install the appropriate `fsspec` backend
2644-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
2642+
`azureml://`, `memory://`. Other than for `memory://`, install the
2643+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
26452644
pn_dir : str, optional
26462645
Option used to stream data from Physionet. The Physionet
26472646
database directory from which to find the required record files.
@@ -2699,8 +2698,8 @@ def wfdbdesc(record_name, pn_dir=None):
26992698
files are searched for locally, or when `record_name` is a
27002699
supported cloud URI, they are read directly from cloud storage
27012700
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
2702-
`azureml://`. Install the appropriate `fsspec` backend
2703-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
2701+
`azureml://`, `memory://`. Other than for `memory://`, install the
2702+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
27042703
pn_dir : str, optional
27052704
Option used to stream data from Physionet. The Physionet
27062705
database directory from which to find the required record files.
@@ -2833,8 +2832,8 @@ def wfdbtime(record_name, input_times, pn_dir=None):
28332832
files are searched for locally, or when `record_name` is a
28342833
supported cloud URI, they are read directly from cloud storage
28352834
via `fsspec`. Supported URIs: `s3://`, `gs://`, `az://`,
2836-
`azureml://`. Install the appropriate `fsspec` backend
2837-
(e.g., `s3fs`, `gcsfs`, `adlfs`).
2835+
`azureml://`, `memory://`. Other than for `memory://`, install the
2836+
appropriate `fsspec` backend (e.g., `s3fs`, `gcsfs`, `adlfs`).
28382837
input_times : str, list
28392838
The desired times (or samples) to be converted and displayed for the
28402839
chosen record. This can either be a single string, or a list of

0 commit comments

Comments
 (0)