Out-of-tree plugins that put MDSplus data behind a Pelican origin, serving the two kinds of client that exist — plus the client-side transport that reaches them:
| Component | For | Gets |
|---|---|---|
libXrdOssMdsplus.so (XrdOss) |
path-based clients — toksearch, curl, pelicanfs |
caching; TDI results as virtual files |
libXrdHttpMdsip.so (XrdHttpExtHandler) |
existing thin-client code holding an MDSplus.Connection |
full protocol compatibility; mdsip relayed over the origin's HTTPS port |
libMdsIpFDP.so (MDSplus transport) |
those same thin clients, on the client side | a one-string change: MDSplus.Connection('fdp://…') |
None of them evaluates anything: all delegate to MDSplus's own mdsip server,
so the first is a path parser and a pipe and the other two are pipes.
The two server-side plugins load into a stock Pelican origin — no fork — by pointing
Xrootd.ConfigFile at a fragment containing:
ofs.osslib ++ /path/to/libXrdOssMdsplus.so prefix=/tdi server=localhost:8000
http.exthandler mdsip /path/to/libXrdHttpMdsip.so prefix=/mdsip,host=localhost,port=8000,auth=xrootd
# Production: the sandbox is rootless, so the relay reaches it over a Unix socket
# rather than a TCP hop through pasta (which stalls replies above ~4 MB):
http.exthandler mdsip /path/to/libXrdHttpMdsip.so prefix=/mdsip,socket=/run/fdp-mdsip/mdsip.sock,auth=xrootd
Note that neither name in the config carries the -5 suffix the files on
disk do: XRootD appends the plugin version itself, and referencing the suffixed
name makes it search for -5-5.so and only succeed via a fallback. The client
transport is the exact reverse — libMdsIpFDP.so must be named precisely that,
because MDSplus resolves it from the URL scheme.
Two of the three ship as conda packages on ga-fdp. For a client, that is the
whole install:
conda install -c ga-fdp -c conda-forge mdsip-fdpfrom MDSplus import Connection
conn = Connection('fdp://fdp-d3d-origin.nationalresearchplatform.org:8443/mdsip')
conn.openTree('efit01', 198873)
conn.get(r'\ipmhd')No LD_LIBRARY_PATH, no XRootD, no Pelican — MDSplus finds the transport
because the package installs it alongside libMdsShr.so, whose RPATH is
$ORIGIN/.. A bearer token comes from $BEARER_TOKEN or ~/.fdp/token.
The relay is xrdhttp-mdsip (origin administrators only). The virtual-file
plugin is not packaged — it has no deployment yet. See
docs/packaging.md.
The rest of this document covers the virtual-file plugin first, since it is where the design work is; the relay has its own section further down: The other half.
The central idea: the object path is the query. A client asks for a path
that encodes (tree, shot, TDI expression), and gets back the evaluated result
as if it were a file. Nothing about the request lives outside the path — not in
a query string, not in a header — because that is the only channel that survives
the federation intact (see Why the path carries everything).
┌────────────────────────────────────────────────────────────────────────┐
│ CLIENT │
│ asks for an object path, deserializes the bytes it gets back │
└───────────────────────────────┬────────────────────────────────────────┘
│ GET /fdp-d3d/tdi/efit01/00/00/19/00/190000/BADWxBAA...
▼
┌────────────────────────────────────────────────────────────────────────┐
│ PELICAN DIRECTOR (stock Pelican, unmodified) │
│ path.Clean() on the decoded path, then 307 to an origin │
│ ⚠ this is why the encoding may contain no '/' and no '%' │
└───────────────────────────────┬────────────────────────────────────────┘
▼
┌────────────────────────────────────────────────────────────────────────┐
│ OSDF CACHE (XrdPfc) ── future; see caveat below ── │
│ keys on URL path only, so the token in ?authz= never fragments it │
└───────────────────────────────┬────────────────────────────────────────┘
│ (miss)
▼
┌────────────────────────────────────────────────────────────────────────┐
│ ORIGIN — XRootD managed by Pelican (stock image + MDSplus runtime)│
│ │
│ ofs.osslib default ← native POSIX storage │
│ ofs.osslib ++ libXrdOssStats.so ← Pelican's own layer │
│ ofs.osslib ++ libXrdOssMdsplus.so ← THIS PLUGIN, injected via │
│ Xrootd.ConfigFile │
└───────────────────────────────┬────────────────────────────────────────┘
▼
┌────────────────────────────────────────────────────────────────────────┐
│ libXrdOssMdsplus.so (src/) │
│ │
│ IsTdiPath(lfn, prefix)? │
│ no ──────────────────────────────► forward to the wrapped Oss │
│ (the raw tree-file archive │
│ keeps working, same origin) │
│ yes ─┐ │
│ ▼ │
│ ParseTdiPath TdiPath.cc split path, validate shot bucket │
│ └─ join chunks ≤249 bytes each (NAME_MAX) │
│ └─ Base64UrlDecode Base64Url.cc → opaque payload, NOT parsed │
│ ▼ │
│ ResultCache ResultCache.cc hit? serve it │
│ │ miss │
│ ▼ │
│ MdsIpClient MdsIpClient.cc ConnectToMds / MdsOpen, │
│ bounded by a per-request timeout│
└───────────────────────────────┬────────────────────────────────────────┘
│ SendDsc x2 + GetAnswerInfoTO(timeout) -> GetManyExecute($)
▼
┌────────────────────────────────────────────────────────────────────────┐
│ mdsip -m (MDSplus's own server, separate process) │
│ │
│ GetManyExecute($) upstream C++: evaluates every item and │
│ returns an ALREADY-serialized dictionary │
│ (mdsobjects/cpp/mdsdata.c:921) │
│ process per connection -m; GetManyExecute uses a static XD and is │
│ not thread-safe │
└───────────────────────────────┬────────────────────────────────────────┘
│ serialized {name: {value|error}}
▼
back up the stack, cached, served as file bytes
The plugin never interprets the payload. It decodes the path, hands the bytes to
GetManyExecute($), and writes the answer out verbatim — MDSplus is the only
thing that understands the request format, so there is one definition of it
rather than ours plus theirs. Semantics therefore match atlas.gat.com by
construction rather than by testing.
On isolation. Evaluation happens in the mdsip process, so a crash or hang in
tree opening or TDI takes down a disposable worker rather than the origin. But
the separation is not total: libMdsIpShr.so directly DT_NEEDEDs libTdiShr,
libTreeShr and libMdsShr, so those libraries are loaded into the XRootD
process even though we never call them. See tests/fed/FINDINGS.md.
XRootD asks for a size before it asks for bytes, and the Pelican director parses
Content-Length with strconv.Atoi — a failure there becomes a 404 rather than
a redirect. So Stat cannot be lazy: it evaluates the expression, caches the
payload, and reports the real length. The Open/Read that follows serves from
that cache.
A cache miss between Stat and Read simply re-evaluates, so correctness never
depends on the cache. That is what lets it be a fixed-size LRU with no
invalidation logic at all.
\ipmhd from efit01, shot 190000, labelled ip:
request apd.List([apd.Dictionary({'name':'ip','exp':'\\ipmhd','args':()})])
.serialize() → 135 bytes
i.e. exactly what MDSplus's GetMany builds and sends. We produce it
with tests/mkpath.py; a real client already has GetMany.serialize().
base64url BADWxBAAAAAAAAABBAAAABQ...ANjEEAAAAAAAAAEYAAAAKAAAA...
path /tdi/efit01/00/00/19/00/190000/vfe49badc9b703486/BADWxBAA...
│ └────────┬───────┘ └───────┬───────┘ └────┬────┘
│ │ │ └─ payload,
│ │ │ chunked at 249 B
│ │ └─ version (see below)
│ └─ shot/100 in digit pairs
└─ tree ('-' means: evaluate with no tree open)
response a serialized dictionary {"ip": {"value": <array>}}; a failed
expression yields {"ip": {"error": "..."}} and the response is
still 200 — GetMany semantics, because it IS GetMany
The digit-pair bucketing mirrors the existing MDSplus archive layout and keeps ~100 shots per directory. It is not decoration: a flat namespace makes XrdPfc's startup scan take hours while holding a global lock on open (xrootd#2804).
XrdPfc never revalidates. Once an object is cached it is served until purged for capacity, and Pelican has no per-namespace "do not cache" flag. So without a version in the name, a re-analysed shot would be served stale forever.
The token is derived from a stat of the tree's .datafile — inode, size and
mtime, hashed — so regenerating the tree changes the object name. Old cache
entries simply age out, and copies of superseded versions stay correct: they
hold exactly what that version was.
The plugin recomputes the current token on every request and refuses anything
else, so a stale path cannot be served afresh. It locates the file through
treepath=, a semicolon-delimited list of templates mirroring MDSplus's own
<tree>_path convention (%T tree, %S shot, %B digit-pair bucket) — a list
because a tree may live under codes/, shots/ or another branch.
A remote client cannot derive the token itself — it has no access to the tree files — so it asks:
GET /fdp-d3d/tdi-version/<tree>/<d1>/<d2>/<d3>/<d4>/<shot> -> vfe49badc9b703486
That answer is deliberately not cached, by us or by anyone: its whole
purpose is to report a value that changes. Fetch it with ?directread. The
natural place to call it is openTree(), once per tree-open, after which every
expression in that session reuses the token.
Without treepath= the plugin refuses every request that names a tree.
Failing closed is deliberate: serving an unverifiable object is exactly the bug
versioning exists to prevent. Requests naming no tree carry version -.
Client-supplied headers do not reach the origin through a cache. On a miss
XrdPfc reads upstream via XrdOucCacheIO with no HTTP request context, and the
XrdCl layer builds its request from URL parameters. That is precisely why
Pelican has http.header2cgi Authorization authz — moving a header into the URL
is the only way to get it across. So there is no side channel: no headers, no
request body, no out-of-band registry. The path is the entire request.
A corollary is that the encoding alphabet matters for correctness, not tidiness.
A / inside a path segment is eaten by the director's path.Clean, and %2F
cannot protect it because the decoded path is what gets normalised. base64url
(A-Za-z0-9-_) survives; standard base64 does not. Measured, both ways, in
tests/fed/FINDINGS.md.
mdsip needs MDS_PATH to include tdi/remote. GetManyExecute is a TDI
function (tdi/remote/GetManyExecute.fun), not a builtin. Without that path the
server answers every request with %TDI-E-UNKNOWN_VAR, which reads like a
client bug and is not one.
Two things that used to be required here are no longer: wrapping expressions in
data(...), and resetting TDI state between requests. Both were compensating
for a bespoke evaluator that no longer exists — mdsip returns unwrapped nodes
correctly and gives each connection its own process. See docs/mdsip-spike.md.
Everything above serves path-based clients — callers who can be told to batch,
and therefore the ones caching pays for. But most existing DIII-D code is not
that. It holds an MDSplus.Connection and calls conn.get() one signal at a
time (55 such call sites in the local repos, against 0 for getMany).
For those callers there is a second plugin, libXrdHttpMdsip.so: an
XrdHttpExtHandler that relays the mdsip protocol over the origin's
existing HTTPS port, TLS and SciTokens.
MDSplus.Connection('fdp://…') stock, unmodified
│ mdsip over the IoRoutines vtable
▼
libMdsIpFDP.so accumulates one COMPLETE call, POSTs it
│ HTTPS
▼
XRootD :443 ── XrdHttpMdsip holds one mdsip socket per session
│ mdsip over TCP or a Unix socket
▼
mdsip server (sandboxed) does the actual work
Both halves are built. Existing thin-client code reaches FDP by changing one
string — MDSplus.Connection('fdp://d3d-origin.gat.com:8443/mdsip') — because
MDSplus loads the transport itself from the URL scheme. See
docs/client-transport.md.
Three POSTs make up the whole protocol:
| Request | Body up | Body down |
|---|---|---|
PUT <prefix>/connect |
— | an opaque session token (authorized here) |
PUT <prefix>/msg + X-Fdp-Session |
one complete mdsip call | exactly one mdsip answer |
PUT <prefix>/close + X-Fdp-Session |
— | — |
POST works too, but only against a directly-addressed origin: the Pelican
director refuses to route POST and routes PUT, so PUT is what the client uses.
See tests/fed/FINDINGS.md.
auth=xrootd is not optional in spirit. An ext handler runs before
XRootD's authorization, so the relay checks for itself by asking the origin's
own policy — otherwise anyone reaching the port could open a session, and a
session is arbitrary code execution. The handler refuses to load without an
explicit auth= setting. See docs/security.md.
Loaded alongside the osslib, again with the unsuffixed name:
http.exthandler mdsip /path/to/libXrdHttpMdsip.so prefix=/mdsip,host=localhost,port=8000,auth=xrootd
# Production: the sandbox is rootless, so the relay reaches it over a Unix socket
# rather than a TCP hop through pasta (which stalls replies above ~4 MB):
http.exthandler mdsip /path/to/libXrdHttpMdsip.so prefix=/mdsip,socket=/run/fdp-mdsip/mdsip.sock,auth=xrootd
Parms are a single XrdOucStream::GetWord() token, so they cannot contain
spaces — comma is the separator. Add +notls before the library path only
when running without TLS, as the test harness does.
Full protocol compatibility, for free: get, getMany, put, setDefault,
anything — because a real mdsip server handles it, and the semantics match
atlas.gat.com because they are atlas.gat.com's, relayed. There is no
reply fabrication anywhere, so there is no class of bug where a mistake yields
plausible wrong data instead of an error.
The cost is caching, and it cannot be recovered: a cache keys on a URL, and a
tunnel has none. Moving the intelligence to the origin does not help either,
because the request that crossed the network was still a POST. The two plugins
are complementary rather than competing — see docs/relay-spike.md.
mdsip is stateful. openTree sets the tree context on the connection, so
a relay that opened a fresh mdsip connection per HTTP request loses it and every
subsequent get() fails. A stateless variant was tried and dies immediately
with %MDSPLUS-E-ERROR.
So the handler holds one mdsip socket per session, which has consequences worth stating plainly:
- sessions are sticky to one origin and cannot be load-balanced
- an abandoned session holds an mdsip connection until the idle reaper takes it
(
idle=, default 300s), andmaxsessions=bounds the damage from a flood - one call at a time per session; a concurrent second call gets
409rather than being interleaved into the byte stream, which would corrupt it - a broken stream retires the session instead of resynchronising — a half-written call cannot be recovered from, and the alternative is handing the next caller misaligned bytes
The relay links no MDSplus library at all (readelf -d shows only
libXrdUtils, libXrdHttpUtils and libc), speaking mdsip over a plain socket.
An origin container serving only the tunnel therefore needs no MDSplus runtime.
Both plugins end at the same place: a client-supplied TDI expression evaluated
by an MDSplus server. That is arbitrary native code execution — measured,
not assumed. MdsShr->system("touch /tmp/x") creates the file, because
image->routine() is a generic FFI and dlsym reaches libc through the
library's dependency chain. So removing spawn() would accomplish nothing, and
there is no subset of TDI that is both useful and safe.
The security boundary therefore cannot be inside the mdsip process. The container is the boundary:
pixi run sandbox-build # -> localhost/fdp-mdsip
pixi run seccomp-gen # -> deploy/mdsip-seccomp.json (allowlist, ~150 calls)
pixi run sandbox-verify # starts it, attacks it, tears it down
MDSIP_SANDBOX=1 pixi run relay-e2e # the relay against the sandboxscripts/mdsip-sandbox.sh runs mdsip in its own container with no route off
the host, no resolver, a read-only root, read-only trees, no capabilities, and
resource limits. It is a separate container from the origin — the single
most important control, because a stolen issuer.jwk mints arbitrary
federation tokens.
Run it under a dedicated service account, not the account that runs the
origin: rootless podman maps container uid 0 to the invoking user, so sharing
the account makes container-root the owner of the issuer keys.
deploy/fdp-mdsip.container is the quadlet unit
for that; docs/security.md has the account setup and one consequence worth
knowing — the two containers can no longer share a podman network, so in TCP mode the
relay must reach mdsip via host.containers.internal; production avoids the hop
entirely with socket= (above).
tests/security/verify_sandbox.sh asserts each control by attacking it through
ordinary TDI, the same channel a client has. It first confirms the client
does have code execution, so a passing run cannot pass for the wrong reason.
Run it on the target host, not just a dev box. The network controls are
properties of podman's network backend rather than of the flags, and podman 5.0
removed CNI — so a result from a podman 4.x/CNI machine says nothing about a
netavark one. The script prints the stack it ran on for that reason. See
the caveat in docs/security.md.
docs/d3d-origin-state.md is the measured state of
the production origin, the deployment decisions and their reasons, and what is
left to do — start there for deployment.
Read docs/security.md before deploying either plugin. It
has the threat model, the measured capabilities, and three findings that
reading documentation would not have produced — including that podman mounts
the host's credentials into the container by default, and that it silently
ignores resource limits on cgroups v1 rootless.
| Piece | Status |
|---|---|
| microVM isolation | not built, and weaker justification since the sandbox moved to its own service account. /dev/kvm is available; see the trade-offs in docs/security.md |
Deployment also needs ops access, which is the actual gate on going live.
| Path | What |
|---|---|
src/ |
Both plugins. The osslib's logic lives in small units (Base64Url, TdiPath, MdsIpClient, ResultCache) with OssMdsplus.cc thin wiring over them; the relay is HttpRelay.cc over MdsipSession.cc. The two share no code. |
tests/ |
doctest unit tests; mkpath.py is the Python-side reference implementation of the path grammar |
tests/fed/ |
Local Pelican federation in podman — see tests/fed/FINDINGS.md |
tests/integration/ |
Standalone XRootD end-to-end, for both plugins |
tests/security/ |
What a client can do (probe_capability.py) and what the sandbox must stop it doing (verify_sandbox.py) |
scripts/ |
mdsip-sandbox.sh — the isolation flags, which are where the containment actually lives |
pixi run build # -> libXrdOssMdsplus-5.so, libXrdHttpMdsip-5.so, libMdsIpFDP.so
pixi run test # doctest suites via ctestEnd-to-end (each starts mdsip and a standalone XRootD for you):
pixi run e2e # virtual files, ~1s, no container
pixi run relay-e2e # the tunnel, with a stock MDSplus.Connection
pixi run bash tests/integration/test_real_tree.sh # needs a staged treerelay-e2e is the one that proves the tunnel: it runs a real
MDSplus.Connection through the handler and compares every result against the
same call made straight to mdsip. Anything short of exact agreement fails.
XRootD is pinned <6 to match the deployed v5 plugin chain. The Pelican origin
image ships XRootD v5.9.2 and the pixi environment resolves to the same version,
so the plugin is built against the ABI it is loaded into.
bash tests/fed/fedbox.sh start # plain federation
bash tests/fed/fedbox.sh start /tmp/extra.cfg # with an osslib fragment
bash tests/fed/fedbox.sh start /tmp/extra.cfg /path/to/plugin.so
bash tests/fed/fedbox.sh stoptests/fed/run_fed_test.sh needs the MDSplus-capable origin image, because
ConnectToMds dlopens libMdsIpTCP.so by name at runtime:
podman build -f Containerfile.runtime -t fdp-origin-mdsplus .
pixi run bash tests/fed/run_fed_test.shThis is the only layer that exercises the director. See
docs/deployment-notes.md if podman complains about
/run/user/$UID.
repos/docs/superpowers/specs/2026-08-06-mdsplus-virtual-file-service-design.md
and the implementation plan alongside it in plans/.