Sidecar: bulk-CAR import endpoint to eliminate per-block HTTP round-trips

[vrogojin]

Context

In sphere-sdk PR #303 we parallelized pinCarBlocksToIpfs from a serial for-loop to a bounded worker pool (default concurrency 10). For a typical 250-block wallet migration this drops wall-clock from ~25 s to ~2.5 s.

The remaining cost is ~250 client → sidecar HTTPS round-trips. Each block is one POST to `/sidecar/submit?cid=...` + one POST to Kubo's `/api/v0/dag/put`. At ~80-150 ms per round-trip from a browser on testnet, even fully parallel this still pays:

```
ceil(N / concurrency) × RTT ≈ 25 batches × ~100ms = ~2.5s
```

That's good enough for the user-visible migration today, but the intrinsic network overhead is one round-trip total if the sidecar accepts the whole CAR in one POST.

Proposal

Add `/sidecar/import-car` (POST) that:

1. Accepts a CAR file in the request body (Content-Type: `application/vnd.ipld.car`)
2. Server-side splits into per-block bytes using the same `CarReader` logic that lives client-side today
3. Pins each block via Kubo's `dag/put` over the LOCAL network (microseconds per pin, not 100 ms)
4. Populates the sidecar's instant-pin cache for cross-device reads
5. Returns a single JSON envelope with per-CID results:

```json
{
"ok": true,
"blocks": [
{ "cid": "bafy...A", "pinned": true },
{ "cid": "bafy...B", "pinned": true },
{ "cid": "bafy...C", "pinned": false, "error": "sha256 mismatch" }
],
"rootCid": "bafy...root"
}
```

Client-side rollout

Once available, sphere-sdk's `pinCarBlocksToIpfs` should:

1. Capability-probe (HEAD `/sidecar/import-car` or a `/sidecar/version` endpoint advertising supported features)
2. Use the bulk endpoint when supported
3. Fall back to the per-block parallel pool when the sidecar predates this endpoint (multi-deploy compatibility window)

Expected impact

Wallet size	Current (parallel-10)	With bulk endpoint
24 tokens (~250 blocks)	~2.5 s	~150 ms (1 RTT)
100 tokens (~1000 blocks)	~10 s	~250 ms (1 RTT + sidecar pin loop)
1000 tokens (~10000 blocks)	~100 s	~1 s (1 RTT + sidecar pin loop)

Caveats

• Request body size cap: 10k-block migrations could approach 50-100 MB. Need an explicit per-request size limit + a streamed parser path on the sidecar.
• Partial-failure semantics: if 1 of 1000 blocks fails the local hash check, return per-CID results so the client can retry the failures individually rather than re-uploading the whole CAR.
• Auth / rate-limiting: bulk endpoint should respect the same throttling as `/sidecar/submit` to avoid bypassing it.

Related

• sphere-sdk PR #303 — parallel pin worker pool (the interim 10x win)
• Existing endpoints: `/sidecar/submit?cid=...` (per-block POST), `/sidecar/blob?cid=...` (per-block GET)
• Kubo's native `dag/import` provides equivalent functionality but is disabled on the public-facing gateway for security; this proposal puts an equivalent on our trusted sidecar API surface.

Origin

User-requested follow-up after the migration-flush root-cause diagnosis on 2026-05-27. Quote: "is it possible to perform bulk calls to the IPFS node with same HTTP GET/POST call? In this manner, we would remove all the http and ipfs logic round-trip network overheads."