Feature Request: Native multi-URL segmented download with sequential merge (DASH/ISM support)

[rador1999]

Feature Request: Native segmented/multi-URL download with binary merge (DASH/ISM support)

Summary

Add support for downloading a list of URLs in parallel and concatenating them in order
into a single output file — the standard pattern for DASH and ISM (Smooth Streaming)
manifest-based video streams.

---

Background / Use case

Modern streaming platforms (Amazon, Apple TV+, Netflix, etc.) deliver video via adaptive
manifests (DASH .mpd / ISM Smooth Streaming) where content is split into hundreds or
thousands of small segments that must be:

1. Downloaded in parallel
2. Concatenated in order into a single output file

Any application downloading adaptive streaming content needs to handle this pattern. Currently, the only way to do this with Surge is to submit each segment as an
individual POST /download call — which completely defeats Surge's performance advantages.

---

The problem: per-segment API overhead kills throughput

When submitting 1000–2000 segments individually:

• Each POST /download is a separate HTTP round-trip to the local API
• Surge's internal scheduler creates a separate download entry per segment
• Connection pools are not shared across segments
• The API itself becomes the bottleneck before the network ever does

---

Benchmark data (same content, same machine, same network)

Content: Amazon VOD stream (ISM/Smooth Streaming)
Segments: 1874 segments × ~65–103 KB each (~2.8 GB total)
Network: capable of 600+ MB/s (verified on single-file DASH with Surge)

Downloader	Speed	Notes
aria2c (-j64 -x64 -s64)	~215–260 MB/s	segmented input file + binary merge
Surge (individual POST ×1874)	~2.7 MB/s	one /download call per segment
Surge (single large DASH file)	600+ MB/s	same CDN, same session, same machine

Surge is ~100x slower than aria2c on segmented content, and ~220x slower than its own
single-file performance on identical infrastructure and the same CDN.

This is entirely due to per-segment API and connection overhead — not network or CDN
limitations. The moment you switch the same content to a single-file download, Surge
hits 600 MB/s with no changes.

---

Proposed solution

Extend POST /download to accept a urls array alongside the existing url field:

{
  "urls": [
    "https://cdn.example.com/seg00000001.mp4",
    "https://cdn.example.com/seg00000002.mp4",
    "https://cdn.example.com/seg00000003.mp4"
  ],
  "path": "/output/dir",
  "filename": "track.mp4",
  "headers": { "Authorization": "Bearer ..." },
  "merge_sequential": true
}

Surge would:

• Download all URLs in parallel using a single shared connection pool
• Track per-segment completion internally
• Concatenate results in submission order once all segments complete
• Expose aggregate progress via the existing GET /download?id= endpoint:
  • downloaded: total bytes across all segments
  • total_size: sum of all segment sizes (once known)
  • speed: aggregate transfer rate
  • status: completed only when merge is done

---

Why not just submit individually?

I tried. Here is what happens in practice with 1874 segments:

• Submission alone takes 15–20 seconds before any real downloading begins
• The Surge API is saturated handling add_download requests instead of downloading
• Even with a thread pool of 256 workers submitting concurrently, throughput peaks
  at ~2.7 MB/s — aria2c does 215–260 MB/s on the same content with zero tuning
• The gap widens with segment count: more segments = more API overhead = slower

---

Alternatives considered

Approach	Result
Submit segments individually via POST /download	2.7 MB/s — 100x slower than aria2c
Throttle submission with a semaphore	No meaningful improvement
Rolling window / queue-ahead submission	Marginal improvement, still far below aria2c
Use aria2c for segmented, Surge for single-file	Works but requires maintaining two downloaders

---

Impact

This is the single most common download pattern in streaming video tools. Every
DASH and ISM stream uses it. Right now Surge is effectively unusable for this
use case despite being dramatically faster than aria2c for everything else.

Native segmented support would make Surge a complete solution for adaptive streaming
downloads, eliminating the need to fall back to other downloaders entirely.

---

References

• DASH spec: https://dashif.org/docs/DASH-IF-IOP-v4.3.pdf
• ISM/Smooth Streaming: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-sstr
• aria2c input file format (existing art): https://aria2.github.io/manual/en/html/aria2c.html#input-file

[SuperCoolPencil]

hey @rador1999

would you mind dming me on reddit/discord. it's supercoolpencil on both.

Would like to understand more about the feature request...

[rador1999]

hey @rador1999

would you mind dming me on reddit/discord. it's supercoolpencil on both.

Would like to understand more about the feature request...

Sent friend request on discord. Please check

_life_is_bitch

[rador1999]

any updates on this?