Merge pull request #6 from vasylenko/fix/http1-fallback-on-cloudflare-block

Default to HTTP/1.1 transport + add --raw passthrough

Author: vasylenko

CHANGELOG.md

@@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-07-06
+
+### Added
+- `--raw` (CLI) / `raw` (MCP): return the unprocessed body verbatim — no Readability, no content-type gate; `MARKFETCH_MAX_BYTES` still applies.
+
+### Fixed
+- Cloudflare-class CDNs 403 valid Chrome headers over HTTP/2 but pass them over HTTP/1.1 — markfetch now defaults to HTTP/1.1.
+- Empty-heading pruning truncated headings containing an inline `#` (`## Step 1 # download` became `# download`).
+
 ## [0.6.0] - 2026-05-14
 
 ### Added

README.md

@@ -7,7 +7,7 @@
 [![node](https://img.shields.io/node/v/markfetch.svg?color=10b981)](https://nodejs.org/)
 [![license](https://img.shields.io/npm/l/markfetch.svg?color=10b981)](https://github.com/vasylenko/markfetch/blob/main/LICENSE)
 
-The built-in fetch tools that ship with AI coding agents return raw HTML, broken markdown, or `403` from Cloudflare more often than you'd like. `markfetch` sends **HTTP/2 with a coherent Chrome header set** so bot-detection systems see a real browser, then runs the response through the **same Reader View pipeline your browser uses** (Mozilla's Readability → turndown). The output is markdown indistinguishable from a human running "Save as Markdown" — on sites that would block a naive curl.
+The built-in fetch tools that ship with AI coding agents return raw HTML, broken markdown, or `403` from Cloudflare more often than you'd like. `markfetch` sends **a coherent Chrome header set** so bot-detection systems see a real browser, then runs the response through the **same Reader View pipeline your browser uses** (Mozilla's Readability → turndown). The output is markdown indistinguishable from a human running "Save as Markdown" — on sites that would block a naive curl.
 
 One command, two surfaces:
 
@@ -61,13 +61,13 @@ gemini mcp add -s user markfetch npx -y markfetch
 | CloudFlare `/markdown` | ✓ | ✓ | – | paid |
 | **`markfetch`** | **✓** | **✓** | **✓ (8 codes)** | **✓** |
 
-- **Real-browser HTTP/2 + Chrome fingerprint.** ALPN-negotiated h2, `User-Agent`, `Sec-CH-UA-*`, `Sec-Fetch-*`, `Accept-*`. A Chrome UA with no client hints is a *stronger* automation signal than curl — `markfetch` sends the full coherent set, derived from the UA at startup so an override stays internally consistent.
+- **Real-browser request fingerprint.** `User-Agent`, `Sec-CH-UA-*`, `Sec-Fetch-*`, `Accept-*` — a coherent Chrome header set. A Chrome UA with no client hints is a *stronger* automation signal than curl, so `markfetch` sends the full set, derived from the UA at startup so an override stays internally consistent. HTTP/1.1 over TLS: some CDNs fingerprint undici's HTTP/2 connection and 403 it.
 
 - **Reader-View-quality extraction.** [linkedom](https://github.com/WebReflection/linkedom) → [@mozilla/readability](https://github.com/mozilla/readability) → [turndown](https://github.com/mixmark-io/turndown) with GFM tables, strikethrough, and task lists. Code fences preserve `language-X` hints. Sphinx-style bare `<pre>` blocks render as code, not escaped prose. Intraword underscores stay un-escaped — no more `list\_tools`.
 
-- **One tool, one shape (MCP).** `fetch_markdown(url, savePath?)` returns markdown in `content[0].text`. No `structuredContent`, no frontmatter, no metadata fields. Several major MCP clients (Claude Code CLI, VS Code/Copilot) forward only `structuredContent` to the model and drop `content[]` when both are present — `markfetch` deliberately stays on the channel your LLM can actually read.
+- **One tool, one shape (MCP).** `fetch_markdown(url, savePath?, raw?)` returns markdown in `content[0].text`. No `structuredContent`, no frontmatter, no metadata fields. Several major MCP clients (Claude Code CLI, VS Code/Copilot) forward only `structuredContent` to the model and drop `content[]` when both are present — `markfetch` deliberately stays on the channel your LLM can actually read.
 
-- **`savePath` / `-o` escape valve.** Pass an absolute path (MCP `savePath`) or `-o <path>` (CLI) and the markdown lands on disk instead of the response channel. Use it when your client's inline tool-result cap would truncate large responses, or to redirect output from a shell pipeline. The file is only ever the markdown of the URL — fetch errors return a `[code]` string and never touch the disk.
+- **`savePath` / `-o` escape valve.** Pass an absolute path (MCP `savePath`) or `-o <path>` (CLI) and the output lands on disk instead of the response channel. Use it when your client's inline tool-result cap would truncate large responses, or to redirect output from a shell pipeline. The file is only ever the fetched output (extracted markdown, or the raw body with `--raw`) — fetch errors return a `[code]` string and never touch the disk.
 
 - **Whole document or honest failure.** No pagination, no truncation. If the document doesn't fit in `MARKFETCH_MAX_BYTES`, you get `too_large` — never a half-truth.
 
@@ -88,6 +88,9 @@ npx -y markfetch https://example.com/article -o article.md
 
 # Pipe into another tool
 npx -y markfetch https://example.com/article | pandoc -o article.pdf
+
+# Fetch JSON / APIs / page source verbatim
+npx -y markfetch --raw https://api.github.com/repos/vasylenko/markfetch
 ```
 
 For repeat use, install once:
@@ -102,7 +105,8 @@ Flags:
 
 | Flag | Purpose |
 |---|---|
-| `-o, --output <path>` | Save markdown to a file (absolute or relative path). Default is stdout. |
+| `-o, --output <path>` | Save the output to a file (absolute or relative path). Default is stdout. |
+| `--raw` | Return the unprocessed response body as UTF-8 text — skips Readability and the content-type gate. For JSON, XML, plain text, or page source (binary is not byte-preserved). |
 | `-V, --version` | Print version and exit. |
 | `-h, --help` | Print usage and exit. |
 
@@ -115,8 +119,8 @@ Errors carry one of eight deterministic codes:
 | `network_error` | DNS / TCP / TLS failure, or an unexpected internal error from the fetcher. |
 | `http_error` | Upstream returned a non-2xx status. |
 | `timeout` | Per-request budget `MARKFETCH_TIMEOUT_MS` exceeded. |
-| `unsupported_content_type` | Response was not `text/html` or `application/xhtml+xml`. |
-| `extraction_failed` | Readability returned no article content (typical for pure client-rendered SPAs). |
+| `unsupported_content_type` | Response was not `text/html` or `application/xhtml+xml` (not raised with `--raw` / `raw`). |
+| `extraction_failed` | Readability returned no article content (typical for pure client-rendered SPAs). Not raised with `--raw` / `raw`. |
 | `too_large` | Response body or extracted markdown exceeded `MARKFETCH_MAX_BYTES`. |
 | `save_failed` | `savePath` was given but `writeFile` failed (parent directory missing, permission denied, etc.). |
 | `save_forbidden` | `savePath` resolves outside the allowed write roots — see [Write sandbox](#write-sandbox). MCP-only; the CLI has no sandbox. |

docs/SPEC.md

@@ -6,13 +6,15 @@ Text processing pipeline:
 
 ```
 URL
-  → undici.fetch       HTTP/2 via ALPN; full Chrome header set; Sec-CH-UA-* derived from UA
+  → undici.fetch       HTTP/1.1; full Chrome header set; Sec-CH-UA-* derived from UA
   → linkedom.parseHTML
   → @mozilla/readability
   → turndown + GFM     HTML → markdown
   → caller             markdown body, or "Saved N bytes to /path" confirmation
 ```
 
+`raw` mode (`--raw` / MCP `raw`) returns the body straight from `undici.fetch`, skipping the parse → extract → convert steps.
+
 Errors throw `MarkfetchError` uniformly from core; adapters catch once. Codes: `network_error`, `http_error`, `timeout`, `unsupported_content_type`, `extraction_failed`, `too_large`, `save_failed`; plus `save_forbidden`, emitted by the MCP adapter only (before `fetchMarkdown` runs — see "Asymmetric write sandbox" under Core Decisions). CLI emits `[code] message` to stderr and exits 1; MCP emits `{ isError: true, content: [{ text: "[code] message" }] }`.
 
 ## Core Decisions
@@ -23,7 +25,9 @@ Errors throw `MarkfetchError` uniformly from core; adapters catch once. Codes: `
 
 - **Core throws, adapters translate.** Seven of the eight error codes surface from `core.ts` — five are thrown explicitly as `MarkfetchError`; `network_error`, `timeout`, and (sometimes) `http_error` are translated by `classifyError` from underlying-API errors (undici TypeErrors, AbortSignal timeouts). The eighth code, `save_forbidden`, is the exception — it's emitted by the MCP adapter before `fetchMarkdown` is invoked (see "Asymmetric write sandbox" below). New core codes need an `ErrorCode` union member + a throw site; adapters don't change.
 
-- **HTTP/2 + coherent Chrome fingerprint.** Wire protocol, headers, and UA must agree — a Chrome UA over HTTP/1.1 or without `Sec-CH-UA-*` is *more* suspicious than curl. `Sec-CH-UA-*` is derived from `MARKFETCH_USER_AGENT` at startup so override-coherence is mechanical.
+- **HTTP/1.1 + coherent Chrome header fingerprint.** Headers and UA must agree — a Chrome UA without `Sec-CH-UA-*` is a stronger bot signal than curl, so `Sec-CH-UA-*` is derived from `MARKFETCH_USER_AGENT` at startup and override-coherence is mechanical. HTTP/1.1 is deliberate: undici's h2 path hands `node:http2` a pre-connected socket whose first-flight frames some CDNs (Cloudflare, seen on `openai.com`) score as a bot and 403; the identical request over h1.1 passes. h2 buys nothing for single-shot GETs.
+
+- **`raw` passthrough.** Returns the fetched body verbatim — no Readability, no content-type gate; the fetch layer and `MARKFETCH_MAX_BYTES` cap stay. Same `fetchMarkdown` code path via a `raw` flag — no second entry point. Body is UTF-8 text; binary is not byte-preserved.
 
 - **Single-channel MCP response.** `content[0].text` only. Several major MCP clients (Claude Code CLI, VS Code/Copilot) forward only `structuredContent` to the model and drop `content[]` when both are present — a single-channel response keeps the markdown reachable from those clients.
 
@@ -40,6 +44,7 @@ Errors throw `MarkfetchError` uniformly from core; adapters catch once. Codes: `
 - **Authentication.** `MARKFETCH_AUTH_HEADER` env var (simple), or Chrome-cookie import for sites where the user is already logged in (frictionless, platform-specific, security-sensitive). Trigger: first useful internal / paywalled doc.
 - **JS rendering fallback for SPAs.** Playwright / headless Chrome as a companion package (`markfetch-heavy`) so the lean package stays lean. Trigger: enough useful sites returning `extraction_failed`.
 - **CloudFlare `/markdown` fallback.** Gated by `CF_AUTH_TOKEN`; fall back when Readability fails. Trigger: extraction failure rate stays high after Readability tuning.
+- **Browser-grade TLS + HTTP/2 impersonation.** Stricter CDN tiers fingerprint Node's TLS (JA3/JA4) and 403 every protocol. Apify's `impit` (BoringSSL-based) ships a Chrome-matching TLS + h2 fingerprint, near-drop-in for `undici.fetch`. Trigger: target sites 403 the current h1.1 + Chrome-header approach.
 - **Cookie reuse across redirects within a single fetch.** Currently none. Trigger: a target serves content only after a session-cookie redirect.
 - **Proxy support** (`MARKFETCH_PROXY_URL`) and **`Accept-Language` control** (`MARKFETCH_ACCEPT_LANGUAGE`). Trigger: corporate proxy / locale-specific content.
 - **Single-binary distribution.** Bun's `build --compile`, Node SEA, or similar. Trigger: `npx` first-run latency feedback, or an offline / airgapped need.

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "markfetch",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Fetch a URL, return clean markdown. MCP server and CLI for AI agents.",
   "license": "MIT",
   "author": {

package.json

@@ -27,9 +27,13 @@ program
   .argument("<url>", "absolute http(s) URL to fetch")
   .option(
     "-o, --output <path>",
-    "save markdown to file (absolute or relative path); default is stdout",
+    "save output to a file (absolute or relative path); default is stdout",
   )
-  .action(async (url: string, options: { output?: string }) => {
+  .option(
+    "--raw",
+    "return the unprocessed response body (skip Readability and the content-type gate)",
+  )
+  .action(async (url: string, options: { output?: string; raw?: boolean }) => {
     // CLI resolves relative output paths against cwd before calling core;
     // core requires an absolute path so the contract is unambiguous regardless
     // of which adapter invokes it. Tilde expansion is intentionally NOT done
@@ -43,9 +47,10 @@ program
       const { markdown, bytes, savedTo } = await fetchMarkdown({
         url,
         savePath,
+        raw: options.raw,
       });
       if (savedTo === undefined) {
-        // Raw markdown body — no added newline, matches MCP content[0].text.
+        // Verbatim output — no added newline, matches MCP content[0].text.
         process.stdout.write(markdown);
       } else {
         // Confirmation message — the only stdout newline the CLI ever adds.