docs: update specs, README, and AGENTS.md for SSRF protection

- specs/initial.md: add SSRF prevention section, update error messages,
  add block_private_ips to tool builder, add SSRF test requirements
- specs/fetchers.md: add dns_policy to FetchOptions, add SSRF section,
  add dns.rs to module structure
- AGENTS.md: add threat-model.md to available specs list
- README.md: add SSRF protection to features, add Security section
  with usage examples

https://claude.ai/code/session_011H7YzDQ8VXbaNT7nXGQgmE

Author: claude

AGENTS.md

@@ -35,6 +35,7 @@ Available specs:
 - `specs/initial.md` - WebFetch tool specification (types, behavior, conversions, error handling)
 - `specs/fetchers.md` - Pluggable fetcher system for URL-specific handling
 - `specs/maintenance.md` - Periodic maintenance checklist (deps, docs, spec-code alignment)
+- `specs/threat-model.md` - Security threat model (SSRF, network, input validation, DoS)
 
 Specification format: Abstract and Requirements sections.
 

README.md

@@ -10,6 +10,7 @@ AI-friendly web content fetching tool designed for LLM consumption. Rust library
 - **Binary detection** - Returns metadata only for images, PDFs, etc.
 - **Timeout handling** - 1s first-byte, 30s body with partial content on timeout
 - **URL filtering** - Allow/block lists for controlled access
+- **SSRF protection** - Resolve-then-check blocks private IPs by default
 - **MCP server** - Model Context Protocol support for AI tool integration
 
 ## Installation
@@ -171,6 +172,26 @@ Errors are returned in the `error` field:
 - `ContentError` - Failed to read body
 - `BinaryContent` - Binary content not supported
 
+## Security
+
+FetchKit blocks connections to private/reserved IP ranges by default, preventing SSRF attacks when used in server-side or AI agent contexts.
+
+**Blocked by default:** loopback, private networks (10.x, 172.16-31.x, 192.168.x), link-local (169.254.x including cloud metadata), IPv6 equivalents, multicast, and other reserved ranges.
+
+```rust
+// Default: private IPs blocked (safe for production)
+let tool = Tool::default();
+
+// Explicit opt-out for local development only
+let tool = Tool::builder()
+    .block_private_ips(false)
+    .build();
+```
+
+DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canonicalized before validation.
+
+See [`specs/threat-model.md`](specs/threat-model.md) for the full threat model.
+
 ## Configuration
 
 ### Timeouts

specs/fetchers.md

@@ -66,6 +66,7 @@ Fetchers receive `FetchOptions` for:
 - `block_prefixes` - URL prefix block list
 - `enable_markdown` - Enable markdown conversion
 - `enable_text` - Enable text conversion
+- `dns_policy` - DNS resolution policy for SSRF prevention (default: block private IPs)
 
 ### Extensibility
 
@@ -82,10 +83,20 @@ Design supports hundreds of fetchers by:
 - `FetchError::FetcherError(String)` for fetcher-specific errors
 - GitHub API errors return response with error field set
 
+### SSRF Protection
+
+Both built-in fetchers integrate resolve-then-check DNS validation:
+- Resolve hostname to IP before connecting
+- Validate IP against blocked ranges (private, loopback, link-local, etc.)
+- Pin validated IP via `reqwest::ClientBuilder::resolve()` to prevent DNS rebinding
+- Enabled by default via `DnsPolicy::default()` (blocks private IPs)
+- See `specs/threat-model.md` for threat IDs: TM-SSRF-001 through TM-SSRF-010
+
 ## Module Structure
 
 ```
 crates/fetchkit/src/
+├── dns.rs               # DnsPolicy - SSRF prevention via resolve-then-check
 ├── fetchers/
 │   ├── mod.rs           # Fetcher trait, FetcherRegistry
 │   ├── default.rs       # DefaultFetcher

specs/initial.md

@@ -48,6 +48,7 @@ Provide a builder to configure tool options, including:
 - Support allow/block list of URL prefixes.
 - Support enabling/disabling request options (feature gating).
 - Support User-Agent override (e.g., `allow_ua`).
+- Support `block_private_ips(bool)` for SSRF prevention (default: `true`).
 
 #### Types
 
@@ -74,6 +75,7 @@ Provide a builder to configure tool options, including:
   - Missing url
   - Invalid url scheme
   - Invalid method
+  - Blocked URL (prefix list or DNS policy)
   - Client build failure
   - Request error (timeout/connect/other)
 - `ToolStatus` (or equivalent)
@@ -134,6 +136,17 @@ Provide a builder to configure tool options, including:
   - If allow list is non-empty, URL must match at least one allow prefix.
   - If block list matches, request is denied even if allow list matches.
 
+### SSRF Prevention (DNS Policy)
+
+By default, FetchKit blocks connections to private/reserved IP ranges:
+- Resolves hostnames to IP addresses before connecting (resolve-then-check).
+- Validates resolved IPs against blocked ranges (loopback, private, link-local,
+  cloud metadata, carrier-grade NAT, documentation, benchmarking, multicast, broadcast).
+- Handles IPv6-mapped IPv4 addresses via canonicalization.
+- Pins validated IP via `reqwest::ClientBuilder::resolve()` to prevent DNS rebinding.
+- Blocked by default; opt out via `ToolBuilder::block_private_ips(false)`.
+- See `specs/threat-model.md` for full threat analysis.
+
 ### HTTP Behavior
 
 - User-Agent: configurable via tool builder or CLI/MCP/Python options
@@ -237,7 +250,7 @@ Content is HTML if:
 - Missing url -> tool error string "Missing required parameter: url".
 - Invalid URL -> tool error string "Invalid URL: must start with http:// or https://".
 - Invalid method -> tool error string "Invalid method: must be GET or HEAD".
-- Blocked prefix -> tool error string "Blocked URL: prefix not allowed".
+- Blocked URL (prefix or DNS policy) -> tool error string "Blocked URL: not allowed by policy".
 - First-byte timeout -> "Request timed out: server did not respond within 1 second".
 - Connect error -> "Failed to connect to server".
 - Other request errors -> "Request failed: <error>".
@@ -263,6 +276,7 @@ Unit:
 - Binary content detection.
 - HTML conversion, entity decoding.
 - Newline filtering behavior.
+- DNS policy IP range blocking (IPv4, IPv6, mapped addresses).
 
 Integration (mock HTTP server):
 - GET/HEAD with expected fields.
@@ -272,3 +286,12 @@ Integration (mock HTTP server):
 - Last-Modified extraction.
 - Size correctness for text and binary.
 - Body timeout truncation.
+
+SSRF security:
+- Private IP blocking (loopback, 10.x, 172.16.x, 192.168.x).
+- Cloud metadata endpoint blocking (169.254.169.254).
+- IPv6 loopback/mapped address blocking.
+- Non-HTTP scheme blocking (file, ftp, data, gopher).
+- Default-blocks-loopback verification.
+- Explicit opt-out verification.
+- Script stripping in converted content.