fix(coding-agents): cap the 429 backoff so a long Retry-After cannot park a drain

Review follow-up on the retry path.

`Retry-After` was honoured without a ceiling, so an hour-long value — an
incident, a misconfigured limiter, a proxy inventing one — would park the drain
for that hour, up to the whole `maxMs`, with the background seed frozen behind
it. The header is a server's hint, not a budget we owe it.

Capped at 60s. That keeps the signal (the floor and the header still lengthen the
wait) without handing over the schedule: if the limit still applies, the next
poll gets another 429 and backs off again.

Also re-syncs the docs page and skill mirror from the README, which the rebase
onto main left stale by a column width.

Author: nicoloboschi

hindsight-docs/docs-integrations/coding-agents.md

@@ -281,7 +281,8 @@ Environment variables are a **fallback**: the file wins wherever it sets a value
 an existing setup changes nothing. `retainTags` takes a comma-separated list
 (`HINDSIGHT_RETAIN_TAGS="project:{gitProject},env:work"`); entries are trimmed and blanks dropped.
 The map-valued settings (`mapPathToBank`, `harnesses`, `banks`, `retainMetadata`) are file-only —
-per-key branching doesn't survive flattening into one variable.
+per-key branching doesn't survive flattening into one variable. `maxParallelRetains` is available
+as `HINDSIGHT_MAX_PARALLEL_RETAINS` for containers and CI.
 
 There is deliberately no repo-carried config file — per-repo bank routing is `mapPathToBank`,
 per-agent differences are `harnesses.<name>`.
@@ -321,6 +322,7 @@ hook by Codex...), so one shared config serves several agents side by side:
 | `surveyModel`           | `haiku`                              | model for the survey — Claude recipe only (`claude -p --model`); other agents use their configured default                                                                                                                          |
 | `surveyBudgetUsd`       | `2`                                  | survey spend cap — Claude recipe only (`claude -p --max-budget-usd`); other agents rely on their read-only sandbox                                                                                                                  |
 | `retainSessions`        | `true`                               | plugin-harness write-back (opencode, Kilo): async upsert of the session transcript every turn, plus an idle flush that captures the reply the per-turn pass can't see (set `false` to opt out; hook harnesses always write on Stop) |
+| `maxParallelRetains`    | `10`                                 | cap on concurrent retain-related requests: drain()'s per-op polls plus deepen's chat/git retain pools. The API rate-limits bursts, not single requests — if you see 429s, lower this rather than raising it                         |
 | `logLevel`              | `"info"`                             | plugin-log verbosity (`"debug"` \| `"info"` \| `"warn"` \| `"error"`); `HINDSIGHT_LOG_LEVEL` env overrides                                                                                                                          |
 | `gitIngest`             | `"message"`                          | git depth for seeding AND staying current (same engine): `"message"` = commit messages only (one doc, re-upserted when HEAD moves); `"full"` = messages + per-commit full diffs (progressive, newest first); `"none"` = git off     |
 | `harnesses.<name>`      | —                                    | per-harness override of any field above                                                                                                                                                                                             |

hindsight-integrations/coding-agents/src/core/hindsight.test.ts

@@ -64,6 +64,23 @@ describe("HindsightClient.drain", () => {
     await p;
   });
 
+  it("caps the backoff however long Retry-After asks for", async () => {
+    // The header is a hint, not a budget we owe the server: an hour-long value would otherwise
+    // park the drain — and the background seed behind it — for that hour.
+    vi.useFakeTimers();
+    const client = new HindsightClient({ apiUrl: "http://x", bank: "b" });
+    const fetchMock = vi.fn(async () => jsonResponse(429, {}, { "Retry-After": "3600" }));
+    vi.stubGlobal("fetch", fetchMock);
+
+    const p = client.drain(["1"], "test", 300_000);
+    await vi.advanceTimersByTimeAsync(0);
+    expect(fetchMock).toHaveBeenCalledTimes(1);
+    await vi.advanceTimersByTimeAsync(60_000); // capped at 60s, not 3600s
+    expect(fetchMock).toHaveBeenCalledTimes(2);
+    await vi.advanceTimersByTimeAsync(300_000);
+    await p;
+  });
+
   it("uses the 10s floor when Retry-After is shorter than it", async () => {
     vi.useFakeTimers();
     const client = new HindsightClient({ apiUrl: "http://x", bank: "b" });

hindsight-integrations/coding-agents/src/core/hindsight.ts

@@ -95,6 +95,17 @@ const POLL_CYCLE_MS = 5000;
 /** Minimum backoff after a 429 that carried no (or a shorter) Retry-After. */
 const RETRY_AFTER_FLOOR_MS = 10 * 1000;
 
+/**
+ * Ceiling on a single backoff, however long `Retry-After` asks for.
+ *
+ * The header is a server's hint, not a budget we owe it: a large value (an incident, a
+ * misconfigured limiter, a proxy inventing one) would otherwise park a drain for as long as it
+ * says — up to the whole `maxMs`, with the background seed frozen behind it. Capping keeps the
+ * signal without handing over the schedule; if the limit still applies, the next poll simply gets
+ * another 429 and backs off again.
+ */
+const RETRY_AFTER_CEILING_MS = 60 * 1000;
+
 /** Bank-level missions the template seeds once and then leaves alone (#2492). */
 const MISSION_FIELDS = ["reflect_mission", "retain_mission", "observations_mission"] as const;
 
@@ -323,10 +334,9 @@ export class HindsightClient {
         try {
           const r = await fetch(this.bankUrl(`/operations/${id}`), { headers: this.headers() });
           if (r.status === 429) {
-            backoffMs = Math.max(
-              backoffMs,
-              RETRY_AFTER_FLOOR_MS,
-              retryAfterMs(r.headers.get("retry-after"))
+            backoffMs = Math.min(
+              RETRY_AFTER_CEILING_MS,
+              Math.max(backoffMs, RETRY_AFTER_FLOOR_MS, retryAfterMs(r.headers.get("retry-after")))
             );
             return; // op stays pending — retried after the backoff
           }

skills/hindsight-docs/references/sdks/integrations/coding-agents.md

@@ -276,7 +276,8 @@ Environment variables are a **fallback**: the file wins wherever it sets a value
 an existing setup changes nothing. `retainTags` takes a comma-separated list
 (`HINDSIGHT_RETAIN_TAGS="project:{gitProject},env:work"`); entries are trimmed and blanks dropped.
 The map-valued settings (`mapPathToBank`, `harnesses`, `banks`, `retainMetadata`) are file-only —
-per-key branching doesn't survive flattening into one variable.
+per-key branching doesn't survive flattening into one variable. `maxParallelRetains` is available
+as `HINDSIGHT_MAX_PARALLEL_RETAINS` for containers and CI.
 
 There is deliberately no repo-carried config file — per-repo bank routing is `mapPathToBank`,
 per-agent differences are `harnesses.<name>`.
@@ -316,6 +317,7 @@ hook by Codex...), so one shared config serves several agents side by side:
 | `surveyModel`           | `haiku`                              | model for the survey — Claude recipe only (`claude -p --model`); other agents use their configured default                                                                                                                          |
 | `surveyBudgetUsd`       | `2`                                  | survey spend cap — Claude recipe only (`claude -p --max-budget-usd`); other agents rely on their read-only sandbox                                                                                                                  |
 | `retainSessions`        | `true`                               | plugin-harness write-back (opencode, Kilo): async upsert of the session transcript every turn, plus an idle flush that captures the reply the per-turn pass can't see (set `false` to opt out; hook harnesses always write on Stop) |
+| `maxParallelRetains`    | `10`                                 | cap on concurrent retain-related requests: drain()'s per-op polls plus deepen's chat/git retain pools. The API rate-limits bursts, not single requests — if you see 429s, lower this rather than raising it                         |
 | `logLevel`              | `"info"`                             | plugin-log verbosity (`"debug"` \| `"info"` \| `"warn"` \| `"error"`); `HINDSIGHT_LOG_LEVEL` env overrides                                                                                                                          |
 | `gitIngest`             | `"message"`                          | git depth for seeding AND staying current (same engine): `"message"` = commit messages only (one doc, re-upserted when HEAD moves); `"full"` = messages + per-commit full diffs (progressive, newest first); `"none"` = git off     |
 | `harnesses.<name>`      | —                                    | per-harness override of any field above                                                                                                                                                                                             |