Skip to content

Skip comment fetches for status-filtered posts; drop HelpScout inter-page sleep - #26

Merged
dkships merged 2 commits into
mainfrom
perf/skip-filtered-comment-fetches
Jul 10, 2026
Merged

Skip comment fetches for status-filtered posts; drop HelpScout inter-page sleep#26
dkships merged 2 commits into
mainfrom
perf/skip-filtered-comment-fetches

Conversation

@dkships

@dkships dkships commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Two speedups for the slow fetch paths.

ProductLift: filter by status before fetching comments. get_feature_requests fetched comments for every post (250ms sleep + one API call each), then applied the status filter afterwards. The filter now runs inside fetchFeatureRequests before the comment loop, so filtered-out posts cost nothing. Same match semantics as before: status name compared case-insensitively, posts with no status excluded. On a portal with 200 posts and 20 matching, that's 180 fewer API calls and ~45s less sleeping per call.

HelpScout: remove the inter-page sleep in fetchConversations. The 250ms sleep was the effective pacer (~90-130 req/min with network time). Removing it lets pages run at network speed; the existing 170/min sliding-window cap and 429 retry/backoff keep us under HelpScout's documented 200/min limit. fetchMailboxes keeps its sleep (PAGE_DELAY_MS is still used there).

Behavior note: pii_categories_redacted may shrink when a status filter is set. It now reflects only PII found in the posts/comments actually returned, not in posts the filter would have discarded anyway. The returned data is scrubbed exactly as before.

Tests: two new vitest cases mock fetch and assert no /comments request fires for posts that fail the filter, and that unfiltered calls still fetch comments for every post. 66/66 passing.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dkships
dkships marked this pull request as ready for review July 10, 2026 00:56
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dkships
dkships merged commit df33f70 into main Jul 10, 2026
2 checks passed
@dkships
dkships deleted the perf/skip-filtered-comment-fetches branch July 10, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant