Skip to content

feat: add SerpBase (Google) search engine via REST API - #197

Open
gefsikatsinelou wants to merge 1 commit into
gitronald:masterfrom
gefsikatsinelou:feat/add-serpbase-engine
Open

feat: add SerpBase (Google) search engine via REST API#197
gefsikatsinelou wants to merge 1 commit into
gitronald:masterfrom
gefsikatsinelou:feat/add-serpbase-engine

Conversation

@gefsikatsinelou

Copy link
Copy Markdown

Summary

Adds a third SearchEngine backend, method="serpbase", which fetches Google results from the SerpBase REST API instead of driving a browser or scraping HTML. Results are rendered into a minimal Google-style SERP document so they flow through the existing parser unchanged — you get the usual general results with the standard schema.

Background

Plan 051 (v0.11.0) kept requests as the no-browser option but noted it is "increasingly blocked for live collection". The browser path (patchright) works but needs an installed Chrome and a display (Xvfb on headless hosts), and both paths hit Google's /sorry/ CAPTCHA wall on shared/VPS IPs. This PR adds a pure-HTTP backend that doesn't scrape at all: it calls the SerpBase Google Search Results API (GET https://api.serpbase.dev/google/search?q=...&api_key=...&num=...) and renders the JSON into the classic #rso > div.g > div.yuRUbf structure the parser already handles.

Changes

  • New: WebSearcher/searchers/serpbase_searcher.pySerpBaseSearcher with the same send_request/cleanup contract as the other backends, plus a _json_to_html renderer mapping organic_results (title/link/snippet) to minimal Google-style markup.
  • Updated: WebSearcher/models/configs.pySearchMethod.SERPBASE and SerpBaseConfig (api_key, base_url, timeout), wired into SearchConfig.
  • Updated: WebSearcher/searchers/searchers.pySearchEngine.__init__ accepts serpbase_config and dispatches method="serpbase".
  • Updated: WebSearcher/demos/cli.pyws-demo search/searches accept serpbase.
  • Updated: README.md — short "API method (no browser, no scraping)" section.
  • Updated: tests/test_searchers.py — no-key graceful path + end-to-end JSON → parsed-results test.

Design decisions

Decision Rationale
API key via serpbase_config / SERPBASE_API_KEY env var Matches the existing per-backend config pattern (requests_config, patchright_config)
Graceful skip when key missing Empty ResponseOutput (response_code 0) + warning — a crawl without a key doesn't crash or block the other methods
Render JSON to minimal HTML instead of a parallel result schema The parser/classifier pipeline stays the single source of truth; saved records are identical in shape to browser/requests methods
Reuses requests No new dependencies

Testing

  • No key set: logs a warning, returns an empty response, no exception.
  • Key set (via env or config): parse_serp produces general results with the correct title/url/text.
  • Existing tests pass unchanged (12 passed).

Related

  • Plan 051's note that the requests path is "increasingly blocked for live collection" — this is a pure-HTTP path that doesn't scrape.

Add a third SearchEngine method ('serpbase') that fetches results from the
SerpBase Google Search Results API instead of driving a browser or scraping
HTML. Results are rendered into a minimal Google-style SERP document so the
existing parser pipeline produces the usual 'general' results with the
standard schema.

- New SerpBaseSearcher backend (requests-based, no new dependencies)
- SearchMethod.SERPBASE + SerpBaseConfig (api_key, base_url, timeout)
- Graceful degradation: missing SERPBASE_API_KEY logs a warning and returns
  an empty response instead of raising, leaving other methods unaffected
- ws-demo CLI accepts 'serpbase' as a method choice
- Tests: no-key graceful path + end-to-end JSON-to-parsed-results
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant