feat: add SerpBase (Google) search engine via REST API - #197
Open
gefsikatsinelou wants to merge 1 commit into
Open
feat: add SerpBase (Google) search engine via REST API#197gefsikatsinelou wants to merge 1 commit into
gefsikatsinelou wants to merge 1 commit into
Conversation
Add a third SearchEngine method ('serpbase') that fetches results from the
SerpBase Google Search Results API instead of driving a browser or scraping
HTML. Results are rendered into a minimal Google-style SERP document so the
existing parser pipeline produces the usual 'general' results with the
standard schema.
- New SerpBaseSearcher backend (requests-based, no new dependencies)
- SearchMethod.SERPBASE + SerpBaseConfig (api_key, base_url, timeout)
- Graceful degradation: missing SERPBASE_API_KEY logs a warning and returns
an empty response instead of raising, leaving other methods unaffected
- ws-demo CLI accepts 'serpbase' as a method choice
- Tests: no-key graceful path + end-to-end JSON-to-parsed-results
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a third
SearchEnginebackend,method="serpbase", which fetches Google results from the SerpBase REST API instead of driving a browser or scraping HTML. Results are rendered into a minimal Google-style SERP document so they flow through the existing parser unchanged — you get the usualgeneralresults with the standard schema.Background
Plan 051 (v0.11.0) kept
requestsas the no-browser option but noted it is "increasingly blocked for live collection". The browser path (patchright) works but needs an installed Chrome and a display (Xvfb on headless hosts), and both paths hit Google's/sorry/CAPTCHA wall on shared/VPS IPs. This PR adds a pure-HTTP backend that doesn't scrape at all: it calls the SerpBase Google Search Results API (GET https://api.serpbase.dev/google/search?q=...&api_key=...&num=...) and renders the JSON into the classic#rso > div.g > div.yuRUbfstructure the parser already handles.Changes
WebSearcher/searchers/serpbase_searcher.py—SerpBaseSearcherwith the samesend_request/cleanupcontract as the other backends, plus a_json_to_htmlrenderer mappingorganic_results(title/link/snippet) to minimal Google-style markup.WebSearcher/models/configs.py—SearchMethod.SERPBASEandSerpBaseConfig(api_key, base_url, timeout), wired intoSearchConfig.WebSearcher/searchers/searchers.py—SearchEngine.__init__acceptsserpbase_configand dispatchesmethod="serpbase".WebSearcher/demos/cli.py—ws-demo search/searchesacceptserpbase.README.md— short "API method (no browser, no scraping)" section.tests/test_searchers.py— no-key graceful path + end-to-end JSON → parsed-results test.Design decisions
serpbase_config/SERPBASE_API_KEYenv varrequests_config,patchright_config)ResponseOutput(response_code 0) + warning — a crawl without a key doesn't crash or block the other methodsrequestsTesting
parse_serpproducesgeneralresults with the correct title/url/text.12 passed).Related
requestspath is "increasingly blocked for live collection" — this is a pure-HTTP path that doesn't scrape.