⚠ DEPRECATED — This project is superseded by moodlehq/wiki-rag, a complete end-to-end RAG pipeline that ingests MediaWiki sites natively via the API, indexes into Milvus, and serves via an OpenAI-compatible API and MCP server. No custom middleware or scraping tooling required.
This repository is kept for reference only. No further development planned.
Section-grained scraper for vanguard-galaxy.fandom.com, built to feed the vrt-cogs Assistant cog (a Discord RAG bot framework). Outputs CSV and JSON in the exact shape the cog's ?assistant importcsv / ?assistant importjson commands expect.
Uses the public MediaWiki API (api.php) — no scraping, no Cloudflare workarounds, no bot password required for reads.
- Enumerates every non-redirect article in namespace 0 (~115 articles for this wiki).
- Fetches each page's wikitext +
revidviaaction=parse. - Splits each page on
==H2==headings so the lead and each top-level section become their own retrieval entry — RAG works better when chunks are concept-sized, not page-sized. - Renders templates as
key: valuelines (so infobox data — faction, location, role — survives) while dropping image-only fields (image: foo.png). - Strips
[[File:…]]/[[Image:…]]wikilinks and any residual*.png|jpg|gif|svg|webptokens. - Flattens to plaintext via
mwparserfromhell.strip_code, normalises whitespace. - Splits sections longer than 4000 chars (vrt-cogs' per-entry cap) at paragraph boundaries.
- Writes a
revidmanifest so subsequent runs only re-emit pages that actually changed. - Fetches
Module:ShipDataonce per run, parses it, and emits one<Name> – Specchunk per ship as natural-language sentences (manufacturer, class, hull/shield/armor modifiers, hardpoints, warp speed, crew, cargo, shipyard requirements, conquest rank where applicable). The Module's revid is tracked under__module_shipdatain the manifest so a Module edit re-emits every Spec card. - Resolves
{{#invoke:Shipbox|field|...}}and{{#invoke:Shipbox|infobox|...}}parser-function calls in article wikitext using a small registry (resolvers.py). Future modules (e.g.Module:ItemData) plug in by registering a handler — no scraper changes.
docker pull ghcr.io/fank/vanguard-galaxy-wiki-scraper:latestMulti-arch (amd64 + arm64). No Python toolchain required on the host.
git clone https://github.com/fank/vanguard-galaxy-wiki-scraper.git
cd vanguard-galaxy-wiki-scraper
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt# incremental run — outputs land in ./out on the host
docker run --rm -v "$PWD/out:/work/out" ghcr.io/fank/vanguard-galaxy-wiki-scraper
# full re-emit + tighter API spacing
docker run --rm -v "$PWD/out:/work/out" ghcr.io/fank/vanguard-galaxy-wiki-scraper --full --sleep 0.05Mount /work/out to the host directory you want the CSV/JSON/manifest written to. The container runs as UID 1000, so on a typical single-user Linux host the files end up owned by your user with no extra config.
Suggested shell alias:
alias vg-wiki-scrape='docker run --rm -v "$PWD/out:/work/out" ghcr.io/fank/vanguard-galaxy-wiki-scraper:latest'
vg-wiki-scrape --fullpython scrape.py # incremental — skips pages whose revid hasn't changed
python scrape.py --full # ignore manifest, re-emit everything
python scrape.py --sleep 0.05 # tighter API spacing (default 0.1s)
python scrape.py --out path # custom output directory (default ./out)
python scrape.py --dry-run # print row count + a sample Spec card and exitFirst run takes ~30s for the full wiki. Subsequent runs touch only changed pages.
out/
├── vg_wiki.csv # name,text columns — for `?assistant importcsv true`
├── vg_wiki.json # {name: {text}} — for `?assistant importjson true`
└── vg_wiki.manifest.json # title → revid, drives incremental runs
Entry naming is stable: <Page Title> – <Section Title>, with lead paragraphs as <Page Title> – Overview. Long sections split into <name> (1/N), <name> (2/N). Stable names mean re-imports cleanly overwrite prior entries — no duplicates, no orphaned chunks.
JSON shape matches the cog's importer exactly:
{
"Damage – Damage calculation": { "text": "Outgoing damage is computed in three stages..." },
"Damage – Practical implications": { "text": "Roughly, every 5 points of CombatPower..." }
}The embedding and model keys are deliberately omitted so the bot re-embeds with whatever model it has configured (local Ollama, OpenAI, etc.).
?assistant importjson true # true = overwrite existing entries with same names
Daily refresh:
# cron entry
0 4 * * * cd /path/to/wiki-scraper && .venv/bin/python scrape.pyThen drop the regenerated out/vg_wiki.json in Discord and re-run ?assistant importjson true. The cog only re-embeds entries whose text actually changed, so the local embed model isn't doing 342 × 768-dim work every day.
For local embedding models (e.g. embeddinggemma, nomic-embed-text) cosine scores run systematically lower than OpenAI's. Defaults will reject every match. Calibrate after first import:
?query stacking penalty # see actual top-N scores for this embed model
?assistant minrelatedness 0.30 # set just below the lowest "good" hit
?assistant topn 5 # widens the funnel
The bot's system prompt also needs to explicitly instruct grounding on the injected context — character-roleplay prompts otherwise dominate and the LLM will say "I don't have the lore loaded" even when context is present.
pip install -r requirements.txt -r requirements-dev.txt
pytestTests run offline against captured fixtures in tests/fixtures/. No network
calls during test runs.
Single-wiki tool today — API_URL is hardcoded at the top of scrape.py. Pointing it at any other Fandom wiki only needs that constant changed (and possibly the USER_AGENT for politeness). The MediaWiki API surface is identical across Fandom.
moodlehq/wiki-rag replaces this scraper entirely:
| This project | wiki-rag equivalent |
|---|---|
scrape.py (custom MediaWiki fetcher) |
wr-load — native MediaWiki API dump, incremental mode |
split_sections() + chunking |
wr-index — fixed/paragraph chunking with overlap, Milvus vector store |
| CSV/JSON flat file output | Vector index (Milvus) + wr-server (OpenAI API) / wr-mcp (MCP protocol) |
| Manual re-import into Discord cog | Built-in query serving with hybrid search, HyDE, query rewrite |
The only features not covered by wiki-rag out of the box are the structured ship/aspect Spec cards and ranking rosters (shipdata.py / aspectdata.py). Those could be migrated into wiki templates or a lightweight pre-processing step if needed.
Built thanks to the work of the Vanguard Galaxy Wiki maintainers. This tool consumes the wiki — it doesn't replace authoring it. If you find this useful, contribute back to the wiki.