Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.
-
Updated
Jun 20, 2026 - TypeScript
Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.
Automate web workflows on real browser tabs without hosting a browser farm.
Curated GEO platforms, repositories, and verification scripts for AI-era discoverability.
Automatically generate llms.txt and llms-full.txt files for Laravel applications to help AI models and LLM crawlers understand your website.
🤖 Generate optimized robots.txt files for AI search engine crawlers (GPTBot, PerplexityBot, ClaudeBot, and more)
See your site the way search engines and AI see it.
Server-side AI crawler & bot traffic tracking for Laravel — reports ChatGPT, Claude, Googlebot, GPTBot & co. to DataFast. Unofficial PHP port of @datafast/ai-crawl.
Claude Code skill that generates and audits ai.txt, robots.txt (AI-crawler rules), llms.txt, and TDM opt-out files for websites.
Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples
GEO SEO agent skill for AI search optimization, answer engine optimization, AI search visibility, technical SEO, structured data, citations, and AI crawler access.
Define AI permissions once. Emit the rules today's AI web actually uses. One manifest → robots.txt, AIPREF headers, X-Robots-Tag, Google-Extended, TDMRep.
Smart prerendering for WordPress — make your content visible to search engines, social bots, and AI crawlers. Zero config. 29+ bots. ~200ms.
Joy-Flow v1.7 | Maternal Cognitive Anchor (MCA) for "Ghost Fleet" Liquidation. Resolving 96% Agentic Failure via Protocol-Native Logic Gates. Deterministic Reliability for Jagged Intelligence. The Stone of Help is permanent.
See which AI crawlers, search bots, and agents hit your site — and block the ones you choose. A fail-open Cloudflare Worker with a private dashboard.
Interactive crawler IP intelligence dashboard for search, AI, and user-triggered fetchers.
MCP server for Googlebot vs AI crawler parity analysis — compare crawl behaviour from Nginx logs with GSC data to detect visibility gaps
AI-Crawler-Blocklist publishes AI crawler blocklists and deployment-ready firewall snippets from official operator-published sources. It separates verified IP ranges, user-agent rules, robots.txt controls, and watch lists so site operators can choose the right enforcement level without mixing signal quality.
Edge-native Akamai EdgeWorker that intercepts AI crawler traffic and serves token-efficient Markdown instead of HTML — no origin changes required
Open source, vendor-neutral telemetry standard + SDKs for LLM/AI features (traces, metrics, logs) via OpenTelemetry.
Evidence-based Claude plugin + A2A v1.0 agent for auditing, generating, deploying llms.txt files. 17 anti-patterns, 18 sectors, 13 languages. Honest empirical framing (three studies say null citation lift). Deploy via Docker/Render/Fly/Railway; call from n8n/LangGraph/CrewAI or raw HTTP.
Add a description, image, and links to the ai-crawlers topic page so that developers can more easily learn about it.
To associate your repository with the ai-crawlers topic, visit your repo's landing page and select "manage topics."