Skip to content
#

ai-crawlers

Here are 42 public repositories matching this topic...

AI-Crawler-Blocklist publishes AI crawler blocklists and deployment-ready firewall snippets from official operator-published sources. It separates verified IP ranges, user-agent rules, robots.txt controls, and watch lists so site operators can choose the right enforcement level without mixing signal quality.

  • Updated Jul 7, 2026
  • Python

Evidence-based Claude plugin + A2A v1.0 agent for auditing, generating, deploying llms.txt files. 17 anti-patterns, 18 sectors, 13 languages. Honest empirical framing (three studies say null citation lift). Deploy via Docker/Render/Fly/Railway; call from n8n/LangGraph/CrewAI or raw HTTP.

  • Updated May 23, 2026
  • Python

Improve this page

Add a description, image, and links to the ai-crawlers topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the ai-crawlers topic, visit your repo's landing page and select "manage topics."

Learn more