Generate llms.txt and doc-content.txt for AI tools

This commit adds the generations of llms.txt (more robust than what was there)
as well a single docs-content.txt that can be fed to an agent or answer engine
to set context of how to work with Flox.

This does add a Makefile (sorry) because I needed something to run multiple
steps when building the documentaiton site.

The README has been updated to explain Makefile usage (and it's completely optional).

The tools directory is also new, with a small python file to generate the AI indexing files.

The robots.txt that helps the crawlers know where to work is maintained in the
flox/floxwebsite repo.

Author: stahnma

.github/workflows/ci.yml

@@ -43,6 +43,7 @@ jobs:
         with:
           command: |
             mkdocs build
+            python3 tools/generate_llms_txt.py ./site
             mkdir -p ./public/docs
             cp -R ./site/* ./public/docs/
             chmod -R +w ./public/docs

Makefile

@@ -0,0 +1,43 @@
+# Flox Documentation Makefile
+
+.PHONY: help dev build clean install
+
+# Default target
+help:
+	@echo "Available targets:"
+	@echo "  dev     - Start development server with live reload"
+	@echo "  build   - Build static site and generate AI files"
+	@echo "  clean   - Clean build artifacts"
+	@echo "  install - Install dependencies"
+	@echo "  help    - Show this help message"
+
+# Development server with live reload
+dev:
+	@echo "Starting development server..."
+	@echo "Site will be available at: http://127.0.0.1:8000"
+	@echo "Press Ctrl+C to stop"
+	mkdocs serve
+
+# Build static site and generate AI files
+build:
+	@echo "Building static site..."
+	mkdocs build
+	@echo "Generating AI files..."
+	python3 tools/generate_llms_txt.py ./site
+	@echo "✅ Build complete! Site available in ./site/"
+
+# Clean build artifacts
+clean:
+	@echo "Cleaning build artifacts..."
+	rm -rf site/
+	rm -rf public/
+	@echo "✅ Clean complete!"
+
+# Install dependencies (if needed)
+install:
+	@echo "Installing dependencies..."
+	@if command -v poetry >/dev/null 2>&1; then \
+		poetry install; \
+	else \
+		echo "Poetry not found. Please install dependencies manually."; \
+	fi

README.md

@@ -4,17 +4,53 @@ Live at: [flox.dev/docs](https://flox.dev/docs).
 
 ## Usage
 
+### Quick Start
+
+```bash
+# Activate the Flox environment
+$ flox activate
+✅ You are now using the environment 'floxdocs'.
+
+# Start development server
+λ (floxdocs) $ make dev
+```
+
+The documentation will be available at `http://127.0.0.1:8000` with live reload.
+
+### Available Commands
+
+```bash
+make dev     # Start development server with live reload
+make build   # Build static site and generate AI files
+make clean   # Clean build artifacts
+make help    # Show all available commands
 ```
+
+### Flox Native Usage
+
+You can still use the original Flox services approach:
+
+```bash
 $ flox activate
 ✅ You are now using the environment 'floxdocs'.
 
 λ (floxdocs) $ flox services start mkdocs
 ✅ Service 'mkdocs' started.
-
 ```
+
 Once mkdocs service started you can preview the documentation at
 `https://127.0.0.1:8000`.
 
+## AI-Friendly Documentation
+
+This documentation site automatically generates AI-friendly files for different use cases:
+
+- **`llms.txt`** - Optimized for AI agents with critical rules, workflows, and organized sitemap
+- **`docs-content.txt`** - Comprehensive content for answer engines and RAG systems
+
+These files are automatically generated during the build process and are available at:
+- `https://flox.dev/docs/llms.txt`
+- `https://flox.dev/docs/docs-content.txt`
 
 ## Guidelines
 

tools/README.md

@@ -0,0 +1,31 @@
+# Tools
+
+This directory contains build tools and utilities for the Flox documentation site.
+
+## Scripts
+
+### `generate_llms_txt.py`
+
+Generates AI-friendly documentation files from the built MkDocs site:
+
+- **`llms.txt`** - Agent-focused file with critical rules, workflows, and organized sitemap
+- **`docs-content.txt`** - Answer engine file with comprehensive documentation content
+
+**Usage:**
+```bash
+python3 tools/generate_llms_txt.py <site_directory>
+```
+
+### `generate_llms_txt.sh`
+
+Convenience script for local development. Generates both AI files after a MkDocs build.
+
+**Usage:**
+```bash
+mkdocs build
+./tools/generate_llms_txt.sh
+```
+
+## Integration
+
+These scripts are automatically run during CI builds in `.github/workflows/ci.yml` to ensure the AI files are always up-to-date with the documentation content.