DOC-3373: Add TinyMCE 8-specific llms.txt files for AI/LLM discoverability

[kemister85]

Ticket: DOC-3373

PR #2: tinymce/8 Branch (TinyMCE 8 Content)

LLM File Generation Automation

Overview

This PR automates the generation of llms.txt and llms-full.txt files for LLM consumption, replacing manual curation with an automated script that ensures consistency and accuracy.

What Was Created/Updated

1. -scripts/generate-llm-files.js (New/Updated)

• Main Node.js script that generates both LLM files
• Sitemap-only approach: Uses only sitemap.xml (no dependency on nav.adoc)
• H1 title fetching: Makes HTTP requests to ~400 pages to get actual page titles
• 404 validation: Ensures no broken links are included
• Automatic categorization: Groups pages by topic (integrations, plugins, API, etc.)
• Title uniqueness: Makes titles unique (e.g., "ES6 and npm (Webpack)" vs "ES6 and npm (Rollup)")
• HTML entity decoding: Handles entities like ’ → '
• Error filtering: Filters out error pages and duplicate URLs

2. -scripts/generate-llm-files.sh (Existing)

• Shell wrapper for convenience (optional)

3. package.json (Updated)

• Added generate-llm-files script (uses local sitemap)
• Added generate-llm-files-from-url script (uses production sitemap)
• Fixed path issues (added ./ prefix)

4. -scripts/README-llm-files.md (Updated)

• Complete documentation explaining the workflow
• Manual regeneration approach (not CI/CD)
• How it works, usage instructions, troubleshooting

Generated Files

llms.txt (~127 lines)

• Curated overview with code examples
• Getting started guides
• Integration references
• Links to complete index

llms-full.txt (~700 lines)

• Complete index of all ~396 documentation pages
• Organized by category
• Uses actual H1 titles from pages
• No duplicate URLs
• Unique, descriptive titles

Workflow

Current Approach

• Manual regeneration after releases (major/minor/patch)
• Run script locally → Review in PR → Commit
• Not automated in CI/CD (too resource-intensive: 400+ HTTP requests, ~4-5 minutes)

Future

• Files moved to root post-build (separate PR)
• On new major version: move old files to version directory, regenerate for new /latest

Pre-checks:

• Branch prefixed with feature/<version>/, hotfix/<version>/, staging/<version>/, or release/<version>/.

Review:

• Documentation Team Lead has reviewed