Add bots and agents overview page (#455)

Add /docs/bots documenting Kernel's two Web Bot Auth identities (the
existing user-directed Agent on kernel.sh and the new Kernel Search
crawler on search.bot.kernel.sh), so site owners can allow/block by
purpose. Also document Kernel Search signing config on the Web Bot
Auth page.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: ulziibay-kernel

bots.mdx

@@ -0,0 +1,45 @@
+---
+title: "Bots and agents"
+description: "Kernel's bots and agents, their purposes, and how to verify them with Web Bot Auth"
+---
+
+Kernel identifies its bots and agents with [Web Bot Auth](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture). Each identity serves its own key directory from its own authority, so site owners can allow or block each one **independently by purpose** — for example, allow search indexing while blocking user-directed agents (or vice versa).
+
+Kernel is listed in [Cloudflare's bots and agents directory](https://radar.cloudflare.com/bots/directory/kernel) and [Vercel's public directory](https://bots.fyi/?query=kernel).
+
+## Kernel Agent
+
+User-directed browser automation. Kernel Agent visits pages on behalf of an end user's request; it is not an automatic crawler. Requests are signed with Web Bot Auth rather than a dedicated crawler user-agent token.
+
+| Field | Value |
+| --- | --- |
+| Purpose | Agent |
+| Operator | Intermediary (end-user directed) |
+| Signature-Agent | `https://www.kernel.sh` |
+| Key directory | `https://www.kernel.sh/.well-known/http-message-signatures-directory` |
+
+## Kernel Search
+
+Crawls pages to build search indexes and retrieval databases. Kernel Search identifies itself with the `KernelSearchBot` user-agent token and follows `robots.txt` directives for that token, including crawl-delay preferences.
+
+| Field | Value |
+| --- | --- |
+| Purpose | Search |
+| Operator | Direct (Kernel-operated) |
+| User-Agent | `KernelSearchBot` |
+| Signature-Agent | `https://search.bot.kernel.sh` |
+| Key directory | `https://search.bot.kernel.sh/.well-known/http-message-signatures-directory` |
+
+## Verifying Kernel traffic
+
+Each identity publishes its public key set (JWKS) at its key directory. To verify a request:
+
+1. Read the `Signature-Agent` header to determine which Kernel identity signed the request.
+2. Fetch the public key set from that identity's key directory and cache it per the `Cache-Control` header.
+3. Verify the `Signature` and `Signature-Input` headers per [RFC 9421](https://datatracker.ietf.org/doc/html/rfc9421).
+
+Most major bot-detection services, CDNs, and WAFs verify Web Bot Auth automatically. See [Web Bot Auth](/browsers/bot-detection/web-bot-auth) for how Kernel signs requests.
+
+## Contact
+
+For questions about Kernel bot or agent traffic, contact [support@kernel.sh](mailto:support@kernel.sh).

browsers/bot-detection/web-bot-auth.mdx

@@ -145,18 +145,49 @@ The directory should contain your public keys in JWKS format:
 kernel extensions build-web-bot-auth \
   --to ./web-bot-auth-ext \
   --key ./my-key.jwk \
-  --url https://yourdomain.com/.well-known/http-message-signatures-directory \
+  --url https://yourdomain.com \
+  --signature-agent https://yourdomain.com \
   --upload my-web-bot-auth
 ```
 
-### 4. Register with Vercel and other Web Bot Auth-aware directories (optional)
+### 4. Kernel Search configuration
 
-If you want Vercel-protected sites to recognize your agent, you can register your key directory with [Vercel](https://bots.fyi/new-bot). Kernel is officially listed in the Vercel directory.
+Kernel Search uses a distinct Web Bot Auth identity from Kernel's user-driven
+agent traffic:
+
+- User-Agent: `KernelSearchBot`
+- Signature-Agent: `https://search.bot.kernel.sh`
+- Key directory:
+  `https://search.bot.kernel.sh/.well-known/http-message-signatures-directory`
+
+To build a browser extension for Kernel Search, use the Kernel Search Ed25519
+private key (JWK) and upload it under a distinct extension name:
+
+```bash
+kernel extensions build-web-bot-auth \
+  --to ./web-bot-auth-search-ext \
+  --key ./kernel-search.jwk \
+  --url https://www.kernel.sh \
+  --signature-agent https://search.bot.kernel.sh \
+  --upload web-bot-auth-search
+```
+
+For host-proxy based signing, configure the Search crawler's host-proxy
+environment with:
+
+```bash
+HOST_PROXY_WEB_BOT_AUTH_ENABLED=true
+HOST_PROXY_WEB_BOT_AUTH_KEY_PATH=/path/to/kernel-search-private-key.pem
+HOST_PROXY_WEB_BOT_AUTH_DIRECTORY_URL=https://search.bot.kernel.sh
+```
 
+The signing key must correspond to the public key hosted by
+`search.bot.kernel.sh`.
 
+### 5. Register with Vercel and other Web Bot Auth-aware directories (optional)
 
+If you want Vercel-protected sites to recognize your agent, you can register your key directory with [Vercel](https://bots.fyi/new-bot). Kernel is officially listed in the Vercel directory.
 
- 
 ## References
 
 - [Vercel's Public Directory](https://bots.fyi/?query=kernel)

docs.json

@@ -147,7 +147,8 @@
                           "proxies/datacenter"
                         ]
                       },
-                      "browsers/bot-detection/web-bot-auth"
+                      "browsers/bot-detection/web-bot-auth",
+                      "bots"
                     ]
                   },
                   "browsers/extensions",