docs: document API server and built-in tools - #1045
Conversation
- Add OpenAI, Anthropic, and native API guidance - Document built-in tools, /tools, and security behavior - Link new guides across existing documentation
aittalam
left a comment
aittalam
left a comment
There was a problem hiding this comment.
TYSM @angpt ! 🙏
I left some comments with suggestions or request for changes. Overall I am very happy to see these docs becoming available, as I think they can help people build agents based on llamafile. However, I suggested to cut some parts for the following reasons:
- some were very easy to generate, but I think are not as easy to digest for a human (and they depended on a feature which is experimental, so I would not create another artifact that then needs to be maintained in sync with upstream code).
- others, I believe, would work way better in a cookbook/tutorial rather than in a document describing an API or a feature
WDYT? I hope you agree but if you have different opinions I am happy to talk about it
| > [!WARNING] | ||
| > CORS controls which browser origins can read responses. It is not | ||
| > authentication. Keep the default loopback binding or configure an API key | ||
| > before making the server reachable from another machine. |
There was a problem hiding this comment.
There are a few different concepts here that might need to be unpacked:
- the possibility to bind to a custom --host (by specifying eg. a LAN address or 0.0.0.0 to completely open it)
- CORS, which I believe works on the Web UI only (it is usually enforced by the browser)
- --api-key to stop either browser or remote clients if they are not properly authenticated
- the security implications of opening a server to provide access more widely than required
I'd suggest to change the sentence to something like: "As a general rule, avoid making your server accessible to more clients than what you actually need. Usually start with the default host binding
(the server listens to localhost only), and change it only if you need to access the server from another
machine. When you do this, add an api key to make sure that even if someone can access the server, they won't be able to actually use it unless authenticated"
| | `get_datetime` | Read | Get the server's current date and time. | | ||
| | `get_info` | Read | Get the runtime operating system and working directory. | | ||
|
|
||
| ### `read_file` |
There was a problem hiding this comment.
I think it's great that we are providing more information about llama.cpp tools, because I personally found them very elusive.
However, I am a bit concerned about going this level of detail. The main reason is that we do not own these tools, they are experimental, and they could change more quickly than we keep this docs up-to-date.
Another reason is that no human should need this API, as their descriptions are brought automatically into the conversation with the LLM and the LLM itself prepares the tool calls. Last but not least, this is information which is very cheap to create (the full signatures are on /tools), but heavy for a reader to digest, and I do not want these docs to be regarded to as slop and ignored.
I think it'd be probably more helpful to say something like "Tool support in llama.cpp is experimental
and thus subject to changes. If you want to know more about each individual tool, http://localhost:8080/tools returns a JSON containing their up-to-date signatures".
| satisfy any configured API key can invoke an enabled tool without using the Web | ||
| UI. Select the smallest useful tool list at startup. | ||
|
|
||
| llamafile's sandbox changes which tools can succeed: |
There was a problem hiding this comment.
I would add a link to security.md here. I am fine with a bit of repetition, but if you see that
after linking the doc the information you have here becomes too redundant feel free to remove (parts of) it
| invoke `POST /tools` or its own implementation, return the result in the | ||
| selected API's tool-result format, and continue the conversation. | ||
|
|
||
| See [API server: Tool calling](api.md#tool-calling) for the complete agent |
There was a problem hiding this comment.
See my comment above (I do not think agentic loop and tool definitions / mappings should be in that doc).
| > [!IMPORTANT] | ||
| > `/tools` is an experimental internal interface between the Web UI and | ||
| > `llama-server`. It can change or be removed without notice. Do not depend on | ||
| > it as a stable downstream application API. Applications should normally use | ||
| > the model's [function-calling API](api.md#tool-calling), execute approved | ||
| > tools in the application, and return the results to the model as tool | ||
| > messages. | ||
|
|
||
| The examples below assume the default address, `http://127.0.0.1:8080`. If the | ||
| server was started with an API prefix, such as `--api-prefix /llama`, prepend | ||
| that prefix: `/llama/tools`. | ||
|
|
||
| The endpoint is available when at least one built-in or MCP tool is configured. | ||
| If tools are disabled, both methods return HTTP `403`. When `--api-key` or | ||
| `--api-key-file` is configured, authenticate in either of these forms: | ||
|
|
||
| ```sh | ||
| -H 'Authorization: Bearer YOUR_API_KEY' | ||
| ``` | ||
|
|
||
| ```sh | ||
| -H 'X-Api-Key: YOUR_API_KEY' | ||
| ``` | ||
|
|
There was a problem hiding this comment.
I think this part is redundant (I read different parts of it in different places already)
| `POST /tools` call executes immediately: the server does not reproduce the Web | ||
| UI's permission prompt. | ||
|
|
||
| ### List tools with `GET /tools` |
There was a problem hiding this comment.
I think this section (from here to the end of the ### sections) is a bit too verbose, mainly because the tool calls are not done manually but in the agentic loop. I like the level of detail, but I think it'd be more useful in a tutorial/cookbook than in this documentation. WDYT?
Co-authored-by: Davide Eynard <davide.eynard@gmail.com>
Co-authored-by: Davide Eynard <davide.eynard@gmail.com>
Co-authored-by: Davide Eynard <davide.eynard@gmail.com>
Co-authored-by: Davide Eynard <davide.eynard@gmail.com>
|
@aittalam I believe I addressed most (if not all) of your comments. Let me know what you think. Feel free to push back if you aren't happy with any of the updates. |
2 participants
Description
PR Type
Checklist
llama.cpp/,whisper.cpp/, orstable-diffusion.cpp/, I also updated the matching*.patches/files.AI Usage Information
When answering reviewer questions, please respond yourself rather than pasting reviewer comments into an AI system and posting the reply back unchanged.