Skip to content

Commit e38c4ed

Browse files
sandijean90sandijean90
authored
docs: Revising the llm proxy page (#2009)
Signed-off-by: Sandi Besen <sandibesen@gmail.com>
1 parent 7300603 commit e38c4ed

2 files changed

Lines changed: 220 additions & 124 deletions

File tree

‎docs/development/agent-integration/llm-proxy-service.mdx‎

Lines changed: 110 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,130 @@
11
---
22
title: "LLM Proxy Service"
3-
description: "Learn how to use the LLM proxy service extension within your agent"
3+
description: "Leverage Agent Stack's model and provider agnostic LLM inference"
44
---
55

6-
When building AI agents, one of the first requirements you might have is to connect your agent to a Large Language Model (LLM). Fortunately, the Agent Stack helps with this by providing built-in OpenAI-compatible LLM inference.
6+
When building AI agents, one of the first requirements is connecting to a Large Language Model (LLM). Agent Stack provides built-in, OpenAI-compatible LLM inference that is model and provider agnostic.
77

8-
The platform's OpenAI endpoints are model and provider agnostic, serving as a proxy to whatever is configured.
8+
In order to effectively implement the LLM Proxy Service there are 3 steps to follow:
99

10-
For you as an agent builder, the usage is extremely simple because we've wrapped the usage into a Service Extension.
10+
<Steps>
11+
<Step title="Add the LLM service extension to your agent">
12+
Import the necessary components and add the LLM service extension to your agent function.
13+
</Step>
14+
15+
<Step title="Configure your LLM request">
16+
Specify which model your agent prefers and how you want to access it.
17+
</Step>
18+
19+
<Step title="Use the LLM in your agent">
20+
Access the optionally provided LLM configuration and use it with your preferred LLM client.
21+
</Step>
22+
</Steps>
1123

12-
<Tip>
24+
<Note>
1325
Service Extensions are a type of [A2A Extension](https://a2a-protocol.org/latest/topics/extensions/) that allows you to easily "inject dependencies" into your agent. This follows the inversion of control principle where your agent defines what it needs, and the platform (in this case, Agent Stack) is responsible for providing those dependencies.
14-
</Tip>
26+
</Note>
1527

1628
<Warning>
1729
Service extensions are optional by definition, so you should always check if they exist before using them.
1830
</Warning>
1931

20-
## Quickstart
2132

22-
<Steps>
23-
<Step title="Add LLM service extension to your agent">
24-
Import the necessary components and add the LLM service extension to your agent function.
25-
</Step>
33+
## Implementing Steps
2634

27-
<Step title="Configure your LLM request">
28-
Specify which model your agent prefer and how you want to access it.
29-
</Step>
35+
### 1. Add the LLM service extension to your agent
3036

31-
<Step title="Use the LLM in your agent">
32-
Access the optionally provided LLM configuration and use it with your preferred LLM client.
33-
</Step>
34-
</Steps>
37+
Import the `LLMServiceExtensionServer` and `LLMServiceExtensionSpec` from the SDK. You will use these within a type hint to let the platform know your agent requires LLM access.
38+
39+
```python
40+
from agentstack_sdk.a2a.extensions import LLMServiceExtensionServer, LLMServiceExtensionSpec
41+
from a2a.types import Message
42+
from typing import Annotated
43+
44+
# The extension is added as an Annotated parameter in your agent function
45+
async def my_agent(
46+
input: Message,
47+
llm: Annotated[
48+
LLMServiceExtensionServer, ...]
49+
):
50+
# agent logic
51+
pass
52+
```
53+
### 2. Configure your LLM request
54+
55+
Use `LLMServiceExtensionSpec.single_demand()` to request a model. By passing a suggested tuple, you tell the platform which model you'd prefer to use.
56+
57+
```python
58+
from agentstack_sdk.a2a.extensions import LLMServiceExtensionServer, LLMServiceExtensionSpec
59+
from a2a.types import Message
60+
from typing import Annotated
61+
62+
# The llm parameter is configured with a specific model demand
63+
async def my_agent(
64+
input: Message,
65+
llm: Annotated[
66+
LLMServiceExtensionServer,
67+
LLMServiceExtensionSpec.single_demand(suggested=("ibm/granite-3-3-8b-instruct",))
68+
]
69+
):
70+
# agent logic
71+
pass
72+
```
73+
74+
When you specify a suggested model like `"ibm/granite-3-3-8b-instruct"`the platform:
75+
76+
1. Checks if the requested model is available in your configured environment
77+
2. Allocates the best available model that matches your requirements
78+
3. Provides you with the exact model identifier and endpoint details
79+
80+
The platform handles the complexity of model provisioning and endpoint management, so you can focus on building your agent logic.
3581

36-
## Example of LLM Access
82+
### 3. Use the LLM in your agent
3783

38-
Here's how to add LLM inference capabilities to your agent:
84+
Once the platform provides the extension, you can extract the OpenAI-compatible configuration.
85+
86+
```python
87+
from typing import Annotated
88+
from a2a.utils.message import get_message_text
89+
from a2a.types import Message
90+
from agentstack_sdk.a2a.extensions import LLMServiceExtensionServer, LLMServiceExtensionSpec
91+
92+
async def my_agent(
93+
input: Message,
94+
llm: Annotated[
95+
LLMServiceExtensionServer,
96+
LLMServiceExtensionSpec.single_demand(suggested=("ibm/granite-3-3-8b-instruct",))
97+
]
98+
) -> None:
99+
# Verify that the optional extension was provided
100+
if llm and llm.data and llm.data.llm_fulfillments:
101+
user_message = get_message_text(input)
102+
103+
# Access the resolved LLM configuration
104+
llm_config = llm.data.llm_fulfillments.get("default")
105+
106+
if llm_config:
107+
# These credentials work with any OpenAI-compatible client library
108+
api_model = llm_config.api_model
109+
api_key = llm_config.api_key
110+
api_base = llm_config.api_base
111+
```
112+
The platform automatically provides you with:
113+
114+
- **`api_model`**: The specific model identifier that was allocated to your request
115+
- **`api_key`**: Authentication key for the LLM service
116+
- **`api_base`**: The base URL for the OpenAI-compatible API endpoint
117+
118+
These credentials work with any OpenAI-compatible client library, making it easy to integrate with popular frameworks like:
119+
- BeeAI Framework
120+
- LangChain
121+
- LlamaIndex
122+
- OpenAI Python client
123+
- Custom implementations
124+
125+
<Accordion title="Full Code Example">
126+
127+
This complete example shows how to receive a user message and respond using the credentials provided by the LLM Proxy Service:
39128

40129
```python
41130
import os
@@ -87,45 +176,4 @@ if __name__ == "__main__":
87176
run()
88177
```
89178

90-
## How to request LLM access
91-
92-
Here's what you need to know to add LLM inference capabilities to your agent:
93-
94-
**Import the extension**: Import `LLMServiceExtensionServer` and `LLMServiceExtensionSpec` from `agentstack_sdk.a2a.extensions`.
95-
96-
**Add the LLM parameter**: Add a third parameter to your agent function with the `Annotated` type hint for LLM access.
97-
98-
**Specify your model requirements**: Use `LLMServiceExtensionSpec.single_demand()` to request a single model (multiple models will be supported in the future).
99-
100-
**Suggest a preferred model**: Pass a tuple of suggested model names to help the platform choose the best available option.
101-
102-
**Check if the extension exists**: Always verify that the LLM extension is provided before using it, as service extensions are optional.
103-
104-
**Access LLM configuration**: Use `llm.data.llm_fulfillments.get("default")` to get the LLM configuration details.
105-
106-
**Use with your LLM client**: The platform provides `api_model`, `api_key`, and `api_base` that work with OpenAI-compatible clients.
107-
108-
## Understanding LLM Configuration
109-
110-
The platform automatically provides you with:
111-
112-
- **`api_model`**: The specific model identifier that was allocated to your request
113-
- **`api_key`**: Authentication key for the LLM service
114-
- **`api_base`**: The base URL for the OpenAI-compatible API endpoint
115-
116-
These credentials work with any OpenAI-compatible client library, making it easy to integrate with popular frameworks like:
117-
- BeeAI Framework
118-
- LangChain
119-
- LlamaIndex
120-
- OpenAI Python client
121-
- Custom implementations
122-
123-
## Model Selection
124-
125-
When you specify a suggested model like `"ibm/granite-3-3-8b-instruct"`, the platform will:
126-
127-
1. Check if the requested model is available in your configured environment
128-
2. Allocate the best available model that matches your requirements
129-
3. Provide you with the exact model identifier and endpoint details
130-
131-
The platform handles the complexity of model provisioning and endpoint management, so you can focus on building your agent logic.
179+
</Accordion>

‎docs/stable/agent-integration/llm-proxy-service.mdx‎

Lines changed: 110 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,130 @@
11
---
22
title: "LLM Proxy Service"
3-
description: "Learn how to use the LLM proxy service extension within your agent"
3+
description: "Leverage Agent Stack's model and provider agnostic LLM inference"
44
---
55

6-
When building AI agents, one of the first requirements you might have is to connect your agent to a Large Language Model (LLM). Fortunately, the Agent Stack helps with this by providing built-in OpenAI-compatible LLM inference.
6+
When building AI agents, one of the first requirements is connecting to a Large Language Model (LLM). Agent Stack provides built-in, OpenAI-compatible LLM inference that is model and provider agnostic.
77

8-
The platform's OpenAI endpoints are model and provider agnostic, serving as a proxy to whatever is configured.
8+
In order to effectively implement the LLM Proxy Service there are 3 steps to follow:
99

10-
For you as an agent builder, the usage is extremely simple because we've wrapped the usage into a Service Extension.
10+
<Steps>
11+
<Step title="Add the LLM service extension to your agent">
12+
Import the necessary components and add the LLM service extension to your agent function.
13+
</Step>
14+
15+
<Step title="Configure your LLM request">
16+
Specify which model your agent prefers and how you want to access it.
17+
</Step>
18+
19+
<Step title="Use the LLM in your agent">
20+
Access the optionally provided LLM configuration and use it with your preferred LLM client.
21+
</Step>
22+
</Steps>
1123

12-
<Tip>
24+
<Note>
1325
Service Extensions are a type of [A2A Extension](https://a2a-protocol.org/latest/topics/extensions/) that allows you to easily "inject dependencies" into your agent. This follows the inversion of control principle where your agent defines what it needs, and the platform (in this case, Agent Stack) is responsible for providing those dependencies.
14-
</Tip>
26+
</Note>
1527

1628
<Warning>
1729
Service extensions are optional by definition, so you should always check if they exist before using them.
1830
</Warning>
1931

20-
## Quickstart
2132

22-
<Steps>
23-
<Step title="Add LLM service extension to your agent">
24-
Import the necessary components and add the LLM service extension to your agent function.
25-
</Step>
33+
## Implementing Steps
2634

27-
<Step title="Configure your LLM request">
28-
Specify which model your agent prefer and how you want to access it.
29-
</Step>
35+
### 1. Add the LLM service extension to your agent
3036

31-
<Step title="Use the LLM in your agent">
32-
Access the optionally provided LLM configuration and use it with your preferred LLM client.
33-
</Step>
34-
</Steps>
37+
Import the `LLMServiceExtensionServer` and `LLMServiceExtensionSpec` from the SDK. You will use these within a type hint to let the platform know your agent requires LLM access.
38+
39+
```python
40+
from agentstack_sdk.a2a.extensions import LLMServiceExtensionServer, LLMServiceExtensionSpec
41+
from a2a.types import Message
42+
from typing import Annotated
43+
44+
# The extension is added as an Annotated parameter in your agent function
45+
async def my_agent(
46+
input: Message,
47+
llm: Annotated[
48+
LLMServiceExtensionServer, ...]
49+
):
50+
# agent logic
51+
pass
52+
```
53+
### 2. Configure your LLM request
54+
55+
Use `LLMServiceExtensionSpec.single_demand()` to request a model. By passing a suggested tuple, you tell the platform which model you'd prefer to use.
56+
57+
```python
58+
from agentstack_sdk.a2a.extensions import LLMServiceExtensionServer, LLMServiceExtensionSpec
59+
from a2a.types import Message
60+
from typing import Annotated
61+
62+
# The llm parameter is configured with a specific model demand
63+
async def my_agent(
64+
input: Message,
65+
llm: Annotated[
66+
LLMServiceExtensionServer,
67+
LLMServiceExtensionSpec.single_demand(suggested=("ibm/granite-3-3-8b-instruct",))
68+
]
69+
):
70+
# agent logic
71+
pass
72+
```
73+
74+
When you specify a suggested model like `"ibm/granite-3-3-8b-instruct"`the platform:
75+
76+
1. Checks if the requested model is available in your configured environment
77+
2. Allocates the best available model that matches your requirements
78+
3. Provides you with the exact model identifier and endpoint details
79+
80+
The platform handles the complexity of model provisioning and endpoint management, so you can focus on building your agent logic.
3581

36-
## Example of LLM Access
82+
### 3. Use the LLM in your agent
3783

38-
Here's how to add LLM inference capabilities to your agent:
84+
Once the platform provides the extension, you can extract the OpenAI-compatible configuration.
85+
86+
```python
87+
from typing import Annotated
88+
from a2a.utils.message import get_message_text
89+
from a2a.types import Message
90+
from agentstack_sdk.a2a.extensions import LLMServiceExtensionServer, LLMServiceExtensionSpec
91+
92+
async def my_agent(
93+
input: Message,
94+
llm: Annotated[
95+
LLMServiceExtensionServer,
96+
LLMServiceExtensionSpec.single_demand(suggested=("ibm/granite-3-3-8b-instruct",))
97+
]
98+
) -> None:
99+
# Verify that the optional extension was provided
100+
if llm and llm.data and llm.data.llm_fulfillments:
101+
user_message = get_message_text(input)
102+
103+
# Access the resolved LLM configuration
104+
llm_config = llm.data.llm_fulfillments.get("default")
105+
106+
if llm_config:
107+
# These credentials work with any OpenAI-compatible client library
108+
api_model = llm_config.api_model
109+
api_key = llm_config.api_key
110+
api_base = llm_config.api_base
111+
```
112+
The platform automatically provides you with:
113+
114+
- **`api_model`**: The specific model identifier that was allocated to your request
115+
- **`api_key`**: Authentication key for the LLM service
116+
- **`api_base`**: The base URL for the OpenAI-compatible API endpoint
117+
118+
These credentials work with any OpenAI-compatible client library, making it easy to integrate with popular frameworks like:
119+
- BeeAI Framework
120+
- LangChain
121+
- LlamaIndex
122+
- OpenAI Python client
123+
- Custom implementations
124+
125+
<Accordion title="Full Code Example">
126+
127+
This complete example shows how to receive a user message and respond using the credentials provided by the LLM Proxy Service:
39128

40129
```python
41130
import os
@@ -87,45 +176,4 @@ if __name__ == "__main__":
87176
run()
88177
```
89178

90-
## How to request LLM access
91-
92-
Here's what you need to know to add LLM inference capabilities to your agent:
93-
94-
**Import the extension**: Import `LLMServiceExtensionServer` and `LLMServiceExtensionSpec` from `agentstack_sdk.a2a.extensions`.
95-
96-
**Add the LLM parameter**: Add a third parameter to your agent function with the `Annotated` type hint for LLM access.
97-
98-
**Specify your model requirements**: Use `LLMServiceExtensionSpec.single_demand()` to request a single model (multiple models will be supported in the future).
99-
100-
**Suggest a preferred model**: Pass a tuple of suggested model names to help the platform choose the best available option.
101-
102-
**Check if the extension exists**: Always verify that the LLM extension is provided before using it, as service extensions are optional.
103-
104-
**Access LLM configuration**: Use `llm.data.llm_fulfillments.get("default")` to get the LLM configuration details.
105-
106-
**Use with your LLM client**: The platform provides `api_model`, `api_key`, and `api_base` that work with OpenAI-compatible clients.
107-
108-
## Understanding LLM Configuration
109-
110-
The platform automatically provides you with:
111-
112-
- **`api_model`**: The specific model identifier that was allocated to your request
113-
- **`api_key`**: Authentication key for the LLM service
114-
- **`api_base`**: The base URL for the OpenAI-compatible API endpoint
115-
116-
These credentials work with any OpenAI-compatible client library, making it easy to integrate with popular frameworks like:
117-
- BeeAI Framework
118-
- LangChain
119-
- LlamaIndex
120-
- OpenAI Python client
121-
- Custom implementations
122-
123-
## Model Selection
124-
125-
When you specify a suggested model like `"ibm/granite-3-3-8b-instruct"`, the platform will:
126-
127-
1. Check if the requested model is available in your configured environment
128-
2. Allocate the best available model that matches your requirements
129-
3. Provide you with the exact model identifier and endpoint details
130-
131-
The platform handles the complexity of model provisioning and endpoint management, so you can focus on building your agent logic.
179+
</Accordion>

0 commit comments

Comments
 (0)