Skip to content

Commit 4242c88

Browse files
NiveditJaingemini-code-assist[bot]
NiveditJain
and
gemini-code-assist[bot]
authored
Adding inference APIs (#582)
* Update mkdocs configuration to include 'navigation.tabs' and restructure navigation hierarchy; add new 'inference.md' documentation for Inference APIs. * Update VSCode settings to enable auto import completions and set type checking mode; fix typo in inference documentation. * Update docs/docs/inference.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/docs/inference.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/docs/inference.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/docs/inference.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/docs/inference.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/docs/inference.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 parent 6f17d64 commit 4242c88

3 files changed

Lines changed: 154 additions & 26 deletions

File tree

‎.vscode/settings.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,5 +17,7 @@
1717
"state-manager"
1818
],
1919
"python.testing.unittestEnabled": false,
20-
"python.testing.pytestEnabled": true
20+
"python.testing.pytestEnabled": true,
21+
"cursorpyright.analysis.autoImportCompletions": true,
22+
"cursorpyright.analysis.typeCheckingMode": "basic"
2123
}

‎docs/docs/inference.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Inference APIs
2+
3+
## Overview
4+
Exosphere provides a single inference API that adapts to your volume, SLA, and cost constraints. Designed for reliable volume inference, it enables you to run inference on any number of tokens with automatic sharding, batching, rate limiting and cost management. All APIs are flexible SLA such that you can choose your SLA window and data volume to optimize cost and performance. Cost in percentage (calculated based on the combination of SLA and data volume) of base price could be up to 70% less.
5+
6+
## Infer APIs
7+
You can use the following inference APIs to run inference on your data:
8+
9+
### `POST /v0/infer/`
10+
This is the main API for running inference. This will send your data to the inference engine and start the inference process. Example request:
11+
```bash
12+
curl -X POST https://models.exosphere.host/v0/infer/ \
13+
-H "Content-Type: application/json" \
14+
-H "Authorization: Bearer <your-api-key>" \
15+
-d '[
16+
{
17+
"sla": 60, // minutes
18+
"model": "deepseek:r1-32b", // model name
19+
"input": "Hello model, how are you?"
20+
},
21+
{
22+
"sla": 1440, // 1 day
23+
"model": "openai:gpt-4o", // model name
24+
"input": "Hello OpenAI, how are you?"
25+
}
26+
]'
27+
```
28+
Example response:
29+
```json
30+
{
31+
"status": "submitted",
32+
"task_id": "2f92fc35-07d6-4737-aefa-8ddffd32f3fc",
33+
"total_items": 2,
34+
"objects":[
35+
{
36+
"status": "submitted",
37+
"usage": {
38+
"input_tokens": 10,
39+
"price_factor": 0.3
40+
},
41+
"object_id": "63bb0b28-edfe-4f5b-9e05-9232f63d76ec"
42+
},
43+
{
44+
"status": "submitted",
45+
"usage": {
46+
"input_tokens": 10,
47+
"price_factor": 0.5
48+
},
49+
"object_id": "88d68bc3-643c-4251-a003-6c2c14f76649"
50+
}
51+
]
52+
}
53+
```
54+
You can track the complete task status using the `GET /v0/infer/task/{task_id}` API or you can track individual object status using the `GET /v0/infer/object/{object_id}` API.
55+
56+
### `GET /v0/infer/task/{task_id}/`
57+
This API is used to track the complete task status. Example request:
58+
```bash
59+
curl -X GET https://models.exosphere.host/v0/infer/task/2f92fc35-07d6-4737-aefa-8ddffd32f3fc \
60+
-H "Authorization: Bearer <your-api-key>"
61+
```
62+
Example response:
63+
```json
64+
{
65+
"status": "submitted",
66+
"task_id": "2f92fc35-07d6-4737-aefa-8ddffd32f3fc",
67+
"total_items": 2,
68+
"objects": [
69+
{
70+
"object_id": "63bb0b28-edfe-4f5b-9e05-9232f63d76ec",
71+
"status": "completed",
72+
"usage": {
73+
"input_tokens": 10,
74+
"price_factor": 0.3,
75+
"output_tokens": 11
76+
},
77+
"output": {
78+
"type": "text",
79+
"text": "Hey user, I am doing great! How about you?"
80+
}
81+
},
82+
{
83+
"object_id": "88d68bc3-643c-4251-a003-6c2c14f76649",
84+
"status": "submitted",
85+
"usage": {
86+
"input_tokens": 10,
87+
"price_factor": 0.5
88+
}
89+
}
90+
]
91+
}
92+
```
93+
94+
### `GET /v0/infer/object/{object_id}/`
95+
This API is used to track the status of an individual object. Example request:
96+
```bash
97+
curl -X GET https://models.exosphere.host/v0/infer/object/63bb0b28-edfe-4f5b-9e05-9232f63d76ec \
98+
-H "Authorization: Bearer <your-api-key>"
99+
```
100+
Example response:
101+
```json
102+
{
103+
"object_id": "63bb0b28-edfe-4f5b-9e05-9232f63d76ec",
104+
"status": "completed",
105+
"usage": {
106+
"input_tokens": 10,
107+
"price_factor": 0.3,
108+
"output_tokens": 11
109+
},
110+
"error": null,
111+
"output": {
112+
"type": "text",
113+
"text": "Hey user, I am doing great! How about you?"
114+
}
115+
}
116+
```
117+
118+
- The output is only available when the item is completed. If no SLA is provided, the item will be completed immediately and the output will be available immediately.
119+
120+
- The price factor is a multiplier for the base price of the model. It is used to calculate the cost of the item.
121+
122+
- The object_id is a unique identifier for the object. It is used to track the status of the object. In case of any failure individual object status will be available to track the failure.
123+
124+
> **Note**: Auto retry policy will be triggered for transient failures without any additional cost.

‎docs/mkdocs.yml‎

Lines changed: 27 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ theme:
3232
# - navigation.instant.progress
3333
- navigation.prune
3434
# - navigation.sections
35-
# - navigation.tabs
35+
- navigation.tabs
3636
# - navigation.tabs.sticky
3737
# - navigation.top
3838
# - navigation.tracking
@@ -141,27 +141,29 @@ extra_css:
141141
- stylesheets/extra.css
142142

143143
nav:
144-
- Introduction: index.md
145-
- Getting Started: getting-started.md
146-
- Exosphere Concepts:
147-
- Overview: exosphere/concepts.md
148-
- Architecture: exosphere/architecture.md
149-
- Fanout: exosphere/fanout.md
150-
- Unite: exosphere/unite.md
151-
- Signals: exosphere/signals.md
152-
- Retry Policy: exosphere/retry-policy.md
153-
- Store: exosphere/store.md
154-
- Triggers: exosphere/triggers.md
155-
- Local Setup:
156-
- Overview: exosphere/local-setup.md
157-
- Docker Compose: docker-compose-setup.md
158-
- State Manager Setup: exosphere/state-manager-setup.md
159-
- Dashboard: exosphere/dashboard.md
160-
- Register Node: exosphere/register-node.md
161-
- Create Runtime: exosphere/create-runtime.md
162-
- Create Graph:
163-
- Overview: exosphere/create-graph.md
164-
- Components: exosphere/graph-components.md
165-
- Validation: exosphere/graph-validation.md
166-
- Python SDK: exosphere/python-sdk-graph.md
167-
- Trigger Graph: exosphere/trigger-graph.md
144+
- Orchestrator:
145+
- Introduction: index.md
146+
- Getting Started: getting-started.md
147+
- Exosphere Concepts:
148+
- Overview: exosphere/concepts.md
149+
- Architecture: exosphere/architecture.md
150+
- Fanout: exosphere/fanout.md
151+
- Unite: exosphere/unite.md
152+
- Signals: exosphere/signals.md
153+
- Retry Policy: exosphere/retry-policy.md
154+
- Store: exosphere/store.md
155+
- Triggers: exosphere/triggers.md
156+
- Local Setup:
157+
- Overview: exosphere/local-setup.md
158+
- Docker Compose: docker-compose-setup.md
159+
- State Manager Setup: exosphere/state-manager-setup.md
160+
- Dashboard: exosphere/dashboard.md
161+
- Register Node: exosphere/register-node.md
162+
- Create Runtime: exosphere/create-runtime.md
163+
- Create Graph:
164+
- Overview: exosphere/create-graph.md
165+
- Components: exosphere/graph-components.md
166+
- Validation: exosphere/graph-validation.md
167+
- Python SDK: exosphere/python-sdk-graph.md
168+
- Trigger Graph: exosphere/trigger-graph.md
169+
- Inference APIs: inference.md

0 commit comments

Comments
 (0)