Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 90 additions & 0 deletions docs/build/private-llms/usage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,96 @@ Once you have [nilAI API access](/build/network-api-access), you can start using
## Getting Started with Private LLMs

<Tabs>
<TabItem value="nucs-flow" label="NUCS" >
## Getting Started with Private LLMs

<Tabs>

<TabItem value="python" label="Python" default>
<Tabs>
<TabItem value="pip" label="pip" default>
```bash
pip install nilai-py
```
</TabItem>
<TabItem value="uv" label="uv">
```bash
uv pip install nilai-py
```
</TabItem>
</Tabs>
</TabItem>
<TabItem value="typescript" label="TypeScript">
```bash
pnpm install @nillion/nilai-ts
```
</TabItem>

</Tabs>

You can either use:
- `API Key` flow as the sole developer / organization or
- `Delegation Flow` to provide permissions to another user / organization.

### API Key Flow

1. Use `https://nilai-a779.nillion.network/nuc/v1` as the BASE URL
2. Check [available models](/build/private-llms/overview#available-models) or query the [`/v1/models`](/api/nilai/get-models-v-1-models-get) endpoint or
3. Select an available model and use it with the [`/v1/chat/completions`](/api/nilai/chat-completion-v-1-chat-completions-post) nilAI node endpoint

With OpenAI compatibility, you can use any OpenAI library. Here's an example for querying the `Llama-3.1-8B` model:

<Tabs>
<TabItem value="python" label="Python">
```python reference showGithubLink
https://github.com/NillionNetwork/nilai-py/blob/main/examples/0-api_key_mode.py
```
</TabItem>
<TabItem value="typescript" label="Typescript">
```typescript
import "dotenv/config";
import { NilaiOpenAIClient, NilAuthInstance } from "@nillion/nilai-ts";

// To obtain an API key, navigate to https://subscription.nillion.com
// and create a new subscription.
// The API key will be displayed in the subscription details.
// The NilaiOpenAIClient class automatically handles the NUC token creation and management.

const API_KEY = process.env.NILLION_API_KEY;

async function main() {
// Initialize the client in API key mode
// For sandbox, use the following:
const client = new NilaiOpenAIClient({
baseURL: "https://nilai-a779.nillion.network/v1/",
apiKey: API_KEY,
nilauthInstance: NilAuthInstance.SANDBOX,
// For production, use the following:
// nilauthInstance: NilAuthInstance.PRODUCTION,
});

// Make a request to the Nilai API
const response = await client.chat.completions.create({
model: "google/gemma-3-27b-it",
messages: [
{ role: "user", content: "Hello! Can you help me with something?" }
],
});

console.log(`Response: ${response.choices[0].message.content}`);
}

// Run the example
main().catch(console.error);
```
</TabItem>
</Tabs>

### Delegation flow

To use the delegation flow, you need to create a delegation token server.

The server then creates the delegation tokens and managing their expiration and usage. Then the delegation token allows you to make requests to the nilAI API.

<TabItem value="python" label="Python" default>
<Tabs>
Expand Down