Implement a dora-openai-websocket server #1093

haixuanTao · 2025-07-30T10:10:37Z

This PR introduces a dora-openai-websocket that makes it possible to connect an openai client through dora in realtime.

The server is able to spawn and connect to dataflows allowing to customize the dataflow from the user client.

Getting started

See README.md at examples/openai-realtime/README.md

phil-opp

Going through the CLI crate and opening a new connection to the coordinator seems a bit complicated, given that we already have a connection to a daemon through the node API. The daemon is already connected to the coordinator, so it could forward some start message to it. This way, we don't need to add the heavy CLI dependency and the code would work in distributed dataflows too.

So how about the following approach:

We add a new DaemonRequest::StartDataflow variant
- First field is dataflow, i.e. the path to the dataflow YAML file
- The second field is uv (optional)
- Third field is name (optional)
- No need for coordinator addr and port fields.
We add a new DoraNode::start_dataflow method that sends a DaemonRequest::StartDataflow request to the connected daemon
The daemon handles the StartDataflow request by reading the dataflow descriptor and session file and then sending a new CoordinatorRequest::StartDataflow request to the coordinator
The coordinator handles this CoordinatorRequest::StartDataflow request in a similar way as the ControlRequest::Start that is sent by the CLI

phil-opp · 2025-07-30T12:20:19Z

node-hub/dora-openai-websocket/src/main.rs

+    dora_cli::command::Command::Start(Start {
+        dataflow,
+        name: Some(node_id.to_string()),
+        coordinator_addr: IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)),
+        coordinator_port: DORA_COORDINATOR_PORT_CONTROL_DEFAULT,
+        attach: false,
+        detach: true,
+        hot_reload: false,
+        uv: true,
+    })
+    .execute()


This is a synchronous call, which might block the async task. This is not recommended since it blocks a full thread of the tokio runtime.

I agree. Although, this is more of a proof of concept. I think it's fine to just leave this sync call for now and rework this later.

phil-opp · 2025-07-30T12:22:18Z

node-hub/dora-openai-websocket/src/main.rs

+        coordinator_addr: IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)),
+        coordinator_port: DORA_COORDINATOR_PORT_CONTROL_DEFAULT,


So this only works when the node is running on the same machine than the coordinator. This seems quite limiting.

Agreed I will make this configurable.

phil-opp · 2025-07-30T13:19:11Z

Going through the CLI crate and opening a new connection to the coordinator seems a bit complicated, given that we already have a connection to a daemon through the node API. The daemon is already connected to the coordinator, so it could forward some start message to it.

The drawback of this approach is that this adds more functionality to the dora-node-api. So this only makes sense if we think that a functionality like this is common enough and also useful for other nodes.

This way, we don't need to add the heavy CLI dependency

An alternative way to make the CLI dependency less heavy could be to make the dora daemon/dora coordinator/dora runtime commands optional features. Nodes like this one don't require it, so this could make the dependency more lightweight.

haixuanTao · 2025-08-10T12:04:48Z

Going through the CLI crate and opening a new connection to the coordinator seems a bit complicated, given that we already have a connection to a daemon through the node API. The daemon is already connected to the coordinator, so it could forward some start message to it. This way, we don't need to add the heavy CLI dependency and the code would work in distributed dataflows too.

So how about the following approach:

We add a new DaemonRequest::StartDataflow variant

First field is dataflow, i.e. the path to the dataflow YAML file

The second field is uv (optional)

Third field is name (optional)

No need for coordinator addr and port fields.

We add a new DoraNode::start_dataflow method that sends a DaemonRequest::StartDataflow request to the connected daemon

The daemon handles the StartDataflow request by reading the dataflow descriptor and session file and then sending a new CoordinatorRequest::StartDataflow request to the coordinator

The coordinator handles this CoordinatorRequest::StartDataflow request in a similar way as the ControlRequest::Start that is sent by the CLI

So I understand where this is coming from and I agree that it is annoying to have to link the whole cli library.

I think the thing is that implementing what you're mentioning is a lot of work as we would need to reimplement the node - daemon connection which I don't want to do at this moment.

I would rather just keep this easy fix to just put those function public for now...

phil-opp · 2025-08-13T11:24:24Z

I'm fine with making the functions public for now, but if this functionality is something that will also be needed by other nodes, I'd prefer a proper solution. The main limitation I'd like to avoid is that the coordinator currently has to run on the same machine as the node. Going through the daemon instead of the CLI would enable this functionality also in distributed settings.

I think the thing is that implementing what you're mentioning is a lot of work as we would need to reimplement the node - daemon connection which I don't want to do at this moment.

I don't think that it's too much work. Let me try to draft a PR against your PR.

phil-opp · 2025-09-03T12:37:27Z

What's the status of this? How does this PR relate to #1122 ?

…ory footprint

Add HOST and PORT environment variable support with defaults: - HOST defaults to 0.0.0.0 (allow external connections) - PORT defaults to 8123 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

haixuanTao force-pushed the make-qwen-llm-configurable branch from 3f0d534 to fa888e8 Compare July 30, 2025 10:26

phil-opp reviewed Jul 30, 2025

View reviewed changes

phil-opp mentioned this pull request Aug 13, 2025

Add support for starting dataflows through API crate and daemon #1101

Closed

haixuanTao force-pushed the make-qwen-llm-configurable branch 2 times, most recently from afd751b to 1a626ad Compare August 29, 2025 09:25

phil-opp added the waiting-for-author The pull request requires adjustments by the PR author. label Sep 3, 2025

haixuanTao force-pushed the make-qwen-llm-configurable branch from 1df28d1 to 42a4c61 Compare September 10, 2025 19:17

This was referenced Sep 16, 2025

[DRAFT] Latest OpenAI websocket #1122

Closed

Making dora_cli::command public #1142

Open

haixuanTao added 17 commits September 17, 2025 15:28

make qwen model configurable

f93c9d3

Default to llama cpp

f7eddb2

Make whisper better by making it output punctuation

3484287

Making command public

d36159f

Add dora-openai-websocket

c35bc70

Add an example using whisper

b33950e

Minor fix and improvements

f48a342

Improvement of the overall language pipeline to be more resilient

d6edb85

make tracing optional

ab678cd

improve dataflow configuration for whisper

b18d816

Improved readme

456796e

add dora install within the readme

34233ce

Use quantized model instead of fp16 for faster response and lower mem…

78c2c7c

…ory footprint

Adding model selection

b8c714e

Add en-core-web-sm as kokoro depedency

1de576c

Add history to the conversation

ef9db39

Add speech started topic

c1edb29

haixuanTao and others added 18 commits September 17, 2025 15:29

Improve speech detection messages

ab0a29d

Improve chinese support

d96ce3a

Make server host and port configurable

09a6af4

Add HOST and PORT environment variable support with defaults: - HOST defaults to 0.0.0.0 (allow external connections) - PORT defaults to 8123 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Add dora-openai-websocket as build specification

e531c1b

Make openai websocket server a pip library

f5017d2

pushing whisper-1-template

167604a

Use pip instead of cargo

012facb

Fixing formatting

f4c08f3

Remove licensing

7b2aa7c

Accumulates audio in vad to not get over saturated by audio chunk

2d66081

Remove 24khz downsampling

3541f18

Fix template with tool call

8a0a6b6

Add max token

a7b21dd

Add better responsedone output

6e3fa62

fix whisper to support tool call

047a1ea

add qwen tooling

34e88db

fix tool call and remove tool call on function output

007af2f

Revert whisper logic

bc38f19

haixuanTao force-pushed the make-qwen-llm-configurable branch from 2c24307 to bc38f19 Compare September 17, 2025 07:29

haixuanTao added 6 commits September 17, 2025 15:32

use $ sign escaping in tool call

feb53e2

Readding ... for better improving whisper

fa192c3

Remove queue from whisper

aa0a60c

Adding system prompt into dataflow

38e8adf

Improve websocket support

20de6af

annotate item_id with unique value

5edad66

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement a dora-openai-websocket server #1093

Implement a dora-openai-websocket server #1093

Uh oh!

haixuanTao commented Jul 30, 2025

Uh oh!

phil-opp left a comment

Uh oh!

phil-opp Jul 30, 2025

Uh oh!

haixuanTao Aug 10, 2025

Uh oh!

phil-opp Jul 30, 2025

Uh oh!

haixuanTao Aug 10, 2025

Uh oh!

phil-opp commented Jul 30, 2025

Uh oh!

haixuanTao commented Aug 10, 2025

Uh oh!

phil-opp commented Aug 13, 2025

Uh oh!

phil-opp commented Sep 3, 2025

Uh oh!

Uh oh!

		coordinator_addr: IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)),
		coordinator_port: DORA_COORDINATOR_PORT_CONTROL_DEFAULT,

Implement a dora-openai-websocket server #1093

Are you sure you want to change the base?

Implement a dora-openai-websocket server #1093

Uh oh!

Conversation

haixuanTao commented Jul 30, 2025

Getting started

Uh oh!

phil-opp left a comment

Choose a reason for hiding this comment

Uh oh!

phil-opp Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

haixuanTao Aug 10, 2025

Choose a reason for hiding this comment

Uh oh!

phil-opp Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

haixuanTao Aug 10, 2025

Choose a reason for hiding this comment

Uh oh!

phil-opp commented Jul 30, 2025

Uh oh!

haixuanTao commented Aug 10, 2025

Uh oh!

phil-opp commented Aug 13, 2025

Uh oh!

phil-opp commented Sep 3, 2025

Uh oh!

Uh oh!