Skip to content

Add support for configurable ONNX SessionOptions across feature extraction and inference modules#73

Open
dalgwen wants to merge 1 commit into
livekit:mainfrom
dalgwen:sess_options_add
Open

Add support for configurable ONNX SessionOptions across feature extraction and inference modules#73
dalgwen wants to merge 1 commit into
livekit:mainfrom
dalgwen:sess_options_add

Conversation

@dalgwen

@dalgwen dalgwen commented May 12, 2026

Copy link
Copy Markdown

Allows, for example, to set up more conservative CPU usage like this:

    import onnxruntime as ort
    opts = ort.SessionOptions()
    opts.intra_op_num_threads = 1
    opts.inter_op_num_threads = 1
    opts.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
    opts.add_session_config_entry("session.intra_op.allow_spinning", "0")
    opts.add_session_config_entry("session.inter_op.allow_spinning", "0")

    model = WakeWordModel(models=model_files, sess_options=opts)

These example session options drastically reduce CU consumption by mono-threading everything and using sleep instead of spinning/busy waiting.

Suitable for a 'softer' background process in a small environment. Cost is theoretically higher latency (not humanly noticeable).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant