Skip to content

Conversation

@tomhaynes
Copy link
Contributor

Bugfixes

  • select otel logging provider according to specified protocol
  • normalise otel paths correctly, for different telemetry types

Relevant issues

No ticket, but previously enabling otel logging caused an AttributeError because it was assuming the wrong logging provider.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • I have added a screenshot of my new test passing locally
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🐛 Bug Fix
✅ Test

Changes

@vercel
Copy link

vercel bot commented Oct 17, 2025

@tomhaynes is attempting to deploy a commit to the CLERKIEAI Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant
Copy link

CLAassistant commented Oct 17, 2025

CLA assistant check
All committers have signed the CLA.

@krrishdholakia
Copy link
Contributor

krrishdholakia commented Oct 20, 2025

hey @tomhaynes do you mind explaining the change you made here?

I don't follow what was causing the previous error - so it'll be helpful to understand the change

@tomhaynes
Copy link
Contributor Author

Hi sure @krrishdholakia, there were two issues I was bumping into. I'm looking to forward logs to an otel-collector, I initially followed these docs https://docs.litellm.ai/docs/proxy/logging#opentelemetry

Initially there were just no logs generated (metrics / traces come through fine). I found a change from last month that seemed to require LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS to be enabled - when I did this I got the following stacktrace:

Exception while exporting logs.
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/opentelemetry/sdk/_logs/_internal/export/__init__.py", line 311, in _export_batch
    self._exporter.export(self._log_records[:idx])  # type: ignore
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/grpc/_log_exporter/__init__.py", line 108, in export
    return self._export(batch)
           ~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 263, in _export
    request=self._translate_data(data),
            ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/grpc/_log_exporter/__init__.py", line 105, in _translate_data
    return encode_logs(data)
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/_log_encoder/__init__.py", line 38, in encode_logs
    return ExportLogsServiceRequest(resource_logs=_encode_resource_logs(batch))
                                                  ~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/_log_encoder/__init__.py", line 60, in _encode_resource_logs
    sdk_resource = sdk_log.log_record.resource
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LogRecord' object has no attribute 'resource'

This sounded similar to this issue open-telemetry/opentelemetry-python#3552 (comment) that this LogRecord object should not be instantiated in this way. I also noticed that the code hardcodes the grpc exporter rather than setting it dynamically based on protocol (although the attribute error manifests with either grpc/http). I tweaked the import to import LogRecord as SdkLogRecord to avoid conflict with native python LogRecord.

The second issue I saw was similar to this #13942 - regarding the endpoint configuration, specifically the path appended to it (/v1/traces or /v1/logs etc) - my changes to normalise the endpoint for each telemetry type should resolve this issue also, although they argue that we should be using the dedicated endpoint config for each OTEL_EXPORTER_OTLP_TRACES_ENDPOINT etc. - this arguably would be slightly better, but my normalisation logic works as long as all telemetry is expected to go to the same collector.

Thanks to @mubashir1osmani who was helping me investigate this, and has confirmed that other customers are also failing to get otel logging through.

@krrishdholakia krrishdholakia merged commit 98f1d63 into BerriAI:main Oct 21, 2025
4 of 6 checks passed
@krrishdholakia
Copy link
Contributor

Thanks for explaining that!

@mubashir1osmani please feel free to submit a PR to update any relevant documentation on our end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants