[8.19] (backport #10819) Always run the otel collector self-monitoring input in the otel runtime

[mergify]

What does this PR do?

Makes the prometheus metrics input we use to collect otel collector metrics always run inside said otel collector, even if the user configures self-monitoring to run in beats processes.

Why is it important?

This input can't run in a beats process because it requires an environment variable which we only set for the otel collector. Fixing this properly would require significant additional work, and we're planning to collect these metrics in a much more robust way in 9.3.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in ./changelog/fragments using the changelog tool
  - [ ] I have added an integration test or an E2E test

How to test this PR locally

Build the agent locally and run it with self-monitoring in process mode and an input in otel mode. For example:

agent:
  logging:
    to_stderr: true
    level: debug
  monitoring:
    _runtime_experimental: process
inputs:
- data_stream:
    namespace: default
  id: unique-system-metrics-input
  streams:
  - data_stream:
      dataset: system.cpu
    metricsets:
    - cpu
  type: system/metrics
  use_output: default
  _runtime_experimental: otel
outputs:
  default:
    username: elastic
    password: xxxxx
    hosts:
    - 127.0.0.1:9200
    type: elasticsearch

Then check status. There should be a prometheus/metrics-monitoring component running as a beat receiver:

{
    "id": "prometheus/metrics-monitoring",
    "name": "prometheus/metrics",
    "state": 2,
    "message": "HEALTHY",
    "units": [
        {
            "unit_id": "prometheus/metrics-monitoring",
            "unit_type": 1,
            "state": 2,
            "message": "Healthy"
        },
        {
            "unit_id": "prometheus/metrics-monitoring-metrics-monitoring-collector",
            "unit_type": 0,
            "state": 2,
            "message": "Healthy",
            "payload": {
                "streams": {
                    "metrics-monitoring-collector": {
                        "error": "",
                        "status": "HEALTHY"
                    }
                }
            }
        }
    ],
    "version_info": {
        "name": "beats-receiver",
        "meta": {
            "build_time": "2025-10-24 17:36:57 +0000 UTC",
            "commit": "66155e4093dcf38ab0e446f83f40ca449c73eb10"
        }
    }
}

Related issues

• Closes https://github.com/elastic/ingest-dev/issues/6295

---

This is an automatic backport of pull request #10819 done by [Mergify](https://mergify.com).

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 8a5bf22

Failed CI Steps

• Unit tests - Windows 10

cc @swiatekm