Feature request: Add Avro serialization for Kafka keys and values

[plamailbox-beep]

Use Case

Our current telemetry pipeline is:

Telegraf -> Apache NiFi -> Kafka

Telegraf collects and enriches the metrics. Apache NiFi then serializes
the Kafka message key and value using Avro and publishes the messages to
Kafka.

The source systems and Kafka are located in different network zones.
Using Avro is important because it reduces the amount of data transferred
between these zones.

We want to remove Apache NiFi and use Telegraf as the only component in
the pipeline:

Telegraf -> Kafka

This would allow Telegraf to collect, enrich, serialize, and publish the
metrics directly to Kafka.

Expected behavior

The outputs.kafka plugin should support Avro serialization for both:

• the Kafka message key;
• the Kafka message value.

The serialized key and value should use binary Avro encoding and the
Confluent wire format.

The key and value should preferably support separate Avro schemas and
schema IDs, since they may use different schemas.

The Avro schema is defined once and reused for serialization.

A possible configuration could look like this:

[[outputs.kafka]]
  brokers = ["kafka:9092"]
  topic = "metrics"

  key_data_format = "avro"
  data_format = "avro"

  schema_registry_url = "https://schema-registry.example.com"

  key_schema_id = 10
  value_schema_id = 11

  # Optional TLS configuration
  # tls_ca = "/etc/telegraf/certs/ca.pem"
  # insecure_skip_verify = false

Telegraf should:

1. Read the configured schema ID for the Kafka key and value.
2. Check whether the corresponding schema is available in the local
   in-memory cache.
3. Fetch the schema from Schema Registry over HTTPS on a cache miss.
4. Store the retrieved schema in the cache.
5. Reuse the cached schema for subsequent messages.
6. Serialize the Kafka key and value using the corresponding schemas.
7. Publish the resulting Confluent-compatible payload to Kafka.

The exact configuration names are illustrative and can be adjusted to
match the existing Telegraf conventions.

Actual behavior

Telegraf currently supports Avro as an input data format, but it does
not support Avro serialization for Kafka output messages.

In particular, outputs.kafka cannot currently:

• serialize the Kafka message key as Avro;
• serialize the Kafka message value as Avro;
• produce Kafka key and value payloads in the Confluent wire format.

As a result, the current pipeline cannot be migrated completely to:

Telegraf -> Kafka

Using JSON or another text-based format instead of Avro would increase
the amount of data transferred between the network zones.

Additional info

The main goal is to remove Apache NiFi and reduce the operational and
network overhead of the current pipeline.

A minimal first implementation could support:

• the Kafka output plugin only;
• binary Avro encoding;
• user-provided schemas;
• separately configured key and value schema IDs;
• fetching existing schemas from Schema Registry over HTTPS;
• in-memory schema caching;
• Confluent wire format.

Schema registration, dynamic schema generation, and advanced schema
evolution support could be added later.

This request is related to issue #1630:

#1630

That issue requested Avro support for Kafka producer and consumer. The
current implementation provides Avro parsing, but Avro output
serialization is still required for this use case.

[srebhan]

@plamailbox-beep this is not Kafka specific. What we would need is an Avro serializer plugin. Happy to review a PR adding such a serializer!