Firebolt Kafka Sink Connector

A Kafka Connect Sink Connector that delivers data from Apache Kafka topics into Firebolt tables.

Requirements

• Apache Kafka Connect 3.2 or later
• Java 11 or later
• Firebolt account with a service account (client ID and secret)

Installation

Download the latest release artifacts from GitHub Releases.

Each release includes:

Artifact	Description
firebolt-kafka-connector-<version>.jar	Uber JAR with all dependencies
firebolt-db-firebolt-kafka-connect-<version>.zip	Confluent Hub archive (for confluent-hub CLI)
firebolt-kafka-connector-<version>-sources.jar	Source code

Option A: Manual JAR install

1. Download firebolt-kafka-connector-<version>.jar from Releases.
2. Copy the JAR into your Kafka Connect plugin directory:

   mkdir -p /path/to/kafka-connect/plugins/firebolt-kafka-connector
   cp firebolt-kafka-connector-*.jar /path/to/kafka-connect/plugins/firebolt-kafka-connector/

3. Ensure plugin.path in your Kafka Connect worker configuration includes the plugins directory:

   plugin.path=/path/to/kafka-connect/plugins

4. Restart Kafka Connect workers.

Option B: Confluent Hub CLI

1. Download firebolt-db-firebolt-kafka-connect-<version>.zip from Releases.
2. Install using the Confluent Hub CLI:

   confluent-hub install --no-prompt firebolt-db-firebolt-kafka-connect-<version>.zip

3. Restart Kafka Connect workers.

Option C: Docker

Mount the uber JAR into your Kafka Connect container's plugin directory:

services:
  kafka-connect:
    image: confluentinc/cp-kafka-connect:latest
    volumes:
      - ./firebolt-kafka-connector-<version>.jar:/usr/share/java/firebolt-kafka-connector/firebolt-kafka-connector.jar
    environment:
      CONNECT_PLUGIN_PATH: /usr/share/java

Confluent Cloud

The connector is available as a verified plugin on Confluent Hub. See the Firebolt documentation for Confluent Cloud setup instructions.

Configuration

Create a connector using the Kafka Connect REST API:

curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d '{
    "name": "firebolt-sink",
    "config": {
      "connector.class": "com.firebolt.kafka.connect.FireboltSinkConnector",
      "tasks.max": "1",
      "topics": "my-topic",
      "topic.to.table.mapping": "my-topic:my_table",
      "jdbc.connection.url": "jdbc:firebolt:my_database?engine=my_engine&account=my_account",
      "firebolt.clientId": "<your-service-account-client-id>",
      "firebolt.clientSecret": "<your-service-account-client-secret>",
      "key.converter": "org.apache.kafka.connect.storage.StringConverter",
      "value.converter": "org.apache.kafka.connect.json.JsonConverter",
      "value.converter.schemas.enable": "false"
    }
  }'

Configuration Properties

Property	Required	Default	Description
connector.class	Yes		Must be com.firebolt.kafka.connect.FireboltSinkConnector
topics	Yes		Comma-separated list of Kafka topics to consume
jdbc.connection.url	Yes		Firebolt JDBC URL (e.g., jdbc:firebolt:my_db?engine=my_engine&account=my_account)
firebolt.clientId	Yes		Firebolt service account client ID
firebolt.clientSecret	Yes		Firebolt service account client secret
topic.to.table.mapping	Yes		Comma-separated mapping of topics to Firebolt tables (e.g., topic1:table1,topic2:table2)
tasks.max	No	1	Maximum number of tasks
ingestion.type	No	sql	Ingestion mode: sql (INSERT via SQL) or binary (Parquet upload)
exactlyOnce	No	false	Enable exactly-once delivery semantics
errors.tolerance	No	none	Error tolerance: none (fail on error) or all (skip and report to DLQ)
post.processing.script	No		Optional post-processing SQL to run after each batch (JSON format)

Building from Source

Prerequisites

• Java 11 or higher

Build

./gradlew build

Run tests

./gradlew test

Build Confluent Hub archive

./gradlew buildConfluentHubArchive

The archive will be at build/confluent/firebolt-db-firebolt-kafka-connect-<version>.zip.

License

Apache License 2.0 -- see LICENSE for details.