Integrations — Kafka

ChartVersion3.0.1TypeapplicationAppVersion4.1.0

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Integration overview

This page describes how to configure integrations with Apache Kafka installed via the tdp-kafka chart.
The chart deploys a Kafka cluster managed by the Strimzi operator and, optionally, Debezium connectors for real-time data capture (Change Data Capture — CDC).

For authentication and security configurations (LDAP, TLS, LDAP in Kafka UI), see Security — Kafka.

Besides CDC via Debezium, Kafka also integrates with three other platform components:

• Trino: see the Kafka connector to query topics via SQL.
• OpenMetadata: Kafka can be registered as a Messaging Service — see Integrations — OpenMetadata.
• Ranger: topic access policies can be centralized via rangerIntegrations.kafka — see Integrations — Ranger.

Kafka Connect and Debezium

The tdp-kafka chart supports the creation of Kafka Connect clusters with Debezium connectors for CDC from relational databases.
Debezium captures change events (INSERT, UPDATE, DELETE) from the database and publishes them to Kafka topics.

Supported database types

Type	Key in kafkaConnects	Database
PostgreSQL	kafkaConnects.postgres	PostgreSQL
MySQL	kafkaConnects.mysql	MySQL / MariaDB
SQL Server	kafkaConnects.sqlserver	Microsoft SQL Server
Oracle	kafkaConnects.oracle	Oracle Database

How to enable a connector

For each database you want to monitor:

1. Enable the connector type with kafkaConnects.<CONNECTOR_TYPE>.enabled: true.
2. Configure the Kafka Connect cluster under kafkaConnects.<CONNECTOR_TYPE>.connect.
3. Configure individual connectors under kafkaConnects.<CONNECTOR_TYPE>.connectors.
4. Set the database credentials using createSecret or secretName.

Internal Kafka cluster address

The bootstrapServers parameter defines the internal Kafka cluster address to which Kafka Connect connects.
When the tdp-kafka chart is installed with release name tdp-kafka in the same namespace, the default address is:

bootstrapServers: tdp-kafka-kafka-bootstrap:9092

If the release name or namespace differs, adjust the bootstrapServers value accordingly.

Credential management

The chart offers two ways to supply database credentials to the connector:

Option	Parameters	Recommendation
Secret created by chart	createSecret: true + user + password	For test environments only; avoid in production
Existing Secret	createSecret: false + secretName	Recommended for production

Do not version passwords in values.yaml

When using createSecret: true, the password is provided as a Helm parameter.
Never commit plain-text passwords in values.yaml files versioned in Git.
In production, prefer createSecret: false with a pre-existing Kubernetes Secret referenced via secretName.

Example: PostgreSQL connector with existing Secret

This example uses a pre-existing Kubernetes Secret for database credentials, avoiding plain-text passwords in values.yaml.

1. Create the Secret with database credentials:

Terminal input

kubectl -n <NAMESPACE> create secret generic <DEBEZIUM_SECRET_NAME> \
  --from-literal=username='<DATABASE_USER>' \
  --from-literal=password='<PASSWORD>'

2. Configure the connector in values.yaml:

kafkaConnects:
postgres:
  enabled: true
  connect:
    name: connect-debezium-postgres
    replicas: 1
    bootstrapServers: tdp-kafka-kafka-bootstrap:9092
    kafkaVersion: "4.1.0"
  connectors:
    appdb:
      enabled: true
      name: debezium-postgres-<APPLICATION_NAME>
      topicPrefix: "<TOPIC_PREFIX>"
      postgres:
        createSecret: false
        secretName: "<DEBEZIUM_SECRET_NAME>"
        host: "<DATABASE_HOST>.<DB_NAMESPACE>.svc.cluster.local"
        port: "5432"
        dbname: "<DATABASE_NAME>"

note

• Replace <TOPIC_PREFIX> with the desired prefix for Kafka topics generated by the connector (e.g., pgapp).
• The host field must reference the Kubernetes service of the database — typically in the format <SERVICE_NAME>.<NAMESPACE>.svc.cluster.local.
• kafkaVersion must be compatible with the installed Kafka cluster version (see appVersion in tdp-kafka's Chart.yaml).

Example: MySQL connector with existing Secret

kafkaConnects:
mysql:
  enabled: true
  connect:
    name: connect-debezium-mysql
    replicas: 1
    bootstrapServers: tdp-kafka-kafka-bootstrap:9092
    kafkaVersion: "4.1.0"
  connectors:
    appdb:
      enabled: true
      name: debezium-mysql-<APPLICATION_NAME>
      topicPrefix: "<TOPIC_PREFIX>"
      mysql:
        createSecret: false
        secretName: "<DEBEZIUM_SECRET_NAME>"
        host: "<DATABASE_HOST>.<DB_NAMESPACE>.svc.cluster.local"
        port: "3306"
        databaseIncludeList: "<DATABASE_NAME>"

Example: SQL Server connector with existing Secret

kafkaConnects:
sqlserver:
  enabled: true
  connect:
    name: connect-debezium-sqlserver
    replicas: 1
    bootstrapServers: tdp-kafka-kafka-bootstrap:9092
    kafkaVersion: "4.1.0"
  connectors:
    appdb:
      enabled: true
      name: debezium-sqlserver-<APPLICATION_NAME>
      topicPrefix: "<TOPIC_PREFIX>"
      sqlserver:
        createSecret: false
        secretName: "<DEBEZIUM_SECRET_NAME>"
        host: "<DATABASE_HOST>.<DB_NAMESPACE>.svc.cluster.local"
        port: "1433"
        databaseNames: "<DATABASE_NAME>"

Example: Oracle connector with existing Secret

kafkaConnects:
oracle:
  enabled: true
  connect:
    name: connect-debezium-oracle
    replicas: 1
    bootstrapServers: tdp-kafka-kafka-bootstrap:9092
    kafkaVersion: "4.1.0"
  connectors:
    appdb:
      enabled: true
      name: debezium-oracle-<APPLICATION_NAME>
      topicPrefix: "<TOPIC_PREFIX>"
      oracle:
        createSecret: false
        secretName: "<DEBEZIUM_SECRET_NAME>"
        host: "<DATABASE_HOST>.<DB_NAMESPACE>.svc.cluster.local"
        port: "1521"
        dbname: "<CDB_NAME>"
        pdbName: "<PDB_NAME>"

Connector validation

After installation, verify that the resources were created correctly:

Terminal input

kubectl get kafkaconnect -n <NAMESPACE>
kubectl get kafkaconnector -n <NAMESPACE>

To see details of a specific connector:

Terminal input

kubectl describe kafkaconnector <CONNECTOR_NAME> -n <NAMESPACE>

To view the Kafka Connect pod logs:

Terminal input

kubectl logs -n <NAMESPACE> -l strimzi.io/cluster=<CONNECT_NAME> --tail=100

Prerequisite: Strimzi CRDs

The KafkaConnect and KafkaConnector resources are Strimzi CRDs.
Make sure the tdp-crds chart is installed before enabling Debezium connectors.
See Installation via Helm for the chart installation order.