Kafka Configuration

ChartVersion3.0.1TypeapplicationAppVersion4.1.0

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Feature SupportLDAPsupportedIngresssupportedGateway APIsupported

The tdp-kafka chart deploys an Apache Kafka cluster managed by the Strimzi operator, as well as a web interface (Kafka UI) and support for Debezium connectors via Kafka Connect.

Overview

Property	Value
Chart	tdp-kafka
Kafka	4.1.0
Chart Version	3.0.1

Related Pages

• Integrations — Kafka — Kafka Connect, Debezium, and client integration
• Security — Kafka — users, authentication, TLS, and policies
• External exposure: Ingress and Gateway API — HTTP/HTTPS Kafka UI exposure

What is Strimzi?

Strimzi is a Kubernetes operator specialized in the Kafka cluster lifecycle. Instead of managing Kafka processes manually, Strimzi translates native Kubernetes resources (like KafkaNodePool and Kafka) into declaratively configured, monitored, and updated brokers.

The 4.1.0 version of the tdp-kafka chart operates in KRaft mode (Kafka without ZooKeeper). In Kafka 4.x the ZooKeeper mode was removed, so metadata management is handled by Kafka itself via KRaft, which simplifies the topology and reduces latency of metadata operations.

To find out more

See Apache Kafka — Concepts for a complete overview of the tool, its architecture and operation.

Deployed components

Component	Description
Kafka Cluster	Brokers and controllers managed by Strimzi via KafkaNodePool
Entity Operator	Manages KafkaTopic and KafkaUser as Kubernetes resources
Kafka UI	Web interface for monitoring, topic and message navigation
Kafka Connect	Infrastructure for Debezium connectors (CDC) — optional

Values structure (Helm)

The tdp-kafka chart organizes configuration in the blocks below:

• Kafka cluster root blocks — name, kafkaVersion, metadataVersion, clusterLabel, listeners.*, security.*, config.*, entityOperator.* and nodePool.*.
• tdp-kafka: — chart enablement.
• kafka-ui: — web UI configuration, authentication, RBAC, cluster connection and Service.
• kafkaConnects: — optional Debezium connectors for databases such as PostgreSQL, MySQL, SQL Server and Oracle.
• registryPushSecret: — optional Secret for connector image push.
• TDP-Settings:, ingress: and gatewayApi: — Kafka UI exposure control through Ingress or Gateway API.

name: tdp-kafka
kafkaVersion: 4.1.0
metadataVersion: 4.1-IV1

listeners:
plain:
  enabled: true
tls:
  enabled: true

nodePool:
name: brokers
replicas: 4

tdp-kafka:
enabled: true

kafka-ui:
enabled: true

kafkaConnects:
postgres:
  enabled: false

TDP-Settings:
gateway:
  ingress:
    enabled: false
  gatewayApi:
    enabled: false

Prerequisites

• Kubernetes 1.32+, Red Hat OpenShift 4.19+, or Rancher Manager 2.10.x+.
• Helm 3.2.0+.
• StorageClass available with sufficient capacity for broker volumes.
• Strimzi CRDs installed in the cluster (see section below).

Strimzi CRD Management

Strimzi Custom Resource Definitions (CRDs) — such as Kafka, KafkaNodePool, KafkaTopic, KafkaUser, and KafkaConnect — are managed separately from the tdp-kafka chart. In TDP Kubernetes, these CRDs are delivered by the tdp-crds chart.

CRDs are cluster-scoped resources and are not removed when uninstalling the tdp-kafka chart.

Installing CRDs before the first Kafka installation

Before installing the tdp-kafka chart for the first time, install the tdp-crds chart in the cluster:

Terminal input

helm upgrade --install tdp-crds \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-crds \
  -n <NAMESPACE> --create-namespace

Strimzi version update

When updating the TDP version, update the tdp-crds chart before running helm upgrade for tdp-kafka.

Troubleshooting: CRD conflict

If helm install or helm upgrade returns an error similar to:

conflict with "helm" using apiextensions.k8s.io/v1: .spec.versions

confirm that the tdp-crds chart has already been updated to the expected version, then repeat the tdp-kafka installation or upgrade.

Installation

Installation (OCI)

For those who install or operate the cluster, the command below is the first declarative step: Helm applies the tdp-kafka chart in the indicated namespace and Strimzi starts to reconcile the desired state (brokers, Entity Operator, UI and, if enabled, Connect). Behavior adjustments — capacity, listeners, replication, passwords — come later, via values.yaml or helm upgrade with the same release.

Terminal input

helm upgrade --install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka -n <NAMESPACE> --create-namespace --timeout 10m

Post-installation validation

Terminal input

# Kafka cluster status (Strimzi resource)
kubectl get kafka -n <NAMESPACE>

# Cluster Node Pools
kubectl get kafkanodepools -n <NAMESPACE>

# Pods managed by Strimzi
kubectl get pods -n <NAMESPACE> -l strimzi.io/cluster=<KAFKA_CLUSTER_NAME>

# Services created (bootstrap, brokers, UI)
kubectl get svc -n <NAMESPACE> | grep kafka

# KafkaConnect instances (Debezium)
kubectl get kafkaconnect -n <NAMESPACE>

# Registered connectors
kubectl get kafkaconnector -n <NAMESPACE>

Main parameters

Main parameters

The following table summarizes keys that are typically reviewed at first installation and upgrades: cluster name, KRaft versions, listeners, replication factors, Entity Operator, Node Pool sizing and Kafka UI options. For exact values of all chart keys, use helm show values in the version of the package you are deploying; here the focus is what each parameter controls in practice for those who operate the release.

Parameter	Description	Default value
name	Kafka cluster name	tdp-kafka
kafkaVersion	Kafka version	4.1.0
metadataVersion	KRaft metadata version	4.1-IV1
clusterLabel	Cluster label	tdp
listeners.plain.enabled	Plain listener enabled	true
listeners.plain.port	Plain listener port	9092
listeners.tls.enabled	TLS listener enabled	true
listeners.tls.port	TLS listener port	9093
config.default.replication.factor	Default topic replication factor	3
config.min.insync.replicas	Minimum synchronous replicas	2
entityOperator.enabled	Entity Operator enabled	true
nodePool.replicas	Number of brokers/controllers	4
nodePool.storage.volumes[0].size	Volume size per node	10Gi
kafka-ui.enabled	Kafka UI enabled	true
kafka-ui.service.type	UI Service Type	ClusterIP
kafka-ui.service.nodePort	NodePort for the UI (when type: NodePort)	30081
config.offsets.topic.replication.factor	Offsets topic replication factor	3
config.transaction.state.log.replication.factor	Transaction log replication factor	3
nodePool.roles	Node roles (controller/broker)	["controller","broker"]
nodePool.storage.type	Storage type	jbod
security.enabled	Enable broker authentication and ACLs	false
security.authentication.type	Strimzi authentication type	scram-sha-512
security.authorization.type	Strimzi authorization type	simple
kafka-ui.connection.bootstrapServers	Kafka UI bootstrap server	tdp-kafka-kafka-bootstrap:9092
kafka-ui.applicationConfig.auth.type	UI authentication type	LOGIN_FORM
kafka-ui.rbac.enabled	Enable Kafka UI RBAC	false
ingress.annotations	Ingress annotations	{}
TDP-Settings.gateway.ingress.enabled	Enables Ingress for Kafka UI	false
TDP-Settings.gateway.gatewayApi.enabled	Enables Gateway API (HTTPRoute) for Kafka UI	false
ingress.ingressClassName	IngressClass to use	nginx
ingress.hosts[0].host	Ingress hostname	kafka-ui.tdp.local
gatewayApi.kafkaUi.enabled	Enables HTTPRoute for Kafka UI	true
gatewayApi.kafkaUi.hostnames[0]	HTTPRoute hostname	kafka-ui.tdp.local
nodePool.name	Node Pool name	brokers
security.listener	Listener protected by authentication	plain
security.users	Declarative list of KafkaUser with ACLs	[]
kafka-ui.service.port	Kafka UI service port	80
kafka-ui.connection.authentication.existingSecret.name	Secret with the Kafka UI technical principal password	""
kafka-ui.rbac.roles	Kafka UI RBAC roles	[]
config.transaction.state.log.min.isr	Minimum ISR for the transaction log	2

Configuration details

Kafka cluster configuration

Node Pool

Node Pools define how many nodes there are, what roles each plays (KRaft metadata and/or data), and how storage behaves when scaling or removing the resource. In small environments, combining controller and broker in the same pool simplifies the operation; As load or isolation requirements grow, separating pools prevents metadata and data I/O operations from competing for the same resources. The choices below directly impact node cost, disk capacity, and what happens to PVCs when the release is removed.

In Strimzi with KRaft, cluster nodes are declared in Node Pools (KafkaNodePool). Each pool defines the number of replicas, the roles played by the nodes and the storage.

nodePool:
  name: brokers
  replicas: 4
  roles: ["controller", "broker"]
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: true
        kraftMetadata: shared

Field	Description
roles	A node can be controller, broker or both. In small clusters, combine the two roles. In large clusters, separate them into distinct pools
storage.type: jbod	JBOD (Just a Bunch of Disks) allows multiple volumes per node, maximizing I/O throughput
deleteClaim: true	PVCs will be deleted when removing the NodePool. Set false in production if you want to preserve data
kraftMetadata: shared	Indicates that this volume also stores KRaft metadata

Production sizing

With 4 replicas combining controller and broker, KRaft quorum and data replication coexist on the same nodes. For heavy loads, consider separate pools: a pool of controllers (3 replicas) and a pool of brokers (as many as needed).

Version and metadata

name: tdp-kafka
kafkaVersion: 4.1.0
metadataVersion: 4.1-IV1
clusterLabel: "tdp"

The metadataVersion field is the KRaft metadata version used by the cluster. Keep it aligned to kafkaVersion during fresh installs. In updates, increase only after all brokers are on the new version.

Listeners

Listeners define the cluster entry points for producers and consumers.

Keeping plain and tls enabled is common: plain simplifies internal clients on the same cluster network (less TLS overhead), while TLS addresses scenarios where security policy or the network requires encryption in transit. The practical decision is which bootstrap and port the applications will use; mixing without criteria can generate inconsistent configuration between services. The bootstrap addresses listed below are the ones you pass on to the application teams.

listeners:
  plain:
    enabled: true
    port: 9092
  tls:
    enabled: true
    port: 9093

Listener	Port	When to use
plain	9092	Communication within the cluster, development environments, or trusted private networks
tls	9093	Communicating over untrusted networks or when clients require encryption in transit

The bootstrap service name follows the Strimzi convention: <CLUSTER_NAME>-kafka-bootstrap. Example with default name tdp-kafka:

• Plain: tdp-kafka-kafka-bootstrap:9092
• TLS: tdp-kafka-kafka-bootstrap:9093

Replication and durability

The default values below prioritize durability (more copies, min.insync.replicas aligned to acks=all on well-configured clients) over tolerating more simultaneous failures without tuning. In return, clusters with few brokers cannot honor replication factors greater than the number of nodes — hence the need to align these numbers with the actual topology before going to production.

config:
  offsets.topic.replication.factor: 3
  transaction.state.log.replication.factor: 3
  transaction.state.log.min.isr: 2
  default.replication.factor: 3
  min.insync.replicas: 2

These parameters determine the balance between durability and availability:

Parameter	Meaning
default.replication.factor: 3	Each topic will have 3 copies distributed in different brokers
min.insync.replicas: 2	A producer with acks=all only receives confirmation when at least 2 replicas are synchronized. Tolerates the loss of 1 broker without data loss
offsets.topic.replication.factor: 3	Controls the durability of consumer offsets (internal topic __consumer_offsets)
transaction.state.log.*	Transaction log durability (transactional production / exactly-once)

Smaller clusters

In environments with less than 3 brokers, reduce *.replication.factor and min.insync.replicas to the actual number of nodes. Maintaining factors greater than the number of brokers prevents the creation of topics.

Entity Operator

entityOperator:
  enabled: true

The Entity Operator is a Strimzi component that looks at KafkaTopic and KafkaUser resources in the namespace and reconciles them with the cluster. With it enabled, topics and users can be managed declaratively as Kubernetes manifests, without direct access to the Kafka administrative API.

Kafka UI

Kafka UI provides a graphical interface for inspecting topics, viewing messages, monitoring consumer groups, and tracking basic cluster metrics.

Treat the UI as an operational and diagnostic tool, not as a replacement for Kafka's own access controls: it exposes insight into the cluster according to the web application's authentication configuration. The following example uses PLAINTEXT in relation to the internal bootstrap — consistent with cluster networking, but an additional warning not to expose the interface without a strong password, adequate networking and, when necessary, the mechanisms described in the security page.

kafka-ui:
  enabled: true
  applicationConfig:
    auth:
      type: LOGIN_FORM
    spring:
      security:
        user:
          name: admin
          password: <PASSWORD>
  connection:
    clusterName: TDP
    bootstrapServers: <CLUSTER_NAME>-kafka-bootstrap:9092
  service:
    type: ClusterIP
    port: 80
    nodePort: 30081

Replace <CLUSTER_NAME> with the cluster name value (e.g. tdp-kafka) and <PASSWORD> with a strong password.
The UI connection to the Kafka cluster is configured via kafka-ui.connection.bootstrapServers.
For LDAP or SCRAM broker authentication, see Security — Kafka.

Internal configuration: yamlApplicationConfigConfigMap

The chart automatically generates a ConfigMap (tdp-kafka-ui-config) from the values in kafka-ui.applicationConfig. The kafka-ui.yamlApplicationConfig field remains null so that applicationConfig takes precedence. No manual changes to these fields are required.

Default password

The admin password (chart default) should not be kept in accessible environments outside the internal network. Swap before exposing the interface.

Kafka Connect / Debezium

The chart supports Debezium connectors for real-time Change Data Capture (CDC).
Configuration is done under the kafkaConnects.* key.

Supported types: postgres, mysql, sqlserver, oracle.

For each type, you can:

• Auto-generate the Secret with credentials: createSecret: true
• Reference an existing Secret: createSecret: false + secretName

Minimal example for PostgreSQL using an existing Secret:

kafkaConnects:
postgres:
  enabled: true
  connect:
    name: connect-debezium-postgres
    replicas: 1
    bootstrapServers: tdp-kafka-kafka-bootstrap:9092
    kafkaVersion: "4.1.0"
  connectors:
    appdb:
      enabled: true
      name: debezium-postgres-appdb
      topicPrefix: "pgapp"
      postgres:
        createSecret: false
        secretName: "db-pg-appdb"
        host: "tdp-postgresql-project.<NAMESPACE>.svc.cluster.local"
        port: "5432"
        dbname: "appdb"
        user: "debezium"

The topicPrefix field defines the prefix of Kafka topics created by the connector (e.g.: pgapp.public.table).

Strimzi creates a KafkaConnect and a KafkaConnector resource for each enabled connector.
Check the status with:

Terminal input

kubectl get kafkaconnect -n <NAMESPACE>
kubectl get kafkaconnector -n <NAMESPACE>

Detailed documentation

See Integrations — Kafka for complete examples of each connector type (PostgreSQL, MySQL, SQL Server, Oracle) and credential management.

Access & Security

Access

ClusterIP / Port-forward (Kafka UI)

The Kafka UI service uses the ClusterIP type by default.
To access locally without exposing the port externally:

Terminal input

kubectl port-forward svc/<RELEASE_NAME>-kafka-ui 8080:80 -n <NAMESPACE>

Access at http://localhost:8080.

Example

Example

kubectl port-forward svc/tdp-kafka-kafka-ui 8080:80 -n tdp-project

NodePort (Kafka UI)

To expose via NodePort, set kafka-ui.service.type: NodePort in the values.
The default port is 30081, configurable via kafka-ui.service.nodePort.

Terminal input

kubectl get nodes -o wide

Access at http://<NODE_IP>:30081.

Ingress (Kafka UI)

To expose the Kafka UI through Ingress, set TDP-Settings.gateway.ingress.enabled: true and configure the ingress.* block with the IngressClass, hostname, and paths.
TDP-Settings.gateway.ingress.enabled and TDP-Settings.gateway.gatewayApi.enabled are mutually exclusive — enable only one.

Kafka UI ≠ Kafka brokers

Ingress publishes only the Kafka UI web interface.
External producers and consumers still require additional configuration to access Kafka brokers (listeners).

See External Exposure — Kafka UI for the complete YAML example, IngressClass options (including OpenShift), ingress.tls, and validation.

Gateway API (Kafka UI)

To expose the Kafka UI through the Kubernetes Gateway API, set TDP-Settings.gateway.gatewayApi.enabled: true and configure the gatewayApi.* block.
The chart automatically creates the Gateway and HTTPRoute resources when enabled.

Kafka UI ≠ Kafka brokers

Gateway API publishes only the Kafka UI web interface.
External producers and consumers still require additional configuration to access Kafka brokers (listeners).

See External Exposure — Kafka UI for the complete YAML example, gatewayClassName, listeners, HTTPRoute, /etc/hosts, and validation.

Bootstrap for producers and consumers

The bootstrap address follows the Strimzi convention: <KAFKA_CLUSTER_NAME>-kafka-bootstrap.<NAMESPACE>.svc.cluster.local.

With the default name: tdp-kafka:

# Plain (PLAINTEXT) — default port 9092
tdp-kafka-kafka-bootstrap:9092

# TLS — default port 9093
tdp-kafka-kafka-bootstrap:9093

UI Authentication

Detailed documentation

See Security — Kafka to configure LOGIN_FORM or LDAP authentication in the web interface.

Troubleshooting

Troubleshooting

Terminal input

# Kafka cluster details and events
kubectl describe kafka <KAFKA_CLUSTER_NAME> -n <NAMESPACE>

# Broker logs
kubectl logs -n <NAMESPACE> \
  -l strimzi.io/cluster=<KAFKA_CLUSTER_NAME> \
  -c kafka

# Kafka UI logs
kubectl logs -n <NAMESPACE> -l app.kubernetes.io/name=kafka-ui

# Connector details
kubectl describe kafkaconnector <CONNECTOR_NAME> -n <NAMESPACE>

Common issues

Issue	Likely cause	Action
CRD conflict on install (spec.versions)	Strimzi CRDs already exist in the cluster	Confirm the tdp-crds chart is at the expected version and retry the tdp-kafka helm upgrade
Pods in Pending state	StorageClass missing or insufficient capacity	kubectl get storageclass and kubectl describe pvc -n <NAMESPACE>
Kafka UI cannot connect to the cluster	Incorrect bootstrapServers or Kafka credentials	Check kafka-ui.connection.bootstrapServers and the Secret referenced in kafka-ui.connection.authentication.existingSecret
Broker fails to start	min.insync.replicas exceeds available broker count	Adjust config.min.insync.replicas to the actual number of replicas
Connector in FAILED state	Incorrect database credentials or host	kubectl describe kafkaconnector <CONNECTOR_NAME> -n <NAMESPACE>
Kafka UI does not start (RBAC)	RBAC resources or actions not supported by the UI image version	Check kafka-ui.rbac.roles and pod logs
Pods with OOMKilled	Insufficient CPU/memory limits	Adjust pod resource limits via the subchart resources field

Diagnostic commands

Terminal input

# Download the chart before validating:
helm pull oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka --untar
helm lint tdp-kafka/
helm template tdp-kafka tdp-kafka/ -n <NAMESPACE>

# Cluster and user status
kubectl -n <NAMESPACE> get kafka
kubectl -n <NAMESPACE> get kafkauser

# Secrets generated by Strimzi
kubectl -n <NAMESPACE> get secret

# Strimzi operator logs
kubectl -n <NAMESPACE> logs deployment/strimzi-cluster-operator

# Kafka UI logs
kubectl -n <NAMESPACE> logs deployment/<RELEASE_NAME>-kafka-ui

# Connectors
kubectl -n <NAMESPACE> get kafkaconnectors
kubectl -n <NAMESPACE> describe kafka <KAFKA_CLUSTER_NAME>

Uninstallation

Uninstallation

Terminal input

helm uninstall <RELEASE_NAME> -n <NAMESPACE>

PVCs

The broker PVCs are not automatically removed by helm uninstall, unless nodePool.storage.volumes[].deleteClaim: true is configured (chart default).
To manually remove PVCs (destructive and irreversible operation):

Terminal input

kubectl delete pvc -n <NAMESPACE> -l strimzi.io/cluster=<KAFKA_CLUSTER_NAME>