TDP Kubernetes Component Configuration

Each component has a configuration page with the Helm chart parameters. Depending on the component, there are also Security, Ingress, and Integrations pages.

Before you start

Read General Configuration to understand the concepts and patterns common to all components: how to pass values, resources, PVCs, Secrets, external PostgreSQL, and S3.

TDP Kubernetes Component Configuration

General Configuration

General concepts for configuring TDP Kubernetes components: namespaces, resources, images and Secrets.

Airflow

Umbrella chart: Airflow with values under `tdp-airflow`; Kubernetes executor by default.

Configuration

Executor, internal or external PostgreSQL, DAG/log PVCs, `TDPConfigurations.s3Connection`.

Security

Optional LDAP via `tdp-airflow.ldap` and `tdp-airflow.extraEnv` (Flask-AppBuilder).

Integrations

TDP helpers: external PostgreSQL (`externalDatabase`) and S3 connection Secret.

ArgoCD

GitOps on the Kubernetes cluster with automatic Application synchronization.

Configuration

Chart parameters, Dex/LDAP integration and options supported by the chart.

Security

SSO/LDAP via Dex with RBAC group mapping.

Ingress

External exposure via Ingress with TLS and custom hostname.

ClickHouse

Configure ClickHouse on Kubernetes with StatefulSets, PVCs and retention policies.

CloudBeaver

Web-based database administration on Kubernetes.

Configuration

Chart parameters and supported datasource configuration.

Security

LDAP authentication with bind DN and user search.

Ingress

External exposure via Ingress with TLS and custom hostname.

Integrations

Datasources: ClickHouse, Trino, PostgreSQL and S3/MinIO.

Delta Lake

Configuration of maintenance routines and parameters supported by the chart.

Hive Metastore

Chart parameters and internal or external PostgreSQL options supported.

Iceberg

Chart parameters and supported maintenance jobs.

JupyterLab

Multi-user notebooks on Kubernetes with Spark and RWX StorageClass.

Configuration

Chart parameters for notebooks, storage and Spark integration.

Security

LDAP authentication via LDAPAuthenticator.

Integrations

Integration with Apache Spark for notebooks with Iceberg support.

Kafka

Kafka on Kubernetes using the Strimzi operator with CRDs.

Configuration

Chart parameters, listeners and additional supported components.

Security

SASL/LDAP authentication for Kafka brokers via Strimzi.

Integrations

Debezium CDC: PostgreSQL, MySQL, SQL Server and Oracle.

NiFi

NiFi on Kubernetes with TLS managed by Cert-Manager and Ranger integration.

Configuration

NiFi/NiFiKop chart parameters supported by the package.

Ingress

External exposure via Ingress with TLS and custom hostname.

Integrations

Centralized access control via Apache Ranger.

OpenMetadata

Data governance on Kubernetes with PostgreSQL and service integration.

Configuration

Chart parameters, database and supported integrations.

Security

LDAP authentication with user and server attributes.

Ingress

External exposure via Ingress with TLS and custom hostname.

Integrations

Datasource registration: Airflow, Trino, ClickHouse, Kafka, Superset.

Apache Ozone

S3-compatible object storage for TDP on Kubernetes.

Configuration

Chart parameters for S3 Gateway, OM, SCM, Recon and S3 authentication.

Ingress

External exposure of the 4 Ozone services via Ingress.

Integrations

Using Ozone as S3 storage for Spark and Trino.

PostgreSQL

PostgreSQL chart parameters, persistence, resources and metrics.

Ranger

Centralized access control for Kafka, NiFi and Trino.

Configuration

Chart parameters, database and supported integrations.

Integrations

Plugin registration and default policies for Kafka, NiFi and Trino.

Spark

Spark on Kubernetes with Spark Operator, Hive Metastore, Delta Lake and Apache Ozone.

Configuration

Chart parameters, `hadoopConfig`, `customSparkConfig` and supported integrations.

Integrations

Delta Lake, Iceberg with S3/MinIO and JupyterLab integration.

Superset

Analytical dashboards on Kubernetes with Redis and PostgreSQL.

Configuration

Chart parameters, secrets and supported database.

Security

LDAP authentication via Flask-AppBuilder.

Integrations

Datasources: Trino and ClickHouse with automatic import.

Trino

Distributed queries on Kubernetes with Hive, Iceberg and Delta catalogs.

Configuration

Chart parameters, catalogs and supported authentication options.

Security

LDAP authentication with TLS certificates for the coordinator.

Ingress

External exposure via Ingress with TLS and custom hostname.

Integrations

Connectors: ClickHouse, Iceberg, Kafka; access control via Ranger.