Configuring TDP Kubernetes components

Each component has a configuration page with the Helm chart parameters. Depending on the component, there are also Security, Ingress, and Integrations pages.

Before you start

Read General configuration to understand concepts and patterns shared by all components: how to pass values, resources, PVCs, Secrets, external PostgreSQL, and S3.

Configuring TDP Kubernetes components

General configuration

General concepts for configuring TDP Kubernetes components: namespaces, resources, images, and Secrets.

Airflow

Umbrella chart: Airflow with values under `tdp-airflow`; Kubernetes executor by default.

Configuration

Executor, internal or external PostgreSQL, DAG/log PVCs, `TDPConfigurations.s3Connection`.

Security

Optional LDAP via `tdp-airflow.ldap` and `tdp-airflow.extraEnv` (Flask-AppBuilder).

Ingress

External exposure via Ingress with optional hostname and TLS.

Integrations

TDP helpers: external PostgreSQL (`externalDatabase`) and S3 connection Secret.

ArgoCD

GitOps on the Kubernetes cluster with automatic Application synchronization.

Configuration

Chart parameters, Dex/LDAP integration, and chart-supported options.

Security

SSO/LDAP via Dex with RBAC group mapping.

Ingress

External exposure via Ingress with TLS and a custom hostname.

ClickHouse

Columnar analytical database on Kubernetes with HA, S3, and consumption by other TDP tools.

Configuration

Installation, main parameters, persistence, resources, and an HA starter example.

Security

Passwords, default user access control, additional profiles, and LDAP.

Ingress

Play UI and Dashboard exposure via Ingress with a dedicated hostname.

Integrations

S3/MinIO, storage policy, and use from CloudBeaver, Superset, and Trino.

CloudBeaver

Web-based database administration on Kubernetes.

Configuration

Chart parameters and supported datasource configuration.

Security

LDAP authentication with bind DN and user search.

Ingress

External exposure via Ingress with TLS and a custom hostname.

Integrations

Datasources: ClickHouse, Trino, PostgreSQL, and S3/MinIO.

Delta Lake

Maintenance routines and chart-supported parameters.

Hive Metastore

Chart parameters and supported internal or external PostgreSQL options.

Iceberg

Iceberg maintenance jobs with Spark runtime and dependency on Hive Metastore and S3.

Configuration

Maintenance jobs, main parameters, and Spark container image.

Integrations

S3/MinIO, Hive Metastore, and use from Spark, Jupyter, Trino, and Airflow.

JupyterLab

Multi-user notebooks on Kubernetes with Spark and RWX StorageClass.

Configuration

Chart parameters for notebooks, storage, and Spark integration.

Security

LDAP authentication via LDAPAuthenticator.

Ingress

External exposure of JupyterHub via Ingress with optional hostname and TLS.

Integrations

Apache Spark integration for notebooks; Iceberg is used via Spark.

Kafka

Kafka on Kubernetes using the Strimzi operator with CRDs.

Configuration

Chart parameters, listeners, and supported additional components.

Security

SASL/LDAP authentication for Kafka brokers via Strimzi.

Integrations

Debezium CDC: PostgreSQL, MySQL, SQL Server, and Oracle.

NiFi

NiFi on Kubernetes with TLS managed by cert-manager and Ranger integration.

Configuration

NiFi/NiFiKop chart parameters supported by the package.

Ingress

External exposure via Ingress with TLS and a custom hostname.

Integrations

Centralized access control via Apache Ranger.

OpenMetadata

Data governance on Kubernetes with PostgreSQL and service integrations.

Configuration

Chart parameters, database, and supported integrations.

Security

LDAP authentication with user attributes and server settings.

Ingress

External exposure via Ingress with TLS and a custom hostname.

Integrations

Datasource registration: Airflow, Trino, ClickHouse, Kafka, Superset.

Apache Ozone

S3-compatible object storage for TDP on Kubernetes.

Configuration

Chart parameters for S3 Gateway, OM, SCM, Recon, and S3 authentication.

Ingress

External exposure of the 5 Ozone interfaces (OM, SCM, Recon, S3 REST, S3 Web UI) via Ingress.

Integrations

Using Ozone as S3 storage from Spark and Trino.

PostgreSQL

PostgreSQL chart parameters, persistence, resources, and metrics.

Ranger

Centralized access control for Kafka, NiFi, and Trino.

Configuration

Chart parameters, database, and supported integrations.

Ingress

Ranger UI exposure via Ingress with optional hostname and TLS.

Integrations

Plugin registration and default policies for Kafka, NiFi, and Trino.

Spark

Spark on Kubernetes with Spark Operator, Hive Metastore, Delta Lake, and Apache Ozone.

Configuration

Chart parameters, `hadoopConfig`, `customSparkConfig`, and supported integrations.

Integrations

Delta Lake, Iceberg with S3/MinIO, and JupyterLab integration.

Superset

Analytical dashboards on Kubernetes with Redis and PostgreSQL.

Configuration

Chart parameters, secrets, and supported databases.

Security

LDAP authentication via Flask-AppBuilder.

Ingress

External exposure via Ingress with TLS and custom hostname.

Integrations

Datasources: Trino and ClickHouse with automatic import.

Trino

Distributed queries on Kubernetes with Hive, Iceberg, and Delta catalogs.

Configuration

Chart parameters, catalogs, and supported authentication options.

Security

LDAP authentication with TLS certificates for the coordinator.

Ingress

External exposure via Ingress with TLS and a custom hostname.

Integrations

Connectors: ClickHouse, Iceberg, Kafka; access control via Ranger.