Configuring TDP Kubernetes Components
Each component has a configuration page with Helm chart parameters.
Depending on the component, there are also Security, External Exposure and Integrations pages.
Read General Configuration to understand the concepts and patterns common to all components: how to pass values, resources, PVCs, Secrets, external PostgreSQL and S3.
Configuring TDP Kubernetes Components
General Configuration
General concepts for configuring TDP Kubernetes components: namespaces, resources, images and Secrets.
Airflow
Chart umbrella: Airflow with values under `tdp-airflow`; Kubernetes runner by default.
Configuration
Executor, internal or external PostgreSQL, DAGs/logs PVCs, `TDP-Settings.s3Connection`.
Security
Optional LDAP via `tdp-airflow.ldap` and `tdp-airflow.extraEnv` (Flask-AppBuilder).
External Exposure
External exposure via Ingress with optional hostname and TLS.
Integrations
TDP Helpers: External PostgreSQL (`externalDatabase`) and S3 Connection Secret.
ArgoCD
GitOps on Kubernetes cluster with automatic Applications synchronization.
ClickHouse
Columnar analytics database on Kubernetes with support for HA, S3 and consumption by other TDP tools.
Configuration
Installation, main parameters, persistence, features and HA example.
Security
Passwords, default user access control, additional profiles and LDAP.
External Exposure
Play UI and Dashboard exposure via Ingress or Gateway API with dedicated hostname.
Integrations
S3/MinIO, storage and consumption policy by CloudBeaver, Superset and Trino.
CloudBeaver
Web administration of databases in Kubernetes.
Delta Lake
ACID storage layer over S3, with maintenance routines managed by the chart.
Hive Metastore
Table metadata catalog for Spark, Trino and Iceberg.
Iceberg
Iceberg maintenance jobs with Spark runtime and dependency on Hive Metastore and S3.
JupyterLab
Multi-user notebooks on Kubernetes with Spark and StorageClass RWX.
Configuration
Chart parameters for notebooks, storage and integration with Spark.
Security
LDAP authentication via LDAPAuthenticator.
External Exposure
External exposure of JupyterHub via Ingress with optional hostname and TLS.
Integrations
Integration with Apache Spark for notebooks; Use of Iceberg happens via Spark.
Kafka
Kafka on Kubernetes using Strimzi operator with CRDs.
Configuration
Chart parameters, listeners and additional components supported.
Security
SASL/LDAP authentication for Kafka brokers via Strimzi.
External Exposure
Kafka UI exposure via Ingress or Gateway API with custom hostname and optional TLS.
Integrations
Debezium CDC: PostgreSQL, MySQL, SQL Server and Oracle.
NiFi
NiFi on Kubernetes with TLS managed by Cert-Manager and Ranger integration.
Configuration
NiFi/NiFiKop chart parameters supported by the package.
Security
TLS security modes, LDAP authentication and cert-manager-managed certificates.
External Exposure
External exposure via Ingress or Gateway API, with custom hostname and optional TLS.
Integrations
Centralized access control via Apache Ranger.
OpenMetadata
Data governance in Kubernetes with PostgreSQL and services integration.
Configuration
Chart parameters, database and supported integrations.
Security
LDAP authentication with user and server attributes.
External Exposure
External exposure via Ingress or Gateway API, with custom hostname and optional TLS.
Integrations
Datasources registration: Airflow, Trino, ClickHouse, Kafka, Superset.
Apache Ozone
S3-compatible object storage for TDP on Kubernetes.
Configuration
Chart parameters for S3 Gateway, OM, SCM and S3 authentication.
Security
AWS Signature v4 without Kerberos, S3 credentials and access control via ACLs.
External Exposure
External exposure of the 4 Ozone interfaces (OM, SCM, S3 REST, S3 Web UI) via Ingress.
Integrations
Use of Ozone as S3 storage by Spark and Trino.
PostgreSQL
PostgreSQL chart parameters, persistence, resources and metrics.
Ranger
Centralized access control for Kafka, NiFi and Trino.
Configuration
Chart parameters, database and supported integrations.
Security
Admin authentication, database credentials and LDAP/AD configuration.
External Exposure
Ranger UI exposure via Ingress with optional hostname and TLS.
Integrations
Plugin registration and default policies for Kafka, NiFi and Trino.
Spark
Spark on Kubernetes with a master/worker cluster, Hive Metastore, Delta Lake and S3 storage (Ozone, MinIO, or another endpoint).
Superset
Analytics dashboards on Kubernetes with Redis and PostgreSQL.
Trino
Queries distributed on Kubernetes with Hive, Iceberg and Delta catalogs.
Configuration
Supported chart parameters, catalogs and authentication options.
Security
LDAP authentication with TLS certificates for coordinator.
External Exposure
External exposure via Ingress or Gateway API, with custom hostname and optional TLS.
Integrations
Connectors: ClickHouse, Iceberg, Kafka; access control via Ranger.