Integrations — Delta Lake
Integrations overview
The tdp-deltalake chart requires access to S3-compatible storage (Apache Ozone S3 Gateway, MinIO, or another S3 endpoint) and can be integrated with the main processing and orchestration tools in TDP.
S3 / MinIO
The maintenance CronJobs require the s3-credentials Secret with S3 access credentials. Create it before deployment as described in Security — Delta Lake.
Configure the S3 endpoint
maintenance:
spark:
config:
"spark.hadoop.fs.s3a.endpoint": "http://<S3_ENDPOINT>.<NAMESPACE>.svc.cluster.local:9000"
"spark.hadoop.fs.s3a.path.style.access": "true"
S3 parameters
| Parameter | Description | Example |
|---|---|---|
spark.hadoop.fs.s3a.endpoint | S3 endpoint URL | http://ozone-s3g.<NAMESPACE>.svc.cluster.local:9000 |
spark.hadoop.fs.s3a.path.style.access | Force path-style access (required for MinIO/Ozone) | "true" |
Trino
Trino can query Delta Lake tables via the Delta connector. The configuration is performed on the tdp-trino chart side — see Trino Configuration.
Spark
Delta Lake tables can be processed directly by Spark. The configuration is performed on the tdp-spark chart side — see Spark Configuration.
Airflow
Airflow can orchestrate pipelines that read and write Delta Lake tables. The connection configuration is performed on the tdp-airflow chart side — see Integrations — Airflow.
Combining value files
helm upgrade --install <RELEASE_NAME> \
oci://registry.tecnisys.com.br/tdp/charts/tdp-deltalake \
-n <NAMESPACE> \
-f my-values.yaml \
-f values-integration.yaml