Security — Delta Lake

The tdp-deltalake chart runs maintenance jobs (VACUUM, OPTIMIZE) on S3/MinIO storage. Access credentials for the bucket are provided via the Kubernetes Secret s3-credentials, read by the CronJobs via env.secretKeyRef.

S3 credentials Secret

Create the Secret in the namespace before deployment:

kubectl -n <namespace> create secret generic s3-credentials \
  --from-literal=access-key="<ACCESS_KEY>" \
  --from-literal=secret-key="<SECRET_KEY>"

Key	Description
access-key	S3/MinIO bucket access key
secret-key	Secret key for S3 authentication

The Secret name must be exactly s3-credentials. The chart's CronJobs reference this name directly.

S3 endpoint configuration

The S3/MinIO endpoint is configured in the maintenance.spark.config block:

maintenance:
  spark:
    config:
      "spark.hadoop.fs.s3a.endpoint": "http://<s3-endpoint>:<port>"
      "spark.hadoop.fs.s3a.path.style.access": "true"

Replace <s3-endpoint> with the internal S3 service address in the cluster. For Ozone installed with TDP, the default address follows the pattern:

http://<release>-s3g-rest.<namespace>.svc.cluster.local:9878

Best practices

Aspect	Recommendation
Credentials	Do not version access-key and secret-key in a Git repository
Rotation	Recreate the Secret and wait for the next CronJob execution cycle
Minimal access	Configure credentials with permission only for the buckets used by maintenance jobs

Troubleshooting

Problem	Probable cause	Solution
CronJob fails with authentication error	Missing Secret or keys with incorrect names	Check Secret s3-credentials in the namespace
S3 endpoint unreachable	Incorrect URL or port	Test connectivity with kubectl exec within the namespace
Permission denied on bucket	Credentials without access to configured path	Review permissions in MinIO/Ozone

For the full list of parameters, use helm show values on the version of the chart you installed.