Security — Iceberg
The tdp-iceberg chart runs Iceberg table maintenance jobs (snapshot expiration, orphan file removal, data file rewriting). These jobs access S3/MinIO storage via Spark and require the Kubernetes Secret s3-credentials containing bucket access credentials.
Credentials and Secrets
Create the Secret in the namespace before deployment:
kubectl -n <NAMESPACE> create secret generic s3-credentials \
--from-literal=access-key="<ACCESS_KEY>" \
--from-literal=secret-key="<SECRET_KEY>"
| Key | Description |
|---|---|
access-key | S3/MinIO bucket access key |
secret-key | Secret key for S3 authentication |
The Secret name must be exactly s3-credentials. The chart's CronJobs reference this name directly via env.secretKeyRef.
Iceberg catalog configuration
Access to Hive Metastore (used as the Iceberg catalog) and the S3 endpoint is configured in maintenance.spark.config:
maintenance:
spark:
config:
"spark.sql.catalog.iceberg": "org.apache.iceberg.spark.SparkCatalog"
"spark.sql.catalog.iceberg.type": "hive"
"spark.sql.catalog.iceberg.uri": "thrift://<RELEASE_NAME>-metastore.<NAMESPACE>.svc.cluster.local:9083"
"spark.hadoop.fs.s3a.endpoint": "http://<S3_ENDPOINT>:<S3_PORT>"
"spark.hadoop.fs.s3a.path.style.access": "true"
Adjust the Metastore address (spark.sql.catalog.iceberg.uri) if the Hive Metastore release name or namespace differs from the default values.
Best practices
| Aspect | Recommendation |
|---|---|
| Credentials | Do not version access-key and secret-key in a Git repository |
| Rotation | Recreate the Secret and wait for the next CronJob cycle |
| Minimum access | Credentials with permission only on Iceberg table buckets |
Troubleshooting
| Problem | Probable cause | Solution |
|---|---|---|
| CronJob fails with 403 error on S3 | Secret missing or keys with wrong name | Check s3-credentials Secret in namespace |
| Connection error to Metastore | Incorrect Hive Metastore URI | Check address and port 9083 accessibility |
| Table not found in catalog | Incorrect Iceberg catalog namespace in query | Check spark.sql.catalog.iceberg configuration |
For the complete parameter list, use helm show values for the chart version you installed.