Security — Iceberg

The tdp-iceberg chart runs Iceberg table maintenance jobs (snapshot expiration, orphan file removal, data file rewrite). These jobs access S3/MinIO storage via Spark and require the Kubernetes Secret s3-credentials with bucket access credentials.

S3 credentials Secret

Create the Secret in the namespace before deployment:

kubectl -n <namespace> create secret generic s3-credentials \
  --from-literal=access-key="<ACCESS_KEY>" \
  --from-literal=secret-key="<SECRET_KEY>"

Key	Description
access-key	S3/MinIO bucket access key
secret-key	Secret key for S3 authentication

The Secret name must be exactly s3-credentials. The chart's CronJobs reference this name directly via env.secretKeyRef.

Iceberg catalog configuration

Access to the Hive Metastore (used as the Iceberg catalog) and the S3 endpoint are configured in maintenance.spark.config:

maintenance:
  spark:
    config:
      "spark.sql.catalog.iceberg": "org.apache.iceberg.spark.SparkCatalog"
      "spark.sql.catalog.iceberg.type": "hive"
      "spark.sql.catalog.iceberg.uri": "thrift://<release>-metastore.<namespace>.svc.cluster.local:9083"
      "spark.hadoop.fs.s3a.endpoint": "http://<s3-endpoint>:<port>"
      "spark.hadoop.fs.s3a.path.style.access": "true"

Adjust the Metastore address (spark.sql.catalog.iceberg.uri) if the Hive Metastore release name or namespace differ from the default values.

Best practices

Aspect	Recommendation
Credentials	Do not version access-key and secret-key in a Git repository
Rotation	Recreate the Secret and wait for the next CronJob execution cycle
Minimal access	Credentials with permission only for the buckets containing Iceberg tables

Troubleshooting

Problem	Probable cause	Solution
CronJob fails with S3 403 error	Missing Secret or keys with incorrect names	Check Secret s3-credentials in the namespace
Metastore connection error	Incorrect Hive Metastore URI	Verify address and port 9083 accessibility
Table not found in catalog	Incorrect Iceberg catalog namespace in query	Check spark.sql.catalog.iceberg configuration

For the full list of parameters, use helm show values on the version of the chart you installed.