Iceberg Configuration

ChartVersion3.0.1TypeapplicationAppVersion1.10.0

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Feature SupportS3supported

What is Apache Iceberg and why does it need maintenance?

Apache Iceberg is an open table format for large datasets, designed to overcome Hive limitations. Like Delta Lake, Iceberg maintains snapshots: immutable versions of the table for each write operation. With continuous use, this tends to accumulate:

• old snapshots;
• orphan files;
• small files that hurt read performance.

The tdp-iceberg chart creates Kubernetes CronJobs to execute these maintenance routines on a schedule, using Apache Spark as the processing engine.

Learn more

See Apache Iceberg — Concepts for a complete overview of the format, snapshots and use cases.

Helm values structure

The tdp-iceberg chart distributes configuration in two main blocks:

• maintenance: — enables jobs, defines the CronJobs Spark image, Spark configuration (catalog, S3) and each job's schedule.
• spark: — controls the upstream Spark subchart (master/worker pods). Independent of the image used by the CronJobs.

maintenance:
enabled: true
spark:
  enabled: true
  image:
    repository: "registry.tecnisys.com.br/tdp-dev/images/spark"
    tag: "4.0.2-0"
jobs:
  expireSnapshots:
    enabled: true

spark:
image:
  registry: registry.tecnisys.com.br
  repository: tdp-dev/images/spark
  tag: 4.0.2-0

Overview

Property	Value
Chart	tdp-iceberg
Iceberg version (runtime)	1.10.0
Chart version	3.0.1
Registry (OCI)	oci://registry.tecnisys.com.br/tdp/charts/tdp-iceberg
Type	maintenance — Kubernetes CronJobs
External HTTP exposure	Not applicable

Runtime compatibility

Component	Version
Spark	4.0.2
Iceberg (Spark runtime)	1.10.0
Scala	2.13

The chart does not replace the catalog or warehouse: it schedules tasks that operate on existing Iceberg tables. You remain responsible for ensuring the CronJob commands point to the correct catalog, metastore and bucket.

Related pages

• Integrations — Iceberg: S3, Hive Metastore, Spark catalog configuration, Trino, Airflow.
• Security — Iceberg: s3-credentials Secret, S3 credentials and best practices.

Prerequisites

• Kubernetes 1.32+, Red Hat OpenShift 4.19+ or Rancher Manager 2.10.x+
• Helm 3.2.0+
• Tecnisys OCI registry accessible from the installation environment
• s3-credentials Secret created in the namespace before deployment (see Security — Iceberg)
• S3/MinIO endpoint accessible from the cluster
• Hive Metastore accessible via Thrift, if the Iceberg catalog uses type: hive

Without these items, the CronJob Spark commands will fail to resolve warehouse, credentials or metadata. Confirm that the metastore and bucket referenced in the values point to the actual catalog of the tables to be maintained.

Installation

Installation

Terminal input

helm upgrade --install <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-iceberg \
  --version <CHART_VERSION> \
  -n <NAMESPACE> --create-namespace

Placeholder	Description
<RELEASE_NAME>	Helm release name
<NAMESPACE>	Kubernetes installation namespace
<CHART_VERSION>	Chart version

OpenShift

The chart configures adaptSecurityContext: force by default, automatically adapting security contexts to OpenShift requirements. No additional parameters are needed for most OpenShift environments.

global:
  compatibility:
    openshift:
      adaptSecurityContext: force

Verify installation

Terminal input

kubectl -n <NAMESPACE> get cronjobs
kubectl -n <NAMESPACE> get jobs
kubectl -n <NAMESPACE> logs job/<JOB_NAME>

Main parameters

Main parameters

Parameter	Description	Default
maintenance.enabled	Enable maintenance jobs	true
maintenance.spark.enabled	Enable Spark dependency	true
maintenance.spark.image.repository	CronJob image	registry.tecnisys.com.br/tdp-dev/images/spark
maintenance.spark.image.tag	CronJob image tag	4.0.2-0
maintenance.spark.resources.requests.cpu	CPU request for jobs	1
maintenance.spark.resources.requests.memory	Memory request for jobs	2Gi
maintenance.spark.resources.limits.cpu	CPU limit for jobs	2
maintenance.spark.resources.limits.memory	Memory limit for jobs	4Gi
maintenance.jobs.expireSnapshots.enabled	Enable expire snapshots	true
maintenance.jobs.expireSnapshots.schedule	Expire snapshots cron	0 2 * * *
maintenance.jobs.expireSnapshots.retentionDays	Retention days	7
maintenance.jobs.removeOrphanFiles.enabled	Enable orphan file removal	true
maintenance.jobs.removeOrphanFiles.schedule	Remove orphan files cron	0 3 * * 0
maintenance.jobs.removeOrphanFiles.olderThanDays	Minimum orphan age	3
maintenance.jobs.rewriteDataFiles.enabled	Enable data file rewrite	false
maintenance.jobs.rewriteDataFiles.schedule	Rewrite data files cron	0 1 * * 6
global.compatibility.openshift.adaptSecurityContext	OpenShift compatibility	force

Configuration details

Maintenance jobs

Each job becomes a Kubernetes CronJob: schedule, retention policy and the Spark command are declared in the values. Monitor successful executions, duration and cluster consumption — long-running jobs may compete with other Spark workloads.

Jobs are configured under maintenance.jobs.*.

note

The examples below show the expected command format. Adjust catalog, endpoint, warehouse, table and retention windows for your environment.

Expire Snapshots

Removes old snapshots to free storage and metadata space:

maintenance:
jobs:
  expireSnapshots:
    enabled: true
    schedule: "0 2 * * *"
    retentionDays: 7
    command: |
      spark-sql \
        --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0,org.apache.hadoop:hadoop-aws:3.3.4 \
        --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
        --conf spark.sql.catalog.iceberg.type=hive \
        --conf spark.sql.catalog.iceberg.uri=thrift://metastore.hive-metastore.svc.cluster.local:9083 \
        --conf spark.sql.catalog.iceberg.warehouse=s3a://warehouse/hive \
        --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
        -e "CALL iceberg.system.expire_snapshots(older_than => TIMESTAMP '$(date -d '7 days ago' '+%Y-%m-%d %H:%M:%S')');"

Remove Orphan Files

Removes orphan files no longer referenced by valid snapshots:

maintenance:
jobs:
  removeOrphanFiles:
    enabled: true
    schedule: "0 3 * * 0"
    olderThanDays: 3
    command: |
      spark-sql \
        --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0,org.apache.hadoop:hadoop-aws:3.3.4 \
        --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
        --conf spark.sql.catalog.iceberg.type=hive \
        --conf spark.sql.catalog.iceberg.uri=thrift://metastore.hive-metastore.svc.cluster.local:9083 \
        --conf spark.sql.catalog.iceberg.warehouse=s3a://warehouse/hive \
        --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
        -e "CALL iceberg.system.remove_orphan_files(older_than => TIMESTAMP '$(date -d '3 days ago' '+%Y-%m-%d %H:%M:%S')');"

Rewrite Data Files

Rewrites and compacts data files to improve read performance. Disabled by default as it is more resource-intensive.

When enabled, you accept jobs that read and write data in volume — higher CPU, memory and I/O usage on storage, and potentially long execution windows on large tables. Enable when there is query degradation due to file fragmentation, after very granular ingestions, or when the maintenance policy includes periodic compaction.

maintenance:
jobs:
  rewriteDataFiles:
    enabled: false
    schedule: "0 1 * * 6"
    command: |
      spark-sql \
        --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0,org.apache.hadoop:hadoop-aws:3.3.4 \
        --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
        --conf spark.sql.catalog.iceberg.type=hive \
        --conf spark.sql.catalog.iceberg.uri=thrift://metastore.hive-metastore.svc.cluster.local:9083 \
        --conf spark.sql.catalog.iceberg.warehouse=s3a://warehouse/hive \
        --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
        -e "CALL iceberg.system.rewrite_data_files(table => 'iceberg.default.<TABLE_NAME>');"

Job parameters

Parameter	Description
enabled	Enable or disable the CronJob
schedule	Cron expression
retentionDays	Retention days for expireSnapshots
olderThanDays	Minimum age for removeOrphanFiles
command	Shell script executed by the container

Spark configuration

This chart uses two distinct Spark image contexts — do not confuse them:

Key	What it controls	Format
spark.image.*	Spark subchart master/worker pods	registry + repository separately
maintenance.spark.image.*	Maintenance CronJob containers	repository with full URL

Updating one does not update the other — when upgrading Spark, also align the tag in maintenance.spark.image to avoid incompatibility.

Spark subchart (master/worker)

spark:
image:
  registry: registry.tecnisys.com.br
  repository: tdp-dev/images/spark
  tag: 4.0.2-0
  pullPolicy: IfNotPresent

Maintenance container image

maintenance:
spark:
  enabled: true
  image:
    repository: "registry.tecnisys.com.br/tdp-dev/images/spark"
    tag: "4.0.2-0"
    pullPolicy: IfNotPresent

note

These two configurations are independent. Changing spark.image.* does not change the image used by the maintenance CronJobs.

Integrations

For S3/MinIO, Hive Metastore, catalog configuration and usage from Spark, Airflow or Trino, see Integrations — Iceberg.

Uninstallation

Uninstallation

Terminal input

helm uninstall <RELEASE_NAME> -n <NAMESPACE>

The tdp-iceberg chart is of type maintenance and does not create PVCs. Uninstallation removes only CronJobs, Jobs, ServiceAccount, RBAC and other release resources — with no risk of data loss in S3/Ozone buckets.