Skip to main content
Version 3.0

Delta Lake Configuration

ChartVersion3.0.1TypeapplicationAppVersion4.0.1
CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+
Feature SupportS3supported

What is Delta Lake?​

Delta Lake is an open table format that adds ACID transactions to object storage (S3/Ozone). Instead of writing Parquet files directly, Delta Lake maintains a transaction log — a history of all operations (INSERT, UPDATE, DELETE, MERGE) performed on the table.

Over time, this model generates two types of accumulation that require periodic cleanup:

  1. Orphan files: Parquet files that exist in S3 but are no longer referenced by any active version of the table
  2. Old versions: the transaction log maintains history indefinitely; without cleanup, it grows without limit

The tdp-deltalake chart is not a continuous application — it creates Kubernetes CronJobs that run periodically to perform this maintenance automatically, using Apache Spark 4.0.2.

Learn more

See Delta Lake — Concepts for a complete overview of the format, ACID transactions, and use cases.

Maintenance jobs provided​

JobDescription
VACUUMRemoves old files not referenced by the transaction log
OPTIMIZECompacts small files into larger files
OPTIMIZE Z-ORDEROptimizes data layout for queries with specific filters
DESCRIBE HISTORYRecords change history for auditing

Helm Values Structure​

The tdp-deltalake chart distributes configuration across two blocks:

  • maintenance: — maintenance jobs (VACUUM, OPTIMIZE, Z-ORDER, DESCRIBE HISTORY), scheduling (cron), retention, and Spark configuration for the jobs.
  • spark: — upstream Spark chart values: base image, labels, master and worker configuration.
maintenance:
enabled: true
spark:
  enabled: true
  image:
    repository: "registry.tecnisys.com.br/tdp-dev/images/spark"
    tag: "4.0.2-0"
jobs:
  vacuum:
    enabled: true
    schedule: "0 2 * * *"
    retentionHours: 168

spark:
image:
  registry: registry.tecnisys.com.br
  repository: tdp-dev/images/spark
  tag: 4.0.2-0

Overview​

PropertyValue
Charttdp-deltalake
Typemaintenance (CronJobs)
Delta Lake version4.0.1
Chart version3.0.1
Registry (OCI)oci://registry.tecnisys.com.br/tdp/charts/tdp-deltalake
Spark4.0.2
Scala2.13

Prerequisites​

  • Kubernetes 1.32+, Red Hat OpenShift 4.19+ or Rancher Manager 2.10.x+
  • Helm 3.2.0+
  • Secret s3-credentials with keys access-key and secret-key created in the namespace before deployment. See Security — Delta Lake for instructions.

Installation​

Terminal input
helm upgrade --install <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-deltalake \
  --version <CHART_VERSION> \
  -n <NAMESPACE> --create-namespace
PlaceholderDescription
<RELEASE_NAME>Helm release name
<NAMESPACE>Kubernetes installation namespace
<CHART_VERSION>Chart version

OpenShift​

On OpenShift, the chart applies force to the Security Context by default:

global:
  compatibility:
    openshift:
      adaptSecurityContext: force

This value is pre-configured in values.yaml. The reason for this default is covered on the General Configuration page, in the Environment decisions that affect configuration tab, under Orchestrator compatibility and security adjustments.

Post-installation verification​

Terminal input
# List created CronJobs
kubectl get cronjobs -n <NAMESPACE>

# Check latest runs
kubectl get jobs -n <NAMESPACE>

# Logs for a specific job
kubectl logs -n <NAMESPACE> -l job-name=<JOB_NAME>