Airflow Configuration

ChartVersion3.0.1TypeapplicationAppVersion3.0.2

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Feature SupportLDAPsupportedS3supportedExternal DBsupportedIngresssupportedGateway APIsupported

What is Apache Airflow?

Apache Airflow is TDP's workflow orchestration tool. It allows you to create, schedule, and monitor data pipelines defined as Python code (DAGs), with KubernetesExecutor as the default executor.

The tdp-airflow chart packages Apache Airflow 3.0.2 for Kubernetes.

For a complete overview of the tool, its architecture, and how it works, see Apache Airflow — Concepts.

Helm value structure

This is an umbrella chart. All Apache Airflow options must be grouped under the tdp-airflow: key (the dependency alias in Chart.yaml).

Respecting this prefix ensures that Helm applies the values to the correct subchart. For GitOps operators, an argocd app diff or helm template will only show CPU, connections, and other keys if they are nested under tdp-airflow:.

tdp-airflow:
  apiServer:
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"

Overview

Property	Value
Chart	tdp-airflow
Airflow version	3.0.2
Chart version	3.0.1
Default executor	KubernetesExecutor
Metadata database	Internal PostgreSQL by default, with an external PostgreSQL option
Persistence	DAGs, logs, and Triggerer can use PVCs
HTTP exposure	UI/API through internal Service, Ingress, or Gateway API

Related pages

• Integrations — Airflow — External PostgreSQL and S3 connection
• Security — Airflow — LDAP and Secrets
• External exposure: Ingress and Gateway API — HTTP/HTTPS UI exposure

Prerequisites

• Kubernetes 1.32+, Red Hat OpenShift 4.19+, or Rancher Manager 2.10.x+.
• Helm 3.2.0+.
• StorageClass available when DAG, log, or Triggerer persistence is enabled.

Installation

Installation

Minimal installation via OCI registry:

Terminal input

helm upgrade --install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-airflow -n <NAMESPACE> --create-namespace --timeout 10m

The additional values file — for example my-values.yaml — is where only the environment-specific adjustments go, without changing the basic procedure:

Terminal input

helm upgrade --install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-airflow -n <NAMESPACE> --create-namespace -f <VALUES_FILE> --wait --timeout 15m

Tip

The first installation may take a few minutes (images, migrations, hooks). The --wait --timeout 15m flags wait for pods and hooks to complete before returning. Increase the timeout if your cluster is slower.

OpenShift

Since release 3.0.1, Airflow is compatible with Red Hat OpenShift 4.19+.

Installation with the adjustment required by OpenShift Security Context Constraints:

Terminal input

helm upgrade --install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-airflow -n <NAMESPACE> --create-namespace --timeout 10m --set tdp-airflow.securityContexts.containers.runAsNonRoot=true

The runAsNonRoot=true parameter is required by OpenShift's Security Context Constraints (SCCs).

For details on compatibility and additional adjustments, see General Configuration, Environment decisions that affect configuration tab, under Orchestrator compatibility and security adjustments.

Main parameters

Main parameters

The table below summarizes the most frequently referenced parameters when configuring Airflow. For a first review, the points that typically deserve the most attention are: database, DAG and log persistence, LDAP authentication, and integrations with shared platform services.

Parameter	Description	Default
tdp-airflow.enabled	Enable Airflow deployment	true
tdp-airflow.executor	Executor	KubernetesExecutor
tdp-airflow.config.core.default_timezone	Default timezone	America/Sao_Paulo
tdp-airflow.apiServer.service.type	API Server Service type	ClusterIP
tdp-airflow.apiServer.service.ports	UI/API Service ports	See helm show values
tdp-airflow.postgresql.enabled	Built-in PostgreSQL	true
tdp-airflow.dags.persistence.enabled	DAGs PVC	true
tdp-airflow.dags.persistence.size	DAGs PVC size	5Gi
tdp-airflow.dags.persistence.storageClassName	DAGs PVC StorageClass	""
tdp-airflow.logs.persistence.enabled	Logs PVC	false
tdp-airflow.logs.persistence.size	Logs PVC size	5Gi
tdp-airflow.triggerer.persistence.enabled	Triggerer PVC	true
tdp-airflow.triggerer.persistence.size	Triggerer PVC size	5Gi
tdp-airflow.ldap.enabled	LDAP	false
tdp-airflow.ldap.apiServerConfig	Flask-AppBuilder snippet	See helm show values
tdp-airflow.extraEnv	Extra variables (e.g. Secret)	""
tdp-airflow.data	Metadata / external DB	Disabled by default
TDP-Settings.externalDatabase.enabled	TDP external DB helpers	false
TDP-Settings.externalDatabase.recreate	Recreate DB/user on install/upgrade	false
TDP-Settings.externalDatabase.externalSecret.releaseName	External PostgreSQL Secret release	<POSTGRESQL_RELEASE>
TDP-Settings.s3Connection.enabled	TDP S3 connection Secret	false
TDP-Settings.s3Connection.secretName	S3 Secret name	tdp-airflow-minio-connection
TDP-Settings.s3Connection.uri	S3 endpoint URI	https://<S3_ENDPOINT>
TDP-Settings.gateway.ingress.enabled	Enable Ingress	false
TDP-Settings.gateway.gatewayApi.enabled	Enable Gateway API	false

Configuration details

Default configuration

By default, the chart prioritizes a functional installation with minimal external dependencies: executor in Kubernetes, metadata in local PostgreSQL, and DAGs in a persistent volume. Reviewing these defaults depends on the operational policy adopted for database, storage, authentication, and observability, as described in General Configuration.

• Executor: KubernetesExecutor
• Database: Embedded PostgreSQL (subchart), with tdp-airflow.postgresql.enabled: true by default
• DAGs: PVC persistence enabled by default (tdp-airflow.dags.persistence.enabled: true, typical size 5Gi)
• Logs: PVC persistence disabled by default (tdp-airflow.logs.persistence.enabled: false); only enable if the StorageClass meets the required access standard (usually RWX)

Database configuration

Airflow requires a relational database to store its metadata: registered DAGs, execution history, connections, variables, and users.

Without a stable database, Airflow cannot maintain the history and metadata that make the service usable on a daily basis.

The choice between built-in and external PostgreSQL is therefore one of the first configuration decisions:

• built-in PostgreSQL simplifies the initial installation;
• external PostgreSQL is usually preferred when the environment already has its own standards for backup, availability, and database administration.

To understand when to use each, see PostgreSQL internal versus external on the General Configuration page.

Built-in PostgreSQL (default)

Suitable for development and testing only:

tdp-airflow:
  postgresql:
    enabled: true

Note

For production, prefer external PostgreSQL.

External PostgreSQL

Disable the built-in database and provide the connection details of the external PostgreSQL. In practice, this is the most common scenario when the client already uses a PostgreSQL from their own platform or a separate managed database service.

For TDP integration helpers:

TDP-Settings:
  externalDatabase:
    enabled: true
    recreate: false
    externalSecret:
      releaseName: "<POSTGRESQL_RELEASE>"

tdp-airflow:
  postgresql:
    enabled: false

  data:
    metadataSecretName: "<RELEASE_NAME>-airflow-database"
    metadataConnection:
      user: airflow
      pass: ""
      protocol: postgresql
      host: "<POSTGRESQL_SERVICE>.<NAMESPACE>.svc.cluster.local"
      port: 5432
      db: airflow
      sslmode: disable

With TDP-Settings.externalDatabase.enabled: true, the chart uses the configured release name to align the database, user, and metadata Secret with the TDP stack.

Tip

Prefer setting metadataSecretName and leaving pass empty when TDP jobs are responsible for generating the connection Secret.

DAG persistence (PVC)

DAGs need to be available to the components that orchestrate and execute the workflows. This section focuses less on "storing files" and more on ensuring that Airflow can reliably find the DAGs in the environment.

By default the chart already creates a PVC for DAGs. Adjust size, StorageClass, or access mode as required by the cluster:

tdp-airflow:
  dags:
    persistence:
      enabled: true
      size: 5Gi
      storageClassName: ""

Access mode

With KubernetesExecutor, multiple pods need to read the same DAGs.

If the scheduler, webserver, and task pods do not share the volume, use a StorageClass with ReadWriteMany (RWX), as permitted by the environment.

Log persistence (PVC)

By default, logs are not stored in a shared PVC. This simplifies the initial installation and avoids depending on a specific storage type at first use.

In practice, with enabled: false, log retention depends on the observability strategy already adopted in the environment. With enabled: true, the cluster must provide a volume compatible with writes from multiple components.

Chart default: enabled: false. If enabling, evaluate the access mode: logs are written by multiple components and RWO is usually inadequate.

tdp-airflow:
  logs:
    persistence:
      enabled: false
      size: 5Gi

Note

With RWO, log persistence is usually inadequate.

Prefer RWX or another log strategy (for example, an external stack) aligned with the environment.

Triggerer persistence (PVC)

The Triggerer is an Airflow 2.2+ component responsible for managing events and asynchronous task executions. By default it stores heartbeat logs in a separate PVC.

Chart default:

tdp-airflow:
  triggerer:
    persistence:
      enabled: true
      size: 5Gi

Adjust the size and StorageClass as needed, following the same recommendations in Storage on the General Configuration page.

S3 connection (TDP)

Configure this section when TDP integrations need S3-compatible connection parameters. The s3Connection configuration creates or references a Secret containing these parameters; it does not replace DAG persistence configuration on its own.

The platform concept is detailed on the General Configuration page, in the Environment decisions that affect configuration tab, under S3-compatible storage (Ozone, MinIO, or another endpoint).

With TDP-Settings.s3Connection.enabled: true, the chart creates a Secret with S3-compatible connection parameters for TDP integrations. Use placeholders; do not commit credentials in plain text:

TDP-Settings:
  s3Connection:
    enabled: true
    secretName: "<S3_SECRET_NAME>"
    name: "<CONNECTION_NAME>"
    type: "aws"
    accessKey: "<S3_ACCESS_KEY>"
    secretKey: "<S3_SECRET_KEY>"
    uri: "https://<S3_ENDPOINT>"

Additional environment variables

Use tdp-airflow.extraEnv to reference Secrets or inject other variables (for example, the LDAP password):

tdp-airflow:
  extraEnv: |
    - name: EXAMPLE_ENV
      valueFrom:
        secretKeyRef:
          name: <SECRET_NAME>
          key: <SECRET_KEY>

Access & Security

Access

Access to the Airflow web interface is provided through a Kubernetes Service, normally of the ClusterIP type.

For local access during validation or testing:

Terminal input

kubectl -n <NAMESPACE> port-forward svc/<RELEASE_NAME>-api-server 8080:8080

For stable external exposure via Ingress or Gateway API, see External exposure: Ingress and Gateway API.

LDAP authentication

LDAP is optional and disabled by default.

When disabled, there is no dependency on a corporate directory and no need for additional Secrets for this purpose. When LDAP is enabled, Airflow authentication depends on the organization's directory definitions: server, search base, users, and sensitive variables stored in a Secret.

Configuration uses tdp-airflow.ldap and variables injected via tdp-airflow.extraEnv.

Detailed documentation

See Security — Airflow for the bind Secret and LDAP configuration examples.

Uninstallation

Uninstallation

To remove the Airflow installation:

Terminal input

helm uninstall <RELEASE_NAME> -n <NAMESPACE>

This command removes all release resources (pods, services, configmaps, secrets). PVCs (volumes) are retained by default to preserve data. To also remove the PVCs:

Terminal input

kubectl delete pvc -n <NAMESPACE> -l app.kubernetes.io/instance=<RELEASE_NAME>