Integrations — OpenMetadata

Integration overview

The tdp-openmetadata chart can run a post-install/upgrade Job that registers external services in OpenMetadata when datasourcesIntegration.enabled=true.

In addition, OpenMetadata can execute ingestion pipelines in two modes: Kubernetes Jobs (default) or Airflow.

Overview

1. Deploy OpenMetadata.
2. The Job waits for OpenMetadata to be ready, authenticates against the API, and registers the enabled services.

Enabling the Job

datasourcesIntegration:
  enabled: true

Access to the OpenMetadata API is configured under datasourcesIntegration.openmetadata.*. If datasourcesIntegration.openmetadata.auth.passwordSecret.enabled=true, provide the Secret name and password key in your values file.

Per-service configuration

Each integration has an enabled flag and a connection block under datasourcesIntegration.*.

Example (Trino + ClickHouse)

datasourcesIntegration:
  enabled: true
  openmetadata:
    apiEndpoint: "http://<openmetadata-service>.<namespace>.svc.cluster.local:8585/api"
    auth:
      provider: "basic"
      username: "admin@open-metadata.org"
      passwordSecret:
        enabled: true
        name: "<openmetadata-admin-secret>"
        key: "<password-key>"

  trino:
    enabled: true
    connection:
      host: "<trino-service>.<namespace>.svc.cluster.local"
      port: 8080
      username: "<username>"
      scheme: "http"

  clickhouse:
    enabled: true
    connection:
      host: "<clickhouse-service>.<namespace>.svc.cluster.local"
      port: 8123
      username: "<username>"
      passwordSecret:
        name: "<clickhouse-secret>"
        key: "password"
      scheme: "http"

Airflow, Superset, and Kafka

Airflow, Superset, and Kafka are supported types by the Job (Pipeline, Dashboard, Messaging): enable the corresponding block and fill in connection and credentials in the values file (see helm show values).

Typical installation with extra values (generic example):

Terminal input

helm upgrade --install <release> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-openmetadata \
  -n <namespace> \
  -f <base-file>.yaml \
  -f <integrations-file>.yaml

Use values files from your Helm package or a single -f with all required content.

Pipeline execution: Airflow or K8S

OpenMetadata can execute ingestion pipelines using two modes:

• Kubernetes Jobs (k8s): default mode, where pipelines run as native Jobs in the cluster, without requiring Airflow
• Airflow (airflow): mode in which pipelines run through REST integration with the Airflow service

Execution mode is controlled by the pipelineServiceClientConfig.type field, which accepts k8s or airflow.

Kubernetes Jobs mode (type: k8s)

When k8s mode is active, pipelines are submitted as Kubernetes Jobs. The most common parameters in this scenario are:

• execution image
• namespace where the Jobs will be created
• service account
• CPU and memory resources
• TTL and backoff policy
• nodeSelector, annotations, and securityContext

Airflow mode (type: airflow)

When airflow mode is active, OpenMetadata delegates pipeline execution to the Airflow service. The most common parameters in this scenario are:

• Airflow endpoint
• username
• password via Secret
• SSL verification

How to switch

• change only pipelineServiceClientConfig.type to switch execution mode
• keep the remaining parameters aligned with the chosen mode
• airflow mode requires the Airflow service to be enabled and reachable in the chart dependencies

Verification

After the Job completes, the services should appear in the OpenMetadata UI under Settings → Services.

Troubleshooting

Terminal input

kubectl -n <namespace> get jobs
kubectl -n <namespace> logs job/<release>-register-datasources