Version 3.0

Integrations — OpenMetadata

ChartVersion3.0.1TypeapplicationAppVersion1.12.4

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Integration overview

The tdp-openmetadata chart can use an external PostgreSQL (tdp-postgresql) as its metadata database instead of the internal MySQL — see OpenMetadata Configuration.

The tdp-openmetadata chart can run a post-install/upgrade Job that registers external services in OpenMetadata when datasourcesIntegration.enabled=true. The Job waits for OpenMetadata to be ready, authenticates against the API and registers the enabled services.

In addition, OpenMetadata can run ingestion pipelines in two modes: via Kubernetes Jobs (default) or via Airflow.

Integration security

For security practices specific to the datasources job — Kubernetes Secrets, RBAC, HTTPS and password rotation — see Security — OpenMetadata.

Enable integrations per service

The registration job runs during install/upgrade when datasourcesIntegration.enabled: true.
To register a service in OpenMetadata, enable the desired service block and fill in the connection details.
Individual blocks are disabled by default — only the service you enable will be registered.

The datasourcesIntegration.openmetadata block defines how the job authenticates against the OpenMetadata API to perform registration.
When datasourcesIntegration.openmetadata.auth.passwordSecret.enabled: true, provide the Secret name and password key in the values file.

Default credentials for Superset and Airflow

Superset and Airflow use admin/admin as default credentials in the example values files. Change them in production via connection.username/connection.password in values or, preferably, via Kubernetes Secret.

Modular values files

The chart provides separate value files for each service, which can be used individually or combined in a single Helm command with multiple -f options.

File	Enabled integration
`values-datasources-integration.yaml`	All integrations at once
`values-trino-integration.yaml`	Trino only
`values-clickhouse-integration.yaml`	ClickHouse only
`values-superset-integration.yaml`	Superset only
`values-airflow-integration.yaml`	Airflow only
`values-kafka-integration.yaml`	Kafka only

Terminal input
# Trino only
helm upgrade --install <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-openmetadata \
  -n <NAMESPACE> -f values-trino-integration.yaml

# Trino + ClickHouse
helm upgrade --install <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-openmetadata \
  -n <NAMESPACE> -f values-trino-integration.yaml -f values-clickhouse-integration.yaml

# All integrations at once
helm upgrade --install <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-openmetadata \
  -n <NAMESPACE> -f values-datasources-integration.yaml

Per-service configuration

Each integration has an enabled field, serviceName/displayName fields (name and label visible in the UI) and a connection block under datasourcesIntegration.*.

Trino and ClickHouse

datasourcesIntegration:
  enabled: true
  openmetadata:
    apiEndpoint: "http://<OPENMETADATA_SERVICE>.<NAMESPACE>.svc.cluster.local:8585/api"
    auth:
      provider: "basic"
      username: "admin@open-metadata.org"
      passwordSecret:
        enabled: true
        name: "<SECRET_NAME>"
        key: "<SECRET_KEY>"

  trino:
    enabled: true
    serviceName: "tdp-trino"
    displayName: "TDP Trino"
    connection:
      host: "<TRINO_SERVICE>.<NAMESPACE>.svc.cluster.local"
      port: 8080
      username: "<USERNAME>"
      scheme: "http"

  clickhouse:
    enabled: true
    serviceName: "tdp-clickhouse"
    displayName: "TDP ClickHouse"
    connection:
      host: "<CLICKHOUSE_SERVICE>.<NAMESPACE>.svc.cluster.local"
      port: 8123
      username: "<USERNAME>"
      passwordSecret:
        name: "<SECRET_NAME>"
        key: "password"
      scheme: "http"

The ClickHouse Secret must exist before deployment: the default name is tdp-clickhouse-tdp-user, key password.

Production environment (TLS)

For TLS-enabled environments, use scheme: "https" and the secure port (e.g. 8443 for ClickHouse). For Trino with TLS, set scheme: "https" and adjust the port. See Security — OpenMetadata for complete examples.

Superset

Superset is registered as a Dashboard Service. The hostPort parameter includes the protocol and port.

datasourcesIntegration:
  superset:
    enabled: true
    serviceName: "tdp-superset"
    displayName: "TDP Superset"
    connection:
      hostPort: "http://tdp-superset.<NAMESPACE>.svc.cluster.local:8088"
      username: "<USERNAME>"
      password: "<PASSWORD>"
      provider: "db"
      verifySSL: "no-ssl"

note

Use provider: "ldap" if Superset is configured with LDAP. For HTTPS, change verifySSL to "validate" and configure the certificate.

Airflow

Airflow is registered as a Pipeline Service. Use the Airflow API server endpoint.

datasourcesIntegration:
  airflow:
    enabled: true
    serviceName: "tdp-airflow"
    displayName: "TDP Airflow"
    connection:
      hostPort: "http://tdp-airflow-api-server.<NAMESPACE>.svc.cluster.local:8080"
      numberOfStatus: 10
      username: "<USERNAME>"
      password: "<PASSWORD>"
      verifySSL: "no-ssl"

Kafka

Kafka is registered as a Messaging Service. Provide the bootstrap server address; for multiple brokers, separate with commas.

datasourcesIntegration:
  kafka:
    enabled: true
    serviceName: "tdp-kafka"
    displayName: "TDP Kafka"
    connection:
      bootstrapServers: "<KAFKA_BOOTSTRAP_SERVICE>.<NAMESPACE>.svc.cluster.local:9092"

For Kafka with SASL authentication, also configure saslUsername, saslPassword, saslMechanism and securityProtocol in the connection block.

Pipeline execution: Airflow or K8S

OpenMetadata can run ingestion pipelines in two modes:

Kubernetes Jobs (k8s): default mode, in which pipelines run as native Jobs in the cluster, without Airflow dependency
Airflow (airflow): mode in which pipelines are executed via REST integration with the Airflow service

The execution mode is controlled by the pipelineServiceClientConfig.type field, which accepts k8s or airflow.

Kubernetes Jobs mode (`type: k8s`)

When k8s mode is active, pipelines are submitted as Kubernetes Jobs. The most common parameters in this scenario are:

image used for execution
namespace where Jobs will be created
service account
CPU and memory resources
TTL and backoff policy
nodeSelector, annotations and securityContext

Airflow mode (`type: airflow`)

When airflow mode is active, OpenMetadata delegates pipeline execution to the Airflow service. The most common parameters in this scenario are:

Airflow endpoint
username
password via Secret
SSL verification

How to switch

change only the value of pipelineServiceClientConfig.type to switch the execution mode
keep the specific parameters aligned with the chosen mode
airflow mode requires the Airflow service to be enabled and accessible in the chart dependencies

Verification

After the Job completes:

Access the OpenMetadata UI (port-forward or Ingress).
Navigate to Settings → Services.
Verify the registered services by type:
- Databases: TDP ClickHouse, TDP Trino
- Dashboards: TDP Superset
- Pipelines: TDP Airflow
- Messaging: TDP Kafka
To test the connection: open the service, click Test Connection and confirm success.
To start metadata ingestion: open the service, click Add Ingestion and configure the workflow.

Terminal input

kubectl -n <NAMESPACE> get jobs | grep register-datasources
kubectl -n <NAMESPACE> logs job/<RELEASE_NAME>-register-datasources

Re-run the Job

The Job is a post-install/post-upgrade hook. To re-run it:

Terminal input
# Delete the existing Job
kubectl delete job -n <NAMESPACE> <RELEASE_NAME>-register-datasources

# Re-run via helm upgrade (the hook recreates the Job)
helm upgrade <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-openmetadata \
  -n <NAMESPACE> \
  -f <VALUES_FILE>

Enable and disable services individually

To disable a service without removing its configuration:

datasourcesIntegration:
  clickhouse:
    enabled: false
  trino:
    enabled: true
  superset:
    enabled: true

To disable the entire Job (no service is registered):

datasourcesIntegration:
  enabled: false

Advanced Job configuration

Retries and TTL

datasourcesIntegration:
  job:
    backoffLimit: 5
    ttlSecondsAfterFinished: 600

Job image

datasourcesIntegration:
job:
  image:
    repository: "docker.getcollate.io/openmetadata/ingestion"
    tag: "1.12.4"
    pullPolicy: "IfNotPresent"

Job resources

datasourcesIntegration:
  job:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 1
        memory: 1Gi

JWT authentication

If OpenMetadata is configured with JWT instead of basic authentication, change the provider in the job authentication block:

datasourcesIntegration:
  openmetadata:
    auth:
      provider: "jwt"
      # The Job must be adjusted to use JWT tokens instead of username/password

Configuration examples

Development environment

Minimal configuration for a local or development environment, with ClickHouse and Trino accessible in the cluster, without TLS and with plain credentials in the values file:

datasourcesIntegration:
  enabled: true

  openmetadata:
    apiEndpoint: "http://<RELEASE_NAME>.<NAMESPACE>.svc.cluster.local:8585/api"
    auth:
      provider: "basic"
      username: "admin@open-metadata.org"
      password: "admin"

  clickhouse:
    enabled: true
    serviceName: "tdp-clickhouse"
    displayName: "TDP ClickHouse"
    connection:
      host: "<CLICKHOUSE_SERVICE>.<NAMESPACE>.svc.cluster.local"
      port: 8123
      scheme: "http"
      username: "tdpuser"

  trino:
    enabled: true
    serviceName: "tdp-trino"
    displayName: "TDP Trino"
    connection:
      host: "<TRINO_SERVICE>.<NAMESPACE>.svc.cluster.local"
      port: 8080
      scheme: "http"
      username: "admin"

Production environment

Configuration for production with TLS enabled, credentials via Kubernetes Secrets and HTTPS OpenMetadata endpoint:

datasourcesIntegration:
  enabled: true

  openmetadata:
    apiEndpoint: "https://<OPENMETADATA_HOSTNAME>/api"
    auth:
      provider: "basic"
      username: "admin@open-metadata.org"
      passwordSecret:
        enabled: true
        name: "<SECRET_NAME>"
        key: "password"

  clickhouse:
    enabled: true
    serviceName: "tdp-clickhouse"
    displayName: "TDP ClickHouse"
    connection:
      host: "<CLICKHOUSE_SERVICE>.<NAMESPACE>.svc.cluster.local"
      port: 8443
      scheme: "https"
      username: "tdpuser"
      passwordSecret:
        name: "tdp-clickhouse-tdp-user"
        key: "password"

  trino:
    enabled: true
    serviceName: "tdp-trino"
    displayName: "TDP Trino"
    connection:
      host: "<TRINO_SERVICE>.<NAMESPACE>.svc.cluster.local"
      port: <SERVICE_PORT>
      scheme: "https"
      username: "admin"

  superset:
    enabled: true
    serviceName: "tdp-superset"
    displayName: "TDP Superset"
    connection:
      hostPort: "https://tdp-superset.<NAMESPACE>.svc.cluster.local:8088"
      username: "admin"
      passwordSecret:
        name: "<SECRET_NAME>"
        key: "password"
      provider: "db"
      verifySSL: "validate"

note

In production, never store passwords directly in the values file. Always use passwordSecret referencing a pre-created Kubernetes Secret.

Troubleshooting

Terminal input
kubectl -n <NAMESPACE> get jobs
kubectl -n <NAMESPACE> logs job/<RELEASE_NAME>-register-datasources
kubectl -n <NAMESPACE> describe job <RELEASE_NAME>-register-datasources

OpenMetadata REST API

The job uses the OpenMetadata REST API to register services. Main endpoints:

POST /api/v1/services/databaseServices — create service
PUT /api/v1/services/databaseServices — update service
GET /api/v1/services/databaseServices/name/{name} — fetch by name

Full documentation: docs.open-metadata.org/swagger.html

Symptom	Likely cause	Fix
Job fails: "OpenMetadata not ready"	OpenMetadata still initializing; the Job waits up to 30 attempts (approx. 5 min)	Check pods: `kubectl get pods -n <NAMESPACE>`
Job fails: "Authentication error"	Incorrect credentials in values file	Confirm `username`/`password` in `datasourcesIntegration.openmetadata.auth`
Job fails: "Secret not found"	ClickHouse Secret missing	Confirm: `kubectl get secret -n <NAMESPACE> tdp-clickhouse-tdp-user`
Job fails: "Service unreachable"	Incorrect hostname or port	Check `host`/`hostPort`/`bootstrapServers` in values
Service not visible in UI	Job did not complete or failed silently	Check Job logs and status

Integration overview​

Enable integrations per service​

Modular values files​

Per-service configuration​

Trino and ClickHouse​

Superset​

Airflow​

Kafka​

Pipeline execution: Airflow or K8S​

Kubernetes Jobs mode (type: k8s)​

Airflow mode (type: airflow)​

How to switch​

Verification​

Re-run the Job​

Enable and disable services individually​

Advanced Job configuration​

Retries and TTL​

Job image​

Job resources​

JWT authentication​

Configuration examples​

Development environment​

Production environment​

Troubleshooting​