TDP Kubernetes

Prerequisites

• Kubernetes cluster 1.32+
• kubectl installed and configured
• Helm 3.2.0+
• Access to Tecnisys OCI registry (registry.tecnisys.com.br)
• Recommended minimum cluster resources:
  • CPU: 16 vCPUs available
  • RAM Memory: 32 GB available
  • Persistent storage: 200 GB available

What is TDP Kubernetes?

TDP Kubernetes is the cloud-native edition of the TDP platform, designed to run on Kubernetes clusters. This edition uses containers and orchestration to provide:

• Horizontal autoscaling of components.
• High availability with automatic failover.
• Declarative deployment via Helm Charts or ArgoCD.
• Resource isolation and efficient management.

This solution is ideal for teams seeking a modern, scalable data platform based on DevOps and GitOps practices.

This guide will help you deploy, configure, and access TDP Kubernetes on a Kubernetes cluster.

1. Environment Preparation

Before starting the installation, ensure that the Kubernetes cluster is configured and accessible.

1.1. Verify cluster access

Execute the command below to validate access to the Kubernetes cluster:

Terminal input

kubectl cluster-info

The output should display cluster information, including the API server address.

1.2. Create namespace for TDP

Create a dedicated namespace for TDP components. The name is flexible — choose an identifier aligned with your namespace policy (for example tdp-staging, data-platform).

Terminal input

kubectl create namespace <NAMESPACE>

Example

Namespace and release names used in illustrative sections below: tdp-project as namespace and releases aligned with the chart name (for example tdp-kafka).

Example

kubectl create namespace tdp-project

1.3. Configure access to OCI registry

TDP Kubernetes Helm Charts are hosted in the Tecnisys OCI registry. Configure access with your credentials:

Terminal input

helm registry login registry.tecnisys.com.br \
  --username <username> \
  --password <password>

note

Replace <username> and <password> with credentials provided by Tecnisys.

2. Installation via Helm Charts

Helm is the recommended method for installing individual TDP Kubernetes components.

2.1. Add Helm repository (if applicable)

If the registry uses a traditional Helm repository, add it:

Terminal input

helm repo add tecnisys https://registry.tecnisys.com.br/helm
helm repo update

2.2. Install TDP components

Each TDP component has its own Helm Chart.

Illustrative installation of Apache Kafka (replace <RELEASE_NAME> with the Helm release name and <NAMESPACE> with the target namespace):

Terminal input

helm install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka \
  --namespace <NAMESPACE> \
  --version <CHART_VERSION> \
  --create-namespace \
  -f <VALUES_FILE>

Example

Example

helm install tdp-kafka oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka \
  --namespace tdp-project \
  --version 3.0.1 \
  --create-namespace \
  -f values-kafka.yaml

2.3. Customize the installation

You can customize the installation by creating a values.yaml file:

values.yaml

kafka:
  replicas: 3
  resources:
    requests:
      memory: "4Gi"
      cpu: "2"
    limits:
      memory: "8Gi"
      cpu: "4"
  persistence:
    enabled: true
    size: 100Gi

Apply the configuration during installation:

Terminal input

helm install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka \
  --namespace <NAMESPACE> \
  --version <CHART_VERSION> \
  -f <VALUES_FILE>

Example

Example

helm install tdp-kafka oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka \
  --namespace tdp-project \
  --version 3.0.1 \
  -f values.yaml

2.4. Verify installation

List the resources created in the namespace:

Terminal input

kubectl get all -n <NAMESPACE>

Check Pod status:

Terminal input

kubectl get pods -n <NAMESPACE>

All Pods should be in Running status after a few minutes.

3. Installation via ArgoCD (GitOps)

ArgoCD allows managing component deployment declaratively via Git.

3.1. Install ArgoCD

If ArgoCD is not yet installed in the cluster, install it with the commands:

Terminal input

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

3.2. Access the ArgoCD interface

Expose the ArgoCD service:

Terminal input

kubectl port-forward svc/argocd-server -n argocd 8080:443

Access the interface at https://localhost:8080.

The initial password for the admin user can be obtained with:

Terminal input

kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath="{.data.password}" | base64 -d

3.3. Create Application in ArgoCD

Create a YAML file describing the TDP application, as shown in the example:

tdp-application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tdp-kafka
  namespace: argocd
spec:
  project: default
  source:
    repoURL: oci://registry.tecnisys.com.br/tdp/charts
    chart: tdp-kafka
    targetRevision: 3.0.1
  destination:
    server: https://kubernetes.default.svc
    namespace: tdp-project
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Apply the configuration:

Terminal input

kubectl apply -f <application-manifest>

Example

Example

kubectl apply -f tdp-application.yaml

ArgoCD will automatically synchronize the desired state with the cluster.

4. Access Components

After installation, components can be accessed via Kubernetes Services.

4.1. List available Services

Terminal input

kubectl get services -n <NAMESPACE>

4.2. Access web interfaces via Port Forwarding

To expose the Kafka UI locally:

Terminal input

kubectl port-forward svc/kafka-ui 8080:80 -n <NAMESPACE>

Example

In common Strimzi installations, the UI service may appear as tdp-kafka-kafka-ui (adjust to your cluster).

Example

kubectl port-forward svc/tdp-kafka-kafka-ui 8080:80 -n tdp-project

Access the interface at http://localhost:8080.

4.3. Configure Ingress (optional)

For permanent external access, configure an Ingress Controller (such as NGINX Ingress).

Generic manifest (replace <NAMESPACE>, <INGRESS_NAME> and <KAFKA_UI_HOSTNAME>):

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <INGRESS_NAME>
  namespace: <NAMESPACE>
spec:
  rules:
  - host: <KAFKA_UI_HOSTNAME>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kafka-ui
            port:
              number: 80

Example

Illustrative values:

Example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kafka-ui-ingress
  namespace: tdp-project
spec:
  rules:
  - host: kafka-ui.tdp.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: tdp-kafka-ui
            port:
              number: 80

Apply the Ingress:

Terminal input

kubectl apply -f <ingress-manifest>

Example

Example

kubectl apply -f kafka-ui-ingress.yaml

Configure local DNS resolution (/etc/hosts on Linux or C:\Windows\System32\drivers\etc\hosts on Windows):

<INGRESS_IP>   <hostname>

Example

192.168.1.50   kafka-ui.tdp.local

5. Main TDP Kubernetes Components

TDP Kubernetes consists of various components organized in layers:

Component	Description	Helm Chart
Apache Kafka	Event streaming	tdp-kafka
Apache NiFi	Data flow orchestration	tdp-nifi
Trino	Distributed SQL query engine	tdp-trino
Apache Airflow	Workflow orchestration	tdp-airflow
Apache Superset	Visualization and BI	tdp-superset
PostgreSQL	Relational database	tdp-postgresql
ClickHouse	OLAP database	tdp-clickhouse

To install multiple components, repeat the installation steps via Helm or ArgoCD for each chart.

6. Troubleshooting

6.1. Pods not initializing

Check the logs of the problematic Pod:

Terminal input

kubectl logs <pod-name> -n <NAMESPACE>

Check namespace events:

Terminal input

kubectl get events -n <NAMESPACE> --sort-by='.lastTimestamp'

6.2. Registry authentication issues

Recreate the authentication secret:

Terminal input

kubectl delete secret <docker-registry-secret> -n <NAMESPACE>
kubectl create secret docker-registry <docker-registry-secret> \
  --docker-server=registry.tecnisys.com.br \
  --docker-username=<username> \
  --docker-password=<password> \
  -n <NAMESPACE>

Example

Example

kubectl delete secret regcred -n <NAMESPACE>
kubectl create secret docker-registry regcred \
  --docker-server=registry.tecnisys.com.br \
  --docker-username=<username> \
  --docker-password=<password> \
  -n <NAMESPACE>

6.3. Insufficient resources

Check resource consumption on the cluster:

Terminal input

kubectl top nodes
kubectl top pods -n <NAMESPACE>

Adjust resource limits in the values.yaml file and run an upgrade:

Terminal input

helm upgrade <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka \
  --namespace <NAMESPACE> \
  --values values.yaml

Example

Example

helm upgrade tdp-kafka oci://registry.tecnisys.com.br/tdp/charts/tdp-kafka \
  --namespace tdp-project \
  --values values.yaml

Next Steps

After completing TDP Kubernetes installation, consult the full documentation for:

• Advanced configuration of components
• Integration with monitoring tools (Prometheus, Grafana)
• Backup and recovery of data
• Updates of component versions
• Security and access control (RBAC, Network Policies)

For more information on TDP Kubernetes concepts and architecture, refer to the Concepts section in the documentation.