TDP Kubernetes

Prerequisites

• Kubernetes Cluster: version 1.27 or higher
• kubectl installed and configured
• Helm: version 3.2.0 or higher
• Access to Tecnisys OCI registry (registry.tecnisys.com.br)
• Minimum recommended cluster resources:
  • CPU: 16 vCPUs available
  • RAM: 32 GB available
  • Persistent storage: 200 GB available

What is TDP Kubernetes?

TDP Kubernetes is the cloud-native edition of the TDP platform, designed to run on Kubernetes clusters. This edition uses containers and orchestration to provide:

• Automatic horizontal scalability of components.
• High availability with automatic failover.
• Declarative deployment via Helm Charts or ArgoCD.
• Resource isolation and efficient management.

This solution is ideal for teams seeking a modern, scalable data platform based on DevOps and GitOps practices.

This guide will help you with deployment, configuration, and access to TDP Kubernetes on a Kubernetes cluster.

1. Environment Preparation

Before starting the installation, ensure the Kubernetes cluster is configured and accessible.

1.1. Verify cluster access

Run the command below to validate access to the Kubernetes cluster:

Terminal input

kubectl cluster-info

The output should display information about the cluster, including the API server address.

1.2. Create namespace for TDP

TDP Kubernetes uses the tdp-project namespace as default. Create the namespace with the command:

Terminal input

kubectl create namespace tdp-project

1.3. Configure OCI registry access

TDP Kubernetes Helm Charts are hosted in the Tecnisys OCI registry. Configure access with your credentials:

Terminal input

helm registry login registry.tecnisys.com.br \
  --username <your-username> \
  --password <your-password>

note

Replace <your-username> and <your-password> with the credentials provided by Tecnisys.

2. Installation via Helm Charts

Helm is the recommended method for installing individual TDP Kubernetes components.

2.1. Add Helm repository (if applicable)

If the registry uses a traditional Helm repository, add it:

Terminal input

helm repo add tecnisys https://registry.tecnisys.com.br/helm
helm repo update

2.2. Install TDP components

Each TDP component has its own Helm Chart. Example installation of Apache Kafka:

Terminal input

helm install tdp-kafka oci://registry.tecnisys.com.br/helm/tdp-kafka \
  --namespace tdp-project \
  --version 3.0.0 \
  --create-namespace

2.3. Customize the installation

You can customize the installation by creating a values.yaml file:

values.yaml

kafka:
  replicas: 3
  resources:
    requests:
      memory: "4Gi"
      cpu: "2"
    limits:
      memory: "8Gi"
      cpu: "4"
  persistence:
    enabled: true
    size: 100Gi

Apply the configuration during installation:

Terminal input

helm install tdp-kafka oci://registry.tecnisys.com.br/helm/tdp-kafka \
  --namespace tdp-project \
  --values values.yaml

2.4. Verify the installation

List the resources created in the namespace:

Terminal input

kubectl get all -n <namespace>

Check the status of the Pods:

Terminal input

kubectl get pods -n <namespace>

All Pods should be in Running status after a few minutes.

3. Installation via ArgoCD (GitOps)

ArgoCD allows managing component deployment declaratively via Git.

3.1. Install ArgoCD

If ArgoCD is not yet installed in the cluster, install it with the commands:

Terminal input

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

3.2. Access the ArgoCD interface

Expose the ArgoCD service:

Terminal input

kubectl port-forward svc/argocd-server -n argocd 8080:443

Access the interface at https://localhost:8080.

The initial password for the admin user can be obtained with:

Terminal input

kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath="{.data.password}" | base64 -d

3.3. Create Application in ArgoCD

Create a YAML file describing the TDP application:

tdp-application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: tdp-kafka
  namespace: argocd
spec:
  project: default
  source:
    repoURL: oci://registry.tecnisys.com.br/helm
    targetRevision: 3.0.0
    chart: tdp-kafka
  destination:
    server: https://kubernetes.default.svc
    namespace: tdp-project
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply the configuration:

Terminal input

kubectl apply -f tdp-application.yaml

ArgoCD will automatically synchronize the desired state with the cluster.

4. Accessing Components

After installation, components can be accessed via Kubernetes Services.

4.1. List available Services

Terminal input

kubectl get services -n <namespace>

4.2. Access web interfaces via Port Forwarding

Example to access Kafka UI:

Terminal input

kubectl port-forward svc/tdp-kafka-ui -n <namespace> 8080:80

Access the interface at http://localhost:8080.

4.3. Configure Ingress (optional)

For permanent external access, configure an Ingress Controller (such as NGINX Ingress):

kafka-ui-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kafka-ui-ingress
  namespace: tdp-project
spec:
  rules:
  - host: kafka-ui.tdp.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: tdp-kafka-ui
            port:
              number: 80

Apply the Ingress:

Terminal input

kubectl apply -f kafka-ui-ingress.yaml

Configure local DNS resolution (/etc/hosts on Linux or C:\Windows\System32\drivers\etc\hosts on Windows):

<CLUSTER-IP> kafka-ui.tdp.local

5. Main TDP Kubernetes Components

TDP Kubernetes consists of several components organized in layers:

Component	Description	Helm Chart
Apache Kafka	Event streaming	tdp-kafka
Apache NiFi	Data flow orchestration	tdp-nifi
Trino	Distributed SQL query engine	tdp-trino
Apache Airflow	Workflow orchestration	tdp-airflow
Apache Superset	Visualization and BI	tdp-superset
PostgreSQL	Relational database	tdp-postgresql
ClickHouse	OLAP database	tdp-clickhouse

To install multiple components, repeat the installation steps via Helm or ArgoCD for each chart.

6. Troubleshooting

6.1. Pods not starting

Check the logs of the problematic Pod:

Terminal input

kubectl logs <pod-name> -n <namespace>

Check namespace events:

Terminal input

kubectl get events -n <namespace> --sort-by='.lastTimestamp'

6.2. Registry authentication issues

Recreate the authentication secret:

Terminal input

kubectl delete secret regcred -n <namespace>
kubectl create secret docker-registry regcred \
  --docker-server=registry.tecnisys.com.br \
  --docker-username=<your-username> \
  --docker-password=<your-password> \
  -n <namespace>

6.3. Insufficient resources

Check resource consumption in the cluster:

Terminal input

kubectl top nodes
kubectl top pods -n <namespace>

Adjust resource limits in the values.yaml file and run an upgrade:

Terminal input

helm upgrade tdp-kafka oci://registry.tecnisys.com.br/helm/tdp-kafka \
  --namespace tdp-project \
  --values values.yaml

Next Steps

After completing the TDP Kubernetes installation, consult the full documentation for:

• Advanced configuration of components
• Integration with monitoring tools (Prometheus, Grafana)
• Backup and recovery of data
• Version updates of components
• Security and access control (RBAC, Network Policies)

For more information about TDP Kubernetes concepts and architecture, see the Concepts section in the documentation.