Spark Configuration

ChartVersion3.0.1TypeapplicationAppVersion4.0.2

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Feature SupportS3supportedIngresssupportedGateway APIsupported

What is Apache Spark in TDP?

Apache Spark is the distributed processing engine used in TDP for data transformations, ETL pipelines, and batch workloads.
While Trino is focused on interactive SQL queries, Spark is better suited for long-running processing, complex transformations, and writing results back to distributed storage.

In TDP Kubernetes, Spark is usually consumed by clients such as Airflow and Jupyter.
Airflow schedules and triggers jobs, while Jupyter enables interactive use through notebooks.

To find out more

See Apache Spark - Concepts for a complete overview of the tool, its architecture, and operation.

Helm value structure

The tdp-spark chart is a wrapper around the upstream Spark chart. Values are distributed across two blocks:

• spark: — upstream chart values: executor, driver, workers, images, sparkConf, sparkContext.
• tdp-spark: — TDP wrapper values: hadoopConfig, customSparkConfig.properties, integrations (Jupyter, Airflow, Delta Lake, Iceberg), ServiceAccount.

spark:
  worker:
    replicaCount: 2

tdp-spark:
  hadoopConfig:
    "fs.defaultFS": "s3a://warehouse"

When in doubt about which block to use, refer to Main parameters or helm show values.

Master-worker architecture

The tdp-spark chart packages and configures an upstream Spark chart, adding TDP-specific configuration and integrations.
The deployment follows the standalone architecture with master and workers:

Component	Role
Master	Manages the Spark cluster, receives applications, and distributes work to workers
Workers	Run executors for Spark applications
History Server	Provides the execution history interface when enabled
TDP ConfigMaps	Render core-site.xml, spark-defaults.conf, and integration configuration when enabled

The Spark master is the cluster submission endpoint, usually reached by clients as spark://<SPARK_MASTER_SERVICE>:7077.
External drivers, such as Airflow or Jupyter notebooks, must reach this endpoint to submit jobs.

Overview

Property	Value
Chart	tdp-spark
Spark Version	4.0.2
Chart Version	3.0.1
Chart AppVersion	4.0.2
Registry	oci://registry.tecnisys.com.br/tdp/charts/tdp-spark

Related pages

• Integrations - Spark: Jupyter, Airflow, Delta Lake, Iceberg, and S3/S3A settings used by integrations.
• Security - Spark: S3/S3A credentials, placeholders, Secrets, and best practices.
• External exposure - Spark: Ingress (reverse proxy required) and Gateway API for the Spark UI.

Prerequisites

Before installing or customizing the tdp-spark chart, confirm:

• Kubernetes 1.32+, Red Hat OpenShift 4.19+, or Rancher Manager 2.10.x+.
• Helm 3.2.0+.
• A StorageClass available for persistent volumes.
• Tecnisys OCI registry reachable from the installation environment.
• When using S3/S3A: endpoint, credentials, bucket, and permissions available to Spark pods.
• When using Jupyter or Airflow: connectivity from those clients to the Spark master service.

Installation

Installation

Example

helm install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-spark -n <NAMESPACE> --create-namespace

Adjust resources, replicas, and Spark settings through your values file or --set, according to the environment.
The chart is a wrapper: some parameters belong to the upstream spark block, and others belong to the TDP wrapper tdp-spark block.

Main parameters

Main parameters

Parameter	Description	Note
spark.enabled	Enable Spark deployment	Controls the upstream dependency
spark.image.repository	Spark image repository	Validate the effective value in the installed version's values.yaml
spark.image.tag	Spark image tag	Aligned with the Spark version packaged by the chart
spark.master.resources	Master resources	Requests/limits for the master process
spark.worker.replicaCount	Number of workers	Use 2 as an initial reference
spark.worker.autoscaling.enabled	Enable worker HPA	Use false as an initial reference
spark.historyServer.enabled	Enable History Server	Use true as an initial reference
spark.sparkConf	Spark configuration map	Includes spark.* and spark.hadoop.* properties
tdp-spark.hadoopConfig	core-site.xml values rendered by the wrapper	Used for Hadoop/S3A properties
tdp-spark.customSparkConfig.properties	spark-defaults.conf content rendered by the wrapper	Used for additional Spark defaults
tdp-spark.integration.jupyter.enabled	Enable Jupyter integration configuration	Renders a ConfigMap when enabled
tdp-spark.integration.airflow.enabled	Enable Airflow configuration block	Stores Spark client defaults
tdp-spark.deltaLake.enabled	Optional Delta Lake block	Does not modify spark.sparkConf automatically
tdp-spark.iceberg.enabled	Optional Iceberg block	Does not modify spark.sparkConf automatically

The chart uses a tdp-spark.serviceAccount block for TDP wrapper templates and a spark.serviceAccount block for the upstream dependency.
Keep both aligned when customizing ServiceAccounts.

Configuration details

Spark configuration

Use spark.sparkConf for properties that Spark itself must receive, such as executor, driver, event log, and spark.hadoop.* keys.
Use tdp-spark.hadoopConfig when the TDP wrapper must render core-site.xml.
Use tdp-spark.customSparkConfig.properties when additional spark-defaults.conf content is required.

This separation matters: core-site.xml is read by Hadoop before many Spark defaults, while spark-defaults.conf defines submission and runtime defaults.
For credential and S3/S3A details, see Security - Spark.

S3/S3A storage

S3/S3A is a central chart configuration point.
Properties can be configured through spark.sparkConf, tdp-spark.hadoopConfig, or tdp-spark.customSparkConfig.properties.

On the general page, keep only the high-level view: Spark needs an endpoint, credentials, and permissions to access storage.
Complete examples and credential best practices are in Security - Spark.

Integrations and table formats

The chart provides blocks for Jupyter and Airflow integration, as well as optional blocks for Delta Lake and Iceberg.
These blocks organize configuration values, but Delta Lake and Iceberg do not automatically modify spark.sparkConf.
Required Spark properties and dependencies must be provided through spark.sparkConf and tdp-spark.customSparkConfig.properties.

See Integrations - Spark for examples and validation steps by integration.

Resources, workers, and autoscaling

The master coordinates the cluster; workers run executors.
Increase spark.worker.replicaCount when fixed execution capacity must grow.
Use spark.worker.autoscaling.enabled only when the cluster supports HPA and metrics are available.

Prerequisites for HPA

• The Metrics Server must be installed on the cluster.
• Worker resources (resources.requests.cpu and resources.requests.memory) must be configured for HPA to work.

Example HPA configuration for workers:

spark:
  worker:
    replicaCount: 2
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      targetCPU: 70
      targetMemory: 80
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"

Adjust master, worker, driver, and executor resources together.
Configuring many executors without enough worker resources can create queues, scheduling failures, or memory pressure.

Access & Security

External exposure

The chart supports external exposure via Ingress (provided by the upstream Bitnami subchart, configured under spark.ingress.*) and via Gateway API (configured under gatewayApi.spark.*).

Reverse Proxy required for Ingress

For the Spark Master UI and worker/application links to work correctly through an Ingress, the native Spark Reverse Proxy must be enabled:

spark:
  master:
    configOptions: "-Dspark.ui.reverseProxy=true -Dspark.ui.reverseProxyUrl=http://<SPARK_HOSTNAME>"
  worker:
    configOptions: "-Dspark.ui.reverseProxy=true -Dspark.ui.reverseProxyUrl=http://<SPARK_HOSTNAME>"
  sparkConf:
    "spark.ui.reverseProxy": "true"
    "spark.ui.reverseProxyUrl": "http://<SPARK_HOSTNAME>"

Without this configuration, "Back to Master" links and worker redirects point to internal cluster addresses that are not accessible externally.

When submitting applications, provide the same parameters in spark-submit:

spark-submit \
  --conf spark.ui.reverseProxy=true \
  --conf spark.ui.reverseProxyUrl=http://<SPARK_HOSTNAME> \
  ...

Without these parameters, navigating through the Spark Web UI tabs redirects you to the home page instead of job information.

See External Exposure — Spark for complete Ingress and Gateway API examples.

Local access

For debugging or temporary access without Ingress, use port-forward for the Spark Master UI and, when enabled, the History Server:

List Services created by the release:

kubectl -n <NAMESPACE> get svc -l app.kubernetes.io/instance=<RELEASE_NAME>

For local debugging, use port-forward to the Spark master UI and, when enabled, to the History Server:

kubectl -n <NAMESPACE> port-forward svc/<RELEASE_NAME>-master-svc 8080:80

Troubleshooting

Troubleshooting

kubectl -n <NAMESPACE> get pods
kubectl -n <NAMESPACE> get events --sort-by=.lastTimestamp

First check that the master and workers are healthy.
Then validate Services, S3/S3A configuration, and logs from jobs submitted by the client used in the environment.

Uninstallation

Uninstallation

helm uninstall <RELEASE_NAME> -n <NAMESPACE>

For the complete parameter list, inspect the values of the installed chart version.