Integrations - Spark
Overview
The tdp-spark chart organizes integrations through values blocks and ConfigMaps.
Jupyter and Airflow receive Spark client settings, while Delta Lake and Iceberg are optional blocks for table-format scenarios.
The most important point is that these blocks do not replace the Spark configuration required by the environment.
When an integration needs catalogs, libraries, an S3 endpoint, or a metastore, also complete spark.sparkConf and tdp-spark.customSparkConfig.properties.
Configuration pattern
There are three main paths for configuring Spark and storage:
| Block | Role |
|---|---|
spark.sparkConf | Spark and spark.hadoop.* properties passed to the Spark runtime |
tdp-spark.hadoopConfig | Values rendered as core-site.xml by the TDP wrapper |
tdp-spark.customSparkConfig.properties | Content rendered as spark-defaults.conf by the TDP wrapper |
Use spark.sparkConf for Spark properties, hadoopConfig for shared Hadoop/S3A settings, and customSparkConfig.properties for additional submission defaults.
Jupyter
When tdp-spark.integration.jupyter.enabled: true, the chart renders an integration ConfigMap for Jupyter environments.
This ConfigMap contains a spark-defaults.conf with the pairs defined in tdp-spark.integration.jupyter.sparkConfig and an environment setup script for notebooks.
Enablement example:
helm upgrade --install <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-spark -n <NAMESPACE> --set tdp-spark.integration.jupyter.enabled=true
Values model:
tdp-spark:
integration:
jupyter:
enabled: true
sparkConfig:
"spark.master": "spark://<SPARK_MASTER_SERVICE>.<NAMESPACE>.svc.cluster.local:7077"
"spark.serializer": "org.apache.spark.serializer.KryoSerializer"
"spark.driver.bindAddress": "0.0.0.0"
The spark.master value must point to the Spark master Service as seen from the namespace where the Jupyter client runs.
If Jupyter and Spark run in different namespaces, validate DNS, NetworkPolicy, and ports before submitting notebooks.
Airflow
The tdp-spark.integration.airflow.sparkConfig block stores Spark client defaults for Airflow environments.
Use this block to keep values such as master, driver memory, executor memory, and core count in one versioned configuration point.
Values model:
tdp-spark:
integration:
airflow:
enabled: true
sparkConfig:
"spark.master": "spark://<SPARK_MASTER_SERVICE>.<NAMESPACE>.svc.cluster.local:7077"
"spark.driver.memory": "1g"
"spark.executor.memory": "2g"
"spark.executor.cores": "1"
In Airflow, align these values with the operator used to submit Spark jobs.
The Spark chart does not replace DAG configuration; it provides defaults and conventions for the client.
Object Storage S3/S3A
S3/S3A is the usual path for integrating Spark with compatible storage, such as Ozone S3 Gateway, MinIO, or another S3 endpoint.
Keys can appear in spark.sparkConf, tdp-spark.hadoopConfig, or tdp-spark.customSparkConfig.properties.
Property model in spark.sparkConf:
spark:
sparkConf:
"spark.hadoop.fs.s3a.endpoint": "http://<S3_ENDPOINT>:<S3_PORT>"
"spark.hadoop.fs.s3a.access.key": "<ACCESS_KEY>"
"spark.hadoop.fs.s3a.secret.key": "<SECRET_KEY>"
"spark.hadoop.fs.s3a.path.style.access": "true"
Sensitive credentials and endpoints should be treated as environment values, private values files, or Secrets managed by the platform.
For best practices, see Security - Spark.
Delta Lake
The tdp-spark.deltaLake block is optional and organizes values related to Delta Lake.
Enabling this block does not automatically change spark.sparkConf; provide Spark properties, dependencies, and storage settings according to the Spark image used in the environment.
Minimal block enablement model:
tdp-spark:
deltaLake:
enabled: true
Use tdp-spark.customSparkConfig.properties to declare additional defaults when the image requires packages, extensions, or catalogs.
Iceberg
The tdp-spark.iceberg block is optional and organizes values related to Apache Iceberg.
When using Hive Metastore and S3/S3A, also configure the catalog and warehouse in Spark properties.
Configuration model:
spark:
sparkConf:
"spark.sql.catalog.iceberg": "org.apache.iceberg.spark.SparkCatalog"
"spark.sql.catalog.iceberg.type": "hive"
"spark.sql.catalog.iceberg.uri": "thrift://<HIVE_METASTORE_SERVICE>.<NAMESPACE>.svc.cluster.local:9083"
"spark.sql.catalog.iceberg.warehouse": "s3a://<S3_BUCKET>/hive"
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
This example is a structural model.
Validate libraries, Spark version, catalog, and metastore according to the image and services installed in the environment.
Validation
Confirm Services for the Spark release:
kubectl -n <NAMESPACE> get svc -l app.kubernetes.io/instance=<RELEASE_NAME>
When Jupyter integration is enabled, check the generated ConfigMap:
kubectl -n <NAMESPACE> get configmap tdp-spark-jupyter-integration
To validate access to the master from another namespace, run a DNS and port test with a debug image approved for your environment.
The expected endpoint follows the format spark://<SPARK_MASTER_SERVICE>.<NAMESPACE>.svc.cluster.local:7077.