Security - Spark

ChartVersion3.0.1TypeapplicationAppVersion4.0.2

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Overview

The tdp-spark chart concentrates security mainly on how Spark accesses S3/S3A storage and how sensitive values are supplied to configuration.
S3/S3A values should be treated as placeholders in versioned files and provided through Secrets or external tooling.

The practical idea is simple: Spark needs an endpoint, credentials, and permissions to read and write data; documentation should show where those values go, but should not store real secrets.

Storage access modes

Mode	Where to configure	Typical use
spark.sparkConf	spark.hadoop.fs.s3a.* keys	Properties passed directly to Spark
tdp-spark.hadoopConfig	fs.s3a.* keys	core-site.xml rendering by the TDP wrapper
tdp-spark.customSparkConfig.properties	spark-defaults.conf lines	Additional defaults used by Spark clients

Use only placeholders in versioned files.
Provide real values through private values files, Secrets, or external secrets management.

Prerequisites

• S3/S3A endpoint reachable from Spark pods.
• Credentials with appropriate permissions for the bucket or prefix used by jobs.
• Bucket or warehouse created before pipelines run.
• TLS/HTTP configuration consistent with the endpoint used.
• NetworkPolicy allowing traffic from Spark pods to the storage service.

S3/S3A credentials

Credentials can be provided through spark.sparkConf when the environment requires Spark to receive keys as spark.hadoop.* properties:

spark:
  sparkConf:
    "spark.hadoop.fs.s3a.endpoint": "http://<S3_ENDPOINT>:<S3_PORT>"
    "spark.hadoop.fs.s3a.access.key": "<ACCESS_KEY>"
    "spark.hadoop.fs.s3a.secret.key": "<SECRET_KEY>"
    "spark.hadoop.fs.s3a.path.style.access": "true"
    "spark.hadoop.fs.s3a.connection.ssl.enabled": "false"

Do not use real credentials in this example.
Replace placeholders with values supplied by a secure mechanism during installation.

core-site.xml configuration via hadoopConfig

The tdp-spark.hadoopConfig block is rendered by the chart as core-site.xml.
This file is read by Hadoop before many Spark defaults, so it is useful for filesystem and Hadoop authentication properties.

Model:

tdp-spark:
  hadoopConfig:
    "fs.s3a.endpoint": "http://<S3_ENDPOINT>:<S3_PORT>"
    "fs.s3a.access.key": "<ACCESS_KEY>"
    "fs.s3a.secret.key": "<SECRET_KEY>"
    "fs.s3a.path.style.access": "true"

The chart also uses this block for settings such as simple Hadoop authentication, fs.defaultFS, and S3A parameters.
Validate the final set with the values applied in the environment.

spark-defaults.conf configuration via customSparkConfig

The tdp-spark.customSparkConfig.properties block is rendered as spark-defaults.conf.
Use this path for additional Spark defaults, such as event log, local Ivy repository, Hadoop properties, and S3A tuning.

Model:

tdp-spark:
  customSparkConfig:
    properties: |
      spark.hadoop.fs.s3a.endpoint http://<S3_ENDPOINT>:<S3_PORT>
      spark.hadoop.fs.s3a.access.key <ACCESS_KEY>
      spark.hadoop.fs.s3a.secret.key <SECRET_KEY>
      spark.hadoop.fs.s3a.path.style.access true

Keep passwords, access keys, and secret keys out of the repository.

Jupyter and Airflow

Jupyter and Airflow integrations can load Spark defaults that point to the master and runtime settings.
If those clients access S3/S3A directly, they also need to receive the required properties and credentials securely.

Client configuration does not replace credential protection.
Review values applied in tdp-spark.integration.jupyter.sparkConfig and tdp-spark.integration.airflow.sparkConfig before exposing notebooks or DAGs to end users.

Main parameters

Parameter	Role	When to change
spark.sparkConf	Defines Spark and spark.hadoop.* properties	When the Spark runtime needs to receive properties directly
tdp-spark.hadoopConfig	Renders core-site.xml	When Hadoop/S3A configuration must be shared by pods
tdp-spark.customSparkConfig.properties	Renders spark-defaults.conf	When clients and jobs need additional defaults
tdp-spark.integration.jupyter.sparkConfig	Spark defaults for Jupyter	When notebooks submit jobs to the Spark cluster
tdp-spark.integration.airflow.sparkConfig	Spark defaults for Airflow	When DAGs submit jobs to the Spark cluster

Troubleshooting

Situation	What to check
Spark jobs fail with S3 access errors	Endpoint, access key, secret key, and bucket permissions
Connection refused to S3	Service name, namespace, port, and NetworkPolicy
AccessDeniedException	Credential permissions on the bucket or prefix
S3A configuration does not appear in the pod	core-site.xml, spark-defaults.conf, and values applied in the release
Notebook or DAG cannot access data	Jupyter/Airflow defaults and credentials available to the client

Best practices

• Do not store access.key, secret.key, tokens, or sensitive endpoints in Git repositories.
• Use private values files or external Secret mechanisms for real credentials.
• Separate values by environment, such as development, staging, and production.
• Rotate credentials periodically and apply helm upgrade when needed.
• Grant credentials only the permissions required for the bucket or prefix used by jobs.