Version 3.0

Integrations - Apache Ozone

ChartVersion3.0.1TypeapplicationAppVersion2.0.0

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Integration overview

Ozone provides S3-compatible object storage that can replace MinIO or AWS S3 in your TDP stack. The table below summarizes which components can be integrated, with priority and complexity indicators:

Component	Integration Type	Priority	Complexity
Spark	Primary storage for data processing	⭐⭐⭐ High	Low
Trino	Catalog backend for Hive/Iceberg	⭐⭐⭐ High	Low
Hive Metastore	Warehouse storage	⭐⭐⭐ High	Low
Delta Lake	Table storage and maintenance	⭐⭐⭐ High	Low
Iceberg	Table format storage	⭐⭐⭐ High	Low
Airflow	DAG storage, logs, XCom backend	⭐⭐ Medium	Medium
NiFi	Data flow storage and content repo	⭐⭐ Medium	Medium
Jupyter	Notebook storage, data access	⭐⭐ Medium	Low
Superset	Query results cache	⭐ Low	Low
Ranger	Audit logs storage	⭐ Low	Medium

Default internal S3 Gateway REST endpoint:

http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

Use Ingress only when the consumer is outside the cluster or when the architecture requires an external hostname, such as https://<OZONE_S3_REST_HOSTNAME>.

S3 endpoint security

The S3 Gateway REST is the Ozone data plane. Before exposing it or configuring consumers, ensure authentication is enabled, TLS is in place for traffic outside the cluster, and appropriate network controls are configured. See Security - Apache Ozone.

Credentials and endpoint

The examples below use AWS Signature v4 in simple mode. Credentials must come from Kubernetes Secrets, secure environment variables, or an equivalent mechanism of the consuming component. Do not commit <AWS_ACCESS_KEY_ID> or <AWS_SECRET_ACCESS_KEY> to Git.

For Ozone, keep path-style access enabled in S3 consumers, as the bucket is normally part of the URL path.

For details on the credential model, rotation, TLS, and network controls, see the Ozone security page.

Spark with S3A

Why integrate

Store input/output data for Spark jobs
Persist DataFrames and RDDs
Share data between Spark applications
Enable data lake architectures

Configuration

Minimum connection properties:

spark.hadoop.fs.s3a.endpoint=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
spark.hadoop.fs.s3a.access.key=<AWS_ACCESS_KEY_ID>
spark.hadoop.fs.s3a.secret.key=<AWS_SECRET_ACCESS_KEY>
spark.hadoop.fs.s3a.path.style.access=true
spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.connection.ssl.enabled=false

For high-volume data workloads, add these tuning properties to the same block:

spark.hadoop.fs.s3a.connection.maximum=100
spark.hadoop.fs.s3a.threads.max=64
spark.hadoop.fs.s3a.fast.upload=true
spark.hadoop.fs.s3a.block.size=128M

In Kubernetes deployments, deliver the properties via a ConfigMap. Create the manifest:

spark-ozone-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-custom-defaults
  namespace: <NAMESPACE>
data:
  spark-defaults.conf: |
    spark.hadoop.fs.s3a.endpoint=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
    spark.hadoop.fs.s3a.access.key=<AWS_ACCESS_KEY_ID>
    spark.hadoop.fs.s3a.secret.key=<AWS_SECRET_ACCESS_KEY>
    spark.hadoop.fs.s3a.path.style.access=true
    spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
    spark.hadoop.fs.s3a.connection.ssl.enabled=false
    spark.hadoop.fs.s3a.connection.maximum=100
    spark.hadoop.fs.s3a.threads.max=64
    spark.hadoop.fs.s3a.fast.upload=true
    spark.hadoop.fs.s3a.block.size=128M

Apply the ConfigMap and reference it in the Spark chart values.yaml:

Terminal input

kubectl apply -f spark-ozone-config.yaml -n <NAMESPACE>

values.yaml (excerpt)
spark:
  master:
    existingConfigmap: spark-custom-defaults
  worker:
    existingConfigmap: spark-custom-defaults

Usage example

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Ozone S3 Example") \
    .getOrCreate()

# Read from Ozone
df = spark.read.parquet("s3a://warehouse/data/input.parquet")

# Process data
result = df.groupBy("category").count()

# Write to Ozone
result.write.mode("overwrite").parquet("s3a://warehouse/data/output.parquet")

Kubernetes Secret creation

Terminal input
kubectl create secret generic spark-ozone-credentials \
  --from-literal=access-key=<AWS_ACCESS_KEY_ID> \
  --from-literal=secret-key=<AWS_SECRET_ACCESS_KEY> \
  -n <NAMESPACE>

Trino with native S3

Why integrate

Query data stored in Ozone via Hive/Iceberg catalogs
Federated queries across multiple data sources
High-performance analytics on object storage

Configuration

Configure the catalogs in the Trino chart values.yaml:

values.yaml (excerpt)
tdp-trino:
  catalogs:
    hive: |
      connector.name=hive
      hive.metastore.uri=thrift://tdp-hive-metastore.<NAMESPACE>.svc.cluster.local:9083
      hive.non-managed-table-writes-enabled=true
      hive.non-managed-table-creates-enabled=true
      fs.native-s3.enabled=true
      s3.endpoint=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
      s3.region=us-east-1
      s3.aws-access-key=<AWS_ACCESS_KEY_ID>
      s3.aws-secret-key=<AWS_SECRET_ACCESS_KEY>
      s3.path-style-access=true

    iceberg: |
      connector.name=iceberg
      iceberg.catalog.type=hive_metastore
      hive.metastore.uri=thrift://tdp-hive-metastore.<NAMESPACE>.svc.cluster.local:9083
      fs.native-s3.enabled=true
      s3.endpoint=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
      s3.region=us-east-1
      s3.aws-access-key=<AWS_ACCESS_KEY_ID>
      s3.aws-secret-key=<AWS_SECRET_ACCESS_KEY>
      s3.path-style-access=true

Adjust Service names, catalogs, and Secrets to the actual releases in the environment.

Usage example

-- Create schema pointing to an Ozone bucket
CREATE SCHEMA iceberg.warehouse WITH (location = 's3a://warehouse/');

-- Create an Iceberg table
CREATE TABLE iceberg.warehouse.sales (
    id      BIGINT,
    product VARCHAR,
    amount  DECIMAL(10,2),
    sale_date DATE
) WITH (format = 'PARQUET', location = 's3a://warehouse/vendas');

-- Insert data
INSERT INTO iceberg.warehouse.sales VALUES
    (1, 'Laptop', 1200.00, DATE '2024-01-15'),
    (2, 'Mouse',    25.50, DATE '2024-01-16');

-- Query
SELECT produto, SUM(valor) AS total FROM iceberg.warehouse.sales GROUP BY produto;

Hive Metastore and warehouse

Why integrate

Store table metadata and warehouse data in Ozone
Central metadata repository for Spark, Trino, and other tools

Configuration

Configure Hive Metastore to use Ozone as the warehouse in values.yaml:

values.yaml (excerpt)
tdp-hive-metastore:
  metastore:
    type: s3
    s3:
      endpoint: http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
      accessKey: <AWS_ACCESS_KEY_ID>
      secretKey: <AWS_SECRET_ACCESS_KEY>
      bucket: warehouse
      pathStyleAccess: true
  warehouse:
    dir: s3a://warehouse/hive

When Spark and Trino share the same warehouse, keep endpoint, region, access style, and credentials aligned on both sides.

Iceberg and Delta Lake

Why integrate

Store Iceberg and Delta Lake tables in Ozone
ACID transactions on object storage
Time travel and data versioning

Configuration

Example locations:

s3a://warehouse/iceberg
s3a://warehouse/delta

Prefer separate buckets and prefixes by domain or environment.

For Delta Lake maintenance jobs (VACUUM, OPTIMIZE), configure the Delta Lake chart values.yaml:

values.yaml (excerpt)
maintenance:
  spark:
    config:
      "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension"
      "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog"
      "spark.hadoop.fs.s3a.endpoint": "http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878"
      "spark.hadoop.fs.s3a.path.style.access": "true"
      "spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"
      "spark.hadoop.fs.s3a.connection.ssl.enabled": "false"
      "spark.databricks.delta.retentionDurationCheck.enabled": "false"

  defaultTablePath: "s3a://warehouse/delta"

To run VACUUM directly via spark-sql:

Terminal input
spark-sql \
  --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
  --conf spark.hadoop.fs.s3a.endpoint=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878 \
  --conf spark.hadoop.fs.s3a.path.style.access=true \
  --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
  --conf spark.hadoop.fs.s3a.access.key=<AWS_ACCESS_KEY_ID> \
  --conf spark.hadoop.fs.s3a.secret.key=<AWS_SECRET_ACCESS_KEY> \
  --conf spark.databricks.delta.retentionDurationCheck.enabled=false \
  -e "VACUUM delta.\`s3a://warehouse/delta/vendas\` RETAIN 168 HOURS;"

Retention, cleanup, and maintenance policies remain defined in the consuming component.

Jupyter and S3 clients

Why integrate

Access Ozone data from analysis notebooks
Share datasets between users
Store notebooks and analysis results

Configuration via values.yaml

Inject credentials as environment variables in the JupyterHub chart:

values.yaml (excerpt)
tdp-jupyter:
  extraEnv:
    - name: AWS_ACCESS_KEY_ID
      value: "<AWS_ACCESS_KEY_ID>"
    - name: AWS_SECRET_ACCESS_KEY
      value: "<AWS_SECRET_ACCESS_KEY>"
    - name: AWS_ENDPOINT_URL
      value: "http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878"
    - name: AWS_DEFAULT_REGION
      value: "us-east-1"

tip

In production, prefer referencing credentials from a Kubernetes Secret rather than literal values in values.yaml.

Notebook examples

Using boto3 with pandas:

import boto3
import pandas as pd

s3 = boto3.client(
    "s3",
    endpoint_url="http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878",
    aws_access_key_id="<AWS_ACCESS_KEY_ID>",
    aws_secret_access_key="<AWS_SECRET_ACCESS_KEY>",
    region_name="us-east-1",
)

# Read data from Ozone
obj = s3.get_object(Bucket="notebooks", Key="data/sales.csv")
df = pd.read_csv(obj["Body"])

# Process
result = df.groupby("category")["amount"].sum()

# Write result back to Ozone
result.to_csv("/tmp/result.csv")
s3.upload_file("/tmp/result.csv", "notebooks", "data/result.csv")

Using AWS CLI:

Terminal input
aws configure set aws_access_key_id <AWS_ACCESS_KEY_ID>
aws configure set aws_secret_access_key <AWS_SECRET_ACCESS_KEY>
aws configure set region us-east-1
aws s3 ls --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

Airflow and NiFi

Airflow

Why integrate

Store remote logs in Ozone
Use Ozone as XCom backend for large data passing
Store DAG artifacts

Configuration

Configure remote logging and the AWS connection in the Airflow chart values.yaml:

values.yaml (excerpt)
tdp-airflow:
  config:
    logging:
      remote_logging: "True"
      remote_base_log_folder: "s3://airflow-logs/"
      remote_log_conn_id: "ozone_s3"

  connections:
    - conn_id: ozone_s3
      conn_type: aws
      conn_extra: |
        {
          "aws_access_key_id": "<AWS_ACCESS_KEY_ID>",
          "aws_secret_access_key": "<AWS_SECRET_ACCESS_KEY>",
          "endpoint_url": "http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878",
          "region_name": "us-east-1"
        }

DAG example

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from datetime import datetime

with DAG("ozone_example", start_date=datetime(2024, 1, 1), schedule_interval=None) as dag:

    def upload_to_ozone():
        s3_hook = S3Hook(aws_conn_id="ozone_s3")
        s3_hook.load_file(
            filename="/tmp/data.csv",
            key="data/input.csv",
            bucket_name="airflow-data",
        )

    upload_task = PythonOperator(
        task_id="upload_to_ozone",
        python_callable=upload_to_ozone,
    )

NiFi

Why integrate

Ingest data directly into Ozone
Process and transform data stored in Ozone
Store flowfile content

Configuration

Use the PutS3Object and FetchS3Object processors with the following settings:

PutS3Object:

Endpoint Override URL: http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
Access Key and Secret Key: credentials created for the NiFi service
Region: us-east-1
Bucket: nifi-data
Path Style Access: enabled

FetchS3Object: same endpoint, credentials, and region settings.

Simple flow example: GenerateFlowFile → PutS3Object (Ozone) → LogAttribute

Bucket organization by component

In S3-compatible object storage, a bucket is the basic unit of organization: it acts as a top-level container where objects (files, data) are stored, each identified by a unique key within the bucket.

Each consumer should have its own bucket (or dedicated prefix) and independent credentials. This simplifies access control and allows revoking credentials per service without affecting others.

Suggested structure for a complete TDP environment:

Bucket	Primary use
`warehouse`	Spark tables, Hive Metastore, Iceberg, and Delta Lake
`airflow-logs`	Airflow remote logs
`airflow-data`	DAG artifacts and data
`nifi-data`	Data ingested or processed by NiFi
`notebooks`	JupyterHub notebooks and data

Create buckets using the AWS CLI pointed at the internal Ozone endpoint:

Terminal input
aws s3 mb s3://warehouse --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
aws s3 mb s3://airflow-logs --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
aws s3 mb s3://nifi-data --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
aws s3 mb s3://notebooks --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

Suggested internal bucket structure:

warehouse/
├── hive/        # Hive tables
├── spark/       # Spark job output
├── delta/       # Delta Lake tables
└── iceberg/     # Iceberg tables

airflow-logs/
└── dag_id=<DAG_ID>/
    └── run_id=<RUN_ID>/

nifi-data/
├── input/       # Raw data
├── processing/  # In-flight data
└── output/      # Processed data

notebooks/
├── users/       # Per-user notebooks
└── shared/      # Shared notebooks

To create independent credentials per component, use the script available in the Ozone chart (see the security page):

Terminal input
./scripts/generate-s3-credentials.sh create spark-service
./scripts/generate-s3-credentials.sh create trino-service
./scripts/generate-s3-credentials.sh create airflow-service
./scripts/generate-s3-credentials.sh create jupyter-service

Quick test with sample data

To validate the S3 integration right after installation, create test buckets and upload a sample file:

Terminal input
# Bucket for Spark test workloads
aws s3 mb s3://spark-data --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

# Bucket for performance testing
aws s3 mb s3://performance-test --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

# Upload a sample file to the spark-data bucket
aws s3 cp produtos.csv s3://spark-data/ --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

Sample content for produtos.csv:

id,nome,valor
1,produtoA,100
2,produtoB,250
3,produtoC,45

Security

For security practices (Secrets, credential rotation, TLS, NetworkPolicy, and audit logging), see the Ozone security page.

Next steps

Recommended integration order:

Spark + Hive Metastore — data lake foundation
Trino — query engine
Airflow — orchestration layer
Specialized tools (NiFi, Jupyter) as needed

Troubleshooting

To check connectivity to the S3 endpoint from a consumer pod:

Terminal input

kubectl exec -it <POD_NAME> -n <NAMESPACE> -- curl http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- aws s3 ls --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

Symptom	Likely cause	Fix
`Name or service not known`	Wrong internal DNS or namespace	Check `http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878` and the release namespace
Connection refused or timeout	Service without endpoints, NetworkPolicy, or wrong port	Check S3 Gateway pods, Service endpoints, and network policies
Virtual-hosted-style bucket error	Client is trying to place the bucket in the hostname	Enable path-style access in the consumer
`SignatureDoesNotMatch` or 403	Credentials, region, endpoint, or clock drift mismatch	Check Secret, `us-east-1` region, exact URL, and time synchronization
TLS error	Client does not trust the certificate or HTTP/HTTPS is inconsistent	Align URL, certificate, truststore, and consumer TLS configuration

Integration overview​

Credentials and endpoint​

Spark with S3A​

Why integrate​

Configuration​

Usage example​

Kubernetes Secret creation​

Trino with native S3​

Why integrate​

Configuration​

Usage example​

Hive Metastore and warehouse​

Why integrate​

Configuration​

Iceberg and Delta Lake​

Why integrate​

Configuration​

Jupyter and S3 clients​

Why integrate​

Configuration via values.yaml​

Notebook examples​

Airflow and NiFi​

Airflow​

Why integrate​

Configuration​

DAG example​

NiFi​

Why integrate​

Configuration​

Bucket organization by component​

Quick test with sample data​

Next steps​

Troubleshooting​

Integration overview

Credentials and endpoint

Spark with S3A

Why integrate

Configuration

Usage example

Kubernetes Secret creation

Trino with native S3

Why integrate

Configuration

Usage example

Hive Metastore and warehouse

Why integrate

Configuration

Iceberg and Delta Lake

Why integrate

Configuration

Jupyter and S3 clients

Why integrate

Configuration via values.yaml

Notebook examples

Airflow and NiFi

Airflow

Why integrate

Configuration

DAG example

NiFi

Why integrate

Configuration

Bucket organization by component

Quick test with sample data

Next steps

Troubleshooting