Security - Apache Ozone

ChartVersion3.0.1TypeapplicationAppVersion2.0.0

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

General security standards

For concepts that apply to all TDP components — such as Secret management, credential rotation, NetworkPolicy, rate limiting, and auditing — see Security — General Standards.

Overview

The Apache Ozone S3 Gateway in TDP uses AWS Signature Version 4 authentication in simple mode, without Kerberos. In this mode, clients sign each request with an access key and a secret key, and the chart injects the credentials into the S3 Gateway via a Kubernetes Secret.

Effective security combines four controls: per-application or per-technical-user credentials, TLS when there is traffic outside the cluster, network restrictions with NetworkPolicy or equivalent controls, and operational audit logging.

Security architecture

┌─────────────────────────┐
│  Cliente                │
│  (AWS CLI / SDK / Spark)│
└──────────┬──────────────┘
           │ AWS Signature v4
           │ (Access Key + Secret Key)
           ▼
┌──────────────────────────┐
│  S3 Gateway (porta 9878) │
│  - Valida assinatura     │
│  - Verifica credenciais  │
└──────────┬───────────────┘
           │
           ▼
┌──────────────────────────┐
│  Ozone Manager           │
│  (Autorização)           │
└──────────────────────────┘

AWS Signature v4 without Kerberos

Available security levels

Level	Description	Recommended use
None	No authentication	Isolated development only
Simple	AWS Signature v4 without Kerberos	Production (recommended)
Kerberos	Full Ozone security	Enterprise environments with existing Kerberos

Configuration in values.yaml

ozone:
  s3g:
    auth:
      enabled: true
      type: simple
      secretName: ozone-s3-credentials
    env:
      - name: OZONE-SITE.XML_ozone.s3g.authentication
        value: "simple"
      - name: OZONE-SITE.XML_ozone.security.enabled
        value: "false"
      - name: OZONE-SITE.XML_ozone.s3g.secret.http.enabled
        value: "false"

Parameter	Description	Default
ozone.s3g.auth.enabled	Enables S3 authentication in the S3 Gateway	true
ozone.s3g.auth.secretName	Secret containing aws_access_key_id and aws_secret_access_key keys	ozone-s3-credentials

With authentication enabled, clients such as AWS CLI, SDKs, Spark S3A, and Trino must use the same region, endpoint, and credentials expected by the S3 Gateway. Use us-east-1 as the default region when no other convention exists in the environment.

Development only

Use ozone.s3g.auth.enabled: false only in isolated development or lab environments. In shared or production environments, keep authentication enabled and protect the S3 REST endpoint with TLS and network controls.

Credential model

Create separate credentials per application, pipeline, or technical user whenever possible. Avoid sharing the same credential between consumers with different access profiles.

Terminal input

kubectl -n <NAMESPACE> create secret generic ozone-s3-credentials \
  --from-literal=aws_access_key_id="<AWS_ACCESS_KEY_ID>" \
  --from-literal=aws_secret_access_key="<AWS_SECRET_ACCESS_KEY>"

The Secret must exist in the namespace where Ozone is installed and must be referenced by the ozone.s3g.auth.secretName value:

ozone:
  s3g:
    auth:
      enabled: true
      secretName: ozone-s3-credentials

Do not commit real credentials to Git repositories, values files, notebooks, or DAGs. Distribute credentials to consumers via Kubernetes Secrets, managed environment variables, or equivalent component mechanisms.

S3 user management

When the environment uses separate credentials per person, application, or technical service, standardize an operational workflow for creating, listing, and removing S3 users.

Creation via script

The scripts/generate-s3-credentials.sh script is located within the tdp-ozone chart.
To use it, run the command from the chart root, or adjust the path to the location where the chart was extracted.
The utility generates access key and secret key pairs and registers them as Kubernetes Secrets.

Terminal input

NAMESPACE=<NAMESPACE> ./scripts/generate-s3-credentials.sh create <APPLICATION_NAME>

Expected output:

AWS_ACCESS_KEY_ID=a1b2c3d4e5f6g7h8i9j0
AWS_SECRET_ACCESS_KEY=aBcDeFgHiJkLmNoPqRsTuVwXyZ1234567890ABCD

Manual creation

If you prefer to create credentials manually without the script:

Terminal input

ACCESS_KEY=$(openssl rand -hex 10)
SECRET_KEY=$(openssl rand -base64 32)

kubectl create secret generic ozone-s3-credentials-<APPLICATION_NAME> \
  --from-literal=aws_access_key_id="$ACCESS_KEY" \
  --from-literal=aws_secret_access_key="$SECRET_KEY" \
  -n <NAMESPACE>

After creating the Secret, register the accessId with the Ozone Manager to associate it with a tenant user — without this step, the credentials exist in Kubernetes but are not recognized by the S3 Gateway:

Terminal input

kubectl exec -n <NAMESPACE> <RELEASE_NAME>-om-0 -- \
  ozone sh tenant assignusertoaccessid \
  --accessid="$ACCESS_KEY" --tenant-id=default --user-principal=<APPLICATION_NAME>

Listing

Terminal input

./scripts/generate-s3-credentials.sh list

To list S3 Gateway Secrets directly in the cluster:

Terminal input

kubectl get secrets -n <NAMESPACE> -l app.kubernetes.io/component=s3g

Deletion

Terminal input

./scripts/generate-s3-credentials.sh delete <APPLICATION_NAME>

For operational auditing, keep track of which Kubernetes Secret distributes each credential and which workloads consume that Secret.

Access control (ACLs)

Ozone provides volume- and bucket-level ACLs. Use the ozone sh CLI to set and query permissions:

Terminal input

# Set bucket ACL (alice with read/write, bob read-only)
kubectl exec -n <NAMESPACE> <RELEASE_NAME>-om-0 -- \
  ozone sh bucket setacl \
  --acl user:alice:rw,user:bob:r \
  /volume1/bucket1

Terminal input

# Query bucket ACLs
kubectl exec -n <NAMESPACE> <RELEASE_NAME>-om-0 -- \
  ozone sh bucket getacl /volume1/bucket1

Rotation and revocation

To rotate credentials, update or recreate the Secret and restart the pods that consume those variables or files. This includes the S3 Gateway and clients that hold credentials in memory.

Terminal input

kubectl -n <NAMESPACE> rollout restart statefulset/<RELEASE_NAME>-s3g

Practical revocation in simple mode is operational: remove or replace Secrets from consumer namespaces, restart workloads, and block network paths when necessary.

LDAP/AD without Kerberos

LDAP or Active Directory can be used as a reference for account lifecycle and naming conventions, for example to standardize technical users.

Important limitation

With ozone.security.enabled=false (no Kerberos), disabling an account in LDAP/AD does not automatically revoke already-distributed S3 credentials. S3 keys are client credentials that you distribute — revocation is an operational responsibility.

Recommended process for lifecycle management with LDAP/AD:

1. Maintain a stable identifier per account (e.g., LDAP uid or AD sAMAccountName).
2. For each LDAP account, generate an S3 keypair and store it as a Kubernetes Secret.
3. Inject the Secret into workloads that need access (env vars or mounted files).
4. When an account is disabled in LDAP, remove the corresponding Kubernetes Secret from all namespaces and redeploy workloads.
5. Rotate keys periodically or immediately after compromise.

Recommended compensating controls:

• Restrict access to the S3 endpoint with NetworkPolicy or security groups
• Use TLS for S3 traffic
• Create separate service accounts (distinct keys) per application
• Enable audit logging in Ozone

TLS and S3 REST exposure

Use TLS for any access to the S3 Gateway REST outside the cluster or on shared networks. TLS is normally terminated at the Ingress Controller or Gateway configured for the hostname https://<OZONE_S3_REST_HOSTNAME>.

tdp-ozone:
  ingress:
    s3g:
      rest:
        enabled: true
        annotations:
          cert-manager.io/cluster-issuer: letsencrypt-prod
        hosts:
          - host: <OZONE_S3_REST_HOSTNAME>
            paths:
              - path: /
                pathType: Prefix
        tls:
          - secretName: ozone-s3-rest-tls
            hosts:
              - <OZONE_S3_REST_HOSTNAME>

Expose OM, SCM, and S3 Web UI only to administrative networks. The S3 Gateway REST is a data plane and requires additional attention to TLS, S3 authentication, DNS, and network policies.

Rate limiting

To protect the S3 REST endpoint against abusive use, configure request limits in the NGINX Ingress Controller:

tdp-ozone:
  ingress:
    s3g:
      rest:
        annotations:
          nginx.ingress.kubernetes.io/limit-rps: "100"
          nginx.ingress.kubernetes.io/limit-connections: "10"

NetworkPolicy and network security

Restrict who can reach the <RELEASE_NAME>-s3g-rest Service on port 9878. In clusters with NetworkPolicy, allow only the namespaces and ServiceAccounts of authorized consumers, such as Spark, Trino, Hive Metastore, Airflow, NiFi, or Jupyter where applicable.

Also review external exposure: if all consumers are inside the cluster, prefer an internal Service over Ingress for the S3 REST endpoint.

Audit and logs

Enable audit logging in values.yaml:

ozone:
  s3g:
    env:
      - name: OZONE-SITE.XML_ozone.audit.enabled
        value: "true"
      - name: OZONE-SITE.XML_ozone.audit.log.file
        value: "/var/log/ozone/s3-audit.log"

To query audit entries in the S3 Gateway logs:

Terminal input

kubectl logs -n <NAMESPACE> statefulset/<RELEASE_NAME>-s3g | grep AUDIT

Use S3 Gateway and consumer logs to investigate accesses, signature failures, and authorization errors. When audit logging is enabled, direct these logs to the observability solution adopted by TDP.

Production checklist

Before exposing the S3 Gateway REST to users or integrations outside the cluster, validate:

• Default credentials replaced with environment-specific credentials
• TLS enabled for the public S3 REST hostname
• cert-manager annotation or manually provisioned certificate
• Rotation policy defined for S3 credentials
• Credential revocation and recovery procedure documented
• NetworkPolicy, firewall rules, or equivalent controls restricting endpoint access
• Rate limiting configured in Ingress (if externally exposed)
• Audit logging enabled and logs directed to observability
• Create, read, write, and delete object test with the final credentials

Migration from open access to authenticated

If the environment is migrating from an installation without authentication (auth.enabled: false) to authentication enabled:

1. Create credentials before enabling authentication:

   Terminal input

   NAMESPACE=<NAMESPACE> ./scripts/generate-s3-credentials.sh create <APPLICATION_NAME>

2. Update values.yaml enabling ozone.s3g.auth.enabled: true and referencing the created Secret.

3. Apply the chart upgrade:

   Terminal input

   helm upgrade <RELEASE_NAME> oci://registry.tecnisys.com.br/tdp/charts/tdp-ozone \
     --namespace <NAMESPACE> \
     -f values.yaml

4. Update all consumers (Spark, Trino, Airflow, etc.) with the new credentials before or immediately after the upgrade.

5. Verify that all applications are operating with the new credentials before removing open access.

Access testing

Verify authentication with and without credentials:

Terminal input

# Without credentials — should return an authentication error
curl http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878/

Terminal input

# With credentials — should return bucket list
aws configure set aws_access_key_id <AWS_ACCESS_KEY_ID>
aws configure set aws_secret_access_key <AWS_SECRET_ACCESS_KEY>
aws configure set region us-east-1
aws s3 ls --endpoint-url=http://<RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local:9878

Troubleshooting

SignatureDoesNotMatch

1. Verify that the credentials in the Secret are correct:

   Terminal input

   kubectl get secret ozone-s3-credentials-<APPLICATION_NAME> -n <NAMESPACE> -o jsonpath='{.data}'

2. Confirm that the node clock is synchronized (AWS Signature v4 is time-sensitive).

3. Verify that the endpoint used by the client is exactly the same as used in the signature.

4. Confirm that the region is configured as us-east-1.

Credentials not recognized (AccessDenied or 403)

Terminal input

./scripts/generate-s3-credentials.sh create <APPLICATION_NAME>

Restart consumer pods after recreating the Secret.

Connection refused

1. Verify that the Ingress is configured correctly.

2. Test internal DNS resolution:

   Terminal input

   kubectl exec -it <POD_NAME> -n <NAMESPACE> -- nslookup <RELEASE_NAME>-s3g-rest.<NAMESPACE>.svc.cluster.local

3. Test via port-forward:

   Terminal input

   kubectl port-forward -n <NAMESPACE> svc/<RELEASE_NAME>-s3g-rest 9878:9878
   aws s3 ls --endpoint-url=http://localhost:9878

AccessDenied with valid credentials

Check volume or bucket ACLs:

Terminal input

kubectl exec -n <NAMESPACE> <RELEASE_NAME>-om-0 -- \
  ozone sh bucket getacl /<VOLUME_NAME>/<S3_BUCKET>

Adjust permissions as needed with ozone sh bucket setacl.

Symptom	Likely cause	Fix
SignatureDoesNotMatch	Incorrect credential, divergent endpoint, wrong region, or clock drift	Check Secret, exact URL, us-east-1 region, and time synchronization
AccessDenied or 403	Credentials not recognized or inconsistent client policy	Validate the Secret and restart consumer pods
AccessDenied with valid credentials	Volume or bucket ACL does not allow the operation	Check ACLs with ozone sh bucket getacl
Client connects via HTTP but fails via HTTPS	Missing certificate, CA, or truststore in the consumer	Configure CA/truststore and use the same URL scheme as published
Works via port-forward, fails via Ingress	Wrong DNS, TLS, IngressClass, or routing rule	Check hostname, certificate, Ingress, and controller logs
Internal consumer cannot reach endpoint	NetworkPolicy or namespace blocking traffic	Allow traffic to <RELEASE_NAME>-s3g-rest:9878

References

• Apache Ozone S3 Gateway — Official documentation
• AWS Signature Version 4
• Ozone Security Architecture