Skip to main content
Version: 2.3.0

Fix 202507001

Inconsistency or Issue Description​

During the use of Apache Spark for writing data to Apache Ozone buckets via the S3A protocol, distinct failures were identified based on the bucket layout and integration settings with Apache Iceberg. These failures prevented file writing and Spark job completion, even with valid configurations.

Additionally, an issue was identified in viewing Apache Ozone buckets and volumes through the Ozone Recon UI when the Ranger plugin was enabled.

Affected Services and Components​

  • Apache Ozone 1.4.0
  • Apache Spark 3.3.4
  • TDP-PATCH-9180
  • BIGTOP-9162

Impact​

It is not possible to write to Apache Ozone buckets using Apache Spark with the S3 protocol or to view Apache Ozone buckets and volumes through the Ozone Recon UI when the Ranger plugin is enabled.

Technical Details​

Writing to Bucket with OBJECT_STORE Layout​

Executing the following command resulted in an error:

PySpark
df.write.mode("overwrite").text("s3a://bucket/keyfile")

Error returned:

javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

This bug was mapped in JIRA HDDS-10399, related to incorrect request handling in the S3 Gateway when managing directories in OBJECT_STORE layout.

Writing to Bucket with FILE_SYSTEM_OPTIMIZED Layout​

Executing the following command resulted in an error:

PySpark
df.write.mode("overwrite").text("s3a://bucket/keyfile")

Error returned:

DIRECTORY_NOT_EMPTY org.apache.hadoop.ozone.om.exceptions.OMException: Directory is not empty. Key:keyfile

Error in the overwrite operation when the key already exists as a directory.

Ozone OFS does not handle nested deletions and overwrites properly when using the S3A driver.

Writing with Iceberg to OFS Bucket via S3A​

Executing the following command resulted in an error:

PySpark
spark.sql.catalog.s3fs = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.s3fs.type = hadoop
spark.sql.catalog.s3fs.warehouse = s3a://benchmark-iceberg-s3-fs/
df.writeTo("s3fs.benchmark_iceberg_s3_fs.customers").createOrReplace()}

Error returned:

DIRECTORY_NOT_EMPTY org.apache.iceberg.exceptions.CommitFailedException: Failed to commit changes using rename
java.io.FileNotFoundException: No such file or directory

Iceberg operations depend on atomic rename(), which fails with the S3A driver in conjunction with Ozone OFS. The intermediate commit is not found during operation finalization.

Required Actions​

Follow the instructions below, according to the Enterprise Linux (EL) version, to install the fix and resolve these issues.

Fix Installation​

Install the fix (fix-202507001) on all machines (cluster nodes) that have Apache Ozone service components installed.

  1. Download the fix using wget or curl:
Terminal
wget --user USERNAME --password PASSWORD https://repo.tecnisys.com.br/yum/tdp/fixes/ozone/1.4.0/202507/140-202507001/ozone-1.4.0-fix-202507001-0.el9.noarch.rpm
note

Replace USERNAME and PASSWORD with your credentials for accessing the Tecnisys public package repository.

  1. Install the fix:
Terminal
sudo dnf install ozone-1.4.0-fix-202507001-0.el9.noarch.rpm
warning

Remember to install this fix on all machines (cluster nodes) that have Apache Ozone service components installed.

  1. Using Ambari Web, restart the Apache Ozone and Apache Spark services.

Ozone Plugin Reinstallation​

Reinstall the Ozone plugin for Ranger on all machines (cluster nodes) that have the Ozone Manager component installed.

  1. Download the plugin RPM using wget or curl:
Terminal
wget --user USERNAME --password PASSWORD https://repo.tecnisys.com.br/repository/yum-tdp/tdp/components/2.3.0/el-9-x86_64/ranger/ranger_2_3-ozone-plugin-2.4.0-2.x86_64.rpm
note

Replace USERNAME and PASSWORD with your credentials for accessing the Tecnisys public package repository.

  1. Install the plugin:
Terminal
sudo dnf install ranger_2_3-ozone-plugin-2.4.0-2.x86_64.rpm
warning

Remember to install this plugin on all machines (cluster nodes) that have the Ozone Manager component installed.