Highlights — TDP Datacenter
Platform
| Version | Highlights |
|---|---|
| TDP 3.0 | JupyterLab added: interactive environment for data analysis, notebooks and visualisations; support for multiple languages and extensions; modern tab-based interface integrating code editing, execution and charts. |
| OpenMetadata added: data governance and catalogue platform; enables lineage, data discovery and quality definitions; integration with a wide range of ecosystem services (databases, lakes, pipelines). |
Table A. TDP Datacenter platform highlights
Components
See the highlights of the updated components:
| Service | Version | Category | Feature/Highlight |
|---|---|---|---|
| Airflow | 2.10.5 | Orchestration | Cleanup tasks guaranteed even on failure or manual success — prevents resource leaks. |
| Multiple bug fixes: mapped trigger rules, ID validation, XCom with "/", event logs and more. | |||
| Helm Chart image updates with newer versions for Airflow, PgBouncer and exporters. | |||
| Delta Lake | 3.3.0 | Optimised table format | Identity Columns: automatic generation of unique values for new rows (Python and Scala) — generatedAlwaysAs and generatedByDefaultAs modes. |
| VACUUM LITE: 5–10× faster cleanup of obsolete files using only the transaction log. | |||
| Row Tracking Backfill: enables row-level lineage tracking on existing tables, including Row Id and Row Commit Version metadata. | |||
| Version Checksums: per-commit checksums reinforcing consistency, performance and debugging. | |||
| UniForm ALTER: enables UniForm format on existing tables without rewriting data. | |||
| Type Widening: type widening support in Delta Kernel (Java and Rust). | |||
| Druid | 32.0.0 | Real-time Analytics | Stability and performance focus: 341+ commits from 52 contributors (~30% more than previous releases). |
| ANSI SQL compliance: legacy null-handling settings removed — ANSI SQL behaviour is now fixed. | |||
| New Projection feature: pre-computes aggregations for significantly improved query performance. | |||
| Great Expectations | 1.3.5 | Data Quality | New: CheckpointFactory.add_or_update and strict_min/strict_max in ExpectTableRowCountToBeBetween. |
row_condition support with datetimes in Pandas and Spark; BigQuery cleanup scheduling; strict parameter in Window type; guaranteed run_id in ValidationDefinition.run. | |||
| HBase | 2.5.7 | NoSQL Database | Switch to avoid region reopening when editing tables — prevents RIT storms. |
isolate_regions command in RegionMover — precise, controlled region isolation and relocation. | |||
| Hive | 4.0.0 | Data Exploration and Analytics | Major leap: ~5,000 commits since version 3.1.3. |
Apache Iceberg integration: improved support including compaction via OPTIMIZE TABLE. | |||
| Enhanced ACID transactions and locking; compaction for ACID and Iceberg tables. | |||
| Docker support: official Hive images on Docker Hub. | |||
| Execution optimisations: anti-join, branch pruning, column histograms, HPL/SQL, scheduled queries, materialized views, Tez and LLAP. | |||
| Improved replication for external and ACID tables; Apache Ozone support as scalable storage. | |||
| Advanced features: native GeoSpatial, Iceberg compaction, JWT auth in Metastore, SAML 2 in HiveServer2. | |||
| Iceberg | 1.8.0 | Table Formats | End of support for Spark 3.3 and Hive Runtime. |
| Deletion Vectors: new spec, APIs and read/write support. | |||
| New types: Variant and UnknownType. | |||
| Operational improvements: fast append, removal of unused specs, useful procedures in Spark. | |||
| Improved AWS/Azure integration; extended compatibility with Spark, Flink and Hive. | |||
| Kafka | 3.4.1 | Data Streaming | ZooKeeper-to-KRaft migration (initial release; not recommended for production) — zero downtime. |
New generation field in the consumer protocol — helps manage partition claims. | |||
| Time-based metadata snapshots (e.g. hourly); rack-aware consumers for local AZ. | |||
| Kafka Streams (KIP-770 & KIP-837): config and cache metric updates; broadcast output records to all partitions. | |||
| MirrorMaker2: reads all offset syncs at startup; publishes syncs during task commit; translates offsets across replication flows. | |||
| Fixed interceptor resource leak. | |||
| NiFi | 1.28.1 | Data Flow Management | Fixed logging of sensitive parameter values in flow sync log — not exposed even with debug enabled (1.28.1). |
| XSS protection: parameter descriptions are now correctly neutralised (1.28.0). | |||
| End of support for NiFi 1.x (08 Dec 2024) — migration to 2.x series strongly recommended. | |||
| Ozone | 1.4.1 | Scalable Object Storage | Prevents race condition in datanode when creating VERSION — improves operational reliability. |
| Refined SCM logs — reduces noise and false errors when handling sequence IDs in closed containers. | |||
| S3 Gateway security: secrets management endpoint restricted to administrators. | |||
| Spark | 3.5.3 | Distributed Computing | Third maintenance release of the 3.5 series — security and bug fixes. Recommended for environments already running 3.5. |
| Superset | 4.1.2 | Data Visualisation | New charts: Big Number with time comparisons, Heatmap, Histogram and Sankey. |
| Dynamic catalogue in connected databases; more intuitive upload UI with validations. | |||
| Time filters and Jinja macros for dynamic dashboards; UX improvements in SQL Lab and permissions. | |||
| Security fixes in 4.1.2: prevents resource takeover and access control bypass. | |||
| Apache Tez | 0.10.4 | Execution Engine | ~28 fixes and improvements — observability, stability and security. |
| ZooKeeper | 3.8.4 | Distributed Coordination | Configurable maximum connections/clients per server. |
| zkCli: new option to wait for connection before executing commands — avoids failures in unstable environments. | |||
| Startup logs with time units; clearer network address display; improved error messages. |
Table B. TDP Datacenter component highlights