Hive Metastore Configuration

ChartVersion3.0.1TypeapplicationAppVersion4.0.0

CompatibilityKubernetes1.32+OpenShift4.19+Rancher2.10.x+

Feature SupportS3supportedExternal DBsupported

What is Hive Metastore?

The tdp-hive-metastore chart installs Apache Hive Metastore 4.0.0 on Kubernetes and provides an internal catalog service, accessed through a Thrift endpoint on port 9083.

This catalog stores metadata for databases, tables, columns, types, partitions, and locations. Table data remains in the configured warehouse; the Metastore records how that data should be found and interpreted by clients.

In practice, when a compatible client wants to read a table named sales, it queries the Metastore to retrieve the table schema, available partitions, and file paths. Hive, Spark, Trino, and Iceberg are examples of clients that can consume this catalog.

In TDP, the chart supports the warehouse in file mode for simple scenarios or validation, and in s3 mode for integration with S3-compatible object storage, such as Ozone. Catalog metadata is stored in PostgreSQL, either internal or external.

Learn more

See Apache Hive — Concepts, Hive Metastore section to understand the role of the metadata catalog in the data ecosystem.

How Hive Metastore fits into TDP

Compatible clients
(Hive, Spark, Trino, Iceberg...)
     |
     |  "Where is the 'sales' table?"
     v
Hive Metastore  -->  PostgreSQL (stores metadata)
     |
     |  "It is in the configured warehouse"
     v
File or S3/Ozone warehouse (where the actual data is)

Hive Metastore operates as data-plane infrastructure. Instead of exposing a web interface for end users, it serves internal clients that need to query or update table metadata.

For this reason, configuration usually focuses on the internal Service, the metadata PostgreSQL database, and the warehouse mode. Consumption happens through the Service Thrift endpoint on port 9083.

Deployed components

Component	Description
Metastore Server	Thrift service on port 9083 that responds to metadata queries
PostgreSQL (internal or external)	Database that stores schemas, tables, partitions and locations
TDP helper jobs	Database validation/creation and auxiliary resource cleanup jobs

Helm values structure

The tdp-hive-metastore chart distributes configuration in two blocks:

• tdp-hive-metastore: — main configuration: image, PostgreSQL (embedded by default, or external), warehouse type (file or s3), resources, security.
• TDP-Settings: — optional configuration used only when postgres.enabled: false; enables the TDP hooks for validation, provisioning and authentication against an external PostgreSQL.

TDP-Settings:
  externalDatabase:
    enabled: false

tdp-hive-metastore:
  postgres:
    enabled: true
  metastore:
    type: file

Overview

Property	Value
Chart	tdp-hive-metastore
Hive Metastore version	4.0.0
Chart version	3.0.1
Registry (OCI)	oci://registry.tecnisys.com.br/tdp/charts/tdp-hive-metastore
Thrift port	9083
External HTTP exposure	Not applicable (ClusterIP access, no Ingress)

Related pages

• Integration - Hive Metastore: Thrift URI and consumption examples for compatible clients.
• Security — Hive Metastore: Secrets, database credentials and password management.

Prerequisites

• Kubernetes 1.32+, Red Hat OpenShift 4.19+ or Rancher Manager 2.10.x+
• Helm 3.2.0+
• Tecnisys OCI registry accessible from the installation environment
• If using external PostgreSQL: instance accessible from the cluster, max_connections ≥ 300, and admin Secret in the format expected by the chart

Installation

Installation

Terminal input

helm upgrade --install <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-hive-metastore \
  --version <CHART_VERSION> \
  -n <NAMESPACE> --create-namespace

Placeholder	Description
<RELEASE_NAME>	Helm release name
<NAMESPACE>	Kubernetes installation namespace
<CHART_VERSION>	Chart version

Verify installation

Terminal input

kubectl -n <NAMESPACE> get all

Expect Deployment/Service for the metastore (and internal PostgreSQL, if postgres.enabled: true), with names derived from <RELEASE_NAME>.

Metastore Service validation

The chart exposes the Metastore through an internal Service on port 9083. After installation, validate that the Service has endpoints before configuring consuming clients.

Terminal input

kubectl get endpoints -n <NAMESPACE> <RELEASE_NAME>-metastore

If <none> appears, wait for the deploy to complete and check the Metastore pods. The Integration — Hive Metastore page shows how to use this endpoint in consumers.

Local test:

Terminal input

kubectl port-forward -n <NAMESPACE> svc/<RELEASE_NAME>-metastore 9083:9083

Main parameters

Main parameters

Parameter	Description	Default / example
tdp-hive-metastore.postgres.enabled	Internal PostgreSQL	true
tdp-hive-metastore.postgres.internal.pvcSize	Internal PostgreSQL PVC size (not defined in values.yaml — recommended example)	1Gi
tdp-hive-metastore.postgres.internal.maxConnections	Internal PostgreSQL max_connections (not defined in values.yaml — recommended example)	500
tdp-hive-metastore.postgres.external.host	External PostgreSQL host	""
tdp-hive-metastore.postgres.external.port	External PostgreSQL port	5432
tdp-hive-metastore.postgres.database	Database name	hive
tdp-hive-metastore.postgres.username	Database user	hive
tdp-hive-metastore.postgres.password	Database password (use Secret in production)	<POSTGRES_PASSWORD>
tdp-hive-metastore.metastore.type	Metastore type (file or s3)	file
tdp-hive-metastore.metastore.image	Metastore image	registry.tecnisys.com.br/tdp-dev/images/hive-metastore:4.0.0-0
tdp-hive-metastore.metastore.file.warehouseDir	Warehouse directory in file mode	/opt/hive/warehouse
tdp-hive-metastore.metastore.s3.warehouseDir	S3 warehouse path	s3a://warehouse/hive
tdp-hive-metastore.metastore.s3.endpoint	S3 endpoint	""
tdp-hive-metastore.metastore.s3.bucket	Warehouse bucket	warehouse
TDP-Settings.externalDatabase.enabled	Enable external database hooks	false
TDP-Settings.externalDatabase.recreate	Recreate database/user on deploy	false
TDP-Settings.externalDatabase.externalSecret.releaseName	PostgreSQL release (Secret with postgres-password)	tdp-postgresql-project

Configuration details

Database (PostgreSQL)

Hive Metastore stores catalog metadata in PostgreSQL. The choice between embedded and external PostgreSQL is one of the first configuration decisions.

For guidance on when to use each option, see Internal versus external PostgreSQL in the General Configuration page.

Requirement	Minimum	Recommended	Example (TDP PostgreSQL)
max_connections	300	400+	400
Concurrent operations	50+	100+	Supported by TDP default

Embedded PostgreSQL (default)

With postgres.enabled: true, the chart provisions a dedicated PostgreSQL for the metastore. The associated PVC stores only relational metadata (table names, partitions, URIs in object storage), while analytical data remains in Ozone/S3. Size disk and backup for this PVC according to catalog growth (number of tables and partitions), not the data files volume in the bucket.

The postgres.internal.* block is not defined in values.yaml — set the desired values explicitly, as shown in the example below:

tdp-hive-metastore:
  postgres:
    enabled: true

    internal:
      image: "registry.tecnisys.com.br/community/images/postgres:15"
      pvcSize: 1Gi
      maxConnections: 500
      sharedBuffers: 512MB
      effectiveCacheSize: 1.2GB
      maintenanceWorkMem: 128MB
      workMem: 32MB
      walBuffers: 16MB

    database: hive
    username: hive
    password: "<PASSWORD>"

External PostgreSQL

With external PostgreSQL, the availability and sizing of the metastore depend on the shared (or managed) instance you point to. The object storage where table files reside does not migrate with the Postgres host change — changing only the database changes where the records pointing to the warehouse are stored, not the bucket contents.

Set tdp-hive-metastore.postgres.enabled: false, configure tdp-hive-metastore.postgres.external.* and enable TDP-Settings.externalDatabase.enabled: true so that TDP hooks validate/provision the external database.

TDP-Settings:
  externalDatabase:
    enabled: true
    recreate: false
    externalSecret:
      releaseName: "<POSTGRESQL_RELEASE>"

tdp-hive-metastore:
  postgres:
    enabled: false
    external:
      host: "<POSTGRESQL_SERVICE>"
      port: 5432

The hook jobs reuse TDP-Settings.externalDatabase: they authenticate to the external PostgreSQL using the Secret associated with the name in externalSecret.releaseName (key postgres-password, in TDP standard).

• Use recreate: true only when you explicitly want to recreate the Hive database/user; for upgrades that preserve data, keep recreate: false (default).
• For Secret and credentials details, see Security — Hive Metastore.

CLI overrides:

Terminal input

helm upgrade --install <RELEASE_NAME> \
  oci://registry.tecnisys.com.br/tdp/charts/tdp-hive-metastore \
  -n <NAMESPACE> --create-namespace \
  --set TDP-Settings.externalDatabase.recreate=false \
  --set TDP-Settings.externalDatabase.externalSecret.releaseName=<POSTGRESQL_RELEASE>

Warehouse storage mode

The tdp-hive-metastore.metastore.type parameter defines how the Metastore references the warehouse area:

Mode	When to use	Main configuration
file	Simple environments or initial validation	tdp-hive-metastore.metastore.file.warehouseDir
s3	Integration with S3-compatible storage such as Ozone	tdp-hive-metastore.metastore.s3.*

Summarized example for S3 warehouse:

tdp-hive-metastore:
  metastore:
    type: s3
    s3:
      warehouseDir: s3a://warehouse/hive
      bucket: warehouse
      endpoint: "http://<SERVICE_NAME>.<NAMESPACE>.svc.cluster.local:9000"
      accessKey: "<S3_ACCESS_KEY>"
      secretKey: "<S3_SECRET_KEY>"
      pathStyleAccess: true
      region: us-east-1

Consumption details for Spark, Trino, Iceberg and other components are on the Integration — Hive Metastore page.

Metastore resources

Typical reference values: memory 1Gi request / 2Gi limit, CPU 500m request / 1000m limit, configurable persistent volume. Confirm with helm show values oci://registry.tecnisys.com.br/tdp/charts/tdp-hive-metastore.

Troubleshooting

Troubleshooting

1. Connection pool exhaustion: max_connections ≥ 300 in PostgreSQL.
2. Authentication: Correct Secrets and mounts.
3. Storage: PV permissions.
4. Network: PostgreSQL service reachable.
5. Thrift endpoint: Metastore Service with endpoints on port 9083.

Useful commands (adapting release/pod names):

Terminal input

kubectl -n <NAMESPACE> exec <POSTGRESQL_POD> -- psql -c "SHOW max_connections;"
kubectl -n <NAMESPACE> exec <POSTGRESQL_POD> -- psql -c "SELECT count(*) FROM pg_stat_activity;"
kubectl -n <NAMESPACE> logs deployment/<RELEASE_NAME>-metastore

To validate credentials directly in the Hive database, use the commands on the Security — Hive Metastore page.

Uninstallation

Uninstallation

Terminal input

helm uninstall <RELEASE_NAME> -n <NAMESPACE>

This command removes the release resources (Deployments, Services, ConfigMaps, Jobs). PVCs for the internal PostgreSQL are kept by default to preserve catalog metadata.

To remove internal PostgreSQL PVCs (destructive and irreversible operation):

Terminal input

kubectl delete pvc -n <NAMESPACE> -l app.kubernetes.io/name=postgresql

warning

Removing PVCs permanently deletes all Hive catalog metadata (table mappings, schemas, partitions). Data in S3/Ozone buckets is not affected, but clients will lose access to tables by name until the catalog is recreated.