We’re excited to announce the Basic Availability of Hive Metastore (HMS) and AWS Glue Federation in Unity Catalog! This new functionality allows Unity Catalog to seamlessly entry and govern tables saved in Hive Metastores—whether or not self-hosted or hosted by Databricks—in addition to AWS Glue. It represents a key milestone in our Lakehouse Federation imaginative and prescient, which brings exterior knowledge sources together with databases, knowledge warehouses and catalogs, collectively below a unified governance framework with Unity Catalog. Now you can effortlessly uncover, question, and handle all of your knowledge from a single, centralized platform, whatever the format and site. This not solely fosters open entry and collaboration throughout your group but additionally extends knowledge intelligence into each knowledge supply.
On this weblog, we’ll discover the advantages of HMS and AWS Glue Federation, clarify the way it works, and supply steerage on getting began.
Why Hive Metastore and AWS Glue Federation?
HMS has been an early normal for cataloging knowledge to be used in large knowledge programs, and whereas it supplies foundational functionalities, they aren’t ideally suited to fashionable knowledge and AI workloads that demand complete governance, together with fine-grained entry controls on rows and columns, lineage, monitoring and auditing throughout all knowledge and AI belongings in a single place.
Unity Catalog addresses these shortcomings by offering the trade’s solely unified, open governance answer for managing all knowledge and AI belongings. It allows organizations to create an enterprise catalog that curates information, tables, ML fashions, AI instruments, notebooks, and metrics, all ruled with fine-grained entry controls, lineage, monitoring, auditing, and cross-platform sharing in a single answer. Over 10,000+ enterprises are actually leveraging Unity Catalog to manipulate their knowledge property.
HMS and AWS Glue Federation present important advantages for organizations with HMS deeply embedded of their knowledge structure. For these with long-standing HMS or AWS Glue deployments, this functionality presents a seamless path to leverage Unity Catalog’s superior options over knowledge saved within the HMS or Glue metastore. It ensures operational continuity by enabling organizations to maintain legacy workflows whereas progressively upgrading present knowledge and workspaces to Unity Catalog.
Key advantages embody:
- Seamless integration: Join your present HMS and AWS Glue catalogs on to Unity Catalog with out requiring handbook metadata migration.
- Simplified knowledge discovery: Entry and discover knowledge from HMS and AWS Glue by a unified interface, alongside all of your knowledge and AI belongings in Unity Catalog.
- Complete governance: Leverage Unity Catalog’s fine-grained entry controls, tagging, classification, lineage, and audit capabilities on prime of the information saved in HMS and AWS Glue
Using HMS Federation in Unity Catalog enabled an incremental migration that minimized disruption to our clients. Our partnership with Databricks was essential in evaluating short-term wants and long-term objectives. Leveraging Unity Catalog’s capabilities early, we created a scalable and environment friendly knowledge platform with improved governance and entry management as a part of our modernization journey
— Praveena Edward, Lead Knowledge Engineer, Nationwide
Now we have years’ price of datasets which might be cataloged in an exterior Hive Metastore. HMS Federation permits us to instantly profit from Unity Catalog solely options like strong entry management and self-serve AI tooling by Genie Areas, with out the overhead of migrating all of those tables into Unity Catalog
— James Davidheiser, Technical Lead, Knowledge Infrastructure, Asana
The way it works
Unity Catalog now consists of federation connectors for Hive Metastore (HMS) and AWS Glue, serving as a translation layer between Unity Catalog and your exterior metastores. These connectors allow you to mount complete HMS catalogs (each inner and exterior) or AWS Glue as overseas catalogs inside Unity Catalog, making them seem as native objects. You possibly can outline fine-grained entry controls, view lineage, carry out audits, and question HMS or AWS Glue managed tables utilizing the Databricks engine. The federation helps each studying and writing to tables in inner HMS inside Databricks workspaces whereas providing read-only entry for tables in exterior HMS and AWS Glue
With this functionality, you possibly can learn all tables in HMS and AWS Glue—Parquet, Delta, and Iceberg (coming quickly in Public Preview)—enabling you to entry and govern all of your tables seamlessly.
Try the video tutorial beneath to discover AWS Glue and HMS Federation in motion.
What’s new with the GA launch?
The GA launch comes with the next extra options:
- Help for Hive Metastore v3.1
- Help for Hive Metastores hosted in a SQL Server or Postgres databases (mySQL assist already offered)
- Entry to DBFS-root backed tables behind a storage firewall (Azure)
- Help for row and column stage masks on Hive tables
We’ve additionally added Public Preview assist for studying Delta shallow clones outlined within the Hive metastore, in addition to Non-public Preview assist for studying DBFS-mount backed tables throughout workspaces (please contact your Databricks account consultant to enroll!).
Get began
By embracing Unity Catalog because the cornerstone of your Lakehouse structure, you possibly can unlock the ability of a unified and open governance implementation that spans your complete knowledge and AI property.
