Asserting first-class help of Iceberg format in Databricks Delta Sharing

With greater than 300% year-on-year utilization development for two consecutive years, Delta Sharing is essentially the most extensively adopted open protocol for Information and AI sharing. Main knowledge suppliers together with SAP, Walmart, Atlassian, and LSEG use Delta Sharing to share knowledge with their companions and prospects throughout clouds and platforms. At this time, we’re excited to announce that Databricks Delta Sharing has first-class help for the Apache Iceberg format.

Information suppliers can now share knowledge securely and dwell from Databricks to any consumer that helps the Apache Iceberg REST Catalog API. Recipients on platforms equivalent to Snowflake, Trino, Flink, and Spark on all clouds can all use this functionality – including to Delta Sharing’s open ecosystem. 

Moreover, we’re launching a Non-public Preview that allows knowledge suppliers to make use of Delta Sharing to share Iceberg tables managed by catalogs outdoors Databricks – together with AWS Glue, Hive Metastore, Snowflake Horizon, and extra.

Collectively, you may share any new or present tables (Delta or Iceberg, managed or international). This builds in the direction of full open interoperability. You may usher in Iceberg tables from any exterior catalog, handle them by Databricks and Unity Catalog, after which share them out to any recipient—whether or not they’re on Databricks, an Iceberg consumer, or a Delta consumer. This lets you leverage Unity Catalog as your unified knowledge governance layer, providing you with one place for all sharing. 

On this weblog submit, we are going to clarify why open knowledge sharing is essential. We can even dive into how Delta Sharing to Iceberg purchasers work by a hands-on demo.

Why This Issues: Open vs. Closed Sharing

Most knowledge sharing options will not be actually sharing—they’re trapping. They’re basically closed and engineered to make sure vendor lock-in, so that you solely get to share with others who’re already inside their closed ecosystems. This limits your choices, stifles innovation, and drives huge, pointless knowledge replication.

Delta Sharing is essentially the most extensively adopted open normal for safe knowledge sharing. Utilized by category-leading knowledge suppliers, it’s constructed to help totally different clouds and platforms. Delta Sharing operates on three core rules:

• Share any asset.
• Share with anybody.
• Share with none friction.

Including Iceberg consumer help strengthens this dedication. It permits you to share a Delta desk whereas recipients expertise it as a local Iceberg desk. Sharing occurs over the Iceberg REST API, so recipients can join from any Iceberg-compatible platform. This lets you get the very best of each worlds: Information suppliers profit from superior Delta Sharing options equivalent to View sharing, whereas recipients obtain native Iceberg tables by way of the Iceberg REST API.

Recipients get safe, dwell entry to the supply knowledge. This eliminates silos and allows you to share knowledge brazenly with anybody.

This function is right for organizations that must share knowledge externally with companions and prospects utilizing Iceberg purchasers, equivalent to these working on Snowflake or integrating with platforms like Trino, Flink, or Spark. Firms with a number of enterprise items working throughout a number of platforms additionally profit by unblocking seamless, bi-directional knowledge alternate in multi-cloud or hybrid environments. Industries already leveraging these patterns embody healthcare, retail, finance, ad-tech, and extra.

Interoperability: Each Supply and Vacation spot 

As a result of we imagine in full open knowledge entry, we do not cease at sharing knowledge to Iceberg purchasers. We are actually creating the subsequent evolution: sharing international Iceberg tables that reside in exterior catalogs equivalent to AWS Glue or Snowflake Horizon. We’re excited to announce the Non-public Preview of Delta Sharing help of international Iceberg tables.

You would possibly ask: Why share an Iceberg desk by Delta Sharing if it resides in AWS Glue or Snowflake? Why not share immediately from inside that platform?

First, by cataloging your exterior Iceberg knowledge in Unity Catalog, you get a unified governance layer in Unity Catalog, permitting you to get full visibility and governance throughout your knowledge property. Moreover, utilizing Delta Sharing means that you can get the better of each worlds: You profit from Delta Sharing best-in-class sharing functionalities, whereas holding your knowledge in Iceberg format. This contains for instance, the power to Delta Share Views for fine-grained entry management, which isn’t supported natively by the Iceberg IRC API.

With this Non-public Preview, the Databricks Lakehouse is open in each instructions. Your Lakehouse can share knowledge to and obtain knowledge from the rising Iceberg ecosystem.

This twin capacity provides you:

• Easy Collaboration: Work collectively irrespective of which open desk format you employ (Delta or Iceberg).
• Ruled Sharing: Unity Catalog controls entry and supplies audit logs.
• Broadest Attain: Share knowledge as each a supplier and a recipient, breaking platform partitions.

How Does It Work?

Think about your organization, Supplier Corp, makes use of Databricks and Delta Lake to handle buyer knowledge. It is advisable to securely share a each day record of product gross sales with Accomplice Inc, which makes use of Snowflake and prefers the Iceberg format.

Earlier than this function: Supplier Corp must manually export the info, remodel it right into a Snowflake-readable format, add it to the companion’s cloud storage, and arrange a posh synchronization job. That is sluggish, expensive, entails vital admin overhead, and dangers knowledge turning into outdated.

With Delta Sharing to Iceberg Purchasers:

1. Supplier Corp permits Iceberg reads on gross sales knowledge by way of UniForm (this could embody managed and exterior Delta tables, views, materialized views, and streaming tables), and shares it by way of Delta Sharing. This supplies dwell entry with no duplication or re-ingestion required.
2. Accomplice Inc units up a easy connection in Snowflake utilizing the offered credentials for safe authentication by way of short-lived bearer tokens.
3. Accomplice Inc’s analysts can instantly question the shared desk utilizing normal SQL, treating it like a local Iceberg desk of their Snowflake atmosphere.
4. The information they see is at all times dwell (zero-copy), and Supplier Corp maintains full safety and governance with auditing and monitoring utilizing Unity Catalog.

This makes knowledge sharing prompt, secure, and completely format-agnostic.