Analyzing your information catalog: Question SageMaker Catalog metadata with SQL

As your information and machine studying (ML) belongings develop, monitoring which belongings lack documentation or monitoring asset registration traits turns into difficult with out customized reporting infrastructure. You want visibility into your catalog’s well being, with out the overhead of managing ETL jobs. The metadata characteristic of Amazon SageMaker supplies this functionality to customers. Changing catalog asset metadata into Apache Iceberg tables saved in Amazon S3 Tables removes the necessity to construct and keep customized ETL pipelines. Your workforce can then question asset metadata instantly utilizing customary SQL instruments. Now you can reply governance questions like asset registration traits, classification standing, and metadata completeness utilizing customary SQL queries by means of instruments like Amazon Athena, Amazon SageMaker Unified Studio notebooks, and BIsystems.

This automated method reduces ETL growth time and provides your workforce visibility into catalog well being, compliance gaps, and asset lifecycle patterns. The exported tables embrace technical metadata, enterprise metadata, mission possession particulars, and timestamps, partitioned by snapshot date to allow time journey queries and historic evaluation. Groups can use this functionality to proactively monitor catalog well being, establish gaps in documentation, observe asset lifecycle patterns, and ensure that governance insurance policies are constantly utilized.

How metadata export works

After you allow the metadata export characteristic, it runs mechanically on a every day schedule:

1. SageMaker Catalog creates the infrastructure — An Amazon Easy Storage Service (Amazon S3) desk bucket named aws-sagemaker-catalog is created with an asset_metadata namespace and an empty asset desk.
2. Each day snapshots are captured — A scheduled job runs as soon as per day round midnight (native time per AWS Area) to export up to date asset metadata.
3. Metadata is structured and partitioned — The export captures technical metadata (resource_id, resource_type), enterprise metadata (asset_name, business_description), mission possession particulars, and timestamps, partitioned by snapshot_date for question efficiency.
4. Knowledge turns into queryable — Inside 24 hours, the asset desk seems in Amazon SageMaker Unified Studio beneath the aws-sagemaker-catalog bucket and turns into accessible by means of Amazon Athena, Studio notebooks, or exterior BI instruments.
5. Groups question utilizing customary SQL — Knowledge groups can now reply questions like “What number of belongings had been registered final month?” or “Which belongings lack enterprise descriptions?” with out constructing customized ETL pipelines.

The export evaluates catalog belongings and their metadata properties within the area, changing them into Apache Iceberg desk format. The info flows into downstream analytics operations instantly, with no separate ETL or batch processes to keep up. The exported metadata turns into a part of a queryable information lake that helps time-travel queries and historic evaluation.

On this submit, we show methods to use the metadata export functionality in Amazon SageMaker Catalog and carry out analytics on these tables. We discover the next particular use-cases.

• Audit historic adjustments to analyze what an asset regarded like at a particular cut-off date.
• Monitor asset development view how the info catalog has grown over the past 30 days.
• Monitor metadata enhancements to see which belongings gained descriptions or possession over time.

Answer overview

Determine 1 – SageMaker catalog export to S3 Tables

The structure consists of three key elements:

1. Amazon SageMaker Catalog exports asset metadata every day to Amazon S3.
2. S3 Tables shops metadata as Apache Iceberg tables within the aws-sagemaker-catalog bucket with ACID compliance and time journey.
3. Question engines (Amazon Athena, Amazon Redshift, and Apache Spark) entry metadata utilizing customary SQL from the asset_metadata.asset desk.

What metadata is uncovered?

SageMaker Catalog exports metadata within the asset_metadata.asset desk:

The snapshot_time column helps point-in-time evaluation and question of historic catalog states.

Conditions

To observe together with this submit, you need to have the next:

For SageMaker Unified Studio area setup directions, check with the SageMaker Unified Studio Getting began information.

After you full the conditions, full the next steps.

1. Add this coverage to our IAM person or position to allow metadata export. If utilizing SageMaker Unified Studio to question the catalog, add this coverage to the AmazonSageMakerAdminIAMExecutionRole managed position.

{ "Model": "2012-10-17", 
"Assertion": [ 
{
 "Effect": "Allow",
 "Action": [ "datazone:GetDataExportConfiguration",
 "datazone:PutDataExportConfiguration"
 ],
 "Useful resource": "*"
 },
 {
 "Impact": "Enable",
 "Motion": [
 "s3tables:CreateTableBucket",
 "s3tables:PutTableBucketPolicy"
 ],
 "Useful resource": "arn:aws:s3tables:*:*:bucket/aws-sagemaker-catalog" 
} 
]
}

2. Grant describe and choose permissions for SageMaker Catalog with AWS Lake Formation. This step may be carried out within the AWS Lake Formation console.
   1. Choose Permissions -> Knowledge permissions and select Grant.
      
      
      

      Determine 2 – AWS Lake Formation grant permission

   2. Beneath Principal sort, choose Principals, IAM customers and roles and the AWS managed AmazonSageMakerAdminIAMExecutionRole execution position.
   3. Select Named Knowledge Catalog sources.
   4. Beneath Catalogs, seek for and choose :s3tablecatalog/aws-sagemaker-catalog.
   5. Beneath Databases, choose asset_metadata database.
      

      Determine 3 – AWS Lake Formation catalog, database, and desk

      

      Determine 4 – AWS Lake Formation grant permission

   6. For Desk, choose asset.
   7. Beneath Desk permissions, test Choose and Describe.
   8. Select Grant to avoid wasting the permissions.

Allow information export utilizing the AWS CLI

Configure metadata export utilizing the PutDataExportConfiguration API. The Amazon DataZone service mechanically creates an S3 desk bucket named aws-sagemaker-catalog with an asset_metadata namespace, and schedules a every day export job. Asset metadata is exported as soon as every day round midnight native time per AWS Area.

The SageMaker Area identifier is offered on area element web page within the AWS Administration Console. Accessing the asset desk by means of the S3 Tables console or the Knowledge tab in SageMaker Unified Studio can require as much as 24 hours.

AWS CLI command to allow SageMaker catalog export:

aws datazone put-data-export-configuration --domain-identifier  --region  --enable-export

Use this AWS CLI command to validate the configuration is enabled:

aws datazone get-data-export-configuration --domain-identifier  --region 
{
    "isExportEnabled": true,
    "standing": "COMPLETED",
    "s3TableBucketArn": "arn:aws:s3tables:::bucket/aws-sagemaker-catalog",
    "createdAt": "2025-11-26T18:24:02.150000+00:00",
    "updatedAt": "2026-02-23T19:33:40.987000+00:00"
}

Entry the exported asset desk

1. Navigate to Amazon SageMaker Domains within the AWS Administration Console.
2. Choose your area and choose Open.
   

   Determine 5 – Open Amazon SageMaker Unified Studio

3. In SageMaker Unified Studio, select a mission from the Choose a mission dropdown record.
4. To question SageMaker catalog information, choose Construct within the menu bar after which select Question Editor. To create a brand new mission, observe the directions within the Amazon SageMaker Unified Studio Person Information.
   

   Determine 6 – Open SageMaker Unified Studio Question Editor

The asset_metadata.asset desk is offered in Knowledge explorer. Use Knowledge explorer to view the schema and question information to carry out analytics from.

5. Increase Catalogs in Knowledge explorer. Then, choose and increase s3tablecatalog, aws-sagemaker-catalog, asset_metadata, and asset.
6. Check querying the catalog with SELECT * FROM asset_metadata.asset LIMIT 10;.

Determine 7 – Question SageMaker catalog

Queries for observability and analytics

With setup full, execute queries to achieve insights on catalog utilization and adjustments. To observe asset development, and think about how the info catalog has grown over the past 5 days:

SELECT 
    DATE (snapshot_time) as date,
    COUNT (*) as total_assets
FROM asset_metadata.asset
WHERE 
     DATE (snapshot_time) >= CURRENT_DATE - INTERVAL '5' DAY
GROUP BY DATE (snapshot_time)
ORDER BY date DESC;

Determine 8 – Question asset development

Use the catalog to trace metadata adjustments to find out which belongings gained descriptions or possession over time. Use this question to establish belongings that gained enterprise descriptions over the previous 5 days by evaluating in the present day’s snapshot with the sooner snapshot.

SELECT
    t.asset_id,
    t.resource_name,
    p.business_description as description_before,
    t.business_description as description_now
FROM asset_metadata.asset t
JOIN asset_metadata.asset p ON t.asset_id = p.asset_id
WHERE DATE(t.snapshot_time) = CURRENT_DATE
    AND DATE(p.snapshot_time) = CURRENT_DATE - INTERVAL '5' DAY
    AND p.business_description IS NULL
    AND t.business_description IS NOT NULL;

Examine asset values at a particular cut-off date utilizing this question to retrieve metadata from any snapshot date.

SELECT
     asset_id,
     resource_name,
     business_description,
     extended_metadata['owningEntityId'] as proprietor,
     snapshot_time
FROM asset_metadata.asset
WHERE asset_id = 'your-asset-id'
     AND DATE(snapshot_time) = DATE('2025-11-26');

Clear up sources

To keep away from ongoing costs, clear up the sources created on this walkthrough:

1. Disable metadata export:

Disable the every day metadata export to cease new snapshots:

aws datazone put-data-export-configuration 
  --domain-identifier 

2. Delete S3 Tables sources:

Optionally, delete the S3 Tables namespace containing the exported metadata to take away historic snapshots and cease storage costs. For directions on methods to delete S3 tables, see Deleting an Amazon S3 desk within the Amazon Easy Storage Service Person Information.

Conclusion

On this submit, you enabled the metadata export characteristic of SageMaker Catalog and used SQL queries to achieve visibility into your asset stock. The characteristic converts asset metadata into Apache Iceberg tables partitioned by snapshot date, so you’ll be able to carry out time-travel queries, monitor catalog development, observe metadata completeness, and audit historic asset states. This supplies a repeatable, low-overhead option to keep catalog well being and meet governance necessities over time.

To be taught extra about Amazon SageMaker Catalog, see the Amazon SageMaker Catalog documentation. To discover Apache Iceberg desk codecs and time-travel queries, see the Amazon S3 Tables documentation.

Concerning the Authors