Granular Utilization Attribution for dbt Pipelines with Question Tags – Cloned

Your dbt venture runs 80 fashions each evening. The warehouse invoice doubled final quarter. Mannequin efficiency varies extensively, and the consequences of the latest optimizations are unclear. Finance asks which workforce is accountable. You open the question historical past and see… 80 similar rows labeled ‘Databricks Dbt.’ Good luck.

With Question Tags (now in Public Preview), knowledge groups can now profit from out-of-the-box auto-injected tags, corresponding to dbt_model_name, which enrich each run. You can even connect your individual customized tags — workforce, price middle, surroundings, something — to each question your pipeline generates.

Tags are recorded in system.question.historical past, making price attribution, efficiency debugging, and workload monitoring a easy SQL question away (full particulars within the documentation).

This weblog walks by means of a whole, open-source dbt venture that demonstrates Question Tags end-to-end: from configuration to price attribution dashboards. Every little thing described right here is accessible as a GitHub repository you may clone and deploy to your individual workspace, or simply ask Genie.

How dbt-databricks integrates with Question Tags

The dbt-databricks adapter (model 1.11+) helps Question Tags natively. There are three ranges at which tags might be utilized, every constructing on the earlier:

Auto-injected tags

Along with your customized tags, dbt-databricks robotically injects metadata about every mannequin execution:

These auto-tags imply you get per-model visibility with zero configuration — the adapter does it for you.

Profile-level tags

The only strategy: add a query_tags subject to a selected goal in your dbt profile. Each question within the venture inherits these tags robotically.

For instance, this single line tags each question with 4 dimensions: who owns it (workforce), the place the price goes (cost_center), which pipeline it belongs to (project_name), and what surroundings it runs in (env).

Mannequin-level tags

For extra granular attribution, you may provide tags on particular fashions in dbt_project.yml or mannequin configuration in its sql definition. 

Mannequin-level tags merge with profile-level tags. If each outline the identical key, the model-level worth takes precedence.

The place tags seem – system.question.historical past

After operating dbt run, each SQL assertion seems in system.question.historical past with the query_tags column populated as a MAP. You’ll be able to question it utilizing commonplace map entry syntax:

This returns each tagged question from the final 7 days, with the customized and auto-injected tags extracted into particular person columns — prepared for aggregation.

You can even discover the Question Tags for the question you ran within the Question Historical past UI or the SQL Warehouse Monitoring UI.

On the underside proper of the Question Profile, you will note the Question Tags you outlined, offering you with all info needed at look.

Price attribution with Question Tags

Question Tags allow granular utilization attribution to be decided straight through SQL queries, eliminating the necessity for guide log evaluation or splitting warehouse assets.

Which dbt fashions devour essentially the most warehouse assets?

You’ll be able to reply this two methods: ask Genie in plain language for ad-hoc exploration or write the SQL your self for a repeatable, dashboard-ready consequence. Each learn from the identical system.question.historical past knowledge.

Possibility 1: Genie

Genie writes and runs the equal question, and you retain drilling in follow-up questions with out touching any SQL.

Possibility 2: SQL

Both path returns the identical image. In our reference venture, the 4 mart tables (materialized as desk) dominate compute time, whereas staging views and metric views are near-instantaneous. This instantly tells you the place optimization effort ought to focus.

Constructing a self-monitoring dashboard

Our reference venture contains an AI/BI dashboard that queries system.question.historical past filtered by the venture’s personal question tags. The consequence: the pipeline that analyzes billing knowledge additionally tracks its personal prices — dogfooding Question Tags on itself.

The dashboard contains:

• KPIs: Complete tagged queries, whole compute seconds, distinct dbt fashions
• Each day exercise: Question depend and compute time per day, break up by surroundings
• Mannequin breakdown: Compute time per mannequin, coloured by materialization sort
• Materialization break up: Pie chart displaying how compute distributes throughout desk, view, and metric_view
• Question element desk: Each tagged question with mannequin, period, surroundings, and executor

In our reference venture, the 4 mart fashions accounted for 92% of compute time — with out Question Tags, that perception was invisible.

Constructing this dashboard your self takes minutes with Genie Code: ask it for compute time per dbt mannequin from system.question.historical past filtered by your question tags, and it writes the SQL and assembles the visuals. If you happen to’d reasonably skip straight to the completed consequence, the dashboard additionally ships within the reference venture and deploys with one databricks bundle deploy alongside the dbt job (see the Github repository for the detailed information).

Tagging metric views

Databricks metric views (obtainable with dbt-databricks 1.12+) are a brand new materialization sort that defines reusable enterprise semantics within the type of dimensions and measures straight in Unity Catalog (see full documentation). They will carry Question Tags identical to another mannequin, utilizing the query_tags config parameter:

Word the excellence: query_tags are connected to the SQL queries that create or refresh the metric view (tracked in system.question.historical past), whereas databricks_tags are Unity Catalog tags on the object itself (for governance and discovery). The previous is for query-level monitoring, whereas the latter one is Unity Catalog object stage for total knowledge discoverability. 

Finest practices for tagging dbt initiatives

In this article, we lined the holistic course of to construct a strong FinOps apply the place Question Tags are foundational for price attribution. Here is what we realized constructing the reference venture and speaking with dbt energy customers:

• Use a constant tag hierarchy. Outline organization-wide tags on the profile stage (workforce, cost_center, project_name, env) and reserve model-level tags for distinctive instances. This retains tags predictable and avoids per-model configuration sprawl.
• All the time tag the surroundings. Use totally different env values for native growth (local-dev) and deployed jobs (dev, staging, prod). This allows you to separate ad-hoc growth queries from scheduled manufacturing runs in your analytics. In our reference venture, the native profile units “env”: “local-dev” whereas the deployed profile units “env”: “dev”.
• Use `project_name` to tell apart pipelines. When a number of dbt initiatives share a warehouse, project_name enables you to attribute prices per pipeline with out splitting warehouses. Mixed with the auto-injected @@dbt_model_name, you get full traceability: venture → mannequin → materialization.
• Do not over-tag. The auto-injected tags already cowl mannequin title, materialization sort, and adapter variations. You not often have to duplicate this info in customized tags. Focus customized tags on enterprise context that dbt cannot infer: workforce possession, price middle, venture id.
• Tag metric views explicitly. Since metric views are a more recent materialization, it is helpful to tag them with a characteristic key (e.g., “characteristic”: “metric_view”) so you may simply filter for metric view creation queries in your price evaluation.

Strive it your self

The whole reference venture is accessible on GitHub: github.com/databricks-solutions/dbt-query-tags

To get began:

1. Clone the repository
2. Create a Python 3.12 digital surroundings and set up dependencies: pip set up dbt-databricks>=1.12.0a1
3. Replace profiles.yml together with your workspace host, SQL warehouse HTTP path, catalog, and customized question tags
4. Run dbt deps && dbt run –profiles-dir . to execute the pipeline
5. Question system.question.historical past to see your tags in motion
6. Replace dbt_profiles/profiles.yml and databricks.yml to level to appropriate configuration.
7. Deploy with databricks bundle deploy for scheduled runs and the analytics dashboard

Swap in your individual workforce and value middle values. The sample works for any dbt venture on Databricks.

Clone the repository right now! It takes one line in your profile to unlock model-level utilization attribution visibility throughout your whole warehouse.