dbt implementation

dbt is a tool for the T in ELT: it transforms data that is already in your warehouse or lakehouse. You write a model as a SQL SELECT statement or a Python dataframe, and dbt turns it into a view or table and handles the DDL.

Models reference each other using ref(), so dbt knows their order and lineage. Tests and documentation sit alongside the code. dbt does not load data from source systems. It is one building block of your data foundation.

dbt Core is the command-line version that you install and run yourself. dbt Cloud is the hosted product, which dbt Labs now calls the dbt platform. It adds a browser-based IDE, job scheduling, CI for each pull request, hosted documentation and notifications. Catalog and the Semantic Layer are available only with certain subscriptions.

The engine now has two generations. dbt v1 is the Python version, still maintained and licensed under Apache 2.0. dbt v2 runs on a new Rust engine and is the default installation; dbt OSS is its Apache 2.0 release.

dbt Labs recommends three layers. Staging creates one clean model per source table, with the prefix stg_. Intermediate combines data and performs calculations for a specific purpose. Marts provide the entities the organisation works with, such as customers and orders. This moves the data from the structure of the source to the language of the business.

Start with views; turn the models that Power BI reads into tables. Use incremental models, which process only new rows, when runs become too slow.

Data tests check the data after a model is built: is a key unique, are any values missing, and does every order refer to an existing customer? Unit tests check your SQL logic against a small, fixed example before building; dbt Labs recommends running them only during development and in CI.

For models other teams rely on, define a contract specifying column names and data types; if the output does not match, the model will not build. You can read about which rules to monitor in improving data quality.

On Databricks, you use the dbt-databricks adapter and run the project as a dbt task in Lakeflow Jobs, on a SQL warehouse. The identity running the job needs permissions in Unity Catalog for the catalogue, schema and tables that the models read and write. Databricks recommends pinning the adapter version. See Databricks implementation.

In Fabric, a dbt job in Data Factory, still in preview, runs dbt Core on a warehouse or lakehouse. See Microsoft Fabric implementation.

With a snapshot. dbt then records each change as a new row, with the columns dbt_valid_from and dbt_valid_to: a type 2 slowly changing dimension. An updated_at column is preferred; if there is none, dbt compares the columns you specify.

If source data arrives too late, source freshness flags this with a warning or an error. Note that dbt build does not run this check; you need to add it as a separate step in the job.

Often, the problem is moving old views across one for one: the same tangle ends up in dbt, just in Git. Without an owner for each mart, the project can also grow into a thicket of models that nobody dares to remove.

Other pitfalls include using incremental models too early, having no CI check so errors only surface in production, and tests that always issue warnings and are therefore ignored. Loading source data belongs in data engineering; we can handle the subsequent upkeep through management and continuous improvement.

If you are creating a few reports from one system and the logic fits in a single Power BI model, dbt adds a layer that someone has to maintain. Likewise, if most of the work involves streaming or heavy processing in Spark, that belongs in pipelines; dbt is suited to the models that follow.

And if nobody on the team wants to write or review SQL, a dbt project offers little value. If a simpler solution is a better fit, we will say so. If you want to compare providers first, see our overview of data platform consultancies.