Get a data lakehouse built

A data lakehouse combines the low-cost, open storage of a data lake with the reliability of a data warehouse. Data is stored as files in an open format, with a layer on top that manages transactions, schema validation and access rights.

This lets reporting, analysis and machine learning use the same validated tables instead of each relying on a separate copy. The lakehouse is part of our data foundation.

A data warehouse provides clean, structured tables for BI and is usually developed in SQL. A lakehouse can also store unstructured and semi-structured data, such as log files and JSON, and is built for Spark and Python alongside SQL.

The distinction is becoming less clear: in Microsoft Fabric, both the lakehouse and the warehouse store Delta tables in OneLake. According to Microsoft, a warehouse is a good fit if you need T-SQL and transactions across multiple tables. If you mainly work with structured data from ERP and CRM systems, read about building a data warehouse.

The medallion architecture divides a lakehouse into three layers. Bronze contains raw, unvalidated data as it came from the source. Silver contains validated, cleaned and deduplicated data. Gold contains tables organised around business questions, for dashboards and models.

Databricks describes it as a recommended practice, not a requirement, and advises against writing directly from ingestion to silver: a schema change at the source could then break your pipeline. Microsoft recommends placing each layer in its own workspace in Fabric to give you more control over each one.

Delta Lake stores data as Parquet files with an accompanying transaction log. This provides ACID transactions, schema validation when writing and a version for every change, so you can use time travel to retrieve an earlier state. It is the default format in both Databricks and Microsoft Fabric.

Because the protocol is open, other tools can read the tables too. With UniForm, Databricks adds Iceberg metadata so Iceberg readers can read the same files without a copy.

On Databricks, you use Unity Catalog: tables, volumes and models receive names such as catalog.schema.table, access rights are assigned through groups, and lineage shows which source feeds into which dashboard. For the Koninklijk Concertgebouworkest, we work with Delta Lake and Unity Catalog.

In Fabric, workspace roles, OneLake security down to row and column level, and sensitivity labels from Microsoft Purview do that work. Agreements on ownership and definitions are part of data governance.

If someone asks for their data to be erased under the GDPR, a DELETE is not enough. Delta Lake retains old table versions for time travel, so the data remains in the files until VACUUM removes it. For tables with deletion vectors, a REORG step is also needed first.

Databricks recommends starting in the bronze layer, then applying the deletion to silver and gold. Document that process before the first request arrives. Your legal counsel or privacy officer should decide on the legal aspects.

Rarely the technology. More often, no one owns each table, every team writes to its own folder, and nobody knows which notebook populates a gold table. The lakehouse then becomes a swamp of files after all.

Maintenance is also overlooked. Many small files slow down queries, and without VACUUM, old versions make storage grow. On Databricks, predictive optimization handles this for managed tables. Agree in advance who will maintain the platform after it goes live; that could be your own team or our management and continuous improvement service.

If you report on a few structured sources, your team works in SQL, and you do not expect to use machine learning or unstructured data. In that case, a data warehouse is simpler to build and manage. If you only need a handful of reports from one system, a Power BI model connected to the source is often enough.

A lakehouse will not fix a messy source either; start by agreeing how data should be entered. If a simpler solution is a better fit, we will say so. If you want to compare providers first, see our overview of data platform consultancies.