Delta Lake as the storage layer
Delta Lake adds a transaction log to Parquet files. This provides ACID transactions, schema validation on writes and a new table version with every change. The protocol is open, so other tools can read the tables too.
Databricks implementation
One lakehouse where data engineering, analytics and machine learning use the same well-governed tables.
Trusted by leading organisations
When implementing Databricks, we set up a lakehouse: Delta tables in your cloud environment, Unity Catalog for access rights and lineage, Lakeflow for data ingestion and orchestration, and MLflow for models. As a Databricks Partner, Data Science Lab does this for organisations with many data sources, large data volumes or the ambition to bring machine learning into production.
From standalone notebooks to a managed platform
Many Databricks environments start with one data scientist and a handful of notebooks. Over time, jobs run on manually created clusters, tables remain in the old Hive metastore, and no one knows which notebook populates the customer table that the marketing report relies on. When that colleague leaves, the knowledge goes with them.
Costs rise too. Interactive clusters stay running after working hours, and without tags, you cannot see which team is responsible for which usage. Tags only apply to usage from the moment you add them, so if you start later, you lose the history.
The platform does what it needs to do. What is missing is a consistent structure for catalogues, an owner for each table, and a path from development to production. That is where our work on the data foundation begins.
What a well-structured lakehouse includes
Governance with Unity Catalog
Tables, volumes, functions and models sit in a single catalogue using the three-part name catalog.schema.object. You assign permissions to groups, and column-level lineage shows which source feeds into each dashboard or model.
Bronze, silver and gold
Raw data remains unchanged in bronze; silver validates and deduplicates it; gold provides tables for reporting and models. Databricks advises against writing directly from ingestion to silver: a schema change could then break your pipeline.
Loading and orchestration with Lakeflow
Lakeflow Connect loads data incrementally from sources such as SQL Server, Salesforce and SharePoint. Declarative pipelines use expectations to monitor quality at the record level, while Lakeflow Jobs schedules tasks and reports errors.
Taking models into production with MLflow
We track experiments, model versions and approvals in MLflow, with models stored in Unity Catalog. Model Serving makes a model available as a REST API, so your applications can call it rather than leaving it in a notebook.
Databricks SQL for analysts
SQL warehouses give analysts and tools such as Power BI and Tableau access to the same gold tables. Databricks recommends serverless warehouses where available because they start quickly and scale back down.
Is your Databricks environment running as it should?
Discuss your catalogues, costs or the move to Unity Catalog with one of our data engineers.
Our step-by-step plan for a Databricks lakehouse
-
Choose the cloud, region and network
We work with your IT department to define the cloud environment, region, network and identities, and choose classic or serverless compute as our starting point.
-
Define catalogues and owners
We organise Unity Catalog by environment and domain, assign an owner to each catalogue and manage permissions through groups from your identity provider.
-
Connect sources to the bronze layer
We load the first sources incrementally using Lakeflow Connect or custom code and retain the raw data so you can reprocess it later.
-
Model the silver and gold layers
For one specific question, we build validated and aggregated tables, with quality rules in the pipeline. Users check the results against figures they know.
-
Control costs from the start
Compute policies limit cluster types and sizes, automatic shutdown stops forgotten clusters, and tags and budget alerts make each team’s usage visible in the billing system table.
-
Deploy through code, then manage
We define jobs and pipelines in Declarative Automation Bundles in Git so every release is repeatable. Then your team takes over, or we manage the platform.
The data platform is the central place where our data comes together. It lets us make data available across the business for business insights, while ensuring there is only one version of the truth.
Choosing between Databricks and Microsoft Fabric
Databricks runs on Azure, AWS and Google Cloud, and the choice usually depends on the cloud you already use. With classic compute, clusters run in your own cloud account; Databricks manages serverless compute itself, in the same region as your workspace. Pay attention to regions: Unity Catalog has one metastore per region, and lineage does not cross regional boundaries.
Databricks suits organisations with many data sources and large volumes, teams working in Python and Spark, and models that run in production with version control and monitoring. If you mainly want a single platform centred on Power BI with little infrastructure management, explore our approach to Microsoft Fabric implementation. You can also combine them: Fabric can mirror the Azure Databricks catalogue, allowing Power BI to read the gold tables without copying them.
For GVB, we set up an MLOps workflow in Databricks with reusable templates, so a model that segments travel behaviour runs in production and can be maintained. For Fokker Services Group, we brought together fragmented data sources on a Databricks platform in Azure for a planning tool that predicts when maintenance is needed.
About Data Science Lab
Any organisation can start an AI pilot. The challenge is what comes next: making AI part of the organisation. That’s where Data Science Lab makes a difference.
Read more about us
Why organisations choose us
10+
years of experience
8.7
client rating
200+
projects completed
Microsoft & Databricks Partner
Further reading
- Data and AI platformHow we choose between Databricks and Fabric and store your data in an open format in your own cloud.
- Management and continuous improvementA managed data and AI platform on Databricks or Microsoft Fabric, including after go-live.
- SecondmentData platform engineers and MLOps engineers who strengthen your team on a temporary or interim basis.
Frequently asked questions
A working lakehouse with an initial use case: incrementally loaded data sources, bronze, silver and gold tables in Delta Lake, permissions and lineage in Unity Catalog, and monitored jobs. We also set up compute policies, tags and budget alerts, plus a Git-based release process. Your team can build on this without having to lay the foundations again.
The core is the same across all three: Delta Lake, Unity Catalog, Lakeflow and MLflow. The difference lies in how Databricks connects to your identities, network and storage, so choose the cloud where your data and management processes already sit. Choose a region deliberately, too: a Unity Catalog metastore belongs to one region. Also check whether serverless compute is available there.
By building controls into the platform from the start. Compute policies limit which clusters someone can start, automatic shutdown stops inactive clusters, and we use serverless SQL warehouses where possible. Tags for each team make usage visible in the billing system table. Budgets trigger alerts, but they are not hard limits: someone needs to act on them.
We migrate step by step. Databricks recommends first federating the old Hive metastore as a foreign catalog, then converting tables in place without moving data. Existing jobs keep running in the meantime. We then migrate queries and jobs and retire the old metastore, so no one can bypass the new permissions.
We track experiments and model versions in MLflow and register models in Unity Catalog, so lineage and permissions apply to models too. Model Serving turns a model into a REST API for your applications, while jobs handle retraining. Once the model is live, we monitor its quality, because it can drift as the data changes.
We are a Databricks Partner and design platforms according to Databricks best practices. For the Koninklijk Concertgebouworkest, we built a data platform using Databricks, Azure, Delta Lake, Unity Catalog and dbt to connect information from multiple systems. For Vivara, we developed a data platform that loads data daily from sources including SAP, Magento and SharePoint into Azure Databricks.
If you have a few data sources, modest volumes and mainly use Power BI for reporting, a Spark platform is often more than you need. A well-modelled database behind Power BI is often enough, or Microsoft Fabric if you add more sources and teams. Smaller organisations can read more about data and AI for SMEs. And without someone to maintain the Python and Spark code, Databricks can become a burden rather than a help.
Questions about Databricks?
Stef Stoelwinder
Business DevelopmentE-mail info@datasciencelab.nl
Phone +31 (0)20 3636 724
Want to set up a solid Databricks environment or clean up your existing one?
Tell us what’s running now and where the sticking points are. We’ll look at catalogues, costs and the first use case that makes a difference.
Message sent
Thanks for your message. We’ll be in touch as soon as possible.