Skip to content

Improving data quality

We make data quality measurable and fix errors at their source. For Vivara, we built a data platform with a single central source for all departments.

Vivara wanted one central place for its data, with the highest data quality

For Vivara, we developed a data platform that retrieves data daily from sources including SAP, Magento and SharePoint and stores it in Azure Databricks. A bespoke data model lets every department, from finance to marketing, analyse the data.

Read the case study

Where do errors arise in our data?

The ERP system and online shop show different stock levels, an item appears twice in the master data, and every Monday someone corrects it in Excel. We measure where things go wrong and address the cause, at the source or during processing.

Book a meeting
Man vergelijkt met gefronste blik twee uitgeprinte overzichten naast zijn laptop

Catch errors in the pipeline before anyone sees them in a report

Discuss your data quality

Rules for each use

For each report or model, we define which fields are critical and when a value is valid, complete and up to date enough.

Tests on every run

Checks for unique keys, empty fields and valid values run alongside the data. Data quality checks are integrated into Fokker Services Group’s data platform.

Visible by source

You can see the result of each measurement by source and by table, for example in Power BI. That way, the owner knows what is going wrong.

Anomalies that stand out

If the data a model receives changes, monitoring raises an alert. For Rijkswaterstaat, we developed a package for drift and outlier detection.

What counts as good enough depends on the report or model that relies on the data

  1. Define the use and the rules

    Which report or model relies on the data, which fields are critical, and when is a value good enough?

  2. Profile and measure

    We profile the sources and implement the rules as tests in the pipeline, measuring the results on every run.

  3. Address the cause and monitor

    We fix errors at the source or during processing. Monitoring then flags changes in quality.

Do you need a data quality approach, or is input validation enough?

Talk it through with us

A good fit when

Several reports or models rely on data from different sources, people correct figures by hand, or an AI application needs demonstrably reliable training data.

Not a good fit when

The errors come from a single input screen in one system. In that case, a required field or dropdown menu in that system is often enough, without adding another layer.

What determines the scope

The number of sources and tables, how many rules are critical, whether you already have a data platform, how often the data is refreshed and who handles the alerts. You provide access and an owner for each source.

Are we tied to a tool?

The rules are stored as code in your own version control system, for example as dbt tests or as expectations in a Databricks pipeline. Your team can then maintain and extend them.

Correct in the report, check in the pipeline or fix at the source?

Correct in the report

Choose this for a one-off analysis. You’ll need to repeat any correction made in Excel or a report formula the next time the data is refreshed.

Check in the pipeline

Choose this if several reports or models use the same data for calculations. Our advice: run tests every time, with an alert to the source owner.

Fix at the source

Choose this if the error occurs during data entry. A required field or dropdown list prevents it; that requires agreement with the source owner. See data governance.

Data quality checks are part of the data platforms we build

Data Science Lab has worked with data and AI since 2016 as a Databricks and Microsoft partner. We built data platforms for Vivara and Athlon, among others, and are building one for Fokker Services Group with integrated data quality checks.

Book an introductory meeting

The first conversation is about one report you don't currently trust

Stef Stoelwinder

Stef Stoelwinder

Business Development

Stef is your first point of contact. In our first conversation, we discuss which data is incorrect and who relies on it — at your offices, in Amsterdam or online.

How can we make our data reliable again?

Describe which data is incorrect and who relies on it. We’ll reply by email or arrange an initial conversation.

Enter a valid email address.

Let us know how we can help you.

Frequently asked questions

Data quality is the extent to which data is fit for the purpose you use it for. NORA, the Dutch government’s reference architecture, describes dimensions including accuracy, completeness, consistency, timeliness and validity in its data quality framework.

The same table may be good enough for a monthly report but too coarse for a predictive model. Improving data quality therefore starts with how you use the data, not with the table. Data quality is one of the building blocks of a reliable data foundation.

You define rules for each field and count how many records meet them, such as the proportion of customers with a valid postcode or orders with an existing item number. NORA recommends starting with data profiling: a statistical picture of each column, including missing values, outliers and duplicate keys.

You can check most rules automatically with queries against the data. Accuracy is often harder to check: you only know whether the stock figure in the system is correct when someone counts what is on the shelf. Focus your measurements on the critical fields that a report or model actually relies on, and track the results with every run.

In dbt (data build tool), which is also used in the data platforms of Vivara and Fokker Services Group, you write tests alongside your transformations. There are four standard tests: unique, not_null, accepted_values and relationships. A test looks for records that break the rule; if it finds none, it passes.

Databricks has expectations in pipelines: for each rule, you choose whether a failing record is allowed through and counted, dropped, or causes processing to stop. In Unity Catalog, data quality monitoring checks whether tables arrive on time and in full. If you work with Microsoft Fabric, Microsoft Purview offers profiling, quality rules and scores.

You decide this for each rule. A missing email address need not block a revenue report: you let the record through and count it. You set aside an order that has no existing customer. If the total in a financial report does not add up, it is better to stop processing than to show an incorrect figure.

The owner of the source receives a notification about rejected records. NORA recommends correcting errors at the source where possible and finding their cause, so you are not treating the same symptom every week.

Yes. Some of the hardest errors occur there. Master data is the core information used across multiple systems, such as customers, items, suppliers and locations. If an item has one code in the ERP system and another in the online shop, any report that combines the two will count it incorrectly.

In a data platform, we link this information and define which source takes precedence for each field. For the Koninklijk Concertgebouworkest, we link, structure and standardise data from multiple systems and archives.

According to the Dutch Data Protection Authority, organisations must ensure that personal data is accurate and update it when necessary. People can ask for incorrect information to be corrected. This is the principle of accuracy, and you must be able to demonstrate that you comply with it.

For high-risk AI systems, Article 10 of the AI Act requires training, validation and test data to be relevant, sufficiently representative and, as far as possible, free of errors and complete. You can read about when this applies to you on our AI Act page. Legal interpretation is a matter for your lawyer or compliance department.

The organisation that creates and uses the data. Each source should have an owner who decides what is good enough, and often a data steward who follows up on notifications. You can read about how to organise ownership and definitions under data governance.

We build the checks, measurements and notifications so your own team can maintain them. If we build the data platform, we can also manage it afterwards through management and continuous improvement.

If the data is only used once, as in a one-off analysis, correct it directly. And if the errors all come from one input screen, a required field or a drop-down list in that system is often enough. You do not need to wait until everything is in order before you start reporting or using AI. Start with the data that one important report or model relies on, and expand from there if it works. If the data is scattered across separate systems and files, a central data platform can help first. If you work in construction and want to bring together costing, planning and project records, take a look at data and AI for construction. One example where data quality matters directly is WPM reporting. If your energy, procurement and HR data needs to be traceable for a sustainability report, take a look at a CSRD reporting dashboard.