Improving data quality

Data quality is the extent to which data is fit for the purpose you use it for. NORA, the Dutch government’s reference architecture, describes dimensions of data quality including accuracy, completeness, consistency, timeliness and validity.

The same table may be good enough for a monthly report but too coarse for a predictive model. Improving quality therefore starts with how you use the data, not with the table. Data quality is one of the building blocks of a reliable data foundation.

You define rules for each field and count how many records meet them, such as the proportion of customers with a valid postcode or the proportion of orders with an existing item number. NORA recommends starting with data profiling: a statistical picture of each column, including empty values, outliers and duplicate keys.

You can check most rules automatically by querying the data. Accuracy is often different: you only know whether the stock level in the system is correct when someone counts what is on the shelf. Focus your measurements on the critical fields that a report or model actually relies on, and track the results on every run.

In dbt (data build tool), which is also used in the data platforms of Vivara and Fokker Services Group, you write tests alongside the transformations. There are four standard tests: unique, not_null, accepted_values and relationships. A test looks for records that break the rule; if it finds none, it passes.

Databricks has expectations in pipelines: for each rule, you choose whether a failing record is allowed through and counted, dropped, or stops processing. In Unity Catalog, data quality monitoring checks whether tables arrive on time and complete. If you work with Microsoft Fabric, Microsoft Purview offers profiling, quality rules and scores.

You decide that for each rule. A missing email address need not hold up a revenue report: you can let the record through and count it. You can set aside an order with no matching customer. If the total in a financial report is wrong, it is better to stop processing than to show an incorrect figure.

Rejected records are flagged to the owner of the source. NORA recommends correcting errors at the source where possible and finding their cause, so you are not dealing with the same symptom every week.

Yes. It is often where the hardest errors occur. Master data is the core information that appears in multiple systems: customers, items, suppliers and locations. If an item has a different code in the ERP system and the online shop, any report combining the two will produce an incorrect count.

In a data platform, we link this information and establish which source takes precedence for each field. For the Koninklijk Concertgebouworkest, we link, structure and standardise data from multiple systems and archives.

According to the Dutch Data Protection Authority, organisations must ensure that personal data is accurate and update it when necessary. People can ask for incorrect information to be changed. This is the principle of accuracy, and you must be able to demonstrate that you comply with it.

For high-risk AI systems, Article 10 of the AI Act requires training, validation and test data to be relevant, sufficiently representative and, as far as possible, free of errors and complete. You can read about when this applies to you under AI Act. Legal interpretation is a matter for your lawyer or compliance department.

The organisation that creates and uses the data. Each source should have an owner who decides what is good enough, and often a data steward who handles alerts. You can read about how to organise ownership and definitions under data governance.

We build the checks, measurements and alerts so your own team can maintain them. If we build the data platform, we can also look after it afterwards through management and continuous improvement.

If the data is used only once, as in a one-off analysis, correct it directly. And if the errors come from a single input screen, a required field or a dropdown list in that system is often enough.

Do not wait until everything is in order before starting with reporting or AI, either. Start with the data that one important report or model relies on, then expand if it works. If the data is spread across separate systems and files, a central data platform can help first.