Get your Azure data platform set up

An Azure data platform is a set of Azure services that bring your data together, process it and make it available for reporting and AI. It typically includes Azure Data Lake Storage for storage, Azure Data Factory for loading data, a compute layer such as Azure Databricks or Microsoft Fabric, and Key Vault and Microsoft Entra ID for secrets and access.

To learn how to organise data into layers, read about building a data lakehouse. The platform is part of our data foundation.

Azure Data Factory is Microsoft’s managed cloud service for ETL, ELT and data integration. Pipelines copy data and trigger notebooks, stored procedures or other steps, either on a schedule or after an event.

If a source is on your own network, such as an on-premises SQL Server, you install a self-hosted integration runtime on a server that can access it. You also need one for sources that require their own driver, such as SAP HANA and MySQL.

Microsoft calls Data Factory in Microsoft Fabric the next generation of Azure Data Factory and recommends that new teams start there. You can migrate existing pipelines.

That involves more than copying them: in Fabric, datasets are defined within the activity itself, connections do not support dynamic properties, and Mapping Data Flows can be migrated to Dataflow Gen2, where that functionality is still in preview. If your Data Factory is working well and you choose Databricks as your compute layer, there is no reason to rush.

Start with your team and your reporting needs. If the team works in Python and Spark and needs to put models into production, Azure Databricks is a good fit. If Power BI is your standard and you want to manage less infrastructure, Fabric is a good fit.

For dedicated SQL pools in Synapse, Microsoft provides a migration assistant for moving to the Fabric warehouse and recommends running the old and new systems side by side for a while. The two are not mutually exclusive: Fabric can mirror the Azure Databricks Unity Catalog so that Power BI can read its tables.

Azure Data Lake Storage is not a separate service, but a set of capabilities for a standard storage account. You enable them with the hierarchical namespace setting. Files are then stored in actual directories, and you can manage permissions with Azure RBAC and ACLs at directory or file level.

The data is encrypted using Microsoft-managed keys or your own keys. Fabric can use a shortcut to read from such a storage account without copying the data.

Passwords and keys are stored in Azure Key Vault, and Data Factory retrieves them using its managed identity. Unity Catalog in Azure Databricks can also use a managed identity to access storage, so there are no secrets to renew.

For the network, you use Private Link. In Data Factory, this works through a managed virtual network with managed private endpoints, which helps prevent data exfiltration. Databricks can run in your own virtual network, with Private Link for users and compute.

Microsoft describes a defined process for Data Factory. Developers work in a development factory connected to Git, each in their own branch. After a pull request is approved, the change moves through Azure Pipelines to testing and then production, using Resource Manager templates for each environment.

We define the Azure resources themselves in Terraform, as we did for Vivara’s data platform. Once it is live, we can look after the platform through management and continuous improvement.

When you report from just a few sources and nobody wants to manage the Azure resources. In that case, Microsoft Fabric is simpler as a SaaS service, or a good Power BI dashboard connected to the source may be enough.

A new platform will not fix a poorly organised source either; you need to start by agreeing how data is entered. If a lighter solution is a better fit, we will say so. If you first want to compare providers, see our overview of data platform consultancies.