Azure Data Factory implementation

You configure the factory so that pipelines load data reliably and repeatably. A pipeline is a series of activities, such as copying data or starting a notebook. A linked service describes a source or compute service, and the integration runtime provides the computing power between them. An Azure Data Factory consultant designs the loading pattern, runtimes, triggers and route to production.

ADF is the loading layer of a data and AI platform. If you want to set up the entire platform on Azure, including storage, networking and compute, read about setting up an Azure data platform.

When a source is on a private network without a direct connection to the cloud, such as an on-premises SQL Server. The self-hosted runtime runs on a Windows machine behind your firewall and makes only outbound HTTP connections. You also need it for sources that require their own driver, such as SAP HANA and MySQL; Java must be installed on the same machine for these.

For availability, you can connect multiple machines to one runtime, operating in active-active mode. In CI/CD, the runtime must have the same name and type in every environment.

A schedule trigger starts pipelines at set times and can start several pipelines at once. It reports success as soon as a run begins, even if that run fails later. A tumbling window trigger works with consecutive time windows, waits for the outcome of a run, can load historical data and retries failed runs.

For incremental hourly or daily loads, the tumbling window trigger is therefore often the safer choice. For new files in a storage account, use a storage event trigger.

By passing parameters with each run. Microsoft gives the example of using one linked service for different databases on the same SQL Server: you make the database name a parameter instead of creating a linked service for each database. A pipeline then reads from a list to determine which tables to load.

Do not parameterise passwords. Microsoft recommends storing secrets in Azure Key Vault and using only the secret’s name as a parameter.

Developers work in a development factory connected to Git, each in their own branch. After a pull request, the change moves through Azure Pipelines to test and production, using Resource Manager templates and parameters for each environment. This means no one needs to change anything in the live factory.

Watch out for settings that are not included in those templates, such as cost reporting per pipeline. You need to enable that separately in each factory.

Set an alert on the failed pipeline runs metric so the team is notified of every error. Data Factory retains run data for only a limited time. Send the logs to Log Analytics through diagnostic settings; you can also query multiple factories together there.

For each pipeline run, you can see usage by meter. If you enable cost reporting per pipeline, Cost Management shows you what each pipeline costs.

Microsoft calls Data Factory in Fabric the next generation of ADF. There are three routes: add your existing factory as an item in a Fabric workspace, upgrade pipelines using the built-in upgrade, or rebuild them. The upgrade classifies each pipeline as Ready, Needs review, Coming soon or Not compatible.

Linked services become connections, datasets are defined within the activity itself, and global parameters become a variable library. Also compare it with Azure Synapse if your pipelines run there.

If you have a few sources that Power BI can refresh itself, or if you are starting from scratch and Fabric is already the standard, a separate ADF environment adds management work without any benefit.

If the problem is not loading data but what happens to it afterwards, such as models, tests and data quality, start with data engineering.