Azure Data Factory (ADF): Build Scalable Data Pipelines

Build, Orchestrate, and Secure Data Pipelines with Azure Data Factory for Cloud, Hybrid, and Multi-Cloud Integration

Description

A warm welcome to the Azure Data Factory (ADF): Build Scalable Data Pipelines course by Uplatz.

What is Azure Data Factory

Azure Data Factory (ADF) is Microsoft’s cloud-based ETL (Extract, Transform, Load) and data integration service. It enables organizations to move, transform, and orchestrate data from multiple sources, whether on-premises, in the cloud, or across different platforms.

It serves as the data pipeline service within Azure, allowing data to be connected, cleaned, and delivered to systems such as data lakes, data warehouses, business intelligence platforms, and machine learning pipelines.

How Azure Data Factory Works

Azure Data Factory follows a workflow approach with four main stages:

1. Connect to Data Sources (Extract)

ADF connects to more than 100 data sources using linked services, such as SQL Server, Azure Blob Storage, Amazon S3, Google Cloud Storage, Salesforce, and SAP. Data is ingested either in batches or real time.

2. Prepare and Transform Data (Transform)

ADF uses Data Flows (a visual, no-code transformation interface) or custom activities such as SQL scripts, Spark jobs, Databricks notebooks, and stored procedures. Transformations may include joins, filtering, aggregations, format conversions (CSV to JSON, Parquet, etc.), and data cleansing.

3. Move and Load Data (Load)

Data is loaded into target systems including Azure SQL Database, Azure Synapse Analytics, Azure Data Lake, Cosmos DB, or external storage systems. It supports full loads, incremental (delta) loads, and streaming ingestion.

4. Orchestrate and Monitor Pipelines

Workflows are organized into pipelines that contain one or more activities. Triggers allow scheduling or event-based execution. ADF includes built-in monitoring and logging to track performance, identify errors, and analyze throughput.

Core Components of ADF

• Pipelines: Logical groups of activities that define a workflow

• Activities: Individual steps such as copy, transform, or execute stored procedure

• Datasets: References to data structures such as tables or files

• Linked Services: Connection details to data sources

• Data Flows: Visual interface to build transformation logic

• Integration Runtime (IR): The compute engine that executes data movement and transformations, available as cloud or self-hosted

Why Use Azure Data Factory

• Fully managed and serverless with automatic scaling

• Supports hybrid and multi-cloud data integration

• Low-code/no-code development experience with option for advanced coding

• Enterprise-grade security and governance through Azure Key Vault and RBAC

• Prepares data pipelines for advanced analytics, reporting, and machine learning workloads

Azure Data Factory – Course Curriculum

Topic 1: Foundations of Azure & ADF

• Session 1 – Introduction to Azure Data Factory

• Session 2 – Cloud Computing Part-1

• Session 3 – Cloud Computing Part-2

• Session 4 – Cloud Services

• Session 5 – Types of Data

Topic 2: Core Components of ADF

• Session 6 – Top Components of ADF PART-1

• Session 7 – Top Components of ADF PART-2

Topic 3: Data Copy & Migration Basics

• Session 8 – Case Study-1: Copying the Data from Blob Storage to Blob Storage

• Session 9 – Azure BLOB Storage to ADLS Gen2 Copy

• Session 10 – Copy Multiple Files from Azure BLOB Storage to ADLS Gen2

• Session 11 – Copy Data from Azure Blob to SQL DB

• Session 12 – Copy Data Tool

• Session 13 – Copy Activity Behaviour PART-1

• Session 14 – Copy Activity Behaviour PART-2

Topic 4: Parameterization in ADF

• Session 15 – Parameterized Linked Services PART-1

• Session 16 – Parameterized Linked Services PART-2

• Session 17 – Parameterized Dataset and Pipeline

Topic 5: Advanced Copy Operations

• Session 18 – Copy Bulk Data from SQL Database to Blob Storage PART-1

• Session 19 – Copy Bulk Data from SQL Database to Blob Storage PART-2

• Session 20 – Copy Activity on the Basis of File Counts in Source

Topic 6: Stored Procedures & Transformations

• Session 21 – Understanding Stored Procedure on Azure Cloud

• Session 22 – Copy of the Data Using Stored Procedure and SQL Query

• Session 23 – Conversion of CSV to JSON Using ADF PART-1

• Session 24 – Conversion of CSV to JSON Using ADF PART-2

• Session 25 – Copy File (JSON to CSV)

Topic 7: Security & Key Management

• Session 26 – Azure Key Vault Service

Topic 8: Loading Strategies

• Session 27 – Full Load and Delta Load PART-1

• Session 28 – Full Load and Delta Load PART-2

• Session 29 – Full Load and Delta Load PART-3

Topic 9: Hybrid Data Integration

• Session 30 – Copy Data from On-Premise to Cloud in ADF PART-1

• Session 31 – Copy Data from On-Premise to Cloud in ADF PART-2

Topic 10: API Integration & Variables

• Session 32 – Integration of API with ADF PART-1

• Session 33 – Integration of API with ADF PART-2

• Session 34 – Pipeline Variable PART-1

• Session 35 – Pipeline Variable PART-2

Topic 11: Multi-Cloud Integrations

• Session 36 – Integration of AWS with Azure

• Session 37 – Integration of ADF with Google Cloud Storage

Topic 12: Scheduling & Orchestration

• Session 38 – Triggers in ADF

• Session 39 – Schedule Trigger in Azure Data Factory PART-1

• Session 40 – Schedule Trigger in Azure Data Factory PART-2

Topic 13: Data Transformations

• Session 41 – Join Transformation Using Data Flows in ADF

Reminder – Rate this 100% off Udemy Course on Udemy that you got for FREEE!!