Momentum One Academy · glossary
Data Pipeline
Definition
A data pipeline in Microsoft Fabric is a logical workflow of activities that moves and transforms data, applies control logic, and runs as a managed unit. Activities can copy data, transform it, or control execution; runs can start on demand, on a schedule, or from an event. 1
Where it sits in Fabric
Data pipelines belong to Fabric Data Factory, the orchestration layer of Fabric’s data integration experience. They connect sources to destinations and can call other Fabric work, including Dataflow Gen2. The pipeline defines sequence, dependencies, parameters, and triggers; it is not itself the storage layer or the semantic model. Dataflow Gen2 is the low-code, Power Query-based item for preparing and loading data, while the pipeline coordinates when that preparation occurs. 2 3
When to use it
Use a pipeline when a repeatable workflow has multiple steps or needs scheduling, event triggers, branching, retries, parameters, or handoffs between items. It is a strong fit for batch ingestion and orchestration. A single visual transformation may be better kept in Dataflow Gen2 without adding unnecessary control flow. For incremental loads, CDC in Copy Job can reduce the amount of data processed, but Change Data Capture is a loading method, not a replacement for orchestration. 1 2 4
Practical example
Each morning, a pipeline copies the prior day’s order changes from Azure SQL into a Lakehouse, invokes a Dataflow Gen2 item to standardize currencies and reject malformed rows, then loads a curated Warehouse table. Parameters make the same workflow reusable across regions. A validation activity checks expected row counts before the pipeline marks the run complete; a failure branch records context for the operator.
Production caution
Do not assume a successful run proves correct data. Define idempotent writes, explicit keys and watermarks, retry limits, and checks for row counts and duplicates. CDC-based Copy Job behavior depends on supported source and destination connectors; Microsoft documents limitations such as mixed CDC and non-CDC tables being treated as watermark-based incremental copy. Monitor runs and failures in the Monitoring Hub, which Microsoft currently labels preview, and retain enough run history and operational context to troubleshoot. 4 5
Module 3 connection
In Module 3, treat the data pipeline as the control plane for ingestion and transformation. Pair it with Dataflow Gen2 for shaping, CDC for change-aware loading where supported, and Monitoring Hub for operational visibility. Make each dependency, retry, and data-quality check explicit before production scheduling.
Ready to connect ingestion, transformation, and monitoring in a governed Fabric solution? Explore Momentum One Academy’s Fabric Production Workshop.
References
Plan your workshop