What it is

Data Pipelines in Fabric (inherited from Azure Data Factory) let you visually build ingestion flows by chaining activities together: copy data, trigger a notebook, run a stored procedure, send an alert on failure…

When to use it

• The source is a relational database (SQL Server, PostgreSQL, Oracle, MySQL…)
• You need to handle conditional logic (if the table is empty → overwrite mode, otherwise → incremental mode)
• The pipeline must orchestrate multiple steps in sequence (ingestion → transformation → quality checks)
• Scheduled runs are required (every day at 2am, every hour…)

Two ingestion modes to know

Incremental mode Loads only the new or modified rows since the last run. Usually relies on a datetime field (creation or modification date) or an incrementing identifier.

Advantages    : fast, cost-efficient, minimal network load
Disadvantages : requires a reliable watermark field in the source
Best for      : transactional tables (orders, sales, events)

Overwrite mode (full reload) Reloads the entire table on every run. Simple to set up, but expensive on large tables.

Advantages    : simple, reliable, no dependency on a watermark field
Disadvantages : slow on large volumes, high network load
Best for      : reference tables (products, customers, stores), small stable tables

Comment ça marche ??

Press enter or click to view image in full size

A typical pipeline structure

Press enter or click to view image in full size

[Lookup] — reads the last load date from a parameters table
    ↓
[If Condition] — incremental if a date exists, overwrite otherwise
    ↓
[Copy Data] — extracts data from the source and writes it to Bronze in the Lakehouse
    ↓
[Notebook Activity] — triggers the Bronze → Silver transformation
    ↓
[Web Activity] — sends a Teams/email alert on failure

💡 Best practice: Store ingestion parameters (table name, last load date, mode) in a dedicated pipeline_config table in the Lakehouse. This makes each pipeline generic and reusable across any source table.

What it is

Fabric Notebooks let you write Python/PySpark code with native access to the Lakehouse, the distributed Spark runtime, and the full Python ecosystem of libraries.

When to use it

• Data volumes are very high (tens or hundreds of millions of rows)
• Ingestion logic is complex: advanced deduplication, non-standard format parsing, API-based enrichment…
• You need fine-grained performance control: partitioning, Delta read/write optimization
• The team is comfortable with Python / PySpark

Example: incremental ingestion from ADLS in PySpark

from pyspark.sql import functions as F
from delta.tables import DeltaTable

# Parameters
source_path = "abfss://raw@mylake.dfs.core.windows.net/orders/"
target_table = "Tables/bronze/orders"
watermark_col = "modified_date"
# Read the last loaded date
last_load = spark.sql("SELECT MAX(modified_date) FROM bronze.orders").collect()[0][0]
# Read only new data
df_new = (
    spark.read.format("parquet").load(source_path)
    .filter(F.col(watermark_col) > last_load)
)
# Append to the Lakehouse
df_new.write.format("delta").mode("append").save(target_table)
print(f"{df_new.count()} new rows ingested.")

💡 Best practice: Use Delta Lake format for all tables in your Lakehouse. Delta natively supports ACID transactions, time travel (querying a previous version of your data), and automatic file optimization with OPTIMIZE and ZORDER.

What it is

Shortcuts are a feature unique to Microsoft Fabric. They let you reference existing data (in Azure Data Lake Storage, Amazon S3, Google Cloud Storage, or another Fabric Lakehouse) without physically copying it.

The Lakehouse sees the data as if it were its own — while the data stays at its original location.

When to use it

• Data is already stored in ADLS Gen2 or S3
• You want to avoid data duplication and the associated transfer costs
• You need to share data across multiple Fabric workspaces without copying
• The data is managed by another team and is already available in the cloud

What Shortcuts don’t do

A Shortcut does not transform data. It gives read access to data as-is. If transformation is needed, it happens downstream via a Notebook or a Dataflow.

Decision Table: Which Option Should You Choose?

Source = SQL database, conditional logic needed
→ Data Pipeline

Source = Excel / CSV files dropped by business teams
→ Dataflow Gen2

Source = REST API with complex pagination and auth
→ Notebook (PySpark)

Data already in ADLS Gen2 or S3
→ Shortcut

Large volume (>100M rows), complex transformation
→ Notebook (PySpark)

Non-technical team, simple transformations
→ Dataflow Gen2

Multi-step orchestration with error handling
→ Data Pipeline

Share data across workspaces without copying
→ Shortcut