Three data pipeline use cases to make your life easier (Microsoft Fabric)

FREE 40-minute Fabric fundamentals course: www.skool.com/microsoft-fabri...
Data pipelines allow you to automate and schedule data processing tasks in Microsoft Fabric. In this video we walk through three common use cases: getting data from Azure, scheduling a stored procedure, and scheduling a Synapse Data Engineering notebook run.
Missed the last video? Intro to Fabric Data Pipelines • Data Pipelines in Micr...
Want to get up to speed with Microsoft Fabric? • What is Microsoft Fabr...
0:00 Intro
1:08 Pipeline 1: Azure Blob to Data Warehouse
4:26 Pipeline 2: Trigger Stored Procedure
8:57 Pipeline 3: Trigger Synapse notebooks
10:31 Outro

[@shafa7668]

Nice one mate, need to see your other videos

[@LearnMicrosoftFabric]

Thanks for watching!

[@jayananair1]

great videos and thanks for sharing. I was searching for a pipeline that reads data from a text file that has a header and footer row that has to be excluded, also add column headers. Is that possible to achieve use pipeline copy activity. I'm able to achieve the same using Spark, but was looking for skip rows option

[@LearnMicrosoftFabric]

There is an option to use first row as header in a data pipeline copy activity, but in general data pipeline is an orchestration tool, for file/data cleaning it’s better to use dataflow or a notebook 👍

[@canaldovargas]

the data movement cost of the fabric is very high. Anyone having problems with this?

[@AndyKabeer]

Nice video, how do we append data while writing to a destination table using a data pipeline in a warehouse (I don't see any option like we have while writing to the table in Lakehouse e.g. "append" or "overwrite"), my source is rest API

[@LearnMicrosoftFabric]

Hi, thanks! Yes I believe this is currently not possible from a fabric data pipeline. It is possible for Azure Data Factory pipelines so I’m sure they will add the functionality to Fabric soon (hopefully before release). if you need to do this in Fabric you can do it using Dataflow Gen2’s

[@DanielWeikert]

Could you send the output of the notebook to the next step in the pipeline? Or just write the data somewhere and read it in another step? thx

[@LearnMicrosoftFabric]

Hey! It would be useful to be able to do that, but I’m not sure it’s possible currently, normally you would just write the data out to a table and pick it up from there. You can pass parameters into a notebook though from a data pipeline. I’ll be doing making some videos on notebooks in the near future, so I’ll make sure to cover these topics!

[@DanielWeikert]

@@LearnMicrosoftFabric Saw your video with parameters for the pipeline. Interesting thanks. Can you cover the other pipeline elements as well? Just would like to see "Script , Lookup, Webhook and co) br

[@LearnMicrosoftFabric]

Absolutely yes it's on my list to go through each element of pipelines in more detail. WATCH THIS SPACE!