Bulk Copy from SQL DB to Data Lake Parquet using Azure Data Factory [ADF]

This video is a step by step demo of how to copy bulk a full schema using a lookup activity and aa for each activity to load SQL tables into Azure Storage (Data Lake) parquet files. #datafactory #sqlserver #parquet #datalake #azure #azurestorage

[@EmilianoEmanuelSosa]

Thankkkks , you helped me a lot! In my first job as a data engineer! Greetings from Mendoza, Arg

[@pytalista]

No worries 😉

[@HimenSuthar]

This is very helpful. Thank You Very much Have you extended this exercise to create partition-based parquet file inside table folder.?

[@pytalista]

Hi, do not intend to go on this topic. This is a way to bring data to the lake form transactional systems. then from there I would ingest in delta tables.

[@CurlyHairedKiddHD]

Excellent tutorial! Made my job much easier.

[@pytalista]

Glad it helped!

[@thiagojxs]

Saudações brasileiras!! Very good video my friend. This is the type of content that I'm looking lately. Keep going!!

[@pytalista]

Valeu Thiago 👍🏻

[@marcosadlercreutz]

Thank you very match! It was very helpful!

[@pytalista]

Thanks 🙏🏻

[@sudhirg8773]

Thanks a lot for sharing

[@pytalista]

Welcome 🙏🏻

[@leafyfist685]

Thank you so much brother

[@pytalista]

Welcome 🙏🏻

[@ajayigeorge1356]

Can you give me a good reason why you would need to copy data from a sql db to Azure data lake?

[@ajayigeorge1356]

Cool video

[@pytalista]

A good one. A company uses Databricks as its analytics platform and need operational data for all their analytics workload like data science, machine Learning, business reporting and etc. SQL server is a good OLTP database but not good to do analytics at scale.

[@ajayigeorge1356]

@@pytalista alright, thank you.

[@KeotshepileMosito]

Hi, Which resources did you create under your resource group? Thanks

[@pytalista]

I don’t remember here but a resource group is just a logical container to organize resources in Azure. Cheers.

[@sphnyhptrs]

Great tutorial!

[@pytalista]

Thank you!