End to end ETL operation in Azure Data Factory| Minor Project 2 | Azure Data Factory

In this video we will create a small project on ADF using Excel and Azure Data Factory.
Project Description:
We will store an Excel file in Azure Blob Storage
Will create a dataflow in Azure Data Factory to fetch the data
Apply filter to get data of all the IT Department Employees
Sink the data in blob storage in form of Excel
Source Data:docs.google.co...
_______________________________________________________________
Want to become Data Engineer?
Only community based channel covering all the skillsets end to end with Project Work.
All Free !!!!!!!!!!!!!!!!
Social Handles:
LinkedIn: / sukhjeevan
Instagram: sukhjeevan__
GitHub:sukhjeevan287
Mail: [e-mail]
_________________________________________________________________
#azure #adf #dataengineer #datafactoryproject

[@noothandairies]

nice explanation bro

[@sherubhaker1257]

simply add filter in excel file

[@dataengineerpro]

Definitely the intention of video was to give an overview how to perform operations in ADF:)

[@manikantareddynukala7929]

Bro suggest some projects related to cloud iot platforms

[@dataengineerpro]

Sure will release a video soon on this

[@SantoshKumar-yr2md]

release some complex ETL operations where you used SparkSQL, large dataset likewise

[@dataengineerpro]

Yes comming soon

[@lakshmiprasannalingaladinne]

hey hi,Dataflow expression builder is not opening ,how should i access it

[@dataengineerpro]

Can you elaborate the issue

[@manum8732]

Hi Bro, getting the below error while trying to preview data in DATASET creaton. Invalid excel header with empty value, filename is 'employee sample data.xlsx', sheetname is 'eployee sample data', the row number is '0', the column number is '14'. any solution for this.

[@dataengineerpro]

Check if the Excel have space in column name, if yes remove that. Ensure header name is not empty

[@clownbaby55-w4g]

Hi bro, I have created an Azure trial account and created resource gorup, storage account and data factory. At first place i am able to create and display them i even upload a file to the storage account. but when i land to home page again none of them are visible. ı have refreshed several times and logged out&in again but still i cant display them. have you ever had the same problem before ?

[@dataengineerpro]

You need to switch profile from top right option

[@akashvyas8351]

Is it possible to become data engineer as a fresher. No prior experience.

[@dataengineerpro]

Yes there are recruitment for freshers as well but are very less.

[@akashvyas8351]

@@dataengineerpro can you please make a roadmap for fresher to crack those job roles.

[@dipayanbhowal7025]

hey bro my schema is showing some error regarding excel header with empty value? what can be done?

[@dataengineerpro]

Go to dataset and tick the option called Row contains header. Let me know if it works for you

[@dipayanbhowal7025]

@@dataengineerpro i did that part the. My pipeline trigger shows error i followed each step acc to your video then also.

[@dataengineerpro]

@@dipayanbhowal7025 ping me the error screenshots on mail id [e-mail]. Will send solution for the fix :)

[@dipayanbhowal7025]

@@dataengineerpro sure will do

[@dipayanbhowal7025]

hey man i did it finally , saved an arm template BTW Is there any videos with python scripting , using git operations?

[@lakshmiprasannalingaladinne]

pipeline was success but final file was not created

[@dataengineerpro]

Are you talking about the parquet file, can you mail me the run status and details on [e-mail] will reply with the resolution

[@AdityaIngle]

Bro can you provide me with any university id as I also need access to free student azure subscription.

[@dataengineerpro]

You can use random id generator online

[@KarthikBhandary]

Really Great video!! It was very helpful!

[@hadassahe3854]

You are a star. Thank you so much!

[@prashanthitirumala554]

Hii Bro just first time i see u r video regarding explanation of this project u explainng is nice,,i have one doubt in this " how to remove the duplicates if any in target in file" and " how to capture the error file while u upload the file in blob storage"

[@dataengineerpro]

Need few details. Answering based on assumption that you are copying data from some source to parquet files using copy activity. In copy activity you will see option to log file as well as to remove duplicate in sink option. Let me know if you have some other question.

[@prashanthitirumala554]

I have 1400 rows of Excel file ,,aa file lo address column okati vundhi dhanni separate chesi ,,malli a separate chesi na file ni blob loki ingest cheyali ....e process lo nenu trigger s use cheyali and remove duplicate s cheyali ,,,,if any error rows vachinappudu error msg anedhi mail ki notification raavali..,.......e senario bro ,,nenu a actives use cheyalo cheppandi

[@sheepay99]

Didn't you listen to the audio before you posted this online?

[@dataengineerpro]

Sorry for audio in this video, it's fixed now in latest videos:)