1. Handle Error Rows in Data Factory Mapping Data Flows

[@namangupta7111]

Thanks Wafastudies , You are making ADF easy for us by sharing videos

[@vinod02791]

Hi, These Real Time scenarios helped me a lot and added advantage to my experience in Azure and helped to get a job along with other technologies. Thank you for your videos!! :) :)

[@WafaStudies]

Welcome 🤗

[@mohitupadhayay1439]

Hi Wafa, I have some Interview Questions I was asked. Please make some videos on the same as a request. 1. How to create a GENERIC pipeline that can be reused again? 2. Write a Single line query to delete data from 10 tables. 3. What is Encoding in COPY activity? 4. What is the limitation of LOOKUP activity? 5. How do you validate your SINK data in ADF whether it is right or wrong?

[@kumarpolisetty3048]

Thanks wafa studies. Please start tutorial on Azure Databricks. Ur vedios are really helpful.

[@jaymakam9673]

This is what exactly is needed to understand ADF better. Thank you so much for the hardwork _/\_..

[@prekshajain5174]

Amazing explanation wafastudies cleared so many things, great work.

[@adityachary2478]

Fantastic bro ur explanation is awesome I learned lot from ur channel

[@vivek05117gece]

Hi Wafa, very well explained video. I have watched multiple videos on ADF, but after watching your videos I feel confident for the interview.

[@akashputti]

thanks, its definitely going to be helpful. better than udemy and courera courses

[@WafaStudies]

Thank you 😊

[@dvsubbaraonerella1449]

Awesome. Thanks alot for all the efforts and your time.

[@WafaStudies]

Welcome 😊

[@mohammedshoaib1769]

This is just awesome explanation. Covering real time scenarios perfectly giving a good insight about ADF. Thanks you guys keep up the good work. Also expecting more about Databricks.

[@AmitSharma-mv5xe]

Really appreciate your contribution brother

[@soumyaprasanna6606]

Thanks for all the efforts that you have put to create the ADF videos. Can you please share the sample CSV files that you have used to explain here?

[@ootec1]

Really I loved it. Thanks for the video buddy.

[@WafaStudies]

Thank you 😊

[@Anonymous-cj4gy]

Hi, If you want to check if there is an error in another column as well then use the below condition : ErrorRows = isNull(toDate(salesDate, 'dd-MMM-yyyy')) || lesser(toInteger(quantity), 0) || isNull(salesItem) || isNull(country) Here if any one of the conditions is true then that row will go to the bad table. Also, remember if you use dd-MMM-YYYY instead of dd-MMM-yyyy it will give you the wrong output.

[@mrrathore55]

perfect!

[@NehaJain-ry9sr]

Awesome video... many lessons in one video , including common errors... Thanks Wafa.. keep it up.. great work

[@WafaStudies]

Thank you 🙂

[@akkuhome9760]

Keep up the good work mate.., good cover of topics with hands on experience.

[@pawanreddie2162]

If we dont know which column has error rows then how to give split condition

[@sriharig9096]

Really helpful...thank you for your efforts

[@saivaraprasadarao6990]

Excellent explanation Thank you so much 😊

[@WafaStudies]

Welcome 🤗

[@ArunKumar-kb7fr]

Thanks for uploading...explanation is very good..

[@pavankumarvarmadendukuri4665]

Great work brother... keep it up...

[@bhaskarnukala902]

I got one question . What if the bad row "01-------2020" doesn't show up in the data preview. It might be a decimal . Then how come 'varchar'datatype will be useful on the DB where we store bad data?

[@thiagocustodio8177]

how to validate if sales item is not null and not empty string? I could not find a way to write an if statement. Adding one condition to each validation will make the flow pretty complex when working with large datasets. Also, is it possible to redirect to the same sink?

[@jesseantony1223]

Hello, Thank you for the videos. Is there any way to get the csv files mensioned in the video so that we can practice it?

[@tandaibhanukiran4828]

Thankyou Brother. LotsOfLove

[@WafaStudies]

Welcome 😊

[@annekrishnavinod5482]

can we use fault tolerance option for skip the bad rows?

[@karthikeyana2120]

hi any idea, how to take latest file from the blob storage

[@vaibhavkumar38]

Tip: In error table have same column names as normal table but the data type is kept as nvarchar- because only that datatype can store any data inside it

[@rahulkr5694]

Nice explanation. I have one question, I have one csv file in that i have one column with 50 length but some how some records came in CSV file with more than 50 length. How to reject these records from ADF.

[@arun06530]

very nice & informative Video.

[@WafaStudies]

Thank you 🙂

[@sraoarjun]

Excellent video !!

[@ramum4684]

If in multiple col in same row have error then how to built expression in the visual expression builder page?

[@ranjansrivastava9256]

Appreciated !!! If you share this concept using copy activity it will be good.

[@jayalakshmia6647]

Is it possible to run multiple pipelines in parallel? if yes which activity please tell me

[@shubhammanjalkar8591]

Amazing. Thanks you so much

[@ManojSairamChandP]

Hi Maheer, I am new learner to ADF, while doing this scenario, i came to know that self hosted IR cannot be used in Data flows, can you clarify the same. i am using sql server as an on prem application Connection failed Linked service with Self-hosted Integration runtime is not supported in data flow. Also can you clarify that do we need to configure Azure DB in ADF as well? So as to get DB tables details

[@meelookaru2888]

Thank you so much for your time. its really helpful. could you please add some more real time scenarios on ADF.

[@SantoshSingh-ki8bx]

Thanks Wafastudies. I have just one request, If you can share data used in tutorial. It will very helpful to follow on video.

[@ashishsinha5306]

Wafastudies could you provide us with the files it would be really helpful. Great videos btw.

[@ShaziyaKhan-xx4uj]

is real time playlist for testing.

[@gokulajith762]

Can we use the same method if we are using a self hosted integration runtime

[@vishnum4892]

i got error saying linked service with self-hosted ir isno tsupported in dataflows. by using self-host ir its not possible to use data debugging.is there any other way.

[@Adub333]

Hello...i am running into an issue where my blob storage input is returning null values where i have not transformed that particular column...Any recommendations?

[@arijitmitra8585]

Great content. Kudos to you

[@srinubathina7191]

Thank You

[@WafaStudies]

Welcome

[@ssbeats677]

Hi, can you please make a video the wat are the errors we can face and types of errors and how to resolve them

[@sivareddy157]

What about to check isnull function on multiple columns? Bro

[@karthikram1625]

can you explain , how did you connect those sql quiries to linked service?

[@sonamkori8169]

Thank You Sir

[@WafaStudies]

Welcome 😊

[@vidyatechtalks9175]

where did you created the tables??

[@ambatikarthik6822]

Hi maheer, can you help me out how can we connect SQL Server to ADF while we connecting we facing error

[@venkatpr9692]

Tq soooo much sir

[@sravankumar1767]

nice explanation bro..

[@WafaStudies]

Thank you 🙂

[@parthasaradireddy6793]

This is useful for etl tester or not

[@shalakapowar0707]

Hi, Thanks a lot for great video, but I have a question, how are you able to access this db from SQL server management as dataflow will not allow this

[@vijaybodkhe8379]

Thanks for Sharing

[@diwakarnrp2092]

how to download the files for practice purpose

[@prasadrasal5001]

thank you so much

[@WafaStudies]

Welcome 😀

[@UmerPKgrw]

i have simple dataflow copy data from on prem Sql server to blob storage and it gives the following error: Linked service with Self-hosted Integration runtime is not supported in data flow. Please utilize the Azure IR with managed vnet using this tutorial

[@balakrishnaganasala5811]

Thank you Wafa for the excellent videos', could you lease share the scripts you used to create the DB tables and files you have used over here.

[@shubhamsonune1490]

Very helpfull

[@WafaStudies]

Thank you 🙂

[@sudheerkumar7100]

sir pl tel about BANKNIFTY

[@annekrishnavinod5482]

One Question @wafastudies , here your showing one column. for example, i don't know which column is data not properly coming or null value. how to do?

[@mahihi251]

add one expression to each column to validate

[@indhumathid1095]

Really Thank You so much for your videos. I am trying to fetch only required records from excel file and load it into Azure SQL Database using ADF can you give me any idea?

[@rajashekerreddydommata537]

We can add an expression to the variable and use that lists only those which we want, orelse list out all the required records in an array and using for each activity you can iterate and fetch only the required from the excel sheet. Hope this will work.

[@amanahmed6057]

could you upload the files which you are using in every video or make a google drive

[@hariomrajpoot966]

What if the number of columns number are different

[@sharadgawade9408]

Thanks for video on real time scenario. If have multiple files in my source location and one of them contain bad record so how to identify the file which holding bad records? Here you are passing file name as hardcoded.

[@WafaStudies]

Great question. Lets do one thing. I will create second video on this scenario. How to get filenames from folder and loop through each file. Pls stay connected

[@sharadgawade9408]

@@WafaStudies thank you so much..

[@dhanyamenon1485]

Thanks for creating videos on real-time scenarios. It's really helpful. I have a question here - how to validate files in Azure Storage container based on validation schema for that data load already in SQL database used for the ETL validation scenario by using Copydata activity and capture the error rows and send an email notification to the user with the error rows data. I want to use Copy data activity here. have some limitations to use Mapping activity, so not able to use that. Please let me know. Thanks in advance.

[@gokulajith762]

Hello, do you know how to carry out the same functionalities using copydata activity

[@UmerPKgrw]

How much it will cost to run an Azure data factory

[@paramkusamsaikiran1018]

Maheer, If possible can you provide us the files you use in your lecture. It will be helpfull for us for practicing.

[@subbaraochereddy7089]

this is work for only one column , how u handling truncation errors, Special characters files issue , pls upload the file

[@anujgupta-lc1md]

Make More and More

[@harishpentela1234]

Hi Wafa , I have a doubt about loading tables from staging to Data warehouse by using dataflow ,if we have 25 dim tables Do I need to create 25 workflows ?Or is there any Dynamic way to do that Thanks in Advance

[@DataWithNagar]

Create a single workflow template that can be parameterized. Parameters can include source table names, destination table names, transformation logic, etc.

[@pranaypandu9268]

Luv u bro

[@WafaStudies]

Thank you 😊

[@Anas_ILoveMyIndia]

Hi Maheer, We have a scenario like to fetch the zip file from FTP server and which is password protected. How to do the extraction and get the data into Blob storage or into SQL DB? Pls make a video. Thanks a lot !

[@vigneshmurali1994]

Pls do on spark

[@dasarianandchandra6890]

bro ur cool

[@WafaStudies]

Thank you 🙂

[@aravindreddyramidi5543]

Please provide csv file. It would be helpful.

[@nallavellivenkatesh9479]

Tq

[@WafaStudies]

Welcome☺️

[@manasmohanty5754]

Kindly provide some other real time scenarios..

[@WafaStudies]

Kindly check my channel Azure data factory real time scenarios playlist. There are 35+ real time scenarios covered

[@mrrathore55]

@@WafaStudies Many thanks :)

[@raficompitative4077]

Pdf material pls

[@valijoneshniyazov]

I wish csv files attached here.

[@ArrowAAA]

pls share the source files

[@gopalreddy9897]

Please stop saying ok

[@WafaStudies]

Ok😂😂😂

[@hariomrajpoot966]

This. Is very limited scenario

[@anusha8129]

Thankyou sir...