#63. Azure Data Factory - Log error information to csv file using data flow parameter

Error logging is a basic requirement in any solution. This video explains the steps required to log the captured error details into a CSV file using azure data factory data flow activity.

[@ShriramVasudevan]

This is one thing I searched and it is presented so nicely

[@AllAboutBI]

Glad to have helped you

[@swaminathankv4486]

Thanks Madam. I was literally searching for this functionality for a while now.

[@AllAboutBI]

Thank you.

[@bitips]

What is the Sink configuration in the Sink tab on the Dataflow ????

[@manohard5158]

madam, is this the only way that how many tables that are manually entering into a text file?? and using lookup, foreach and copy activities are fine. But, if I want read/load hundreds of tables then what should we use? instead of text file. please clarify to avoid the ambiguity on text file. Thanks, Manohar

[@oriono9077]

Thanks for knowledge sharing !!

[@AllAboutBI]

You are welcome Orion 👍 👍

[@sunilpatil4393]

Very Nice

[@manojkumarchallagund]

Hi, I couldn't clearly see the characters you have replaced in the expression. could you please share those characters to be replaced please? I am getting an error as --> " Internal Server Error: An error occurred while executing the data flow activity. Please try again later." Any idea about the error? is it due to missing replacement characters in my expression?

[@AllAboutBI]

I'll try .. as i don't have any of those pipelines now

[@manojkumarchallagund]

@@AllAboutBI I have created multiple pipelines and data flow activities as part my practice. How can export or save those somewhere outside ADF so that although I lose my Azure account I can import when ever i wanted to refer back ? Could you advise pls.

[@AllAboutBI]

@@manojkumarchallagund use git repo

[@MrSamujosh]

Hi Madam, Could you please elaborate more on how you would create an activity to create one error log file on a daily basis? We should be able to create new file daily and keep updating it till EOD. Regards, Sameer

[@AllAboutBI]

We can't update same file. Appending is not allowed

[@justaboutdata6537]

Thanks for sharing this. I also tried to write to a csv file. Every thing works fine but CSV file has only the last item written. It overwrites the previously written lines in it. Any ideas?

[@AllAboutBI]

With files, It doesn't support appending yet.

[@technologyhunter2460]

Not sure why this is so complex to simply read a variable or an error message and write to a file. Hope microsoft fixes it soon.

[@AllAboutBI]

True. We can expect some updates soon

[@hasinib628]

Cool

[@bitips]

She gets too lost when explaining things that are already done, she should do along with the videos, because none of that is working.

[@RajeevKumar-zf8ox]

I followed the step but unable to data in synapse dw. Could you please share your mail id. So I can share the csv file.

[@AllAboutBI]

[e-mail]

[@RajeevKumar-zf8ox]

Thanks for sharing. I sent the mail with CSV. Kindly let me know what is the issues

[@RajeevKumar-zf8ox]

@@AllAboutBI shared the table scripts in your mail id

[@AllAboutBI]

Thanks looking into it