ETL | AWS Glue | AWS S3 | ETL Job | Detect and remediate personal identifiable information PII

[@rahulpanda9256]

This is great demonstration. Thanks a lot. I have some of the below follow up queries wrt this. 1) I see in most of the cases you are using Visual ETL. In the actual Production scenarios, is Visual ETL sufficient all the times for complex logics? What do you prefer based on your experience? 2) In most of your demos (the ones that I have see so far), you are dealing with structured data in csv format as a source. Have you covered any demo video with semi structured data as source (such as json)? If yes, can you help me with that video link? If no, would be great if you can come up with one of such demo. Thanks a lot!

[@cloudquicklabs]

Thank you for watching my videos. Please find my response sequentially. 1. In production I do suggest to keep the data pipeline definition with Infrastructure as Code ( IaC) with either Terraform or CFN. I shall make video on this soon. 2. Indeed this very good suggestion to make JSON as source data. I shall create new videos on it. Thank you very much for your suggestions , I shall work on it.

[@basavarajpn4801]

Can you please make an scd type 2 implementation using glue

[@cloudquicklabs]

Thank you for watching my videos. And thank you for providing valuable input here. I shall make video on this soon.

[@basavarajpn4801]

@@cloudquicklabs Thanks 😍for the reply,Waiting for the video soon,and Please make end 2end Project as automated.

[@cloudquicklabs]

Sure..

[@basavarajpn4801]

@@cloudquicklabs thank you 😊