Data Engineering Expert Reacts to AWS Architecture at Veritiv

[@iitravindra]

At 4:25 he is explaining how lambda pulls metadata of file from dynamo db. They are using it as metastore for the files they are suppose to receive from s3. Most of pipeline do these validations to make sure the process validates the schema of files before getting processed. This act as a contract for data lambda supposed to recieve.

[@gpavanb1]

Great work! I was watching your videos along with the AWS series for a lot of my data engineering knowledge. What was missing in the AWS videos was the reason for their choices and you are very nicely filling the gap. I'm open to collaborating if you want to do this for more of the series also 😀

[@andreaskayy]

Great point. Usually they leave out some important information and why they do it this way. Going to make sure I address this in future videos

[@andersboje]

I don't hear they are adding the file content into DynomoDB as you mention. My guess is that this is only the metadata of the file they storei DynomoDB. I have done this my self and seen others do this for Data Governance. Store info like when was file received , filename, md5 etc In the data lake I would assume they concat the file into bigger chunks

[@andreaskayy]

Aah, you are right. That makes totally sense!

[@andynelson2340]

Love this type of content Andreas!

[@andreaskayy]

Thanks Andy!

[@SeattleDataGuy]

I second Andy! Keep this content up.

[@arockianirmal9814]

Really awesome! Looking forward for more similar reaction videos!

[@andreaskayy]

Thanks Arockia

[@ChristFan868]

This was excellent. I learned a lot. I hope you do more like this.

[@AbdurrahmanKocukcu]

This reaction/review concept is so valuable and pleasant. Please continue :)

[@andreaskayy]

Thanks a lot!

[@farPositive]

Instead of SQS, can we have SNS?

[@rohitkumarz]

Very insightful.. as an enthusiast.. i find videos like this very great from the point of learning... I can see a lot of students and new graduates finding the breakdowns of these architecture useful and helpful.. thankyou Sir

[@andreaskayy]

Awesome! These videos are also fun to record 😊

[@Larry21924]

This is unmatched. I recently enjoyed a similar book, and it was unmatched. "Mastering AWS: A Software Engineers Guide" by Nathan Vale

[@anthonyloganhall]

This is great.

[@shauryajain4851]

Is future of de bright since lots of tools are coming up

[@andreaskayy]

That is very true. New tools make engineering a lot easier

[@shauryajain4851]

@@andreaskayy will it threaten the de Or still it's good, plz make a video on this

[@justsomeguy1408]

I really appreciate the insight in these videos. I'm currently learning data engineering in AWS so I can try to assist the supply chain company that I work for with projects like these.