Schema Evolution in AWS Glue using Glue Crawler | AWS Athena

[@AkshayaReddyKoduri]

but in real time scenario, if we are getting new columns from source, what will be populated for the existing data for those new columns? Will it be populated with nulls?

[@manojt7012]

Wish you happy new year and really impressed with your dedication 👏

[@KnowledgeAmplifier1]

Thank You Manoj for the inspirational comment ! Wish you a very Happy New Year :-)

[@niranjanjamkhande3773]

Thanks for the video. The problem I am facing is that, my crawler is creating versions when files are added in source folder having same schema as of earlier. Crawler should create version only when there is change in schema. But in my case, versions are getting created whenever crawler runs. I am badly stuck at this problem, please someone help. Thank you.

[@sriadityab4794]

Hi, Thanks for this channel. I have a use case where I need to read the files in S3 incrementally like whenever the files come to S3 and glue job should read and process where I should only process the data of the file dropped on a given day where I have date partitions. How should I write my glue job to process only the new data and write to S3? Can you please help me with this?