How Databricks Leverages Auto Loader to Ingest Millions of Files an Hour

[@CHERANGA123]

Hi, Thanks for this! Can you please provide the link for the notebook please? Also if you could kindly provide what are the security permissions to setup on the service principal that would be great!

[@jennylam1935712995]

Hi appreciate this video. I see that after using the addNewColumns and the stream failed with UnknownFieldException, you manually restarted the stream with an updated schema. So how can I create a loop to continue to restart the stream until the no more UnknownFieldException pop up?

[@prar_shah]

Can you please zoom in,ca nt see the notebook properly, great video

[@aks541]

Nice presentation - if giving the link of the notebook is not feasible, pls share the code with comments. Thanks in advance!

[@ndbweurt34485]

Hi, have a question, i didn't understand how schema evolution is an issue in structured streaming...as we can set mergeschema to true.

[@shanhuahuang3063]

Can you please provide the link for the notebook please? Also if you could kindly provide what are the security permissions to setup on the service principal that would be great!

[@shanhuahuang3063]

what happen on change of column length? does that will cut off the column? thanks

[@shubhamaggarwal3676]

hi can you please provide the notebook link as video is not that clear to understand the code easily.

[@nithints-sp1uc]

without working notebook invisible text this is not useful even if content is good