Advancing Spark - Azure Databricks News Feb 2023

[@rengarlee1000]

Nice!

[@vishnurajendren691]

Big fan of your content❤️

[@stevequan7306]

Hi Simon, could you please make a video about Databricks’ Dolly ?

[@aaronpanpj]

Love your content Simon.wondering if the reason why you can’t use Alias in where is because the sequence of how a select statement is executed: from -> where-> select. Have you seen it in action somewhere else?

[@AdvancingAnalytics]

But that's now how spark interprets it - the execution plan does the project, then filters (unless it pushes via partition filter etc). But really, it's spark, so order /shouldn't/ matter? It's a good step forward anyway, I just want THE WORLD 😅

[@allthingsdata]

It should be mentioned that file arrival triggers require that the workspace has Unity Catalog enabled. Again an example of a feature where it's not clear to me why UC is required. Increasingly seems that DBX is pushing client to UC by making such requirements.

[@RodrigoBocanegraCruz]

Ey Simon! Thanks for the video as usual. The discount code for the training is not working :( I understand I can find there some best practices and patterns and not only a standard technical course, is that right?

[@AdvancingAnalytics]

Whooooooops, I had the discount expiring in Feb. I'll fix that now! The course has some best practices, and the first sessions spend time putting Spark in context, talking about architecture evolution etc. It doesn't go into deeper engineering patterns & practices, that's a separate course I've not yet recorded yet! It's mainly me explaining these different concepts as we go through the fundamentals, putting those building blocks in place!

[@RodrigoBocanegraCruz]

@@AdvancingAnalytics Thanks! I'll keep an eye on the engineering module as well. We certainly need courses with real use cases using well architected approaches and not disconnected samples of technical skills that are not even following engineering practices (yeah, I've seen your video about analytics engineers too). Thanks for all these efforts!