Advancing Spark - Introduction to Databricks SQL Analytics

[@SukhwinderSingh-er1ck]

Great intro, thanks. Can you please help me to understand, how can we run SQL Queries on SQL analytics remotely using Java or Python codes? I also have gone through the documentation but didn't get the solution in detail.

[@faridkhelf2459]

Hello Sir, can we add filter to the Databricks SQL Analytics Dashboard like power BI?

[@Praveenkumar-zn5wi]

Great video thanks... How I can connect cloudera hadoop cluster from azure databricks?

[@shamsuddinjunaid30]

They introduced spark in synapse......and now they introduce SQL in Databricks!!

[@AdvancingAnalytics]

Yep, everyone's heading to the same conclusion that you need a mix of both!

[@phy2sll]

@@AdvancingAnalytics 3, 2, 1... www.zdnet.com/article/snowflake-introduces-snowpark-a-new-developer-environment-for-data-programming/ 🤣

[@AdvancingAnalytics]

Hah! It's gonna be an interesting year or two!

[@zycbrasil2618]

but...Synapse (external tables) cannot read delta parquet files as Delta controls the logs..

[@prashanthxavierchinnappa9457]

Once again a great video!

[@VLM234]

Great introduction, loved the visualization flexibility and all. Is SQL analytics available for community edition??

[@AdvancingAnalytics]

Hey! Don't believe so - it's in limited public preview so I'd doubt we would see it yet regardless. The community workspace usually has full feature-parity, so I'd assume it will be in there eventually, but I've not seen anything to confirm that. If I get a chance, I'll ask the Databricks guys :) Simon

[@phy2sll]

Congrats on the award fella!

[@AmbiguousAbhi]

How can i start a SQL Analytics? Do i have to setup an Azure instance? Can you put a video on how to setup SQL Analytics? Thank you

[@AdvancingAnalytics]

Hey! Yeah, you need an AWS or Azure subscription before you can start. You then need to deploy a databricks environment, which can have SQL Analytics enabled within it. SQL Analytics itself is still a gated preview, you need to apply for access before you can try it out: databricks.com/p/event/sql-analytics-waitlist

[@sid0000009]

In Multi cluster load balancing, do we add in sets of driver + worker nodes when we scale up. ( and not just worker nodes only ). Is my interpretation correct. You have been quite quick to post this video :) as the Summit is still in progress !! thanks a ton Also when we execute a query do we have a provision to select a particular active end point to be used to give it a compute power ? I only saw the Execute button ...did I miss something ?

[@AdvancingAnalytics]

Yep, it's all hidden away, preview stuff currently, but yeah, the idea is that it's making automatically making whole separate clusters, with the load balancer directing queries to different clusters according to demand. Super cool!

[@PersonOfBook]

Is there a way to use partition by and zorder while creating a table?

[@AdvancingAnalytics]

Partition By can be part of the table create statement in SQL, python, scala etc, if you're partitioning, you HAVE to do that during table creation. Z-order is only ever in the optimize statement, so cannot be done during table creation. So yes and no :) Simon

[@haztha999]

Is this available only in premium version? I don't see SQL Analytics in my workspace

[@AdvancingAnalytics]

It's a gated preview currently, you need to apply to Databricks to enable it. The form is here: databricks.com/p/event/sql-analytics-waitlist

[@akhilannan]

Very good intro! Will be interesting to see how much a data engineer can leverage SQL Analytics. Technical, is it possible to run a create table as select statement in queries ? Or it always expects a select? If we can do that, next question would be if we can trigger those queries from ADF for example :)

[@reynoldxin4180]

You can run DDL/DML statements as well, but the focus is more for selects.

[@AdvancingAnalytics]

It's a good question whether the "queries" we create in SQLAnalytics will be visible to other tools via the endpoint, or if we'd need to build those out as hive views via the normal workspace... Essentially - how reusable can we make the SA queries?

[@louiehudson9466]

I know im randomly asking but does anyone know of a way to log back into an instagram account? I stupidly forgot my account password. I love any tips you can offer me

[@caidengrant1101]

@Louie Hudson Instablaster :)

[@louiehudson9466]

@Caiden Grant i really appreciate your reply. I found the site through google and Im waiting for the hacking stuff now. I see it takes a while so I will get back to you later when my account password hopefully is recovered.

[@Yi5Zhou]

Why do I need this? If I am using power bi

[@AdvancingAnalytics]

If you're using Power BI, you likely don't need the query/visuals/dashboards elements of Databricks SQL Analytics. However, you can make use of the SQL Endpoints as the source for your Power BI reports - with Photon & Multi-cluster load balancing coming soon, the Power BI experience should really start to improve. There are a few bits that make it slightly easier for end users to connect, if you don't want business users to be constructing data set connection urls etc!

[@zycbrasil2618]

It sounds like a battle between Synapse & Databricks... :)

[@leoafurlongiv]

Seems like this will have a similar price point to Synapse Serverless

[@zycbrasil2618]

@@leoafurlongiv yes but Synapse SQL On-Demand does not support delta tables. If you link a external table to the same set of parquet files it returns wrong results. Delta implements Delta_Logs and SQL On-Demand cannot understand it..

[@leoafurlongiv]

@@zycbrasil2618 Serverless will support Delta eventually, right

[@zycbrasil2618]

@@leoafurlongiv essential feature

[@zycbrasil2618]

@@leoafurlongiv it would be great as we can expose data as classic sql for reporting, etc

[@jamescunningham8092]

Great. So now our data scientists can go back to copy and pasting a SQL spaghetti of ad hoc rules and patterns because we can't programmatically customize how they access data (since SQL Analytics doesn't appear to be the slightest bit extensible). And because you're pricing it so aggressively, I'm sure at some point management will push us to use it for ad hoc work. I thought I was done with this when we kicked Oracle to the curb.

[@AdvancingAnalytics]

Well, anything that can be registered in hive is accessible to the SQL endpoint. So you can have an extensible framework creating the objects, securing them etc, it's just the reporting/analytical queries that you'd have here. But I agree the security model is a lot less visible & extensible through SQL analytics directly. To clarify, we don't work for Databricks, so for pricing strategy, you'll need to take that up with them ;) Simon