Advancing Spark - Power BI Databricks Connector with Native SQL

[@hubert_dudek]

The native query is an excellent solution in big projects/companies where you can not easily create another view or table in the databricks catalog (especially when you have read-only access :-) ).

[@evgeniyivanov4253]

Thank you for the good content! Very relevant for real world scenarios, were End2End workflow includes BI tools, like PowerBI.

[@trodenn4977]

Would it be possible to do a new guide on DLTs? this time explaining the details with regard to delta live table readStream or read

[@rajkumarv5791]

Two questions as I watch this, 1. Even with an assumption that Query folding is not possible, is that big of a deal if you expect the users to put in the filter conditions/grouping/aggregations in the Query itself? 2. For Power BI reports with low latency requirements, (.i.e. users cant wait for longer to get the results after a click) isn't doing calcs in Power BI a better option?

[@manuelgpri]

Great video! I had this intuition, and for me the main reason I didn't like to put a query in there is that it will always be more hidden and easier to forget there might be a data transformation in there. Anyway for me the big question is: why don't they just add to the connector the capacity to connect to an SQL query in databricks? It is easier to have access to create a query than a view. The connector to SAP BW for example connects to BW queries. Why can't we have the equivalent in Databricks? Is there any problematic downside I'm not seeing?

[@henrikLundberg-xy6pn]

One related question... (maybe it is a stretch :) )... We are having the medallion architecture, what is the best option of authoring tabular models to be deployed as Power BI datasets? I want to use tabular editor, but not connecting to Databricks views in gold layer use the Simba ODBC. There is a way to gather metadata in Power BI Desktop, and paste it into Tabular editor. But it is cumbersome development expericence. One option is to use Power BI Desktop to author the model, deploy it, and through Tabular Editor get the xmla. Then you can add that to repo etc. But still.. adding tables you need that cumbersome workaround producing table metadata in Power BI Desktop. Do you have any suggestions? Or is it up to Tabular editor to add this functionality?

[@mainuser98]

How to import a large dataset , i.e 800 million records into powerbi? Can you make a video about this how to work with this?

[@gardnmi]

I would think spark optimization engine would optimize you native query even without folding enabled.

[@GreyskullAnalytics]

The thing with query folding is that if it's broken, the predicates aren't passed through to Databricks, so it doesn't know what to optimise. The fact that this folds queries by default at least gives the spark engine a chance

[@UstOldfield]

I went into this video with my mind already made up - Native queries are bad. Nice to have my mind changed. Not sure I'll use the functionality, but at least I know that it exists and it's not useless.

[@LittleJimmy2001uk]

not what I expected either, another tool in the belt I guess