Advancing Spark - Delta Sharing

[@evogelpohl]

Nice work as always, Sir. It's clear that the bones of the sharing-ecosystem, Delta based, are here. Excited to see UI/UX's over top ala new layered products.

[@kuldipjoshi1406]

Hi, If you can make a detailed video about table access control, hierarchy of how it works in databricks and best practices , That would be great. Awsm video btw.

[@chittillavenkataviswanath1389]

You are truly amazing! Best learning experiences to start the new year.

[@aqlanable]

Since we are talking about delta sharing, its worth to have a look at alert destinations and alerts in sql persona

[@danhorus]

23:05 I have the exact same question. If ADLS is in a VNET with no public internet access, I don't suppose Delta Sharing would work because the recipient must be able to query the data directly from ADLS, right? This can be quite a deal breaker for building secure meshes

[@ArcaLuiNeo]

I assume for such scenarios one has to start looking at a self hosted delta sharing server...

[@gabrielcohensabban4968]

Could you please include a link to the notebook used in this video. Thanks amazing video!!

[@aqlanable]

Delta sharing still not mature to be in enterprise level, however im waiting for post-ga regards to delta sharing and data marketplace provided from databricks

[@seyma4479]

that would be great if you make a video how to build delta sharing server on our localhosts serving the data from S3 🙂🙂

[@AprenderDados]

And who processes the data? PowerBI is reading delta? Do I need to provide cluster or any computing resource?

[@AdvancingAnalytics]

Delta Sharing essentially just returns a payload of keys to access the underlying cloud files - so your client still does the reading/processing etc! The server part of Delta Sharing doesn't currently require any kind of cluster/compute etc

[@nayan001ujjain]

Hi, Thanks for sharing the knowledge about delta sharing. Can you please explain how costing work in delta sharing and how many hits user can do . Is there any limit? Databricks charging on the basis of IOPS ?

[@AdvancingAnalytics]

Good question - at the moment I've not seen any costs associated! There will be the underlying cost of storage access, data egress etc, but I've not seen a cost model from Databricks yet!

[@nayan001ujjain]

@@AdvancingAnalytics Thank you 😊

[@drummerboi4eva]

Amazing ! Thanks for making these detailed videos Simon ! Do you know if dynamic data masking for GDPR is possible with delta sharing ?

[@aqlanable]

Its possible with unity catalog, u can mask row level, colum level, data level and in powerbi, it will be masked

[@aqlanable]

Unfortunately, you will have to create views and delta sharing doesn't support dynamics view at current time that we are talking, so mostly you need to go with unity catalog then create dynamic view and provide sql endpoint to the powerbi

[@ddarkings]

Is there an advantage to setting up delta share for PBI as opposed to linking PBI direct to SQL Endpoint in Databricks as shown in the Partner connect demos. I guess its a way of limiting which tables can be seen in PBI. Are there other benefits as there is more to set up doing the delta share way

[@AdvancingAnalytics]

Couple of reasons: 1) Delta share doesn't use Databricks compute (aka, it's cheaper) albeit with some limitations, 2) It's primarily focused on users outside of your AD Tenant, who would not be able to connect to your DBX endpoint

[@rostislawkrassow7385]

Thanks for sharing the review. Could a view also be part of a share?

[@danhorus]

The documentation on GitHub mentions support for views. I hope Simon can test it and let us know if there are limitations for views with joins, etc. I would also be a little worried about the security aspect of these views, because perhaps the recipient is able to retrieve the underlying SAS Key and access the unmodified table(s) in ADLS instead of a filtered view with row-level security

[@rostislawkrassow7385]

That's exactly the point. A view with row-level security or join inside requires creation of new physical files to share them on file level with SAS tokens. Only in case of materialized views (new announced feature) this would work on already persisted set of files.

[@aqlanable]

Dynamic view/views still in the post-ga, currently only table supported.

[@rostislawkrassow7385]

@@aqlanable thank you for sharing the insight! Curious to see how that will work

[@rickrofe4382]

Thanks for the preview. Do you know if the same integration with Power BI still work in AWS?

[@AdvancingAnalytics]

Yep! From the recipient's point of view, the Delta Sharing Server could be in Azure Databricks, AWS, a local web server, anywhere! That's the beauty of it being an open protocol!

[@rickrofe4382]

@@AdvancingAnalytics Super cool!

[@vinodhkumarganesan6778]

@@AdvancingAnalytics Hi, Did you see or experience a performance improvement with Power BI running with delta share rather than on SQL warehouse

[@akhilannan]

Can you add a view to the share? Or it has to be table?

[@aqlanable]

Currently only tablr supported, they are working on view in post-ga, u have to wait couple of months