Advancing Spark - Dynamic Data Decryption

The ability to determine whether a user sees encrypted or decrypted values of a record has been incredibly powerful in meeting data protection requirements. There are established patterns from the warehousing world that have been very tricky to adopt in the Lakehouse approach... until now.
In this video, Simon walks through the new aes_encrypt() and decrypt() functions in Databricks Runtime 10.3, before showing how they can be combined with row level security to dynamically display decrypted data to only those with the relevant access.
As always, if you need help establishing a next generation Data Lakehouse architecture, get in touch with Advancing Analytics

[@VishwajeetPol]

Thank you Simon for sharing the video, Honestly, I refer your channel to know what was launched in recent release of DBR rather than going through the databricks article. It was definetly informative to understand RLS with encyption and decryption on the fly

[@drummerboi4eva]

Super vidéo Simon . Row level security is simple but effective and encryption is really a need of the hour with GDPR use cases

[@briancuster7355]

I agree with the other comments in that this is monumental!

[@MajdiSAADANI]

Hello Simon, thank you for your video, could you show please how did you generate your encryption key, I tried get_random_bytes(32) and when I pass the key to the spark native function aes_encrypt I receive this aes_encrypt`/`aes_decrypt` is invalid: expects a binary value. Thanks

[@Vishal-q8e]

How to use aes_encrypt for boolean or integer data types and get encrypted values in same format?

[@rajdeepsinghborana2409]

Informative ❤️

[@dmitryanoshin8004]

Great video! Can you please make a video about local IDE development for Databricks. It works fine with a single notebook using databricks-connect. But it is not working with calling other notebooks in Repo. Maybe we should use a wheel package for this. Thank you!

[@hubert_dudek]

I think it is better to use databricks or azure key vault for keys (better than lookup table) and than rest of logic like in Simon video

[@AdvancingAnalytics]

Oh if you have a single decryption key, certainly. If you need a different key per record, that would get awkward to code without preemptively looking up keys, or writing a very slow UDF!

[@hubert_dudek]

@@AdvancingAnalytics it could be useful that dbutils.credentials is supported through SQL functions (other dbutils as well) so than implementation would be less ugly :-)

[@becavas]

@@AdvancingAnalytics nice, but how to read secret scope (encryption key) from sql?

[@becavas]

Nice. the cypher algorithm is AES256?

[@guilleromero3762]

Super interesting! but shouldn't it be better to store the decryption keys on Azure Key Vault in order to gather all sensitive data in it? Thanks in advance, I'm a big fan of your videos!

[@ivantang5795]

Thinking out loud here. AKV have rate limiting and likely will throttle retrieval requests if we were to store those encryption keys in AKV for a fairly large table.

[@singhrakeshr]

Unless a unique encryption key is used per customer this solution of encrypt/decrypt cant be used for gdpr. Why would you throw away key for a group if only one customer needs to be removed. Also the storage layer will still have customer info even if access via delta table is restricted/obfuscated.. so data needs to be deleted from everywhere..

[@AdvancingAnalytics]

Well... Yes, in this simple example I used group for the decryption, if you were using this for customer info your lookup would be on CustomerID or something similar? The data in storage is encrypted, so the storage layer is safe, not the Delta layer? Give it a try :)

[@zycbrasil2618]

@@AdvancingAnalytics yes encrypted at rest and in transit.

[@paulheadey265]

I was thinking the same thing - seems a bit of an anti-pattern but perhaps you can automate generating each unique key with Pycrypto

[@aragornguan7692]

Great video Simon, thanks! Can I have your notebook somewhere please?

[@norbertczulewicz1695]

You can easily restore the key from decrypto table history.

[@AdvancingAnalytics]

Yep, of course. If you're using Delta for security-style workloads, you need to have a vacuum policy that reflects that - with the right process & controls it's easily manageable Simon