97. Databricks | Pyspark | Data Security: Enforcing Column Level Encryption

[@tanushreenagar3116]

GREAT EXPLANATION SIR

[@rajasdataengineering7585]

Thanks, keep watching!

[@sravankumar1767]

Nice explanation 👌 👍 👏

[@rajasdataengineering7585]

Thank you 🙂

[@SravanthiP-v3o]

Sir After creating the encrypted and decrypted column.. how do we hide the original ssn table? and when we run the command select ssn from dimeployee how do we ensure that we get the encrypted format ?

[@sanjayr3597]

Good video.I have a question regarding KEY, how is that value stored ..? can we use the same function another notebook with the same cluster? any draws using this method?

[@rmrz2225]

Good job, but,I have a question, when we encrypt information aren't we not supposed to be able to decrypt it?

[@rajasdataengineering7585]

Thanks. Yes we need to decrypt it when we need to use it later

[@NagarjunaSunguluru]

Can we apply those functions on nested json data also??

[@rajasdataengineering7585]

Yes we can do. At dataframe level, we can apply encryption and write in json format which will keep encrypted data within json file

[@lavanijavidalikhan3844]

Can we apply same thing to encrypt a csv file

[@rajasdataengineering7585]

Yes we can do

[@ramreddy1138]

Good one. But, how do we filter the data and apply comparisons?

[@rajasdataengineering7585]

In order to filter and compare, we need to decrypt the data on the fly using decrypt method

[@ramreddy1138]

​@@rajasdataengineering7585 It will impact performance too much..

[@revjr1284]

I am getting the below error while encrypting the data. 'TypeError: encoding without a string argument' Kindly help

[@phanisrikrishna]

Hi Raja, This particular video is great. I have one question. will the size of df increase by creating encryption on some of the columns? How do we take care of memory while designing? Thanks in advance.

[@sravankumar1767]

What is md5, could u please explain this one

[@rajasdataengineering7585]

Md5 is one of the hashing function

[@nagamanickam6604]

Thank you

[@rajasdataengineering7585]

You're welcome

[@sabesanj5509]

Raja sir, Will they these kind of questions in Spark interviews??

[@rajasdataengineering7585]

Yes, data security is one of must topic in interviews

[@azureadi-q3y]

good Explain, could you please explain delta lake live table (today i loaded 1 table with 20 columns and same table next day i am getting with 2 more extra columns how to handle in delta loads in ADB and how to manage delta merge command (In production))

[@rajasdataengineering7585]

Sure, will create a video series for delta live table

[@gattureddy3796]

Hi Raja, could you please share a notebook link or DBC file