Advancing Spark - Managing Files with Unity Catalog Volumes

[@vincentdelbaen8815]

Thank you sir! I'll try it out right away and probably include it to our ways of working. I feel it can reduce the burden and avoid creating external locations for each data analysts projects.

[@datawithabe]

Great video,! as always, best place to learn new Databricks features :)

[@MariusS-h2p]

Does this also replace DBFS access in general?

[@atulbansal8041]

How can I get the access of data ricks environment for learning. I know there is a community edition available but somehow I am not able to load my raw files into that

[@petersandovalmoreno5213]

we may write on this volumens?

[@ErikParmann]

So with mounts we can have the dev workspace mount the dev containers, and the prod environment mount the prod containers, and they both get mounted to the same path. So the notebook don't have to 'know' if its running in dev or prod. How will that work in this new world? I noticed that the path contains "dev". Does each notebook have to figure out what environment it is in, and then read/write from the right paths and catalogs based on some string manipulation?

[@neelred10]

Exactly my thought. Maybe environment variable can store dev/qa/prod value and use it to dynamically generate path string.

[@AshleyBetts-h7t]

Love your work Simon. Do you know if it is possible to have a credential that is not associated with same cloud provider as the Unity Catalogue instance? I have Databricks environment deployed on Azure but one of the ingestions is via an S3 bucket. I would love to be able to set this up as an external volume.

[@nachetdelcopet]

I think you will need to create a Access Conector in your AWS, then go to your Databricks workspace and create the storage credentials using the AWS Access Conector ID. Then you can replicate everything he has explained in the video for AWS

[@coleb1567]

Great video. One unrelated question: how do you guys manage deployments with databricks? I come from an airflow +Jenkins background as an engineer. Would you recommend Jenkins for databricks deployments?

[@mc1912]

I remember Simon mentioning they use Terraform for infrastructure deployment, but maybe he can tell us more 😅