Deletion vectors are amazing. They improve concurrency as well which is detailed on the page about isolation and serialization. If you need to delete data about customers for compliance it’s great. Also if you need to replicate your data to another region you won’t be creating as many extra files that need to be transferred and stored so you can get good savings from that as well. Imagine if you have big gigabyte parquet files in a huge table and you need to delete a record here and there it will make a massive difference.

❤❤❤

How come the base layer shows what is in the source system when you have applied cleansing?

I was asked this medallion architecture in job interview and i didnt know except ive heard the name. The interviewer explained it enthusiastically and i just commented that its the same as any other process, he didnt seem to like my response. I didnt get picked for the role and also didnt want to as the interviewer seemed arrogant. Its not rocket science.

Only available for F64 or higher SKUs

Is this working in databrick SQL mode?

awesome demo! I dont see where the online tables were ever used, can you clarify this? thank you 😊

Brilliantly put together, thanks Simon!

clickstream_raw is mapped with BRONZE layer? clickstream_cleaned is mapped with SILVER LAYER? how could I map each delta table with the medallion layers?

oomph... throwing shade at the SQL editor 😵

Where would you use a variable instead of a widget?

Great video, will really come in handy! Can these features only be used on SQL pools or also on Spark clusters?

That syntax leaves you confused as to if the identifier is a column or a variable I think I prefer the syntax with the $ that snowflake has.

Nice name no servers here. Can’t wait to try the serverless for jobs.

Great job bro.

Great video. Where can we get the sample files?

Mazing video - subbed

What a great video - again! You're a legend. Well spoken and explained.

Finally a good, solid video that explains it well. Thanks! I would love to see a follow-up where you actually land some data in Bronze and transform it to Silver in development. How does the data look like in the containers? How does the tab "catalog" show the data in Databricks? How are they related? I want to know these things but can barely find anyone explaining it well. I basically want to build an enterprise lakehouse from scratch. Thanks!

This is a great step in the right direction for the browser UI! Happy days ahead

Before each recipe provide a short life story that is vaguely related to the topic 😂

seems speed is the focus on all the new models, e.g. the brand new fast and technically teading opensource llama 3 . I have to say it's quite fun to see leaders knocked off their perch every few weeks.

Hi, it is possible enable Change Data Feed on the mirrored delta table?

first

Hate notebooks and hate to do it on browser :D

I love that they would take such a powerful tool and open source it in the middle of a AI gold rush. Excited to use it and also to see what it prompts in the future 😃

Hmm, this means I want to learn more a lot. Thanks so much. Joining the Data world makes me learning every minute in my life 😫

I was looking forward to your video on this! We do have a use case where we want to perform some analysis on our data, it’ll be interesting if DBRX can do this for us

I’m curious how expensive it would be to have a custom dbrx model and what are the advantages versus a rag approach. One nice thing with rag is that it’s much easier to control access to proprietary information as if you don’t put it in the prompt the model doesn’t have it. With a custom model it might leak information to the wrong users so then you probably need a custom model for different permissions.

Your videos are great. I am attempting a selective overwrite on delta tables but can not get it to work with unity catalog but I can with hive metastore. Any advice or pointers?

Great and simple explanation! Thanks for sharing this!

Nice off to browser to try the debugger!

Just pronounce HIPAA as hip-ah, much easier. Thanks for the updates, these are so useful.

Great thoughts. Good to know I wasn't the only one thinking in this direction. Wondering if can we go one step further and only have a 2 tier architecture? 1.Staging/Bronze layer 2. combine Silve/Gold layer and directly building star schemas delta tables?

Thanks Simon, love the news series :)

The BROWSE permission did not fly well with one of our teams fearing that now everyone sees what they have and it's gonna rain access requests.

Title says Dec 2023- Jan 2024". Maybe meant to end with "Apr 2024"?

Hey great video , quick question, when I'm doing the merge in Synapse using python I actually get duplicated rows, both the original and the updated on do you know why this is happening ?

Can you explain the difference between using Auto-loader and Structured Streaming (readStream/writeStream) Also, when/how to use foreachBatch in Databricks ??

Is there a way to use Databricks Connect on clusters that do not have Unity Catalog enabled? I am using Runtime 13.3, if that is relevant.

Thank you. You solved a million dollar question I had.

we may write on this volumens?

Thanks for the update. I am considering fabric, but having trouble understanding the consumption estimates that Microsoft provided in their docs. In particular, they claim that "Every 4MB, per 10,000" will cost 306 CU seconds per read on One Lake. Do y'all have any idea what this means, or can you potentially bring it up to some of your contacts and push for clarification? I ask because depending on how you interpret it, fabric warehouse reads can cost 250x more than sql serverless pools on synapse, or 40x less.

Many great announcements. From Arun's blog - the most underrated: External sharing and Metric layer (as mentioned).

MGM Vegas to announce folders! 😂

"Not folders. Probably subfolders." 🤣

I'd love to be able to use this from Java. I'm have a TAXII server to provide access to Cyber security data in Databricks. This potentially opens the door for lots of external services.

Great video!

Looks Exciting 🎉

Hi Simon, I having trouble is sending data to Domo using pydomo lib. As I am using external location in UC as source path the OS.list or other OS functions are not able to read the files in abfss path. Is there any solution to this ?