Advancing AI - DBRX & The AI Playground

There have been huge waves recently after Databricks announced their new open-source MoE (mixture of experts) large language model... but what does that actually mean? If you're already happily chatting away with your favorite LLM, why would you care? And how do you even get started with using it?
In this video, Simon takes a look through the major points of the DBRX announcement and what the sheer focus on speed tells us about LLM maturity. We then take a dive into the new Databricks AI Playground to test out DBRX Instruct and show you how you get can get started with it.
The main DBRX announcements can be found here: www.databricks.com/blog/intro...
With the technical breakdown and benchmarking here: www.databricks.com/blog/accel...
If you're setting out on a GenAI-powered journey, give Advancing Analytics a call to shepherd you on your way!

[@SteveMayzak]

I’ve been using this model for a bit now and it’s giving some of the best answers I’ve seen. That said, I had to upgrade my PC to 96gb ram with a 4090 to get it to work in my lifetime :-). Seriously memory intensive but I’m excited about this model and others like it that make it on the scene

[@DanKeeley]

seems speed is the focus on all the new models, e.g. the brand new fast and technically teading opensource llama 3 . I have to say it's quite fun to see leaders knocked off their perch every few weeks.

[@AdvancingAnalytics]

Ironically, my video production was too slow to avoid the Llama3 announcements the day after I filmed this 😅

[@mkrichey1]

I love that they would take such a powerful tool and open source it in the middle of a AI gold rush. Excited to use it and also to see what it prompts in the future 😃

[@josephjoestar995]

I was looking forward to your video on this! We do have a use case where we want to perform some analysis on our data, it’ll be interesting if DBRX can do this for us

[@nidhigupta7606]

This feature doesn't give the flexibility to interact with our Databricks schema and delta tables. Do you have any suggestions or resources for this feature?

[@alexischicoine2072]

I’m curious how expensive it would be to have a custom dbrx model and what are the advantages versus a rag approach. One nice thing with rag is that it’s much easier to control access to proprietary information as if you don’t put it in the prompt the model doesn’t have it. With a custom model it might leak information to the wrong users so then you probably need a custom model for different permissions.

[@AdvancingAnalytics]

There will certainly be a tipping point of simply having a fine-tuned model on reserved capacity rather than pay-per-token. If you're repeatedly throwing thousands of tokens at a model all day, it'll get fairly pricey. Definitely a huge amount of access/responsibility consideration when you are fine tuning though. Sounds like a video idea 😅

[@SteveMayzak]

RAG is more important than just controlling data leakage. Also enables doing date based searches, geo based, RBAC and limiting hallucinations with tuning. Until a new architectural pattern comes out that solves these things and more, I don’t see RAG going anywhere. It’s the reason we can take advantage of LLMs right now for anything that requires the right answer the first time.

[@katetuzov9745]

Before each recipe provide a short life story that is vaguely related to the topic 😂

[@katetuzov9745]

Seriously though, 1 million tokens seems like much, until you factor in that each question+answer take 600 tokens.

[@AdvancingAnalytics]

1mil tokens, assume 600 p/request so 1,666 requests. I think in DBU costs that puts it around £0.01 per request in British money. Not bad at all, but could get expensive if you were using this for per-row calculations in ETL etc