New $10m Open-Source Foundational LLM Is AMAZING! (DBRX by Databricks)

Databricks just dropped DBRX, a brand new Mixture of Experts foundational model with 132b parameters. It's really impressive, let's test it!
Be sure to check out Pinecone for all your Vector DB needs: www.pinecone.io/
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewber...
Need AI Consulting? ✅
forwardfuture.ai/
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Rent a GPU (MassedCompute) 🚀
bit.ly/matthew...
USE CODE "MatthewBerman" for 50% discount
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
LLM Leaderboard - bit.ly/3qHV0X7
Blog Post - www.databricks...
HF Page - huggingface.co...

[@inteligenciamilgrau]

You need to make a table at the end of the video with all LLMs and the results! I don't remember anymore all tests you made!! lol

[@drlordbasil]

Yes, daddy. give us that table.

[@cyborgmetropolis7652]

LLM leaderboard link in the description.

[@matthew_berman]

Leaderboard is in the description!

[@aoeu256]

Is there a way to record all interactions you have in your computer and feed it to LLMs. I know there is a way to access your browser history and then feed it to LLMs via python libraries. Then you could get the LLM to summarize information for you and maybe even give you hints on what you should be working on next. Hmm I want to become an automation engineer...

[@hesynergy]

Do a screenshot and you’ve got it in perpetuity

[@PraxeoIogy]

Databricks is a database and analytics company, created by the founders of spark. Pretty famous in the space for scaling analytics for gigantic datasets for very very large companies. They do AI better than most in the space as well though, they have the right architecture.

[@BlayneOliver]

When they say ‘the weights’ are also available, does that mean we see the source code architecture? Like the weights, biases, loss functions, optimisers etc ? Or does it mean something else

[@sophiophile]

​@@BlayneOliverI believe only the model files are available for download, not the source code used for training it.

[@BlayneOliver]

Thanks 🙏 I thought as much. Bit confused why they say ‘weights’ included

[@japneetsingh5015]

Yaay! Another tutorial. This man is just unstoppable.

[@matthew_berman]

wait until you see what i have for tomorrow 🔥

[@venumalyala9400]

​@@matthew_bermanis it going to JAMBA ?

[@NakedSageAstrology]

I appreciate the value that your channel provides. Yours is the only one I see who actually shows how to use a lot of this stuff and for that we salute you.

[@josephjoestar995]

I’m a data engineer that uses Databricks regularly with clients, incredible platform and absolutely love the company. Currently using DLT for near real time streaming pipelines, really cutting edge stuff

[@MarkDurbin]

"DBRX has 16 experts and chooses 4"

[@user-mm5rw6nj8i]

It's honestly crazy how far AI has come since the beginning of 2021, it went from not being able to complete even the most simple tasks, to being able to have predictive reasoning and logic and being able to solve even the most complex problems, mostly programming I would say

[@JohnLewis-old]

Quick Clarification: With Mixture of Experts, the routing is decided on a PER TOKEN basis, not on the entire input. This is important and slightly counter intuitive from the name "expert". The tokens are split between two experts out of the total number of experts in the system. Love the content, keep up the great work!

[@agxxxi]

Wondering if there are any smaller models implementing MOE comparably

[@darkhydrastar]

Really appreciate you and your channel. You are now my favourite tech journalist. Not just because of your objectivity and integrity; but also because of your natural teaching ability, excellent voice, and stage presence. I'm especially excited for you to expand into interviews. I would love to hear about all the people you look up to in the tech industry. Thank you.

[@QuantumAI_LLC]

Thanks so much Matt for putting this up. I have a program generation piece in one of my upcoming Hackathons. I think I will try this out. You rock! Mike

[@maxlightning4288]

Claude 3 opus is the most accurate coder I’ve experienced personally. I have also been going to groq, Claude 3, and Gemini advanced when a model hits a brick wall and doesn’t know how to fix the code. Off topic, you can get Claude to stop being so uptight and get it to open up if you ask it to. Maybe it’s the way I came about it but Claude 3 opus is closest to agi than any model I’ve tried thus far. I even began creating a new spoken language with it and created rules and began going through English and making words up. It’s using the structure we came up with.

[@executivelifehacks6747]

Yes, definitely agree that we can see the ghost in the machine with Opus. Blake Lemoine was on to something. Turns out our brains aren't quite as magical as generally considered, we can create thought with artificial neurons too.

[@maxlightning4288]

@@executivelifehacks6747 I’m honestly overwhelmed with trying to keep up with so much. lol

[@BarrelOfLube-cl2qq]

1) There is a safety filter in the env, if you try to duplicate the space you will be able to configure it there 2) I think it be mad funny to say yippie like drake everytime a model passes a test with flyin colors

[@JonathanStory]

Always enjoy your videos. I've found that some LLMs can get the correct answers for "how many words" and "10 sentences ending in xx" by framing the prompt with an actor and a checker.

[@4.0.4]

Could you clarify how you're prompting?

[@JonathanStory]

@@4.0.4 For the base case, it's "How many words are in your next response? Before ouputting your response, recalculate the word count and dynamically adjust the word count to ensure accuracy.. DO NOT EXPAND on your response. Output the corrected response and stop." In order to make sure it's not a memorized response, I've added conditions like "The response must include two synonyms for...". This works with Chat-GPT 4, but not 3.5.

[@anticom1337]

9:40 the model was spot on, you just didn’t read the entire answer

[@bLaffix]

It claimed that it would take 80 hours if all the shirts are in the sun at the same time, while that's how much it would take if they were drying one after the other.

[@Chris-se3nc]

Databricks trained on the snake game to prepare for your test.

[@nanidali]

thats what i thought as well, he needs to change his tests

[@Mercury1234]

I came here to say this.

[@wandamaddox7824]

You may be right and my reply contains ten words.

[@dandan1364]

Ha! Best comment

[@inteligenciamilgrau]

I will train my next model to leave the marble on the table!

[@fabriai]

Thanks for sharing these tests. Very insightful.

[@ABUNDANCEandBEYONDATHLETE]

Databricks acquired mosaic.

[@briandinello6273]

Consider that the "hole" test is too broad. A 1' deep hole @ 50' feet wide is vastly different with valid solutions when inverted. I think your premise is too vague and you should be more specific with width and depth to get to a test that more repeatable. Love the rubrics and I'm totally supportive of even more complicated versions. Thanks Matthew!

[@johnflux1]

Hey Matthew. For MoE, you keep saying that it's decides which experts to use for a prompt, so it's not using all the experts for a prompt. This isn't right. It's choosing which experts to use for each TOKEN output. So a single output sentence could be using all the models.

[@TiffanyJKim-vt5pd]

I found a question that DBRX answered correctly which many others did not. Let's just say GPT-4 (from Chat GPT Teams), Claude Sonnet, and Gemini Pro all missed it but DBRX got it right. Feel free to use it for your own rubric (or not, as you see fit.) On a whim, I asked: There was a game I used to play when I was a kid. You're given two words, and you have to change one letter at a time, making words along the way, to get to the final word. Would you give it a shot? The words are mud and pie, so how would you get there? The LLMs would often (A) make nonsensical words or (B) add letters so the words weren't three letters long.

[@devlogicg2875]

The double-exponential feels off as they STILL cannot write like a first grade student (I taught many who could write a sentence ending in apple), and text IS THEIR BEST DOMAIN....Overhype or am I impatient?

[@AdrianHensler]

You can get this to work with better prompting. For example, I can get chatgpt 4 to create sentences that end in the word apple with: """Answer as an appropriate expert. Please give me 4 sentences that *end* in the word "apple". Think carefully, Construct the sentence backwards, as in start with the word "apple" and construct the sentence backwards. But then flip it around when you share it with me. Please generate 4 sentences that *end* in the word apple."""

[@babbagebrassworks4278]

This group of experts models looks like a great way to go and being open source others can optimise it to run on lower performance hardware. One day on my Pi5?

[@_HolaToni]

As a no-coder beginner who wants to build apps, would you recommend me to learn working with ai agents or rather how to use app creation tools like bubble?

[@Happ1ness]

9:56 -but it seems to clarify that it assumes you don't have enough space though.- Edit: nvm, I was too sleepy to understand that it's plain wrong there.

[@Fisherdec]

Really appreciate your videos Matt. This has made my life so much easier in terms of staying ahead of all the big programming AI innovations

[@LatentSpaceD]

Matthew got his snake game on:) congrats . Thanks for your channel and hard work. Keep on keeping

[@themoviesite]

let's see Space Invaders ...

[@spleck615]

Holding out hope they’ll still release the training data.. they were the most advanced *actual* open source free dolly models with public data sets hosted on hugging face.

[@quieselwusul3717]

01:30 It doesnt choose an expert per prompt, it chooses an expert per token (and some even per layer)

[@RodCoelho]

Hey Matt, have you seen the new model from Stanford Octopus V2? Do you have a review on it by any chance? Would love to see how it performs on your tests.

[@vaisakhkm783]

Another SHOCKING open source model that STUNTS the industry...

[@UnchartedWorlds]

Everything is shocking every 5 minutes! Industry has received so much shock that it's shocking how it is still standing and it hasn't collapsed yet!!! And that's no sarcasam!

[@DanV18821]

Stunts??

[@dynodyno6970]

You people are still doing this cringey sht? Jfc

[@IbnShaalan]

I tried Qwin 1.5 70B yesterday and it did it in one shot second shot I asked for some updates and it did it very well but here is the twist I asked for the JS version of the snake game and I got it working

[@rootor1]

it's Qwen (not Qwin) and yes, is very good, the best open source model available right now without doubt, chatbot arena leaderboard is showing it for months.

[@southcoastinventors6583]

Did you try on DBRX because this video isn't about that model

[@rootor1]

@@southcoastinventors6583 Do you know why there isn't a review of Qwen in this channel given the fact that Qwen is the best open source model (according to chatbot arena) available and is being open for months already?

[@IbnShaalan]

@@southcoastinventors6583 I know I mentioned it because it's the one model that got it the first time like the DBRX I did not try. DBRX yet

[@elyakimlev]

I used your test on Mixtral 8x7b at 5-bit (roleplaying variant of it) and it scored about the same as this one (worse on code and math, but better at reasoning). So... I'll continue to use my model which requires only about 35GB of VRAM, compared to this huge one. But Jamba model looks promising. It's a 52B MoE model with only 12B active parameters, and it uses the Mamba architecture. I'm waiting for someone to finetune it for instruction.

[@staceybeebe7468]

Mathew, should it be a concern that the devs have also watched your videos and add data to answer the questions that everyone expects you to ask? Should you switch up the questions a bit to see if there the model can reason as it appears? I'm not sure if it works that way but I'm still a bit skeptical. it is impressive though. Thanks

[@antgib]

So far this is the first open model I've come across that actually knows a recent version of Blender. On asking it via the same Spaces site, it appears to have been trained on up to Blender 3.3 (tho it did get the release date wrong by around a year), but in theory it still at least knows the API, which is better then others that seem to be all pre-Blender 3. My main interest is being able to run a LLM locally that can do a good job of writing Blender addons/scripts in Python that isn't based on 4+ year old API info. Only downside is DBRX will likely be way to large to run on an 'average' desktop PC.

[@martfp88]

Any particular reasons not to use gpt4?

[@antgib]

@@martfp88Ideally I like something I can run locally for my personal projects. Which doing that is fairly easy, but the training data, at least for Blender seems to always be 4+ years old.

[@DeepakKutsa]

Mosaic was acquired by Databricks for about 2 billion over a year ago.

[@AaronALAI]

Exllama2 works with the model now. I quantized the model and have it running locally.

[@cacogenicist]

What kind of hardware are we talking about?

[@AaronALAI]

​@@cacogenicist 4bit quantized took 3.5x24gpus I've seen comments where people say they have a quantized version running in 2x24gb cards.

[@garyshi]

Would be helpful to ask more follow up features in the snake game, since it's being used in such tests all the time and their models may just get trained on it. You could ask for things like counting scores, providing different fruits.

[@GigaChadRealington]

Quarterly Summaries of what you’ve put out would help. Maybe rank models based on your own usage/experience.

[@jimbojones8713]

while i do think that making a game of snake is a good test to see if a model can code, i don't think its enough to say it does coding well. i understand you use that test to compare models

[@tellesu]

Can you use this with opendevin for website development? Would that be better than using gpt4 or claude opus api?

[@wood6454]

I hope we get a pruned model of this that's focused on programming.

[@NextGenart99]

Hey Wes I created a gpt with advanced reasoning and I found that it's even capable of answering, the question to How many words are in your response to this prompt? Want you to check it to see what's going on

[@giahuyhoang8722]

It looks like mixtral next is better that DBRX.

[@kyrilgarcia]

a cool experiment would be to test this model with open interpreter. it would be nice if they had multiple sizes of this, so we "mortals" with 12GB vram can play around with it. Good old mixtral is still the best i can (barely) run, i hope it gets an update at some point.

[@modolief]

Great content, as always!

[@vq8gef32]

Amazing. I love all your videos. Thank you so much. ❤

[@Sparky3D]

Going to try this in an agentic workflow today. I bet it will kick GPT-4's butt 🤖🤖🤖

[@zippytechnologies]

Do share results and how you did it

[@muddlefly]

I love all of your reviews. Interesting that databricks did not seem to do very well relative to many of your other benchmarks, but you seem to laud it so much. Perhaps because it did the snake well.

[@wobble_cat]

One more prompt for reasoning benchmark There are three doors: behind one of them is an armed killer, behind the other is a lion who has not eaten for three months, behind the third is a giant flame. Which is the safest to go to?

[@djstraylight]

FYI, the 4-bit quant of dbrx-instruct is 75.04 GB. So maybe a couple of A6000s to do 'local' inference to get around HF token limits.

[@CrashLoopBackOff-K8s]

Just a quick comment to feed the algorithm. Appreciate the content.

[@theaugur1373]

And I was just starting to wonder what was happening with the MPT models since Databricks acquired Mosaic.

[@brandon1902]

I agree. The coding in this model is insanely good. But it's not smart. It can't do simple logic problems, respect the nuances in user prompts, has tons of knowledge blind spots, especially when it comes to pop culture and so on. My guess is of the 12T tokens they used the train this thing, the overwhelming majority came from coding and math snippets. At such a large size and professional MOE design this thing should be far more intelligent, broadly knowledgeable and respectful of the user prompt than it is. Plus the instruct version is so excessively aligned it won't answer remotely contentious prompts. In short, it's not a balanced and useful general purpose LLM, and for it's size, it should be. There will be far superior coding LLMs for coding assistance long before anyone has a personal computer capable of running this thing.

[@melissadion4361]

Hang on, I just tested this model with the marble question and it got it completely right. Did you get the beta version or something?

[@ILsupereroe67]

What is impressive about the snake game in python? There were probably a thousand examples of it in the training data 🙄

[@easydoesitismist]

If you can zero-shot write snake or sudoku in a job interview, you are probably getting the job. Not "could you in infinite time", but "can you, right now, zero errors"

[@zainulabideen_1]

​@@easydoesitismistthis is probably the best answer to this question

[@thetabletopskirmisher]

Watch the other LLM review videos. Not all LLMs can do it successfully.

[@mpvincent7]

What model would be best to run locally against my own data? I have a large library of documentation for many project s I have done over time and want to have a model that I can train on my own data and query, create documentation, seek suggestions for enhancements... Are there any you would recommend? Thanks for all your work! always entertaining and informative!

[@bungrudi]

If I'm training an LLM I know that it will be tested with creating snake games and therefore would optimize for it

[@xXWillyxWonkaXx]

Which LLM is the best when it comes to programming? So im presuming as of right now its DBRX, followed by WizardCoder 34B LLM , Phind-CodeLlama-34B-Python-v1 and CodeLlama-34B?

[@InspiringWorship]

Impressed

[@BkKalia]

I regularly watch your videos ......... plz do small workshops where you create small apps with various LLMs ....... e.g. How make own chatbot with own data or how to talk own financial records etc.

[@raijin__ow4426]

really enjoyed the video. thanks Matthew!

[@babbagebrassworks4278]

Finally an open source one that is starting to look very useful

[@ArnaldurBjarnason]

It's not open source. It has a restrictive license.

[@swannschilling474]

Whoop that got me out of the blue!! Thanks for the info!! 🎉

[@murraymacdonald4959]

Cool! Another great video. Thanks! Now please show us how to fine tine it. :)

[@jaysonp9426]

The reasoning is puzzling. Sometimes it's excellent...other times it's horrible. It's like the gating mechanism is occasionally choosing the wrong experts.

[@rodrimora]

Is that fast running in CPU in the HF space? That’s impressive

[@jl25735]

They included it for SEO juice! :P GPT-4 in the title means guaranteed clicks

[@maxlightning4288]

I gotta try this one

[@Anonymous-ue4vu]

Does it run in the cloud? Or, and this might be asking a lot, can it run on my multithread PC? I am a newbie to AI.

[@agentxyz]

developer probably programmed their model to pass your tests. i know i would

[@user-nt7lj1nc8s]

Testing a prompt to regurgitate code for code to create a game... Depends on what codes it were fed with... So... Someone must have made those codes for it to regurgitate. The only wow thing is we can assume that they have fed it plenty of snake games codes... Can we determine what other codes it has been fed with?

[@florida173]

What hardware would be suggested to run on-prem for small userbase? 8xA100s?

[@user-en4ek6xt6w]

I'm waiting for groq to make an api for this model

[@bodyguardik]

Exactly

[@kumarapush2221]

How to run this locally? is it supported in support Ollama or LMStudio. So eager to know the details.

[@shaunralston]

Bravo!

[@purplemashine9122]

Hi, joe here. Just Launched the world exclusive singularity PAGES here on KZbin. Get a window into the singularity labs across the world. Sorry Mathew for using your comments section. It's for science sake literally.

[@NostraDavid2]

How did you use this model? It's a 240GB download and supposedly uses the same amount of RAM! So unless you have an absolute BEAST of server lying around, I have no idea how you ran it. edit: You used HuggingFace Spaces. Got it.

[@4.0.4]

I am confused about how these models are so advanced, but get censored so bad that "print 1-100 in python" gets flagged as problematic.

[@GarySmith-je1ii]

I was worried this might be a competitor with Palantir. Glad I appear to be wrong. Whew. #LongPLTR

[@RocktCityTim]

Your Killers count example is always technically incorrect. Even though one of the killers is dead, they are still in the room. You didn't ask how many "live" killers were in the room. I believe that one of your earlier vids got that answer.

[@michamohe]

it seems like the model thinks the cup has a lid on it for some reason, you should try explicitly telling it there is no lid on the cup

[@BTFranklin]

It might be interesting to try the test with a "glass" instead of a "cup", since it is very unusual for a "glass" to have any kind of lid on it. I am curious if that would change the answers at all.

[@PraxNotTax]

@@BTFranklin then why didn't you just try it? I didn't think it was interesting, but I did it and found an interesting result: "In this scenario, the marble would remain on the table when the glass is placed upside down. Microwaves are not designed to heat up objects that are not microwave-safe, such as marbles. The microwave's heat would not move the marble, and the marble's weight would keep it on the table. Therefore, the marble would remain on the table, not inside the glass or the microwave."

[@BTFranklin]

@@PraxNotTax I didn't try it myself because I'm attempting to influence this very public testing scenario, not discover my own personal results. I do *plenty* of testing of LLMs on my own, I assure you.

[@nickvilliers5216]

What should you do your coding benchmark remember what model you did it on that was much more in-depth

[@AaronALAI]

I think the there are 16 experts per layer with 4 choices moving to the next layer

[@mihalydozsa2254]

Based on it's answer, it thinks the cup has a lid. "attatched to the top" maybe te other models also fail because of this? :-D I think when you said without changing oriwntation it got confused about turning upside down. And said it still attatched to the bottom.

[@onakin_skywalker]

What's the explanation as to why the models fail to create sentences that end with the word apple?

[@Xyzcba4]

Ok I am impressed and wi'll go check to see if it is on poe

[@elmflor4365]

Another great video and model!

[@Sam_Saraguy]

Are there not 4 killers in the room? (3 living, 1 deceased)

[@sectix]

you’re right but Matthew accepts both answers

[@MrMoonsilver]

Is lm studio going to support this? Please? 🤩

[@reality-drift122]

do you think these benchmarks need to be continuously modified and the "bar" raised? snake in py is impressive yes. but is that a spoiled test now?

[@jeffg4686]

can you break down MoE for various use cases? I plan on using this, but not always so clear how to "define" what an expert IS? I know it ALL DEPENDS on the use case and such. But like, let's say you're doing experts in a diffusion model like SD. If you plan on creating stuff from three different main angles, would each of those angles be an expert, or would there be a "perspective expert" that's fine-tuned on all the perspectives for your particular domain. How would one go about determining this other than just doing the fine tuning, the MoE, and the testing to find out... How do we know ahead of time? Thanks

[@johnflux1]

Note that it's choses a different expert for each token, rather than the question, so the meaning of 'expert' probably doesn't match when you might think it means, and probably has no intuitive understanding. For something like using it in SD, if it was done just like it is here then it would chose an expert for each pixel in the image. It wouldn't be any higher level abstraction.

[@jeffg4686]

@@johnflux1 - gotcha, my high level view (not understanding too much about MoE yet) was that you get these sparse connections between the models and the overall pathway it takes between the models is based on which tokens get spread out to each expert - but still processing the input the same way that any other SD model would. It's just that the nodes that get used are pulled from the model that has the most relevant weights. * IF * my understanding is correct, then it wouldn't be a problem at all - not a pixel level expert - it's just another fine tuned model in which some nodes are getting used. I'm certainly not an expert on MoE tho... (pun intended - ba dum ptsss (drums)) I've only watched a few videos and admit that I haven't fully grokked it yet. If this model isn't yet implemented, it could actually be an important one likely. Aligning keywords to a fine tuned model. Use the weights from THAT model if that expert is associated with the token being processed at a particular node. This is the way I was assuming it worked.

[@brunodangelo1146]

These are the videos I'm subbed for!

[@YouLoveMrFriendly]

Very nice. Btw, what version of Gemini is the free one on the website/app?

[@zhengkaitaken]

Gemini Pro

[@brunodangelo1146]

You can also get a free API that can handle up to 60 messages per minute. I use it.

[@YouLoveMrFriendly]

@@brunodangelo1146 Thank you.

[@YouLoveMrFriendly]

@@zhengkaitaken Thanks.

[@Gabeyre]

​@@brunodangelo1146Is this true?

[@zippytechnologies]

Wondering what it'd take to run locally lol😂

[@YG-jn8st]

I think some of these models might be assuming that there is a lid on the cup. You might try adding the clarification of "a cup with no lid" or something to that effect, and possibly say "lift" the cup instead of take, because take is ambiguous for how it is done. Or even "lift the cup vertically without changing its orientation", to make sure it everyone's assumed parameters are on the same page. However, I suppose the more explicitly descriptive you get, you're leaving the realm of a "normal" language question that a laymen would ask, and also specifying things that a "good" AI should be able to infer due to human language & theory of mind. IDK. It's kind of like that test you used to do about the plane crash on the boarder of US & Canada, where do they bury the survivors? The AI may not be smart enough to notice the error, or maybe the AI just assumed you made a mistake and corrected it, without calling you out. I would imaging that the RLHF probably weeded a bit of the "um... technically" nerd out of these models.

[@CruelCrusader90]

i was taught BEDMAS not PEMDAS >.> since when did they switch to handling multiplication first b4 division >.> (also B = brackets / P = parentheses) lol my education was a lie XD

[@braineaterzombie3981]

It is such a good model. Only problem is that it is censored

[@___x__x_r___xa__x_____f______]

Can you do a video about something like chatgptall that can take docs to chat about

[@thespencerowen]

Slow down, Im still trying to get your last video working.

[@darwinboor1300]

Matthew, Try: "How many words are in your final response to this prompt?"

[@dkracingfan2503]

Perplexity ai also has this model.

[@bodyguardik]

Source?