Mixture of Models (MoM) - SHOCKING Results on Hard LLM Problems!

  Рет қаралды 18,428

All About AI

All About AI

Күн бұрын

Пікірлер: 114
@tylerhatch8962
@tylerhatch8962 5 ай бұрын
I loved the "crucial insights" into gravity 😂
@AllAboutAI
@AllAboutAI 5 ай бұрын
😂😂
@YT_Jx
@YT_Jx 5 ай бұрын
Kris, That’s an extraordinary project with surprising results. Thank you for sharing your talents.
@shawnfromportland
@shawnfromportland 5 ай бұрын
the holy grail problem to prompt it with is one in which each contributor gives different advice, and the decider model produces output that's different from all the contributors and different than what the same model would have produced alone without MoM
@mwdcodeninja
@mwdcodeninja 5 ай бұрын
Give each actor its own motivation. Like a murder mystery. The king is the detective. Have it play Clue.
@techpiller2558
@techpiller2558 5 ай бұрын
Different personalities would be good.
@EinarPetersen
@EinarPetersen 5 ай бұрын
I assume this is all on the GitHub repo to play with
@EinarPetersen
@EinarPetersen 5 ай бұрын
Wrong thread
@tubaguy0
@tubaguy0 5 ай бұрын
This was really interesting, I really like your thought process. One suggestion I have to validate your results is to run each of your tests 3-5 times and see how consistent each architecture is, since you’re going to get different outputs each time. I also didn’t see whether you were controlling model temperature in your setup, and I wonder if putting a temp 0 for at least one participating model as a sanity check would improve output (another opportunity to run a batch test battery). This cuts really close to what I’ve been imagining for agent framework cooperative problem solving, so thank you again for showing us how to put ideas into code.
@AllAboutAI
@AllAboutAI 5 ай бұрын
yeah good idea, thnx :)
@trud811
@trud811 5 ай бұрын
Good, but as noted in the comments it will be interesting to see the results for "plain" GPT4 and Claude for the same set of problems
@AllAboutAI
@AllAboutAI 5 ай бұрын
yeah, kinda regret not inc a basetest
@jamesjonnes
@jamesjonnes 4 ай бұрын
​@@AllAboutAIYou can add another video. I'm very interested.
@DarrenAllatt
@DarrenAllatt 5 ай бұрын
You could combine tree of thought into this mixture of models Where instead of having one model pretend to be three different experts, you use a tree of thought process across these different architectures to generate a more accurate output across the entire task
@AllAboutAI
@AllAboutAI 5 ай бұрын
great idea :) tnx
@DefaultFlame
@DefaultFlame 5 ай бұрын
A note: the marble problem has actually been solved in Matthew Berman's testing by WizardLM-70B-V1.0-GPTQ, Mistral 7b OpenOrca, Mixtral, Mistral Next, Mistral Large, and LLaMA 3 70b Groq. That is however only 6 of the 32 models he's tested. I will note that in my own testing Reka Core passed the marble test while it failed when Matthew was testing it.
@stormyRust
@stormyRust 5 ай бұрын
Really interesting! Like others mentioned, even if the added reliability of a MoM architecture, it may not be worth the computing power and additional context length, and GPT-4 might be strong enough to figure most of these out on its own. I think a good compromise would be using less LLMs in the hierarchy
@AllAboutAI
@AllAboutAI 5 ай бұрын
yeah, or you could just use OS models with llama.cpp or ollama i guess
@justtiredthings
@justtiredthings 5 ай бұрын
Cool experiment! I'd like to see a sort of "deliberative democracy" architecture--where the models have a chance to discuss back-and-forth or to see one another's initial answers before providing a final answer and then voting
@frankjohannessen6383
@frankjohannessen6383 5 ай бұрын
The democratic arch should probably have a step where you have one strong model that decides which answers are identical and remove all but one of them before voting. Otherwise you will get a situation where there are maybe 7 correct answers and one wrong, then the wrong answer has an unfair advantage since the models will spread their votes on the 7 correct answers, but not the one wrong answer. Also, solutions like this is where Groq with 300 tokens/s really shines.
@AllAboutAI
@AllAboutAI 5 ай бұрын
great idea, will try this. tnx
@footube3
@footube3 5 ай бұрын
Amazing idea, and excellent result. Well done!!
@AllAboutAI
@AllAboutAI 5 ай бұрын
thnx :)
@john849ww
@john849ww 5 ай бұрын
Thanks for sharing your research! I had an idea about pairing a large model with a tiny model, where the tiny model is asked a question and the large model evaluates the response. The large model identifies any issues and iteratively adjusts the prompt to help the tiny model get the right answer. Not sure if it would work in practice, but if it did maybe a tiny (and cheap) model could be made effective to solve more problems than it otherwise could have?
@DesignDesigns
@DesignDesigns 5 ай бұрын
Excellent...It'd be great if the models could have been run in parallel. This way, less time would be needed.
@AllAboutAI
@AllAboutAI 5 ай бұрын
great idea, will def check into this!
@prolamer7
@prolamer7 5 ай бұрын
Very interesting you are not only thinking about such solutions but actually programming and testing them. Which is really great! What about this architecture: King architecture BUT with a twist that you will execute question batch with king being first GPT4, then Opus, then llama3 70b and you will then chose only answers with majority of votes ie if GPT, Opus agree.
@tubaguy0
@tubaguy0 5 ай бұрын
And if they don’t agree, loop back and resolve the differences via Socratic method until there’s consensus on either a solution or a recommended plan of action to improve the input for the models to try again. I wonder if the local models could reason out the problem super cheaply if given enough reflection and conversation iterations, or if the general models are just too effective to leave out. Probably use case dependent.
@AllAboutAI
@AllAboutAI 5 ай бұрын
cool idea, noted :) tnx
@prolamer7
@prolamer7 5 ай бұрын
@@AllAboutAI :-)
@elevated_souls
@elevated_souls 5 ай бұрын
Seems like the second and third method are both kind of getting the best answer of each model, which makes it kind of average. The first architecture on the other side takes all answers into a consideration, adds all insights and creates the best answer from them all. So in a sense it gets only the good stuff and excels the most. This is the way.
@justtiredthings
@justtiredthings 5 ай бұрын
You could prob achieve that with a more deliberative democratic architecture
@Pregidth
@Pregidth 5 ай бұрын
Excellent idea! I understand Opus and OpenAI as the overseer, but if local LLM could proof being the heroes here, would be supberb!
@AllAboutAI
@AllAboutAI 5 ай бұрын
true!
@VR_Wizard
@VR_Wizard 5 ай бұрын
Next projekt is a Mo(MoM) model where we compare the result of all 3 MoM systems with these systems again. Think of recursively going down the rabbit hole.
@YT_Jx
@YT_Jx 5 ай бұрын
lol
@hypercoder-gaming
@hypercoder-gaming 5 ай бұрын
Perhaps it would be even more powerful to make a Chain of Models/Chain of Experts architecture that has a line of models and each model builds upon the previous models' responses. Making a Network of Models would be even better, where you have layers of multiple models and each next layer gets the previous layer's outputs and ultimately combines everything in the end to form an answer to your question.
@AllAboutAI
@AllAboutAI 5 ай бұрын
very interesting, noted, tnx:)
@c.d.osajotiamaraca3382
@c.d.osajotiamaraca3382 5 ай бұрын
If just one agent is wrong the entire work product crumbles. Wouldn't you need a hallucination check, or some kind of redundancy at each stage?
@hypercoder-gaming
@hypercoder-gaming 5 ай бұрын
@@c.d.osajotiamaraca3382 Yeah, I didn't think of that. Maybe in the system prompt, it could say there is a chance that the output of the previous model(s) is incorrect and to consider that before making a decision. Or between each layer it could have an internet search if applicable. Not sure how I would program that as I don't know how to do HTTP requests but it seems possible.
@USBEN.
@USBEN. 5 ай бұрын
I love these experiments soo much.
@AllAboutAI
@AllAboutAI 5 ай бұрын
thnx :)
@youtuberschannel12
@youtuberschannel12 5 ай бұрын
You should've add a baseline with just gpt-4 turbo and compare.
@AllAboutAI
@AllAboutAI 5 ай бұрын
100%
@LeeBrenton
@LeeBrenton 5 ай бұрын
that's epic man! well done.
@nftawes2787
@nftawes2787 5 ай бұрын
Add a gamifying token layer to learn which models' votes should be heeded/ignored in which situations
@AllAboutAI
@AllAboutAI 5 ай бұрын
yeah cool idea, noted
@MrSuntask
@MrSuntask 5 ай бұрын
Great idea! Like your experiments. Keep up the great work!
@AllAboutAI
@AllAboutAI 5 ай бұрын
thn mate :)
@HistoryIsAbsurd
@HistoryIsAbsurd 5 ай бұрын
Damn this was a beautiful experiment! Really really can see this being used effectively on a grand scale. I mean even think of the little LLMs if we had those all deciding and the larger one is Llama3 or something we can run 100% locally. This can be really powerful for those with less GPU capabilities and low funding if they run them sequentially.
@GetzAI
@GetzAI 5 ай бұрын
Great experiment!! I wonder if adding in reasoning to their answer, if democracy would vote differently? Or if the answers could be discussed before voting. Having a bunch of AI/people voting on bad assumptions is never good. "before voting, discuss each answer for a) meeting all the requirements and b) being the most accurate".
@AllAboutAI
@AllAboutAI 5 ай бұрын
interesting =) noted for future itteration
@bennie_pie
@bennie_pie 5 ай бұрын
This was a fantastic video thank you...brought me up to speed on the various models and structures of their hierachys was fascinating. Not dabbled in agents for a while so was nice to be surprised at progress in this area vs my early attempts that spent tokens and produced nonsense.... question....do you have any teams of agents setup doing actual real world useful stuff? (I'm sure I'll find out by watching more). Thanks for the video.
@JohnDoe-zx8bu
@JohnDoe-zx8bu 5 ай бұрын
I think that it would be better to estimate performance when you have some simple model at the output level. GPT-4 can operate with all of provided problems without previous models solutions
@AllAboutAI
@AllAboutAI 5 ай бұрын
yeah true, will try diff configs for this setup
@peterkonrad4364
@peterkonrad4364 5 ай бұрын
actually, the most likely place for the marble to be is on the floor, because it is no easy feat to turn a cup with a marble in it upside down and put it on a table without the marble falling out before.
@AllAboutAI
@AllAboutAI 5 ай бұрын
that would be the big brain human answer yes hehe
@jiyuhen
@jiyuhen 5 ай бұрын
🤣😂 I just have to say it. Monarchy seems to be the solution for most problems. However jokes aside, this experiment turned out really great, and fascinating how the models behave in the different setups and interactions.
@someverycool4552
@someverycool4552 5 ай бұрын
Monarchy would be best only if the king is also by far above others in mental capacity, in all areas.
@jiyuhen
@jiyuhen 5 ай бұрын
@@someverycool4552 Agreed
@youtuberschannel12
@youtuberschannel12 5 ай бұрын
​​@@jiyuhenHis demo is not really a monarchy. In a real life a king is able to have his own ideas or solution, he has the freedom to completely ignore proposal put forth to him. But in this demo the king doesn't have any of those freedoms and powers. He has to judge and decide what others put forth to him. Let the king have his own ideas and the freedom I mentioned earlier and I bet the results will be different.
@MuplerHi
@MuplerHi 5 ай бұрын
Looks like the Ai jury is about to be seated in a courtroom. and voilà!
@Dabes88
@Dabes88 5 ай бұрын
Makes sence we create theoretically using something between fast and slow thinking. Maybe something like q* needs to use Claude 3
@Not_A_Robot_LOL
@Not_A_Robot_LOL 5 ай бұрын
22:38 it only got one vote because each model provided 10 unique answers. This shows that the democracy model isn’t a good architecture for all kinds of problems (specifically, the ones where the answer is determinate).
@settlece
@settlece 5 ай бұрын
how exciting what an awesome video thank you
@AllAboutAI
@AllAboutAI 5 ай бұрын
thnx :)
@igord7272
@igord7272 5 ай бұрын
Great content!
@EinarPetersen
@EinarPetersen 5 ай бұрын
I would probably choose to do the tally locally using an optimized solution a simple python script bypassing any AI that way you eliminate the models sucking at math and use architecture specialized for solving the math problem of counting 😊
@jamesroth7852
@jamesroth7852 5 ай бұрын
King is King!
@tomtyiu
@tomtyiu 5 ай бұрын
wow, you have an innovative mind. 🤓
@abhishekrp.ai2002
@abhishekrp.ai2002 3 ай бұрын
quite impressed by these MoM / MoE model architectures, however, I am not satisfied with the leetcode hard problem used in the test set. Testing the llm generated response on just 3 test cases is not a valid determinant for the architecture's capabilities. I could see the solution is itself in the order of 4 ( O(N^4)) time complexity, which essentially is Brute Force. A better bench mark would be testing how well it can optimise the code and handle diverse test suits IMO.
@marcuscronan6658
@marcuscronan6658 5 ай бұрын
How much did each "tournament" cost in terms of tokens?
@AllAboutAI
@AllAboutAI 5 ай бұрын
since 8 models was local on ollma, not much =)
@EinarPetersen
@EinarPetersen 5 ай бұрын
So this is all in the GitHub repo to play with I assume?
@EduardoJGaido
@EduardoJGaido 5 ай бұрын
Hello! Thank you for a great video. I ask you or the community, I have a hard problem to solve: i want to make a chatbot using a local LLM with RAG (keep reading please!) BUT i want to use it for my business so the clients of my physiotherapy clinic can ask it and it responds JUST WITH the information that is fed. Otherwise, it says "Oh, i don't know, wait please" so the secretary can answer instead. Just with that, I would be happy. I have a lot of FAQ listed with the answers (in a json friendly format). I can't find this answer anywhere. If you have any information, would be apreciated. Cheers from Argentina.
@Copa20777
@Copa20777 5 ай бұрын
I was following the explanation In the beginning about the different architectures very well, until he opened the code😂
@AllAboutAI
@AllAboutAI 5 ай бұрын
haha, its not that bad
@jekkleegrace
@jekkleegrace 5 ай бұрын
thats great can you please give us paying users access to the github
@OumarDicko-c5i
@OumarDicko-c5i 5 ай бұрын
Waw this much be so expensive lo but great
@smilebig3884
@smilebig3884 5 ай бұрын
The api call cost will go out of roof.
@yngeneer
@yngeneer 5 ай бұрын
lovely
@immortalityIMT
@immortalityIMT 5 ай бұрын
You feed the answer into the next model and ask it is true or false
@skylineuk1485
@skylineuk1485 5 ай бұрын
I came to the same conclusion that arbiter currently works best.
@haileycollet4147
@haileycollet4147 5 ай бұрын
All seems rather pointless without comparing to the results by gpt4/opus on their own ...
@AllAboutAI
@AllAboutAI 5 ай бұрын
yeah that was my regret, should have done a "basetest" to comp
@squiddymute
@squiddymute 5 ай бұрын
are you sure chatgpt4 wouldn’t just solve all these problems without the other models
@MarkoTManninen
@MarkoTManninen 5 ай бұрын
I was thinking same. So the problems should be more adequate. But it was a proof of concept, very nice work. I planned aomething similar with websocket agents last summer, but it became too expensive to test for a hobby. Now it is right time because ollama!
@YT_Jx
@YT_Jx 5 ай бұрын
That’s a great question. In the past at least, ChatGPT4 failed the marble and the apple questions.
@justincase4812
@justincase4812 5 ай бұрын
Do you cut wood with a hammer?
@jichaelmorgan3796
@jichaelmorgan3796 5 ай бұрын
Different models have slightly different training, context, and perspective, so this creates a broader perspective to approach the problem at hand. This will not be practical for everyday prompts.
@DarrenAllatt
@DarrenAllatt 5 ай бұрын
@@jichaelmorgan3796 I disagree on the practicality point. The way it is Setup in this video from a testing the methodology perspective yes it’s not practical.. But behind it is actually very practical. Because all of this stuff that is going on can be done in code so the user doesn’t see everything that goes on behind-the-scenes and just gets the final output
@pensiveintrovert4318
@pensiveintrovert4318 5 ай бұрын
The marble problem proves nothing. Every model vendor cheats on some problems and sticks these problems into the training datasets. Same for other problems. You need to use new problems every time. Otherwise these reviews are b.s.
@michaeltse321
@michaeltse321 5 ай бұрын
means that autocratic systems are better compared to democratic - lol
@AllAboutAI
@AllAboutAI 5 ай бұрын
haha lol
@TheHistoryCode125
@TheHistoryCode125 5 ай бұрын
This video is a whole lot of fluff for a very simple concept: combining outputs from multiple LLMs to get better answers. The creator makes it seem revolutionary with fancy names like "King," "Duopoly," and "Democracy," but it's just basic ensemble methods. They feed the same problem to a bunch of LLMs (including weaker ones like Llama), then use those answers as context for a stronger LLM (GPT-4) to make the final decision. The "Duopoly" is just GPT-4 and Claude arguing about the best answer, and the "Democracy" lets all the LLMs "vote" on the best answer. He doesn't go into detail about how the voting works or what constitutes a good vote. The results are a mixed bag, sometimes better, sometimes worse than just using GPT-4 alone. The real takeaway is that ensemble methods can be useful, but this video overcomplicates it with theatrical names and presentation.
@finalfan321
@finalfan321 5 ай бұрын
you need to go deeper into their reasoning because some of the wrong answers are actually right
@william5931
@william5931 5 ай бұрын
your mom
@YT_Jx
@YT_Jx 5 ай бұрын
@william5931, your dad. :)
@aaronpainting5643
@aaronpainting5643 5 ай бұрын
Why is nobody talking about "Where's Daddy?" or "Lavender" or "Gospel"? these are the most advanced AI systems and are already committing mass murder!
@tellesu
@tellesu 5 ай бұрын
Why do you bots make the same stupid fearmongering comment on every post?
@aaronpainting5643
@aaronpainting5643 5 ай бұрын
@@tellesu I'm not a bot, I just want to hear smart people like this guy break it down. more people need to be talking about this, it is AI development towards Armageddon
@illuminated2438
@illuminated2438 5 ай бұрын
@@tellesu that's not fear mongering it is simple fact. A certain tribal state is using advanced AI to engage in the mass murder of civilians without any human oversight. The where is daddy program literally watches targets leave so-called military locations and orders an attack only after they are at home with their family. It is deliberate mass murder of civilians, and it is powered by AI.
100% Local Super Easy Private Email RAG Setup | Ollama - Gmail ++
13:04
这是自救的好办法 #路飞#海贼王
00:43
路飞与唐舞桐
Рет қаралды 79 МЛН
Who’s the Real Dad Doll Squid? Can You Guess in 60 Seconds? | Roblox 3D
00:34
When mom gets home, but you're in rollerblades.
00:40
Daniel LaBelle
Рет қаралды 110 МЛН
[1hr Talk] Intro to Large Language Models
59:48
Andrej Karpathy
Рет қаралды 2,2 МЛН
How To Build Feature Flags Like A Senior Dev In 20 Minutes
20:33
Web Dev Simplified
Рет қаралды 52 М.
5 CHALLENGES for Claude Computer Use: Here's What Happened
21:18
All About AI
Рет қаралды 114 М.
100% Local Tiny AI Vision Language Model (1.6B) - Very Impressive!!
15:05
Gamba Jamba Cobra
1:20:07
hu-po
Рет қаралды 2 М.
Ray Kurzweil: Does AI Make Immortality Possible?
54:03
Center for Natural and Artificial Intelligence
Рет қаралды 31 М.
AI Pioneer Shows The Power of AI AGENTS - "The Future Is Agentic"
23:47
Build your own Local Mixture of Agents using Llama Index Pack!!!
12:12
host ALL your AI locally
24:20
NetworkChuck
Рет қаралды 1,2 МЛН
Проверил, как вам?
0:57
Коннор
Рет қаралды 976 М.
iPhone Flip станет ХИТОМ!
0:40
ÉЖИ АКСЁНОВ
Рет қаралды 1,2 МЛН
Gaming phone
0:19
Zain Artifolio
Рет қаралды 8 МЛН
Месяц собирал ПК. И все ещё не собрал…
22:58
Игорь Линк
Рет қаралды 234 М.