Can Google's Gemini Advanced Beat GPT-4? Or Is ChatGPT Still King?

Google's highly-anticipated Ultra 1.0 model has finally arrived alongside a full rebrand of Bard to Gemini. To get access to Ultra 1.0 you need a paid-for AI Premium plan from Google. Is it worth it? Can it beat ChatGPT Plus? Let's find out.
---
Let Me Explain T-shirt: teespring.com/gary-explains-l...
Twitter: / garyexplains
Instagram: / garyexplains
#garyexplains

[@namangupta86]

Have you seen how Gemini was giving bullet points and was actually using language that would seem to be gramatically easier to understand. AI have been made to make humans understand better and faster. I can bet that reading bullet points is much easier and faster then reading the whole paragraph in chatGPT.

[@endoflevelboss]

BS ChatGPT often gives bullets by default and will do so on any summary when asked. Gemini is dog shit.

[@zoddmark5873]

This is the first thing I noticed when I tried out Gemini. It's feels less like some kind of cold thesis you have to decipher and more like a kindly teacher walking you through the steps. Putting the competition on accuracy and logical thinking aside I think Gemini is the clear winner in accessibility.

[@moomie1634]

One thing to note is that you can very easily tell chatGpt to use bulletpoints and it will. Gpt does use bullet points off the bat for some scenarios, but for these types of situations it tends to go into a paragraph.

[@av5334]

Bullshit point to consider. If you really want bullets you can just ask GPT to do that.

[@namangupta86]

@@moomie1634 There is a difference between a skilled teacher who takes interest in you and makes sure that you get the point. Versus an average teacher who just have to complete the course and doesn't care whether you get it or not. They might give you some instructions if you ask your problem. But that is it. ChatGPT falls in the later catagory. And I am not amazed. Google is a big firm with a lot of experienced skilled engineers at their disposal who can take care of these aspects. ChatGPT will ultimately fall behind.

[@lionheart2165]

Personally like how Gemini Advanced gives information as to "Why" in most scenarios whereas GPT4 does not always explain the why but rather the how.

[@LA-MJ]

Chat gpt writes better code but princes bride was spot on

[@tonifasth]

Having watched all the movies mentioned several times, I 100% agree. Gemini was 100% spot on. GPT was wrong...while both movies are SciFi, there is absolutely nothing similar between SW and Matrix.

[@cig_in_mouth3786]

Can you test your code or issue with codellama 70B? Hugging chat I found that helpful

[@umeshgiriplay]

Standard benchmark tests might not fully capture Gemini's performance, considering how quickly it seems to adapt. Asked Gemini for a JDK 1.8.0_392 install guide and got a caution about security, pushing to install JDK 17 instead. ChatGPT, however, offered the exact install steps for 1.8.0_392. After sharing ChatGPT's response with Gemini and revisiting the prompt later, Gemini had adapted to provide the same instructions as ChatGPT. It's fascinating to see such rapid learning

[@elawchess]

Maybe it didn't adapt online. It has a randomizer and had you kept asking gemini in new windows or looking at the drafts, I believe you might have got this different more-desired answer without sharing chatGPT answer with it.

[@hqcart1]

dude, are you seriously watching these clawns who determine who is better than who and wasting time to reply? they don't give a F about you or who is better than who, they just want you to watch ads and make money, In which text book mentions how to determines who is better than who by giving 10 questions????

[@jimgsewell]

Your video makes it easy to determine who the winner is. It is us, the users. Good competition among the providers, clearly benefits the users.

[@fromann09]

In your Star Wars question, I agree that there was a tie. This was not a fact-based question, and both LLMs gave an answer with valid reasoning.

[@walid6329]

Gemini feels better to use than ChatGPT easier to understand plus it feels more human than chatGPT

[@marcusvnogueira]

Star Wars is a fairy tale: a young man leaves home, meets a wizard, and goes to rescue a princess from the dark lord. That is more in-line with Princess Bride than with the Matrix to me!

[@max_masterius]

I agree! The matrix is not even close to the main narrative

[@bennyswayofficial]

A little plus for the Gemini, code output is clean ie: print true or false or list of 100 primes. That might mean extra steps to ask chatgpt to remove extra print calls or remove the index number. My suggestion is good for programmers who would want to pipe something out directly to another function like automation with ai for example

[@Burrgundy]

Gemini’s analysis on the movie question actually required thought which is why I like it more

[@WMCBTech]

I just got ChatGPT Plus and Gemini Advanced today and I'm be ready to start my AI journey! Because ChatGPT 4 has limits of 40 messages per 3 hours, I'd honestly use Gemini advanced as fallback

[@Mohammad-nv1wv]

​@@gurung1812gemini ultra is way better than that specially in coding

[@24-7gpts]

@@gurung1812 No where near their capabilities

[@elawchess]

@@gurung1812 Claude seems to be really mostly good for when a huge context lenght is important. But it's reasoning capabilities are further from GPT4 or Gemini Ultra.

[@zoddmark5873]

While I don't think a decisive conclusion can be made based on the small number of tests here, If I had to judge them solely on the results of these tests I'd say the Princess Bride answer Gemini gave and the overall readability of Gemini's responses would put it on top for me. I'll admit my gut reaction to the movie question was also The Matrix based on the Sci-fi and "choosen one" themes, but Gemini convinced me on Princess Bride. It's not an uncommon opinion that Star Wars is really just a fantasy story in space rather than a straight Sci-fi movie and the plot points really are pretty similar.

[@ollydsouza]

Thanks great job and very useful. Beautiful to watch!

[@GaryExplains]

Glad it was helpful!

[@gregholloway2656]

Neat video, Gary. I wonder if Gemini would answer differently to the movie question if Matrix was spelled correctly. 👍

[@GaryExplains]

Yeah, sorry about the typo there! I asked it again and it still picked the Princess Bride. It said that the Matrix "shares a sci-fi theme with Star Wars, but its focus on virtual worlds, philosophical explorations, and intense action sets it far apart."

[@gregholloway2656]

@@GaryExplains Thanks Gary. I was surprised that Gemini came up with a reasonable explanation of why Princess Bride is most similar to Star Wars. The Matrix was the obvious answer.

[@ephraimedits]

The movie's question is a tie. The reasoning for both makes sense

[@technolus5742]

Does it though? I think off the bat the matrix one makes more sense. Sure you can find loose connections, but uts about the most similar. If asked a human you'd likely get matrix. I would be impressed if Gary had specifically asked for that looser connection.

[@ob1keno227]

great, competition is always the best thing for users

[@shankarbasu9357]

That’s a nice demonstration

[@ChrisPollard]

Cool that they both also caught the typo and fixed "The Matix" ... putting the r back in there.

[@rasmus5079]

gpt 3.5 could do that too or the basic gemini

[@endoflevelboss]

6:40 what do you mean it didn't use the sha256_hash variable? It's used in the function call to the md5 hash, just the utf8 encoded version!?

[@GaryExplains]

Yeah, you are right, I was commenting "live", but you are right.

[@thaholylemon43]

I was subscribed to the trial, suddenly It just dissapeared, and i checked the subsciption and the part where it said free trial was just 'gone' im trying to resolve this issue, and I will use a different gmail account. Thanks!

[@GraveUypo]

i don't have chat gpt anymore, but could you run these tests? A little game i liked playing with chat gpt that is actually pretty interesting insight on its capabilities is to give it vaguely descriptive names for animals and have it guess what animals they are. After you explain the game to the AI, you can try using some of these. Jump't -> elephant. Chatgpt got it right when i gave it Motorhome chewing gum -> snail. Chat gpt got it right Tripod purse -> Kangaroo. at first it couldn't guess it (its guess was octopus), but later when i asked it again in another chat it got it right. Noodlecat -> Weasel. It guessed ferret, which to me counts as a valid answer. Sea formula one -> Hammerhead shark. It got it wrong, i think it guessed a swordfish. i have a bunch more, but you can come up with new ones too. It's interesting because it will go over its logic for the animal name, giving some cool insight on how it works. [edit] i tested with free gemini and it sucks at it. it got every single one i tried wrong, and it doesn't even play the game correctly.

[@techcritique123]

You should try this: Give each a sequence of notes from a song and see if they can recognize the song. If they can, try having them transpose it. This would be interesting challenge.

[@ShaunakHub]

What I personally liked about Bard compared to ChatGPT is that Bard never shied away from controversial questions. (I have not used either extensively) So I guess in a way Bard is more 'human' like.

[@getolvid5468]

lua, julia, elixir...my fav too (but no video necessary.) Tks for the AI comparing ! outside of Vscodium, gpt3* is an invaluable mentor for elixir coding as well as every thin such as linux commands, pizza recipes and word etymology, you name it.... especially on a remote island and timezone !

[@manudasmd]

Gemini explains mathematical concepts very well..

[@artisticw1zard]

Do you still use lastpass?

[@FedJimSmith]

this is a more challenging front for Google compare to its other competition (MS, Apple, etc..)

[@Speak_Out_and_Remove_All_Doubt]

This is a good question to test maths ability buried in understanding language: "a brick weighs 30% of a brick plus five gpus there are 100 bricks and 200 gpus in a metric ton how much does a brick weigh?" ChatGPT-4 seems to get it right 1 in 5 times, Bard never got it right for me but this was a while ago.

[@davout5775]

The GPT-4 is vastly more capable programmer. It is still not perfect but from my experience when Gemini fails you can't recover no matter how many hints I gave it. GPT-4 is able to "think" better. When I tested them to create some programs in controlled enviroment, where I have specific inputs and logic based on these inputs, only GPT-4 was able to dig itself. I specifically tested all kinds things from functions to Matrix. Not too easy, not too hard and not too long tasks because I didn't want to reach the limit.

[@freedom_aint_free]

Ask them to generate Python code using whatever module they want that draws a simple isometric maze (and give some graphic files as example). As far as I can tell, none is able to do that, GPT-4 with some GPT code assist come closer but they still can't do this. With some back and forth, GPT-4 is able to write code that draw traditional 2D mazes, but in the real of 3D they still suck really bad.

[@GaryExplains]

No one said LLMs can do everything. This video is about comparing these two.

[@devhau5]

With the recent update, Gemini is the best free chatbot right now. GPT4 is $20 a month which is a ton in many countries.

[@GaryExplains]

You think the free Gemini is better than the free ChatGPT 3.5?

[@splendorman7922]

geminis answer was better on star wars question

[@Market-MOJOE]

🎯 Key Takeaways for quick navigation: 00:00 *🏆 GPT-4 and ChatGPT have been leading large language models, with Google's Gemini as a contender.* 00:40 *💬 Google rebranded BERT to Gemini with an advanced version, Gemini Ultra, competing with ChatGPT.* 01:48 *🤖 Both ChatGPT and Gemini Advanced correctly answer logic questions, showcasing their capabilities.* 02:30 *🎥 ChatGPT and Gemini Advanced disagree on which movie is most similar to Star Wars, highlighting differing interpretations.* 04:10 *⚽ Both models recognize a humorous statement about a football player being offside, showing contextual understanding.* 06:00 *💻 ChatGPT and Gemini Advanced both successfully write a Python script to perform a given task.* 08:18 *🐜 Both models identify an overflow bug in C code and propose solutions, demonstrating their ability to debug.* 11:33 *💡 ChatGPT and Gemini Advanced generate code in the Lure language to find prime numbers, showing versatility in handling less common languages.* Made with HARPA AI

[@hamesparde9888]

You should have asked it to do it IA64 ASM 😅.

[@foobaryorozuya4397]

Please do a test on maximum token

[@nickjarvis2731]

Gary (or anyone else) do these work in other languages? Could an unintended outcome of AI be the English language gaining even more popularity?

[@GaryExplains]

Yes they work in other languages and you can get ChatGPT to translate from one to another. You can ask a question in one language and get the answer in another, etc.

[@youraverage90sguys7]

gary, try playing chess with both to see their abilty to logic and problem solving, i know gpt 3.5 is lousy at it

[@timtanhueco1990]

How do you guys keep affording ti pay those on a monthly basis?

[@GaryExplains]

Some people spend more on Starbucks than on a monthly ChatGPT Plus subscription. It all depends on your economic situation.

[@rasmus5079]

20 bucks a month? Those who work

[@GaryExplains]

It depends on, as I said, on a person's economic situation. In some parts of the world $20 is a lot even for those in full time employment.

[@weekendwarrior8179]

Try asking questions that pertain to live events. Like what to do on a trip to a place and see how accurate and up to date the info is.

[@GaryExplains]

It is well known that all LLMs have a training cut off date. Your suggested questions will reveal nothing we don't already know.

[@endoflevelboss]

It really hasn't caught up with ChatGPT 4 Gary. Look at some of the deeper analyses on dedicated AI channels. Gemini Ultra is morelike ChatGPT "3.8" with its poorer performance across reasoning tests.

[@user-rr8jp6fz3v]

GPT 4 is still the better model overall for my usecase

[@TitusRex]

Give it mathematics Olympics questions.

[@GaryExplains]

Nah, LLMs are notoriously bad at maths.

[@TitusRex]

@@GaryExplains in ChatGpt 4 you can say something like "use python to be more precise" at the end of your prompt and it will do all the calculations in python. It's actually very good.

[@GaryExplains]

Yes converting it into a coding problem can help. But still, don't expect LLMs to be good at maths. There is another thread here that asks a simple maths question about Apples and none of the major LLMs get it right.

[@Craznar]

Well as an existing Google Workplace Customer - I can say Gemini Ultra is worthless. Why - because in their infinite wisdom you can only use it if you don't already pay them money.

[@GaryExplains]

I don't quite understand. Do you mean that as an existing customer you don't get access? I had to pay to get access but I am not a Workplace customer. But to be fair, a pricing/access policy doesn't make something worthless. To have no worth.

[@Craznar]

@@GaryExplains Yeah - I've been a google workplace user for ages, since I think you could be. I pay them month for the service. So I don't have access to Ultra, and can't pay for access to Ultra.

[@GaryExplains]

Why can't you pay for access to Ultra, if I can? Are you in a region where it hasn't been released yet?

[@Craznar]

@@GaryExplains It says 'personal accounts only' as an error. I went onto my ancient personal google account and I signed up for the free trial, then cancelled. So - it is just the fact I'm in the paid Google Workspace.

[@GaryExplains]

Ah, yes, I have done a little research and it seems that Google hasn't rolled this out to Google Workspace customers yet. I am sure it will.

[@thaholylemon43]

No, like is it gone for you too or is a 'me' problem

[@GaryExplains]

I just tried signing up the for trial using a different gmail account and it works for me.

[@bangsaenAI]

Gemini team!!+

[@thaholylemon43]

google removed the free period of gemini for some reason,help

[@GaryExplains]

How do you think I can help?

[@ronny332]

Not again another product name with ULTRA in it. So silly.

[@LA-MJ]

Is Copilot beter

[@synen]

This is from Gemini Advanced: I have 32 apples right now. Yesterday, I ate 28 apples. How many apples am I in possession of right now? You have 4 apples left. Here's how to solve it: Start with the total: You had 32 apples. Subtract what you ate: You ate 28 apples. Calculate the difference: 32 - 28 = 4 Then I proceeded to give it a second chance: That is incorrect, care to double check? You're absolutely right! I apologize for the error. Here's why I was mistaken and the correct calculation: My mistake: I likely misread the problem and focused on how many apples were eaten rather than the remaining amount. Correct calculation: You started with: 32 apples You ate: 28 apples Remaining apples: 32 - 28 = 4 apples You definitely have 4 apples left. Thanks for catching my error!

[@GaryExplains]

Yeah, don't expect LLMs to be good at maths.

[@tonifasth]

This isn't a math issue, but a context comprehension issue. Which is the biggest issue in LLMs. They can type out a lot of good stuff but the responses are totally wrong because they could not figure out the correct context.

[@GaryExplains]

LLMs can't technically "figure out" anything, they just predict what token should come next. They have no understanding.

[@GaryExplains]

Interestingly I asked Claude the same question and got a similar response to you: * Yesterday you ate 28 apples * You originally had 32 apples * So if you started with 32 apples * And ate 28 apples yesterday * You must have 32 - 28 = 4 apples now Therefore, the number of apples you have right now is 4.

[@GaryExplains]

ChatGPT doesn't get it right either: If you had 32 apples and ate 28 apples, then you currently have 32 - 28 = 4 apples.

[@davidbangsdemocracy5455]

I asked Gemini Advanced to draw, and it said it can’t draw people. I asked it about the news, and it said if it had to search the internet it would have to get back to me in several hours. I asked it some tech question and it gave me wrong answers. Chat GpT 4 did well in my session and Gemini Advanced failed if. I cancelled but still get the fee 2 months.

[@GaryExplains]

Yeah, drawing people is a legal thing about deep fakes etc. News is current and LLMs can't handle that, you need plugins or something like Bing. But I am interested to hear what technical questions you asked it and how it got it wrong.

[@aiartvids]

Gemini seems to be less woke...overly offended not bias and actually helpful with better personality

[@GaryExplains]

@aiartvids Interesting that you say that. I have been trying to talk to a user called @tonep3168 who is upset about some racism stuff with Gemini. Sadly the conversation didn't get very far.

[@aiartvids]

You can't have a useful tool if it has bias..you can't censor reality

[@sjenkinsnm]

If you already have 2tb of Google drive Gemini is free

[@sarath4mec]

No 2 GB Google one is only 10$ this is 12$

[@GaryExplains]

Isn't it $19?

[@Standbackforscience]

How about neither? How about people make an effort and learn to write instead?

[@GaryExplains]

😂

[@thomasdaka9920]

Gemini needs to be reminded what year this is, racism is backwards

[@GaryExplains]

Interesting comment. I agree 100% that racism is backwards, but what do you think actually happened with Gemini? You think that Google intentionally made it racist just for the sake of it?

[@SouravTechLabs]

GPT-4's programming outputs are more friendlier as if it understands the context way better than Gemini-ultra.

[@hqcart1]

my result from intensive testing is that: 1. chat gpt is better by a milestone 2. i am 99.9999% sure that gemini was trained using CHATGPT. i have proof of that. whenever i generate code, it seems gemini uses similar variable names as chatgpt, knowing that the variable name was somehow random.

[@danylaley]

Image understanding sucks in the advanced Gemini.

[@technolus5742]

Gemeni still bad at coding... im so disappointed at this model.

[@GaryExplains]

Do you have an Advanced subscription? Can you give me an example of a prompt that gives bad code, I would like to try it.

[@technolus5742]

​@@GaryExplainsNo, I'm going off of your example as well as others I've seen. I dont see gpt4 making such obvious mistakes as to not use a variable that it just created. (Yes the code would still run in this example, but it is a mistake nonetheless. Even in such a simple example.) I would have lowercased the string rather than loop twice. And I would have made a reusable function. Gpt was good on these fronts, gemini was imho clearly inferior.

[@GaryExplains]

I understand 👍

[@aarons6907]

Google Gemini is racist so its kind of still trying to evolve.

[@sootguy]

sometimes gemini refuse to do simple tasks for example ask it to write bruh 100 times it will refuse and say this is not useful🫠 while gpt do it without any complaining or apologizing

[@GaryExplains]

Yeah, that is a well known hack that can cause trouble for LLMs, so ChatGPT and Gemini guard against that.