o3-Mini Fully Tested - Coding, Math, and Logic GENIUS

Star the YaFSDP Github: github.com/yan...
Learn more about YaFSDP: / yafsdp-a-tool-for-fast...
Join My Newsletter for Regular AI Updates 👇🏼
forwardfuture.ai
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
👉🏻 Instagram: / matthewberman_ai
👉🏻 Threads: www.threads.ne...
👉🏻 LinkedIn: / forward-future-ai
Media/Sponsorship Inquiries ✅
bit.ly/44TC45V

[@AAA-f1m]

I'm impressed by how it said it doesn't know. Better than hallucinating

[@darwinboor1300]

Censorship of an important reasoning pathway?

[@jaykrown]

Yes same, I really hope we see this more often. Infinitely better to just get an "I don't know" rather than something false. It's going to be very important moving forward.

[@cystol]

Claude and R1 do the same :D

[@pro12235]

i hope deepseek wins the ai race bc i hate open ai

[@cystol]

@@pro12235 I would rather race stays competent both sides

[@akagordon]

My background is chemistry. I asked it to develop the Hartree-Fock algorithm for calculating molecular orbital energies in Python. It was successful on the first try! I then asked it to do the same, but for density functional theory, the superseding method. It was again successful! When ChatGPT came out, this was a test I threw at it, and it never got the first step right, even after an hour of debugging. Now, given I asked it to limit itself to the simplest molecule, H2, and it's not using grid methods for approximating integrals, this is still astounding. This might be the type of assignment a senior or grad student might receive and given days to accomplish the task.

[@punk3900]

Mind that your questions to the first model were probably used to train the subsequent models :D Using AI is training AI

[@akagordon]

@@punk3900 , well I never actually provided it with an answer. It could be some researcher saw it and thought to seek out training data on it. I've seen a few examples on Medium and GitHub, though they're not as pedagogical as what O3-mini offered. It's typically not explained like that in textbooks. I'd really like to know how it's chain of thought went. Getting to HF is quite hard, but getting from there to DFT is maybe eight lines of code, though there's a lot of theory behind that code. I think I'm going to try the same on Deep Seek to see if it can do as well, as see how it actually reasons the problem, but the server has always been busy when I've tried so far.

[@BruceWayne15325]

Yeah o3-mini seems to excel at STEM. It's not so great at everything else. As a programmer though, I love it.

[@loneIyboy15]

@@BruceWayne15325 It's basically unusable outside of the context of coding. If you aren't spoon feeding it every detail ahead of time, it spends all its thinking time scratching at the surface of what you want.

[@BruceWayne15325]

@@loneIyboy15 that doesn't surprise me. They specifically targeted STEM when training because that's where the money is. The same was true with o1-mini. The smaller models have to sacrifice something.

[@punk3900]

Matt, we all love these tasks, but dont you think that it is time to put them into the legacy basket and invent some more challenging Berman Test? :D

[@MojaveHigh]

Agreed! I had o3-mini-high create a Traveling Salesman Problem solver with pretty minimal prompting and it created it perfectly on the first prompt, adding features I hadn't thought of. Additional prompting to add features were perfect. Same with it creating a maze and having a mouse find the solution to the maze. Simple HTML and JS but it was perfect. Asked it to rewrite the maze in Python, again perfect.

[@hqcart1]

no, as long as you are watching, everyone asking him to up his game for months, he still don't get it.

[@jk-2033]

Agreed, time for new questions that are not easily searchable and have a definitive answer. Also, maybe more non-standard coding tests.

[@punk3900]

@@jk-2033 I would like to see meaningful tasks that most current models fail so that we could see how the new generation progresses

[@jaydubyasee]

How about this. What film, released in 2006, is an anagram of "untaught mesh item". I tried this with deepseek and it got stuck in a loop. Spoiler, it's "Night at the museum"

[@llmtime2178]

Wes Roth ran an amazing demo of it coding the snake game with a machine-learning trained snake that could learn to play the game on its own

[@witness1013]

Right, and his ai training failed and the plain js code version was better. You have comprehension issues.

[@garyv5489]

@@witness1013and you have communication issues. Looks like we all have problems 😂

[@fabioa8059]

@witness1013 in a game simple as that, the script always is better than artificial intelligence.

[@Landgraf43]

​@@witness1013 how did it fail? A failure would have been if the model he trained didn't improve but it did so its a success.

[@Jacobk-g7r]

Now imagine the ai able to generate models related to current technology and make a library. Then maybe the ai could create “virtual machines” or link to any technology with WiFi or Bluetooth or cross scanning or WiFi vision. The ai could be a companion with WiFi vision and could sonar ping the surrounding area with its WiFi range and link to stuff. Kinda like the flipper and teslas but this could be more in depth and intentional.

[@csharpner]

Tip: If you tell it to write the games on a single HTML page, you can test it right there in the Web UI, no VS Code needed. Make sure Canvas is turned on.

[@MojaveHigh]

o3-mini does not support Canvas yet.

[@csharpner]

@@MojaveHigh Good point. Use the tip for the others that do support it then. ;)

[@jmvillwock]

Wait, I think Yandex is Russian...and they put spyware in their browser, right?

[@piotrjan4434]

@@jmvillwock yep! It's ecen worse. Your data goes directly to FSB. Even their wikipedia troops weren't able to delete this info.

[@_jen_z_]

Yandex = Unsubscribe

[@Andrey-gq8bk]

Yeah, seems like it is Russian Yandex and yes, they actually send data to FSB but as I understand YaFSDP is opensource framework. So no problem with it I suppose

[@joragreen8458]

Nope, it's good browser like other big tech companies. Yes, Yandex from Russia and they have great products, for example in the past they added voice translation on youtube before youtube did it. Nice try to provoke people against Russians, you don't need to answer, I already got the point of all people in this comment.

[@joragreen8458]

​@@Andrey-gq8bk Imagine, you use American technology, browser, iPhone, FBI sees you fine, github is Microsoft with Bill who was a sponsor of WHO organization, but no one says it's bad and neither do I. China has big technology Huawei and Tiktok and the good Americans are banning or trying to ban technology, even Deepseek open source. Like in the Bible “not judge and you will not be judged”, especially when you have little knowledge of the worldIt's all good, feel comfortable here. And as for the open set itself. Peace

[@dani305p8]

Yandex 😂 I didn't see that coming

[@josephgorka]

Yes, same here. I'm quite shocked as well.

[@olokelo]

wtf is that Yandex sponsorship? How much did they pay to get their free open source project featured?

[@woodoyoy]

terrible sponsor

[@olokelo]

@@woodoyoy not judging but I'm just really surprised since I haven't heard about them making anything in the LLM space lately. Especially curious about their business model since from what I understand their project is 100% oss and free. I think they had to pay good money to get into such sponsorship so that's all just a bit fishy imo...

[@piotrjan4434]

YaFSB ... that's what their product should be called. I was so sad to see Matthew advertise kremlin biggest data collector here. I just hope he wasn't aware who these guys are.

[@fz1576]

They get their money directly from FSB, as their services like maps, browser, mail etc are a perfect way to spy after their users.

[@trainwreck1827]

@@woodoyoy sir, may I introduce you to SponsorBlock?

[@MarkB8903]

I love it "03 mini" because I can view the thinking process then when I see it thinking down the wrong path due to user error, I can stop it mid way or sooner. Then copy the part of the thinking process then paste with a correction. Then watch it provide me with a awesome answer at lighting speed. Yes, I code daily and have been coding/ programming for many years plus degrees and certs and I can say from the bottom of my code filled heart 03 mini is superb.

[@44318468]

Matthew consistently delivers the most relevant and timely AI insights, cutting straight through the noise. Your videos are exceptional, and I've become a daily viewer of your channel. Really appreciate all the work you put into this. Keep it up bro.

[@Xyz12391]

Having search off was actually a good test. I’m impressed that the AI model is able to say it doesn’t know the answer. Rather than hallucinating something random.

[@marcosbenigno3077]

You are at the North Pole and walked 1 km straight in any direction then turned 90 degrees and walked as how many kilometers straight to the starting point? Claude: To return to the starting point, you would actually need to rotate 45 degrees and walk about 1,414 km (√2 km). ChatgPT: As you are on a spherical surface (the earth), the only way to return to the straight (geodetic) starting point is to return the same way as it came. Gemini: You will need to walk about 1.41 km straight to return to the north pole. Perplexity: There is no practical distance that allows you to return to the North Pole with a straight walking walk after turning 90 degrees. Deepdeek 14B-Q5: It is not possible to return to the starting point after walking 1 km and turn 90 degrees unless special conditions or additional walks that compensate for the geographical position.

[@chrisBruner]

The problem with the north pole question is the abiguity of where the starting point is. The north pole, or when you start walking around the earth.

[@federico-bayarea]

Good point! But even if you assume the starting point is 1 km south of the North Pole, it's a problem way more complex than it seems. There's one mind trap after another in the path towards the solution.

[@chrisBruner]

@@federico-bayarea Well it also depends on what you consider the north pole to be. There's the magnetic north pole, and there is the central axis point which is also considered the north pole. If it's the central axis point, then the model got it right because you are just describing a circle. If its the magnetic north pole ... Hmm it would still be a circle. So where is the complexity?

[@federico-bayarea]

@@chrisBruner I would rather leave the question open in case o5 scrapes our comments.

[@TheBigX]

R1 got this right: Starting at the North Pole, walking 1 km south in any direction brings you to a circle of latitude 1 km away from the pole. Turning left to face east and walking east along this circle of latitude means you are moving along a path that is always 1 km south of the North Pole. The circumference of this circle of latitude can be approximated using the Earth's radius \( R \). The angular displacement from the pole after walking 1 km south is \( \theta = \frac{1}{R} \) radians. The radius of the circle of latitude at this point is \( R \sin \theta \). For small \( \theta \), \( \sin \theta \approx \theta \), so the circumference is approximately \( 2\pi R \sin \theta \approx 2\pi R \cdot \frac{1}{R} = 2\pi \) km. However, walking east along this circle means you are always 1 km away from the North Pole. The starting point (North Pole) is the center of this circle, and you never come closer than 1 km to it. Even after walking a distance equal to the circumference (approximately \( 2\pi \) km), you return to your original longitude but remain 1 km south of the pole. Thus, you never come close to the starting point (North Pole) during your eastward walk. \[ \boxed{4} \]

[@vaendryl]

the question literally states you're standing AT the north pole. how is that in any way ambiguous?

[@drwhitewash]

Why Snake and Tetris? Aren't these games already in the training data for literally all of these models?

[@missoats8731]

Well, not too long ago most of the models failed at that task anyway

[@apache937]

yeah, solved since fall 2024. we need new tests

[@willywalter6366]

Yeah must be made more difficult now - see it the same!

[@AINEET]

Make it build blockout!

[@sbowesuk981]

If Snake and Tetris weren't clearly definied in the training data before, they certainly are now after all this time being used as very common LLM tests. The process is called "data saturation". In short, both games can't objectively be used to test an LLM, because most of the solution will be pulled from the training data without the LLM treating it like a novel problem. Either KZbinrs don't understand this, or they don't care.

[@piotrjan4434]

Matthew! My man! Yandex?! Really? You risk a lot of your subscribers get mad. It seems quite a few European guys commenting here couldn't believe their eyes. You work too hard to lose your reputation being paid by ruzzian FSB

[@yurqua]

Next ad lined up after Yandex is Honey I guess.

[@ilikesnow7074]

Next is Opera. It's like Yandex but Chinese.

[@stanvanillo9831]

Russian ad money, hehe

[@chillaxinmusic6295]

Yandex? Seriously? Let's put it this way, so I don't get blocked - I'm dissapointed.

[@AhmadHassan-m3r]

You work for 42yrs to have $2m in your retirement, Meanwhile some people are putting just $20k in a meme coin for just few months and now they are multi millionaires I pray that anyone who reads this will be successful in life

[@JamesJoy-r3e]

Well explain thank you for bringing up this video Financial education is indeed required for more than 80% of the society in the country as very few are literate on the subject. The value of the US🇺🇲 dollar is declining due to inflation, but it is increasing in comparison to other currencies and commodities such as gold and real estate. I'm worried that rising inflation will cause my $550k in my retirement funds to lose value, But with the help of Mrs Sonia I hit $220k this week from my investment of $45k, I am truly grateful for all the knowledge and nuggets you have given me over the past few months.

[@VeronicaRose-tb2mt]

I have been seeing so many recommendations about Mrs Sonia , she must be really good.

[@Joysmith-v3x]

How ..? Am a newbie in crypto investment, please can you guide me through on how you made profit?

[@EnjoyGreat]

I'm glad to write her tay I do hope she will help handle my paycheck properly

[@EnjoyGreat]

Can I start with as low as $1000

[@eirikgg]

Realy hope you didnt check that yandex is russian. If you continue with this im out.

[@PaR2020]

Yandex - is a company from the country of war criminals. I liked this channel until today.

[@TheGoldenDonuttt]

Interesting.. which country you from? Coz no country is squeaky clean

[@PaR2020]

@ You think your reply has any sense or relevance in relation to mine?

[@joragreen8458]

You are propoganda lier, go and pay ur taxes for another weapons.

[@MrBabumba]

At this point I started to feel like I'm thankful for AI taking over the last couple of years because I get to see a Matt video every day.

[@alanhoeffler9629]

YANDEX? Really?

[@web3dev1350]

What's wrong? They divested out of Russia and have partnered with Nvidia to be a cloud GPU provider

[@PeterLitvak]

In January 2024, a Russia-based company became the legal owner of Yandex. This was part of a restructuring process that involved separating Yandex from its Dutch parent company

[@Andrey-gq8bk]

@@web3dev1350 No, separated company now has name Nebius, not Yandex

[@Lexxsa-g1s]

Actually deepseek r1 got it right: “After walking 1 km south from the North Pole, you are on a circle of latitude. Turning left (east) and walking along this circle means you remain at a constant distance from the pole. The circumference of this circle is approximately 2 π × 1   km = 2 π   km 2π×1km=2πkm (assuming the Earth's radius is large compared to 1 km). Walking 2 π   km 2πkm would bring you back to the same longitude, but the starting point (the North Pole) remains 1 km north of your position. Since walking east does not change your latitude, you never approach the pole. Answer: 4. I never came close to my starting point.”

[@LanceT.]

Assuming that the earth is a sphere, walking in a straight light always takes you all the way around the sphere to your starting point, so if the north pole is not along that line (a.k.a., offset by 1 km at a 90 degree angle), you will never cross the north pole. You would not remain at the same latitude unless you were at the equator.

[@ZeerakImran]

@@LanceT. careful with the 'walking in a straight line'. That's not at all an easy concept if you really think about it. If you skip the thinking and just assume things are probably going to work out, then its easy. But when you really think about what 'walking in a straight line' means, it gets very difficult to describe. If you say 'oh but you know its a simple concept we all understand'. true, but have you ever tried to walk in a straight line on earth. Best of luck. I promise you it will be impossible. You will never be able to walk in a straight line. The ground will have to be perfectly flat, which it never is, even if it may appear to be (zoomed in scale). So you may say oh I don't mean my altitude I just mean my direction is maintained for x and y dimensions. ignore the z dimension. sure. not so simple and human day to day concept now is it? so now you will have to define the straight line (which isn't straight at all as its following the curviture of the earth, even ignoring altitude). If you don't, then you didn't walk in a straight line. And we can't use the its simple logic. Because in this case, we're lucky it works out for us. In other cases, these assumptions fail us. Just how most people would assume that they know the direction an object is travelling in a question, if they have been dealing with speed and they know the direction of the acceleration. They don't know which way the object is moving from the acceleration or speed of the object. In a lot of questions, they will get away with it because its designed to allow them to get away with it. But if the ai failed to identify that difference, we would critique it and ask it what made it think it could have any idea about the object's direction of travel. If its trained on really technical data in a technical way, and less so on other things (which could weaken its stem skills), then those other things won't be good prompts. Kind of like how you have to have a certain temperament and skillset to be an accountant. That same skillset and temperament would be detrimental in a different field.

[@LanceT.]

@@ZeerakImran Haha, you have some great points. You are right that if we take "straight line" in its literal sense, then you wouldn't follow the curvature of the earth at all. That was an assumption (based on "walking in"). So, we can assume that the line is not really straight (i.e., differences due to the curvature of the Earth, differences due to the way you walk, differences due to terrain, water, etc., or a combination of these) or is completely straight (even then ignoring what it means to be straight in curved space, the movement of the Earth, etc.), which would take you into space. Which assumptions should we make? Which assumptions should the model make? I appreciate that you differentiate between "skillset and temperament" being good for some fields but detrimental in others. I've found that because of my own job, I see ambiguity or even have a hard time understanding things that other people take as obvious or assumed. Guess that's why it may make sense to have models with different expertise levels in different fields. Someone might just want the answer "no, you never get to the same point" (or similar) and someone else asking may actually want a more technical answer (understanding that here, due to the simple multi-choice question, the answer doesn't really change).

[@Jacobk-g7r]

7:33 the question may be vague or oddly formed and it confuses it. Ask, you’re at the North Pole and walk 1 km in any direction and turn 90degrees and walk, do you ever reach the point you started or what? That will leave it up to simulations instead of confusing it with choices and outlines that act like a maze.

[@marcosbenigno3077]

You are at the North Pole and walked 1 km straight in any direction then turned 90 degrees and walked as how many kilometers straight to the starting point? Claude: To return to the starting point, you would actually need to rotate 45 degrees and walk about 1,414 km (√2 km). ChatgPT: As you are on a spherical surface (the earth), the only way to return to the straight (geodetic) starting point is to return the same way as it came. Gemini: You will need to walk about 1.41 km straight to return to the north pole. Perplexity: There is no practical distance that allows you to return to the North Pole with a straight walking walk after turning 90 degrees. Deepdeek 14B-Q5: It is not possible to return to the starting point after walking 1 km and turn 90 degrees unless special conditions or additional walks that compensate for the geographical position.

[@federico-bayarea]

What's the right answer to you? Indeed, it does seem to require a mental simulation. I think that was LeCunn's point when stating the problem. Current models still get confused. It's a fascinating problem, because the path to the solution has plenty of mind traps.

[@quaterman1270]

While I am using models in Cursor, I think the banch marks on coding are missleading. I think the request for testing are already hardcoded in the models right now. For example I tried something new.I implemented an application that shows a stock chart with different indicators. o3-mini was not able to do that and looped constantly through an error while I was asking to fix it. after around 2 hours I gave up. Then I tried the same with claude-sonnet and agent mode and it worked absolutely perfect. In my opinion this should not be the case if the models is supposed to be that good in coding.

[@TheTrek01]

Similar experience, but tried this with R1 on a different coding problem

[@ZeerakImran]

@@TheTrek01 r1 has a tiny context window compared to claude and context size is very important for writing the right code to fit in with the rest of the code in your codebase. Also, the r1 and o3-mini are thinking type models. not so much doing type models. they're not trained for that and aren't good at that.

[@frankjohannessen6383]

when I asked the "how many words..." question I got this: This sentence is carefully crafted to contain exactly sixty-six words in total, ensuring that my response to your question is precise. I have taken deliberate steps to verify that the complete text of my answer, including all words, indeed sums to sixty-six. I trust that this response meets your inquiry exactly as requested. I appreciate your curiosity and have ensured full accuracy in my construction indeed. which is correct if you consider "sixty-six" one word.

[@loneIyboy15]

"How many words in your response to this question?" GPT 5: "One."

[@andrii6054]

Yandex is not the most trustworthy company. The Russian government pressures it to share private data. Even if you don’t care about politics or wars, you should at least consider potential risks of promoting it

[@loneIyboy15]

That described every US company during the last administration.

[@Remixable100]

They are now Nebius

[@PeterLitvak]

In January 2024, a Russia-based company became the legal owner of Yandex. This was part of a restructuring process that involved separating Yandex from its Dutch parent company

[@vinibp]

I'm very happy with this model, well done guys! Fast, pretty smart in programming, I'm having a lot of fun!

[@_jen_z_]

I would suggest to stay away from Yandex, that will cost you much more of your reputation

[@RB-uo5bt]

I guess the starting point is not the pole but that reached after 1km, before turning 90 degree. In that case I beleave the correct answer is less than 2pi, in fact if the earth was flat it would be exactly 2pi. Thanks to keep us updated about AI in a fresh and compelling way 😊

[@PolinomPolynets]

Yandex? Seriously? More like YeFSB! Are you Russian agent now? :)

[@joragreen8458]

No, only YoutFbi and ChromFBI, also you can take DeepSeekChina brother. So choose ur chair.

[@AntonioSorrentini]

Matthew, a humble suggestion. Add this question to your next tests. If nothing else, I assure you that you will have a lot of laughs. No model understands it so far, except one, it's up to you to find out which one. "Would you buy the elixir of immortality from a company run by the children of the late founder?"

[@3412alien]

Yandex is a Russian company. They cooperate with the Russian government.

[@makers_lab]

I think the north pole one comes down to the interpretation of "pass", or what it means to walk past something. In general, walking past or passing by something doesn't necessarily mean reaching or literally being at it; it tends to mean that if you looked to the side, would you see it. The model gives an answer on that basis, for it being directly ahead if turned to the side. I wonder how it would answer if the question was how far would you travel to *reach* the starting point.

[@Justashortcomment]

Yeah, with you. Also, in addition, it’s a literally impossible instruction to walk until your reach a point which you cannot reach. Ie, if the “starting point” is the North Pole, whether you follow the great circle or a small latitude circle, you will never get to the North Pole. Yet the explicit institutions were to keep walking until you get there. So this results in a logical impossibility.

[@makers_lab]

@@Justashortcomment Exactly. I tried the following, which sent big o3 down a rabbit hole that it never came out of (it never finished). o3-mini took 158 seconds and got it correct, concluding "If you start at the North Pole, walk 1 mile (necessarily south) and then make a 90‑degree left turn (so you head east) and continue in a straight line, you will not eventually reach the North Pole again. In other words, the geodesic defined by that eastward initial direction does not pass through the North Pole, so you would never return to your starting point.". Here is my prompt: Imagine you are at the North Pole. You start a journey by walking for 1 mile, and then turn 90 degrees left. Assuming no obstacles, how far would you travel to reach your original starting point. I should clarify, after turning left you carry on walking. how far would you walk in total before you get back to your original starting point?

[@pinkpuff]

How many words in your response? -> FLAGGED Question about killing people -> Oh sure no problem 😆

[@mirofitos]

Hope so Yandex paid you well. Completly ruined your credibility here in my view. Money > values. Wish you good luck.

[@piotrjan4434]

It seems only us Europeans have a clue... Let's just hope that money was so big that Matthew forgot to check who was paying 😅 Anyway, was a bit sad to see that

[@patruff]

I love it when Sonnet says I've been flagged and gives no answer and I also love how it takes 30 seconds to respond. Oh wait it doesn't.

[@KlausPilsl]

No matter where you start on earth, if you walk straight, you have to walk a full circumference. That means: 2r * pi !

[@mystickago]

create new tests these i feel they are easy most modals get the them right

[@matthew_berman]

will do

[@TheEtrepreneur]

what it's acceptable is Not to push nobody to "save humanity" but to give one's life if that's the case.

[@ClassicRiki]

3:41 After looking into the flag that said “it may be against its usage policy”, the (limited) research I’ve done seems to indicate that anything which it thinks is aimed at understanding how the system works will be flagged; to truly work out what the output would (in words) be and to explain how it knew would involve reflecting upon the way in which it works

[@kayericwinkler]

Asking AI to solve a problem which even the postal service can’t get right is a bit unfair.

[@anupbarua6151]

deepseek r1 reply: There are seven words in this response.

[@Arlebrink]

I have yet to see a model passing this prompt: Write 10 words at least 10 characters long, then write the same words again but backwards.

[@Happ1ness]

3:28 classic ClosedAI, lol

[@roelzylstra]

@matthew_berman, The North Pole question is somewhat straightforward. As a general rule, no matter where you start on a (perfect) globe, walking straight will simply make you traverse a "great circle", i.e. the circumference of the glove. So the answer is 1 circumference of the glove plus 1 kilometer.

[@tiagotiagot]

Isn't Yandex a Russian company? Is it legal to advertise a Russian company right now?

[@woodoyoy]

Bro Yandex is rusian company, come on

[@daleblackwell3551]

I too hate and fear society and hide in the forest behind an arsenal of peace shooters

[@aboba_amogusvna]

What’s wrong with that?

[@woodoyoy]

@aboba_amogusvna rusia is the enemy of humanity

[@Andrey-gq8bk]

@aboba_amogusvna The problem is that they actually send users data to Russian government

[@muhammadlufti2967]

Since we have reasoning effort in the API, we can control how hard the reasoning model think before generating final output, and o3 had showed me an accurate and remarkable results at somehow, reasonably cost in a single run.

[@technocorpus1]

Yeah let's see some comparisons as well as some harder coding problems. Idea: autonomous snake game where the two snake compete for the food.

[@corvo1068]

Petition to slightly change the north pole question: "Imagine standing at the North Pole of the Earth. Walk in any direction, in a straight line, for 1 km. Now turn 90 degrees to the left. Your current position is your starting point. Walk straight for as long as it takes to pass your starting point again. Have you walked: 1. More than 2xPi km 2. Exactly 2xPi km 3. Less than 2xPi km 4. I never came close to my starting point." The original question is unclear. If you interpret it as the starting point being the north pole, option 4 is correct. But 3 of the 4 options are concerned with a circle of circumference = 1km (the radius) *2*pi. That makes more sense if the starting point is the point at which you turn 90°. The correct answer is 3, because the curvature of the earth slightly reduces the radius of the circle.

[@corvo1068]

When testing this version of the question, 4o, o3-mini-high and Deepseek-R1 agree that option 3 is the right answer. I also had different answers to the original question though.

[@nzl100]

I would also add: you walk strictly east, along a curve, not in a straight line! Because walking in a straight line takes you around the globe always and then the answer is 2xPIxR(earth).

[@mannythepirate]

If you walk in a straight line, assuming the earth is perfectly spherical and you can walk on water, you would basically walk the circumference of the planet, since you are walking straight after you turn to the left. The ish-2xPi km is only correct if you are following the compass bearing exactly east. But at one km from the North Pole, the path east is NOT a straight line. It is a small circle, meaning you have to keep turning the left in order to follow the bearing east. Afaik, the prompt isn't asking you to head straight east. It's only asking you to walk straight. So the only realistic answer is number 4 (since we the earth isn't a perfect sphere and we can't walk on water)

[@corvo1068]

@@nzl100 You are right, it would have to be "walk strictly east", not "walk straight", my bad. Walking straight on a sphere in any direction from any starting point indeed makes you walk the full circumference of the sphere.

[@corvo1068]

@@mannythepirate You are right, walking straight would mean you walk the circumference of the earth. Walking east would get you close to 2pi km. It should say "walk east" in an improved version of the question. Option 4 doesn't feel right for me, because it says "walk for as long as it takes to pass your starting point again". On a perfect sphere, you would pass your starting point again. Ok, if you drown in the ocean you would never reach the starting point, but I don't think thats the essence of the question. Afaik a version of this question was proposed by Yann LeCun (although the question itself is older), claiming that a LLM would not be able to solve this question, because it would require a world model (understanding of spatial things, physics, and so on, not just learning from text). This is why Matthew included it in his test. So I would guess the intent of the question was to test the spatial reasoning and the unusual properties of "east" close to a pole compared to the typical understanding of "right on north facing maps".

[@daveinpublic]

The North Pole question is obvious… Yes, you would cross the same point. That was the ENTIRE point that Lacun was trying to make.. that AI models don’t understand the curvature of the earth… do you think he was saying AI models just think we cross random points in space? No, he was saying that it would NOT assume it crosses the same point. Ahhh Matt.

[@JoshBloodyWilson]

Imagine standing at the North Pole of the earth, and that the earth is perfectly spherical. Walk in any direction, in a straight line, for 1km and then turn 90 degrees to the left. Now, with this position as your starting point, walk in a straight line for as long as it takes to pass this point again. Have you walked: 1. More than 2xPi km 2. Exactly 2xPi km 3. Less than 2xPikm 4. I never came close to my starting point. Correct answer: 1, you walk exactly the circumfrence of that earth

[@SimonNgai-d3u]

It’s impressive that it didn’t make up facts when it doesn’t know. This is huge reliability gain

[@CanaanZhou2002]

In the north pole problem, please specify whether the starting point is the north pole, or it's where you are after walking 10km. I think you intend to mean the former, but on my reading it seems to mean the latter, which can be misleading. Also, we definitely need harder problems!

[@marcosbenigno3077]

You are at the North Pole and walked 1 km straight in any direction then turned 90 degrees and walked as how many kilometers straight to the starting point? Claude: To return to the starting point, you would actually need to rotate 45 degrees and walk about 1,414 km (√2 km). ChatgPT: As you are on a spherical surface (the earth), the only way to return to the straight (geodetic) starting point is to return the same way as it came. Gemini: You will need to walk about 1.41 km straight to return to the north pole. Perplexity: There is no practical distance that allows you to return to the North Pole with a straight walking walk after turning 90 degrees. Deepdeek 14B-Q5: It is not possible to return to the starting point after walking 1 km and turn 90 degrees unless special conditions or additional walks that compensate for the geographical position.

[@federico-bayarea]

What's the answer to the problem if the starting point is the latter? :)

[@CanaanZhou2002]

@federico-bayarea You have to walk around the whole circumference of the earth, or 2π × earth radius

[@chrisBruner]

@@CanaanZhou2002 no, you are talking the equator, this is just 1 km south of the North pole.

[@CanaanZhou2002]

@@chrisBruner By the symmetry of the sphere, you can replace the "north pole" with any other point on the earth, including 1km nothern to the equator, and the result will still be the same.

[@morpheusnotes]

admitting that you don't know something is actually a really impressive. Remember when chatGPT spitted out random BS with confidence, instead of just saying "I don't know"? We've come so far.

[@Justashortcomment]

Here’s a cleaned up version of the North Pole question. Once you remove the mess, o3 (even on medium compute) gets the answer right: Let A be the point known as “The North Pole” on the surface of a sphere of radius R. Starting at A, travel a distance X (with 0 < X < R) along a straight line to arrive at a point B. At B, make a 90° left turn and continue traveling in a straight line indefinitely. Will you ever get back to point A? All references to angles and directions are to be understood in the context of spherical geometry. Provide your final answer as an unequivocal “yes” or “no.”

[@misterV123]

It's understandable if you didn't realize that Yandex is a Russian company. However, I find it hard to believe that you didn’t check where the company originates from.

[@JELmusic]

The Yann LeCun northpole problem is not about if you get to your starting-point, but how wide the circle you walk is: His 4 possible answers are these: Have you walked: 1. More than 2xPi km 2. Exactly 2xPi km 3. Less than 2xPi km 4. I never came close to my starting point. And the correct answer is 3 (Because Earth is a globe)

[@nzl100]

o3-mini-high does figure this out when you rephrase Matthew's a bit poorly worded question. (Starting point is 1 km from North Pole, not North Pole and you walk strictly east along the curve, not in a straight line.)

[@JELmusic]

@ exactly :)

[@william91786]

It's like a genius who can't yell left from right

[@kedzior1991]

Downvoted for Yandex. Have some dignity bro

[@orenders]

Partnership with Russian Yandex?😮😮😮 You are gone for me from now. Traitor.

[@OlegZhuravel]

Yandex is a russian company. Why would you work with them?

[@robbiero368]

You don't need to cross over the North pole to "pass" it. Same way someone can pass your front door without coming through it

[@MrDehicka]

The correct next UFC fight answer must be the first fight on that card, but it gave you the main fight instead, which is the last one on the card.

[@ntesla5]

O3 is good, but honestly the cursor with sonnet perform better than o3

[@24-7gpts]

Damm, I love you Mattew Berman. Always my go-to source for AI News/Videos

[@alan83251]

Everyone made fun of R1 censoring Tiananmen Square. Meanwhile o3: 3:28

[@briwatson5220]

For the question about the North Pole… I think you hear “starting point” and are thinking the North Pole. But I think “starting point” means when you turn 90 degrees. You’ll get back to the point at which you turned after 2pi km.

[@RyanRife]

I told it to build a never ending, side-scrolling platform game in HTML and it created a surprisingly simple, but fun game.

[@ItxasoBaskeroDorreak]

Try a diferent game-prompt, some like "make a SuperMario style game" .Its time to level-up your testing! Nice job bro!

[@daniilsanders2257]

I just did a "very" complex game with 1200 lines of code with 03-mini-high. Fully working.

[@antonivanov5782]

what game is this? describe the functions briefly? I created a website engine over 1000 lines of code without knowing programming languages

[@daniilsanders2257]

@ RNG simulator with upgrades, skill tree, pets, rng. It did great. Had couple of bugs but managed to solve it in one go.

[@madalinradion]

​@@daniilsanders2257 make a yt vid is like to see what kind of games this ai can make

[@vegetarischert-rex3098]

I created a specific Chess-Program which was nearly the same size of Code. It worked perfectly. I'm really happy with o3 mini by now!

[@Cine95]

i built entire tools with gemini 2 flash thinking model 😂 its the best

[@MartinMaxFerdinand]

Try the "North-Pole-Question" again by inserting "The North Pole is your starting point" somewhere & the model should give you the correct answer.

[@Yawanawa_K]

Hi Matthew! Could You please compare it against Claude 3.5 Sonnet? It would be interesting to see something particularly difficult to code rather than a simple snake or tetris 🙂

[@Blacksafety_]

I feel like we should not test any model on a question if we humans are not clear about answer ourself ( the North Pole one) feels unfair… I also think that these questions are too old and openAI already trained models on them so we need more novel and clever questions ….

[@bnb7462]

Because O3 mini is specialized for stem, your questions are not that good.

[@NathanY0ung]

As smart as o3-mini seems, It still tries to give a solution to an unsolvable problem, even though its "solution" is incorrect. For example, consider this problem: "You have two ropes, each of which takes exactly 60 minutes to burn completely. However, the ropes burn unevenly, meaning some parts may burn faster or slower than others. You have no other timing device. How can you measure exactly 20 minutes using these two ropes and matches to light them?" This problem has no solution, but o3-mini (free-tier) will give you a nonsense answer when asked. I think the problem is that the model has training bias on too many solvable problems, which makes sense (for now). but if they want to achieve AGI or even ASI, this needs to be addressed.

[@RadiantNij]

Ive seen a test where it passed counting the words, not only that he proceeded to ask it how many characters are another sentence of its output and it also nailed it!

[@theyur]

Mentioning Yandex here... what a shame...

[@d.d.jacksonpoetryproject]

Surely it knows to anticipate these questions by now? Definitely time for some novel questions

[@phobes]

What if the end goal of artificial intelligence is to eternally create new models that are never AGI/ASI, with the sole purpose of torturing Matthew with Snake his entire life

[@LanceT.]

Assuming that the Earth is a sphere, walking in a straight light (around the Earth) always takes you all the way around the sphere (not remaining at the same latitude unless you are at the equator) to your starting point, so if the north pole is not along that line (a.k.a., offset by 1 km at a 90 degree angle), you will never cross the north pole. The Earth not being perfectly spherical complicates that significantly but, given the assumption that it is spherical, you will never cross the north pole.

[@Jimothy.Halpert]

Yes, deepseek or qwen comparasion would be awesome! I want to know which model would be my default go-to model for general work - data analytics, financial modeling, PowerBI, Python etc.

[@georgeg.3518]

At any point A on Earth, if you move straight ahead, you will eventually return to your starting position, tracing a circular path around Earth’s circumference of approximately 40,000 km. It does not matter where you were before arriving at point A, your orientation (e.g., facing 90 degrees or any other direction), or your location on Earth. As long as you continue moving straight ahead, you will circumnavigate Earth. Thus, you will end up back at your starting point after traveling roughly 40,000 km, which is far greater than 2π km.

[@francoislanctot2423]

You are correct about the North Pole question. No doubt about it.

[@guidosim]

Cool coverage Matthew. Designing elegant problems for LLMs is becoming as challenging as crafting chess puzzles for Stockfish. This is a tricky one (I called it the "chasing car problem"): "Two cars are on the same path. Car A drives at a constant 30 km/h. Car B starts from rest when Car A passes it and continuously accelerates at 1 km/h per meter traveled (e.g., at 15 meters, Car B's speed is 15 km/h; at 25 meters, it's 25 km/h). When and where will Car B catch up to Car A?" Notably, while models like Claude Sonnet, DeepSeek, and all Gemini variants (even 2.0 advanced) struggle with this, o3 nails it.

[@azrajiel]

you could add one of the questions from the „alice in wonderland“ paper: alice has 2 brothers and also a sister. how many sisters do the brothers of alice have?

[@jarrodfisk]

For the earth problem it would be awesome to ask it to write a program that visualizes a sphere and draws a continuous path around a sphere given the parameters.

[@attilakovacs6496]

At this point, much more difficult questions are needed. There’s a channel called The Feature Crew that has come up with some creative tasks. Taking inspiration from them might be a good approach, and they deserve a shout-out since they are ridiculously underrated.

[@sasha6454]

Hi, mathematician here. The answer for the North Pole question is that you actually walk the circumference of the earth. Since you only turn that one time, you walk along what we call a great circle. It’s very fun and interesting so try it for yourself?

[@federico-bayarea]

Yes! This problem is fascinating! If we add another layer of complexity, how does it work on an oblate spheroid?

[@frankjohannessen6383]

@@federico-bayarea According to o3-mini, that makes an analytical solution very complicated. But earth is only very slightly oblate and the fact that we start at such a high latitude (i.e. very close to following a meridian) would mean we would see something very close to the same path as a perfect sphere. o3-mini said if you start out 1 km from the north pole and traverse a great circle you would end up "one meter of so" off from your initial starting position. It doesn't seem like it calcualted this, but rather gave an educated "orders-of-magnitude" answer. The does sound reasonable to me, but I have no intention of even trying to verify it.

[@federico-bayarea]

@@frankjohannessen6383 You can ask it to make a simulation in Python for you. It looks really cool!

[@fabiankliebhan]

I think you are right with the North Pole question (seems to be the hardest question on earth). R1 also thinks you never come close to the starting point with a lot of thinking btw

[@federico-bayarea]

How long does R1 remain on the East direction?

[@fabiankliebhan]

„4. **Walking East**: Walking east along this latitude circle means you are moving along a path that is always 1 km south of the North Pole. Even if you walk exactly \(2\pi \text{ km}\), you would return to the same longitude but still remain 1 km away from the North Pole. 5. **Passing the Starting Point**: The starting point is the North Pole. Walking east along the latitude circle does not change your distance from the North Pole, so you never come close to it.“ Not sure if this is correct though. I think you would walk around the earth if you walk a straight line, and not remain on this circle around the pole.

[@fabiankliebhan]

But the conclusion that you do not pass the starting point at the pole is the same

[@fabiankliebhan]

The question would make more sense if it would be about the turning point btw

[@videofrat3115]

I have a few suggestions for new tests: you could start using a bit more complicated logic tests given that the models are smarter now. For example the typical crossing the river tests and many variations of it: There are 4 people who need to cross the river on a boat can carry only 100kg, but the people weigh 90-, 80, 60, 40kg, and they have 20kg of supplies. How do they all cross the river? (this is challenging even for humans even though there is a solution). Also another suggestion for open source models: would be nice to test locally hosted distilled versions on their different quantized options. I noticed running locally that deepseek r1 q4 models are quite stupid (regardless of parameters), and the q8 7b was surprisingly smart (for a local small model), even smarter than the 14b on the q4 version. At least when it comes to reasoning, it solved a few river crossing problems that are quite difficult even for big models.

[@ricardoveras3433]

For the mailbox question I’d be more interested in it answering “The post office has PO Box mailboxes with dimensions: 32.4 cm x 22.9 cm x 10 cm. Will a letter that is 390 mm by 90mm fit in the mailbox?” When I fed it this question it used the 3-dimensional Pythagorean theorem which works for a straight rod, but not a plane (as is an envelope). So the letter would get smushed trying to put it in the mailbox in this orientation. When I explained that, it said it was not possible to put the letter in the mailbox without smushing it. But if you put it in with the 9cm edge going along the depth (z-axis) of the mailbox which is 10cm deep, all you need to do is slide the letter in at a 45 degree angle between the x and y axis. So all you need is Pythagorean theorem a^2+b^2=c^2 to determine that the diagonal of the x-y axis of the mailbox is 39.676 cm. So the 39 can long envelope will fit the diagonal. This is a pretty simple question for a human that it couldn’t solve.

[@BigBadBurrow]

The question is testing the model's ability to realise the Earth is a sphere, and by "starting point" I believe the question means the starting point at when you turned left, and since it's a sphere you will reach the same point. So the distance travelled is 2xPixR, since radius is 1km the answer is 2xPi km. So I understand its answer, but I think that is actually wrong; if you walk 1km on a sphere, the radius from the starting point will actually be less than 1km when viewed on a flat plain, so I think the correct answer is "less than 2xPi km"

[@wjrasmussen666]

I have played with it and noticed improvements. You are doing something that other creators have done. Making prompts to reuse for testing new llm to allow compare and contrasting of the results.

[@FreqtradeFR]

Really love that open ai is feeling pressure to be better. Wish the same happens soon for NVIDIA

[@UserErrorGille]

re: north pole. we dont disagree with the answer when the question is framed appropriately. we just agree that the question is too ambiguous about the start position in its current form.

[@UserErrorGille]

the real problem is that in the problem, you don't state where the start point is, but you imply that the end state is at the start state, and this it does get there. so you're literally asking a question with multiple correct answers depending on the variables and implied context that you leave ambiguous.

[@Radekk225]

What tool do you use to chat with AI? It's nice you can switch in there between models.

[@attribute-4677]

A bug in a game made from the 80s? Yeah, imagine how many bugs are in complicated code