AMD says these are the same... We DISAGREE. - Testing 12 of the same CPUs for Variance

Remove your personal information from the web at joindeleteme.co... and use code LTT for 20% off.
NexiGo is hosting product giveaways every month! Enter for your chance to win here: viralnation.li... (Contest is open to anyone 13+ in the United States)
Do you really know if your CPU is performing the same as the ones we review? We don’t know. But we know that if we want to increase our testing capacity, we need to PARALLELIZE. But that means we need nearly identical test benches. And trying to make that happen sent us down a far deeper rabbit hole than we could have anticipated.
Testing Process Doc and Forum Post: linustechtips....
Buy an AMD Ryzen 7 7800X3D CPU: lmg.gg/s8Y1K (Canada: lmg.gg/yZnVo ) (Micro Center: lmg.gg/EkzYG )
Buy a NVIDIA GeForce RTX 4090 FE Graphics Card: geni.us/Won6A
Buy a GIGABYTE X670E AORUS XTREME Motherboard: geni.us/FADHuEG
Buy G.Skill Trident Z5 NEO 6000MT/s CL30 2x16GB DDR5 RAM: geni.us/TYRaJf
Buy a Samsung 980 Pro 2TB NVMe M.2 SSD: lmg.gg/aL0BL
Buy a Noctua NH-D15 CPU Cooler: geni.us/lK6GI9O
Buy a MSI MEG Ai1300P PSU: geni.us/NRCy
Purchases made through some store links may provide some compensation to Linus Media Group.
Purchases made through some store links may provide some compensation to Linus Media Group.
► GET MERCH: lttstore.com
► GET EXCLUSIVE CONTENT ON FLOATPLANE: lmg.gg/lttfloa...
► SPONSORS, AFFILIATES, AND PARTNERS: lmg.gg/partners
► EQUIPMENT WE USE TO FILM LTT: lmg.gg/LTTEqui...
► OUR WAN PODCAST GEAR: lmg.gg/wanset
FOLLOW US
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
Twitch: / linustech
MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova
Video Link: • [Electro] - Laszlo - S...
iTunes Download Link: itunes.apple.c...
Artist Link: / laszlomusic
Outro: Approaching Nirvana - Sugar High
Video Link: • Sugar High - Approachi...
Listen on Spotify: spoti.fi/UxWkUw
Artist Link: / approachingnirvana
Intro animation by MBarek Abdelwassaa / mbarek_abdel
Monitor And Keyboard by vadimmihalkevich / CC BY 4.0 geni.us/PgGWp
Mechanical RGB Keyboard by BigBrotherECE / CC BY 4.0 geni.us/mj6pHk4
Mouse Gamer free Model By Oscar Creativo / CC BY 4.0 geni.us/Ps3XfE
CHAPTERS
---------------------------------------------------
0:00 Intro
2:25 Why Same Model ISN'T Same Performance
5:32 Sources of Variance
8:21 Gaming Results
10:43 CS:GO is Wonky
11:41 Gaming Results Cont.
12:39 Productivity Results
13:59 Our Selections
15:15 Testing Mobos and RAM
17:16 Final Discussion
21:26 Outro

[@XenFPV]

This is LTT using their higher budget compared to most tech channels for something genuinely useful. This is really good to see! Well done LTT, this deserve genuine praise.

[@zypher5876]

Gamers nexus did the same thing a while ago

[@joaomendes3907]

@@zypher5876 Gamer's Nexus not listed as other media. Funny

[@raawesome3851]

@@zypher5876 yeah, true. But Linus has a far greater scale than Gamers Nexus, as of now, reviewing everything from CPUs, GPUs, laptops, etc, so it's cool to see that level of standardization on a large scale to make sure the testing is consistent.

[@frankwong9486]

Well , those overclocker do lots of binning and probably run through more samples than the average LTT benchmark review

[@raawesome3851]

@@joaomendes3907 I'm pretty sure that's not out of malice, since Linus' not the editor for it 🤷‍♂️

[@Joshfarmpig]

glad to see factorio used as a gauge for cpus

[@Phil_AKA_ThundyUK]

As it should be lol it's mental intense on ticks. Causes some real aggro on multiplayer systems with different spec.

[@Hugs288]

haha, guage, i get it

[@demondoggy1825]

Factoriohno.

[@whatistruth_1]

The issue is all of these reviewers are doing the factorio benchmark wrong. The game has a 60 tps cap as that's what the engine runs at. What needs to be tested are heavier maps that push the cpu BELOW 60. These above 60 tps benchmarks are as good as a synthetic benchmark in that they're not realistic. And yes, results absolutely do change and so does the cpu hierarchy.

[@nepnep6894]

​@@whatistruth_1not to mention Factorio scales more with memory performance than core perf or cache.

[@spacen3rd950]

As a data scientist, I absolutely love this video and greatly appreciate this perspective. The silicon lottery is absolutely real, and sample size of one is far from sufficient. Glad to see someone doing more in depth analysis. I don't expect the methodology of ltt labs to dive into frequentist vs bayesian stats, but it would be interesting to use public benchmarks (for an incredibly nerdy perspective/audience) as a prior distributions and see how the results differ. Regardless of the depth of it, just seeing more stats in hardware/software benchmarks is a fantastic breath of air. Keep up the good work!

[@ChristianRoseroF]

La muestra es real, en el consumidor final la variación será más grande pero le darán la culpa a los otros componentes como cantidad de ram, ssd u otros factores, AMD solo garantiza ka velocidad del reloj en ciertss condiciones, en el hogar la temperatura ambiente también ocasiona ruido en los datos. Lo cierto es que no hay un tercero que obligue a cumplir un estándar legal porque es complejo medir que los productos sean iguales

[@kytechnelson]

Agreed! Would also be interested in seeing a bayesian analysis with public benchmarks as a prior (though I'm also one of those nerds who would love more of this kind of stuff). I also appreciate how they detailed their similarity metric of using euclidean distance, along with explaining what that looks like in a higher-dimensional space for those not accustomed to seeing it used that way since it's such a common approach for any sort of clustering problem.

[@jannejohansson3383]

When public is put in test, there's always possible cheating or possible than manufacturer goes in game too.. if you understand what I mean..

[@diegoberan7883]

LTT should hire a data team to take care of all this data. It would be great to have it properly stored and analyzed, maybe you guys could ge into testing neural network performance of GPUs too. I find it hard to find data on how well GPUs run AI tasks.

[@kytechnelson]

@@diegoberan7883I second the request for GPU benchmarks for AI tasks. Looking at some of what Puget Systems has done for their AI/ML benchmarks would probably give a good starting point for the LTT team.

[@4RILDIGITAL]

I never realized how much work goes into ensuring consistency in benchmark testing. It's worrying to think about the lack of oversight in computer hardware compared to other industries and the issues that could ensue.

[@the_undead]

Now imagine real scientific testing and trying to get perfectly consistent results and not within 0.25%

[@michaelbailey9857]

As nice as it'd be, it kind of admittedly makes sense to an extent - Cars aren't tested and regulated because they're expensive, they're tested and regulated because they can kill people

[@squatzandoatz92]

i love the fact that you come out and say "yes, our tests are inconsistent, and here's why" in such a fascinating, entertaining fashion. there's a reason i always watch every LTT video that pops up in my feed.

[@AK-tf3fc]

you do know gamer nexus is better then this

[@lilcheaty]

​@@AK-tf3fcnot comparable lmao, this video specifically shows us a big difference in gamers nexus testing vs ltt

[@pituguli5816]

@@lilcheatyIn that case lets throw all reviewers performance results out the window because of a few percent variance, I do agree with you though there is a huge difference in GN and LTT testing and that difference is GN can at least be trusted with their numbers but LTT cannot. "Look guys there is a variance so its not our fault we put out shonky reviews and fudged numbers". Silicon lottery has always been a thing but of course Loonus spins into this to obfuscate the fact their testing methodology is bogus at best. Wanna see a good correction video? HUB's latest video where Steve apologised for his mistakes and moved on, LTT "Silicon lottery cause number to go boom".

[@the_undead]

​@@GsrItaliabecause gamers Nexus has to the best of my knowledge never came out and explicitly confirmed their testing method and if they have, it's not even comparable to this, no one else in the industry is doing what LTT is doing because they don't have the money to just buy 11 top of the line. CPUs and at least 10 4090s just for data accuracy reasons. This isn't necessarily a knock against anyone else on this platform, they just don't have the money to do this

[@Lurch-Bot]

Yeah, how many times are they gonna have to say it before they realize they don't understand scientific method?

[@smayaan2479]

Genuine praise to linus for taking time to address this and how confusing and weird these companies are becoming launching their cpus and gpus in today’s market

[@Blast_HardCheese]

The silicon lottery's always been a thing. I didn't think it would still be this bad though, and it's nice to see it tested

[@FcoEnriquePerez]

Yeah, this video was like the old LTT content we used to get.

[@n_core]

@@Blast_HardCheese I think it's always this bad but the manufacturers just locked the maximum potential performance so then it won't vary once it reaches the end consumer (so your typical consumer won't feel aggrieved because of it,) and you can always unlock it through overclocking. But nowadays they use a different sophisticated method which almost makes overclocking obsolete and it is already unlocked for the end consumers, so we can see the various results between each chip.

[@alexandruilea915]

​@@Blast_HardCheese It's not that bad, all they had to do was find the maximum frequency that all 3 CPUs could run at and fix them at that speed. Then the testing should result in minimal difference. For customers, you should be glad if you get a golden sample and don't care if you didn't because you got what you paid for which are the guaranteed speeds. Everything over the guaranteed speeds is a bonus that you may or may not get.

[@Blast_HardCheese]

@@alexandruilea915Agreed, I just assumed they would have it fully reined in by now

[@thegamechanger7352]

Seems like Gamers Nexus did not make the list of media outlets in 17:24

[@Capellix0001]

LTT Saltmine be like

[@Not.Your.Business]

LTT staying "classy"

[@yutt]

Why would they? Gamers Nexus is a backpack warranty review channel, not technical.

[@nomadicdragon7157]

Did I miss something between Gamers Nexus & LMG?

[@thegamechanger7352]

@@nomadicdragon7157 umm.. assuming you are not joking, just look up the accusations against LTT on Gamers Nexus channel about 5 months ago. Shit went down

[@kaizen_unknown]

I'm glad to see linus and his team working towards doing better, especially after that whole thing where they were admitting lack of accuracy and that they failed their community. Seeing them be so passionate about this really brings a smile to my face.

[@the_undead]

You say that like they're the only ones that don't have perfect accuracy, You say that like putting in the tens of thousands of dollars in equipment and man hours is something that they can just do overnight, You say that like gamers Nexus was at the time he made that disgusting disingenuous video was in the habit of retesting products to use that data in comparison charts for new product launches. But no, none of those things are true and I still don't think gamers Nexus retests. Although if you can provide me with real evidence that he does retest stuff I will happily change this opinion

[@mllrman08]

GN doesn't have the money to do real hardware testing at any kind of scale anymore. Steve himself admits that they put a huge deal of money into the "investigative journalism" they do. When most of his sources we just have to take his word for it because "they cant be revealed" which is fine. Its great that he focus hard on shit companies doing shit things. The the recent EKWB crap, Yes they are garbage, yes the management is terrible and they deserve everything coming. But be honest the only reason his Tech news is successful is because it gives people someone/something to hate. Also, the one thing in common with his investigative videos? He always contacts the company for comments prior to the video even being released. Did he do that with LTT...No. He took what he could from LTTs videos and forums and put them on blast for people to hate, gaining him views. The world today thrives on hatred and in some ways Steve is using that for his own gains. So yeah everyone on the internet, LTT or GN play on peoples emotions and thoughts to get your views. None of them are innocent. but also none are guilty. If LTT failed so did GN. One for testing data the other for using hate to garner views.

[@brendan9335]

Incredible leveraging youtube videos to finance an independent lab to preform tests on all kinds of tech hardware and then making vids showing and explaining all of the data collected which in turn can finance the next round of tests. Bangin job LTT I love it.

[@Lurch-Bot]

Yeah, they have the equipment but no clue how to use it properly, as shown in this video. They pulled 11 tray CPUs, probably from the same tray. Does not reflect the consumer experience at all. Of course enterprise hardware will be more consistent, even if it is going in a consumer product. Imagine if a company like HP had people doing side by side tests of the same model and getting a 20% variance in benchmark scores... In other words, the major manufacturers of complete systems get the best and what goes in the consumer boxes is like a random grab bag of everything else that wasn't good enough for HP or Dell. This is why I benchmark test new CPUs and GPUs and have and will send them back solely for unreasonably low benchmark scores. If everyone did that, it would be a completely different market and the variance would become statistically insignificant for everyone, not just HP, Dell and LTT. Oh, and don't think you can get around it buying a tray CPU from AliExpress. Those will be bottom of the barrel, just consistently so. You're a dreamer if you think that was some leftover from HP's order. Those were ordered by some no name company with the name produced by randomly mashing a few keys on the keyboard and were binned and priced accordingly. I'm sure you can buy 5600X CPUs there all day long that are really a 5500. And it is AMD that is the one running the scam there.

[@ElNeroDiablo]

@@Lurch-Bot Did you miss the part where 11 of the CPU's came from 11 different stores across 3 countries (Canada, USA, and looked like UK)? These 12 chips (including LTT's pre-launch test sample) are going to be from different batches across the 6-month window they were collected, which will emphasis the differences between wafers let alone between chips on the same wafer.

[@treborrrrr]

@@Lurch-Bot 22 seconds from the start of the video, that's how long it took for them to tell you how the CPUs were acquired, and you somehow still managed to make this comment. Wow.

[@MariuszChr]

Ummm, that's literally how businesses work. Spend money to make something, sell it so you can make more, and sell more.

[@Axeiaa]

@@Lurch-BotYou're in many statements. 1) They're not from the same tray, they're from various sources, even from various countries. 2) Companies that get AMD products are more likely to remedy this by restricting performance rather than only getting 'the good stuff'. You're talking about companies that often don't enable XMP, Only use one stick of RAM for dual-channel supporting platforms AND often use RAM that is slower than what even the CPU manufacturer recommend (although that last point has gotten better in the last few years). Don't believe me, just see anyone looking into this (LTT, Gamer's Nexus, David does Tech Stuff etc). Not to mention insufficient cooling which can have a i9 performance than an i7. Consumer level stuff is absolute trash from companies like HP and Dell. Workstation level it's just okay, Server level you're actually getting some sampling/additional testing/verification but you're paying through the nose for it. 3) It's within your right to send things back if you don't like them but you're basically finding your own golden sample that way, if everyone started doing the result would not be what you suggest. Instead they'd be forced to lock down things to the absolute minimum performance any chip can do. 4) There really isn't any sampling as deep as you think there is. Good chips go towards higher end products bad chips go towards lower end products. But lower end is higher volume so high quality chips can end up there just because there isn't enough demand for the high end. (They might even do a rough pre-selection and only select the chips originating from the middle of a wafer for the high end low volume stuff to avoid having to test the rest in depth. The middle of the wafer has a much better chance of churning out high quality chips). 5) AMD isn't running any scams and Aliexpress CPUs can be of just as fine quality as anything else nor are tray CPUs any worse or better on average from any testing I've ever seen done.

[@TechDaddyFr]

Looks like the transparency we wanted. You clearly stated the challenges, how you worked around them and the variables you can't really control. It was very interesting, at least to me. Good job guys!

[@cheredia13]

Statistically,from experience in semiconductor industry, It will take at least a sample size of 30 cpus to establish proper sigma. Ideally you would want to do this experiment with 1 motherboard first and repeat with other boards to gauge variation. The next step would be to test about 100 cpus from different time stamps (hopefully getting different lots),to gauge the variability of their process.

[@vyor8837]

Also at JEDEC to remove differences in memory controller overclock quality.

[@sindrisuncatcher653]

Of course those hundred CPUs are gonna cost about $35,000... for every model of CPU that you test.

[@the_undead]

Even this relatively small scale test was probably a lot for LMG So as much as I'm sure Linus would love to get to that point, they're a long ways off from having that kind of money

[@Grandwigg]

I love seeing this kind of testing across multiple channels. And another shining example of why you should always check across multiple review sources. (Loved the Pokemon referencing of the processors)

[@Grandwigg]

(and I've only known a couple years now that Raikou is long o sound (ish). And i have the gbc Crystal bundle. Best Christmas ever, lol)

[@Just_another_tom]

As PhD researcher, this is my favorite video that you guys have published in a while. Love the data and scientific process used here guys. Keep it up, go labs go!!

[@zaxbit]

Waiting for the first LTT Labs paper to hit Science

[@shinjisan2015]

It's great to see. All this work will make sure their internal comparisons are reliable. You can look at all LTT Labs testing when you want to compare DIFFERENT products for performance and know it's just the products you're looking at that vary, then look across different reviewers testing for the SAME products to get a glimpse the actual market quality variances for that one product. LTT Labs is actually benefiting the whole techtuber community, not just LMG.

[@chairface1859]

lol "PhD researcher"

[@TheRockybulwinkle]

@@chairface1859yes, one who researches PhD candidates 🧐

[@grumpyratt2163]

​@@chairface1859 now you look a bit daft. See working in a University I know this is a proper job title. Just because you have little understanding of job roles within a University doesn't make a job title any less real. I have a very unassuming job title I work within the Estates/Operations department of the University that I work at my job title is Building Support. But dig a little deeper and Im actually part of a team that run three of my uni's flagship research buildings one doing research into neuroscience ( like finding a cure for dementia Huntington's) and that has the Compound Semiconductor research Institute ( yes we have the ability to fab our own silicon wafers) also in the same building is the Chemical Catalysts Institute. This build brings together the school of chemistry school of physics and school of engineering. The final building houses the Social sciences with departments within it that research help form public policies which work with The British Government also the business side of the university. Out of the 5000plus staff at my university about 1000 are academics

[@dahahaka]

you can maybe work around the inconsistencies in red dead 2 by using cheat engine with the unrandomizer, it forces the random number generator to always return the same values, so you could end up with a deterministic benchmark run :)

[@aaaafireball9322]

that's a great idea, do it for every game if possible or matters

[@danymend5797]

I don't think they can automate it

[@Deveyus]

I came down here to check to see if someone had suggested similar. As a former game dev (not for RDR2): This will have a minor impact on performance of games that utilize system random number generators especially (because of the syscall overhead) but importantly the result should be changed the same (incredibly tiny) amount. In exchange, you will get drastically more consistent runs from the behaviors in game. We actually had a debug build we did that replaced all the calls to random things like this with static values, so that we could apply internal test harnesses, especially during building our tutorials. I am not intimately familiar with how Cheat Engine goes about derandominzing, but all the sane ways I can think of it would do it, give you more in repeatability than you lose in cost to run it, or in deviance from player system behavior. (You're simply selecting one player system behavior and repeating it)

[@dieterblancke1756]

@@Deveyus this may be a stupid question, but would the (albeit minor) performance difference be mitigated if you leave in the random number generation but just don’t use it and use the static value instead (therefore still creating the “load” of generating a random number while also keeping your consistency)

[@ErulianADRaghath]

I am not sure if a result gathered with the default randomized behavior turned off is representative of the average gamer experience. Since the goal of the labs is to provide statistically significant testing for the masses, introducing irregularity is maybe not the right solution.

[@AaronShenghao]

This is absolutely what Gamers Nexus loved to see LTT's testing become, even though LMG seems forgot to mention them at 17:24. (It can be just oversight, but not including Gamers Nexus, the most through on testing tech reviewer before LMG have the Lab, seems a bit intentional.)

[@Hathos9]

The issue is that GN isn't trustworthy.

[@martine5923]

@@Hathos9how, cause they called out your best boy linus? go toutch grass please

[@Hathos9]

@@martine5923 That is the issue. GN has a legion of toxic sheep that love drama and attack whoever GN says to. I like technology, not drama.

[@BoraHorzaGobuchul]

​​@@Hathos9 that was very funny, please continue. People who love tech-not-drama don't watch ltt much, the whole point of this show is ~ 70% entertainment 30% info. It's basically a discovery channel tv show talking to the lizard brain off its TA. Nothing bad in it per se, some videos are pretty entertaining, but that's what it is primarily - a tech-related entertainment business.

[@Hathos9]

@@BoraHorzaGobuchul I'm fine with entertainment/tech. It is better than the non-stop drama and negativity of GN. LTT seeks views by making fun videos. GN seeks views by attacking others.

[@Lmmolsen]

@17:24 discretely showcasing bad blood between Gamers Nexus and LTT 😅😂

[@mikebutterface8583]

I thought the same thing

[@GULIwer1980]

@@minijag972Never trust pirate w/o ship. Such a big company, such a small sample.

[@jjlortez]

​@@minijag972 Yeah I always recommend you check out my competitor's shop after he smashes my windows 😂

[@andersonmcintosh7214]

Gamers Nexus incident made this video possible IMO. LTT forced to up their game

[@raawesome3851]

​@@minijag972you really think that it's genuinely him holding a grudge between that? Linus is not an editor for it 💀

[@jamesmatthews291]

Well done. Possibly the most thorough and well-explained video on the effects* of the Silicon Lottery I've ever seen. *Not covering much on the causes, or it would've been a two hour video! In fact, can we have an updated video on that please?

[@winebartender6653]

It's a 1 minute explanation: Each transistor within the silicon will be electrically different. Differences in needed gate voltage (what vCore is) for current to pass, differences in leakage (the amount of current loss through electron migration, aka heat), differences in output capacitance (the inevitable result of having current paths in parallel), etc. With these differences, we have a different required voltage to drive the transistor to operate properly given a clock speed, different heat output given a voltage and different power usage given both. Each modern clockspeed boosting algo for Intel and AMD take these into account to be both stable and maximize performance (aka, clockspeed) given the current load on the CPU. So with all of that in mind, you can see why CPU A and CPU B will perform slightly differently given the same workload. Their clockspeeds will differ because of power, voltage and temperature differences down to a transistor level.

[@the_undead]

​@@winebartender6653You're 1 minute explanation assumes a fairly deep level of knowledge about how a CPU works and seeing as LTT's target audience is not scientists and engineers where these kinds of assumptions could be made, your 1 minute explanation with likely turn into a five or 10 minute exclamation if they were being insanely information dense, at the average information density of an LTT video, I would say the hour-long prediction is not unreasonable given all the other stuff they would have to then cover to make it a complete video

[@winebartender6653]

@@the_undead ? Absolutely not. A simple preface of "A CPU is made up of billions, or even trillions, of transistors, which are essentially tiny electrical switches". You don't need to know anything about voltage, current, capacitance or anything else that you think is a low level concept. The point of the 1 minute explanation is to touch on the fact a transistor will always be different when made in a CPU, that's it. You could absolutely expand it out, but it is unnecessary to get the point across

[@ツッ]

The silicone lottery strikes again, It's crazy how much variation there can be between CPUs, especially within the same model

[@AdventuresOfDetroit]

Yep, Linus dose all these videos just to keep the best stuff for himself.....You think the stats is for us but it's not.

[@nonamenameless5495]

Disagree, it s more like: it s crazy how good the production process for such delicate tech has become that there s so little deviation and the results being so insanely close!

[@NightKev]

@@AdventuresOfDetroit l o l

[@BoujaXD]

Theres not Much variance its +-2%.

[@renchesandsords]

especially for more bottom bin chips, since the premium ones are all reserved for the higher end chips

[@s3rit661]

1:00 yes, silicon lottery, btw it happened the same with 5K series with AM4, they simply overclock themselves until 90c

[@joshbudnik2241]

Variation is normal in any process. I worked with steel for 9 years, you can order a structural grade 50 and get anywhere from a 53 yield strength to a 65. You can't produce everything the exact same and the tighter they hold their tolerances of what's acceptable to ship, the more expensive it is for you to get their cpu. Good companies will improve their process parameters to tighten that up but you don't want almost no variation in cpu

[@thedorsetflyer9113]

As a professional test engineer, I am truly impressed with the level of detail, thought and effort that has gone into the LTT Labs. 👏👏👏👏 Now if you could just coming and explain this level of dedication to some of my colleagues in other disciplines of engineering that would be great. They do seem to think that everything will just work, and don’t appreciate that us test professionals have to think around corners sometimes…

[@raawesome3851]

@@GreySectoid what's your issue?

[@JacobsKrąnųg]

@@GreySectoid yeah its pretty terrible

[@ncg8224]

Majority of the population do engineering for money, not because they are overly passionate. It would be useless to explain

[@Daunlouded]

Your comment is not constructive and is worthless. Please tell your own brilliant idea to improve the testing methods. Else don't type anything.@@GreySectoid

[@danniemck]

What's pretty Terrible about it? ​@@JacobsKrąnųg

[@hardyhousinger7426]

Always excited for labs stuff! That was exceptionally well presented and explained. Really great how you adressed inconsistencies and the multitude of possible causes. Probably one of the best ways I've seen to explain testing methodology, the reasoning behind it and the resulting discrepancies when compared to real world applications. Loved it!

[@luisjalabert8366]

The thing I'm not sure about, is why they couldn't just completely fix the clock frequency to a set value, instead of letting the CPU run AFAP. Sure, you won't get the best performance out of it, but if you want to use them to test a GPU, then it shouldn't be part of the bottleneck either...

[@the_undead]

​@@luisjalabert8366probably because that isn't very indicative of real world usage, they're not trying to get the most accurate numbers. They're trying to get the most accurate representation of how the average person buying these things is going to use it

[@GT5Player]

From collab to no mention at all... Where is GamersNexus?

[@dimitrimichaux461]

It look like you missed something 😅

[@Not.Your.Business]

the saltmine runs deep at LTT

[@CarlizWang]

I lost the silicone lottery with my 7950X3D, out of the box it performs 10% worse than most of the major reviews online. It doesn't like undervaluing, runs hotter than expected and even after much tinkering it barely matches the performance of most with stock settings. Worst of all, sometimes the system refuses to post at all during start up even though it's completely stable in all my stability tests.😢

[@MrThevirus512]

HUGE respect to all members of the team. It must have been really exhausting finishing all these tests.

[@MegaLokopo]

It is mostly automated at this point.

[@robertt9342]

How many months did they work on it?

[@AK-tf3fc]

@sierra2632 it's like you are a naive who watch cringe linus instead of gamer nexus

[@the_undead]

​@@AK-tf3fcYou say that like gamers Nexus is dramatically better, does gamer is Nexus upload videos showing their processes that they used to get the performance specs. Has gamers Nexus spent over a million dollars on scientific equipment and learning how to use said equipment to get better more accurate numbers, I don't f****** think so, feel free to say he makes more entertaining videos all you want, because that is an entirely subjective point. But the purpose of this video was never entertainment, It was to show their processes and the struggles they are having getting data accuracy to a point where they are happy. Where as far as I can tell gamers Nexus doesn't even care about accuracy

[@AK-tf3fc]

@@the_undead gamer nexus video is educative while linus video is entertainment only. Not to mention only one of them exploit people and sell their merchandise

[@eaturfeet653]

When presenting Euclidean distance data you can use a dendrogram to present the pairs of CPUs that are closest to one another. Might help for intuitive visual interpretation of the data

[@frederikex4545]

Or using something like PCA plots or tsne bonus points for showing the audiance how to read scientific plots made for this kind of data ^^

[@moonik665]

@@frederikex4545 This, or the new kid on the block - UMAP. Makes graphing any kind of multidimensional data a breeze

[@VuLamDang]

@@moonik665 or their older more linear cousin, tSNE

[@Solocoyote49]

​@@frederikex4545PCA also would potentially highlight which benchmarks/games have significant impacts on differences between performance. Add in a clustering algorithm like DBSCAN that can catch outliers and you're set

[@ColaBeGaming]

This guy diagrams

[@Victus-ih7tx]

As a computer engineer student myself, I want to say something to help Linus while he creates these type of videos. those processor's are exactly the same but they aren't exactly the same at the same time and there is a multiple reason but ill just tell eligible ones so you guys understand. First one is DIE-Connecter contact with the motherboard, which is sometimes even the slight changes on contact surface between CPU die and motherboard pins are eligible to create those type of problems, that's why we don't have CPU extension units but we have PCIe extensions? we do but it doesn't mean they don't take any performance cuz they do take some performance. and the second thing is when you test CPU the neutrino's (beams come from sun that goes through everything) is effecting CPU itself. yes i know its so rare or sometimes nearly impossible but its not as you might think. the reason behind this is neutrinos can change bits on registers or even ram. so there is a lot to say about it. We can trust about CPU manufactories about those type of things since it provides enough performance (nearly)…

[@XTRLFX]

I never got lucky in silicon lottery but once. That was an Northwood P4 that can be pushed from 1.8 to 2.8 Ghz. For RAM I'm usually happy if I get the values printed on the sticks to run stable.

[@claytonstanley2416]

Loved this video as a statistician, and really like the approach you’re taking. It may be a bit much, but if you did an equivalence check pre/post for any gpu lineup you’re reviewing, that would be pretty solid evidence that any differences you found in the tests across the gpus is due to differences in the gpus. The other option would be to model the specific test rig in the regression, but then you’d need to put each gpu in each test rig, which would defeat the purpose of parallelizing the test in the first place.

[@falcie7743]

Inb4 copyright strike by Nintendo for using Pokemon names

[@XxZannexX]

Nahhh, Linus's 4D chess move of that Corsola pronunciation will evade the Nintendo ninjas.

[@kuolettavaVids]

They're currently overburdened with Palworld, so nows the time to do this.

[@Respectable_Username]

Nah wouldn't be a copyright strike. A trademark violation cease-and-desist maybe though!

[@GeorgeJFW]

@@XxZannexX😂

[@quantum5661]

nah nintendont is busy with palworld this month

[@whistlingoat801]

you can isolate a lot of the problems with pbo by setting the clock rates and voltages, then you can measure a lot of the inconsistent impacts of software (OS and benchmarks) and then you can give a reasonably confident statistical deviance due to those factors and find the actual impact of binning

[@nikeshkunwar24]

crazy to think that the 1-2 % sillicon lottery difference could be the power of thousands of old 6502 cpus

[@Pete292323]

Oh boy, people are going to be loosing their minds again for the average 1% differences.

[@Eric79-f7i]

​@@jake20479I have a 7800x3d. Am I the minority here for buying a chip, and if it runs my programs and games at an acceptable speed that I wouldn't notice a difference unless I lined it up with 11 other chips, that I'm just happy with my purchase and enjoy my games? (Run on sentence, I ran out of breath trying to read it. I'm also not fixing it though.)

[@DanVibesTV]

@@jake20479 you know that that is basically impossible, in almost every industry. no one can build 100% the same product every time

[@NikolaguitarREZB]

@@jake20479 Except you'd be comparing that 'any percentage less' to reviewers who most certainly will not share the same test bench, software variables, temperatures, humidity and several other variables. So, this is good for LTT to keep data similar, but for your average consumer, the rule of 'within margin of error' very much applies.

[@manunen00]

@@jake20479 I mean, I'd much rather a cheap cpu than a consistent one. Do you have any idea how expensive it would be for chip makers to get their consistency down under 1%? I guarantee whenever you buy anything at a gorocery store theres at least a 1% variance to someone else who bought that same thing, it's just pointless to make stuff that consistent.

[@glenwaldrop8166]

​@@jake20479if you want the best you buy the best. We all know they bin the chips, we all know they don't all turn out identical. This has been the norm for decades.

[@duckbilldaniel]

(Tech horror story) There's an Azure server out there with a haunted CPU. We had a lot in the cloud, on one cluster we always had more errors with exactly the same deployments. We got around it for months by updating to a new cluster. Just a few months later the errors were back. I think that the old compute server got moved to the new cluster when the old cluster got decommissioned. And to this day.... I moved to ARM. Just because i know that x86 box is going to get me.

[@Jimmy_Jones]

Change to a different availability zone?

[@duckbilldaniel]

@@Jimmy_Jones company policy had reasons I can't remember for keeping availability zones the same. (I think there was a pre purchasing compute agreement ) Fortunately I moved companies and it hasn't found me yet.

[@tiagobelo4965]

Try getting some techpriests to help

[@lapislazuli2896]

​@@duckbilldaniel"it hasn't found me yet" lol

[@oblivion_2852]

Would it be more expense than its worth to run a profiler on the containers to check performance? Can your company seek compensation for being placed on faulty hardware?

[@middle_pickup]

18:28 To answer your question, Linus, all regulation is written in blood. The reason we have car safety ratings is because before we did corporations were happy to "self-regulate," and sell cars that killed people. Regulation tends to only come in the critical aftermath of such loss.

[@MikkoRantalainen]

Superb work! The only thing missing was repeating the testing on another motherboard to check if the CPU performance depends on individual CPU or the combination of motherboard power delivery and CPU. Would the same CPUs perform the best in any motherboard or did those CPUs simply happen to work the best with that specific motherboard?

[@p_serdiuk]

17:24 Kinda weird not seeing Gamers Nexus... wonder if that relationship will ever recover.

[@Winnetou17]

I was sadden by that too. It would have buried the hatched and allow everybody to move forward. But it's so hard to have nice things today :(

[@Mavis847]

It makes sense, they have no obligation to mention GN and given what happened it's understandable

[@ItsMe-oi9dy]

@@Winnetou17 GN pretty clearly used the situation and drama to drive traffic and profit and weren't entirely honest or genuine in their handling of it. LTT denying them further attention is only appropriate.

[@ItsMe-oi9dy]

​@@waldolemmer It keeps filtering my comment, they likely have keywords in place to prevent discussion of it anymore, but there are videos out there with analysis of it, one rhymes with mechmechfotato.

[@ItsMe-oi9dy]

​ @waldolemmer (We'll see if this goes through) As an example, they were dishonest by not actually giving an unbiased or full picture on the events they were reporting on, they did not contact LTT to get the full picture, and presented his one sided view as fact, then he refused to respond to further discourse on the subject, claiming to be holier and denying any poor handling of it on his part.

[@Byefriendo]

11:53 Colours on 1% low and Avg are inverted

[@-Ninnux-]

Gamers Nexus about to make a third rant video

[@romulodrumond3526]

Hey just avoid Euclidian distance to measure dissimilarity on high dimensional spaces, usually the norms get pretty bit, l2-norm ends up not making much sense (such as k-means clustering)

[@raskalthefirst]

Maybe I'm wrong, but wouldn't the simple solution just be to disable all the dynamic frequency scaling & such and lock the CPU's to the same frequency when testing GPUs. That would probably get rid of 90% of the variance

[@alexlowe2054]

This is industry defining data. Genuinely. Anyone notice at ~9:50 when the charts fly by, that Corsola goes from being in the top 3 CPUs for cyberpunk at 1080p, to suddenly being the worst CPU in the test by a wide margin at 1440p. I expected different games to stress the CPUs in different ways, but I did NOT expect a single game to completely change the order of a single CPU. I guess this goes back to the performance controversy with Halo Infinite that Hardware Unboxed covered, where they found that the specific part of the game you benchmarked could completely change the order of the top performing GPUs. I suppose the same logic applies within the same game, for each resolution. That's crazy. Also, good catch with CS GO. Showing the differences between CPUs is great, but at ~700 frame per second, a microscopic difference in latency has an outsized effect on the FPS value. That's because the time between frames is actually the inverse of the FPS count. Which is fine when you're roughly around AAA game refresh rates between 30 and 120, but that inverse relationship starts skewing your data more and more the higher your FPS number is. High FPS games always have a larger impact on average performance, just because of the way math works. I'd like to see someone normalize using "time between frames" to measure those differences instead of the actual FPS number. That would probably give a number that better represents how different it feels to use two CPUs. It would also give more weight to games with lower FPS values, which is actually where performance matters the most. Almost no one can notice the difference between 400 or 500 FPS, but people can definitely notice the difference between 40 and 50 FPS, even though the "percent difference" is exactly the same. Absolute values matter, and the percentage difference calculation everyone uses can obscure very important data that actually matters. When I'm looking at buying parts, I don't care if a CPU gets 5% more FPS in a game that's already maxing out my monitor's refresh rate. Worst case games generally matter more, since the esports titles with high FPS counts are always going to be easy to run.

[@wohnai]

People dog on KZbin comments... But then sometimes you get good ones like this.

[@Renegade605]

Uhh... Except, because of the way math works, the inverse relationship between FPS and frame time doesn't affect the percent difference at all, and regardless of which way you show it our perception is a logarithmic scale. ie. The difference 40 to 50 FPS is 25%, and the difference 400 to 500 FPS is also 25%. Invert that and the difference 20 to 25 ms is still 25%, and the difference 2.0 to 2.5 ms is even still (surprise) 25%.

[@ridwaan2443]

​@@Renegade605you're getting it but you're not getting it. Say an anomaly increased latency in testing by 2.5 ms. The 20ms case would increase to 22.5ms, i.e. 12.5% and the 2ms case would increase to 4.5ms, i.e. 125%

[@Renegade605]

​@@ridwaan2443 what you just said is the exact opposite of the point OP was making. You're correct that a 2.5ms hitch will feel much worse at 2ms average frame times (400 fps) than at 20ms, except that OP said small differences in frame rate matter more at low average frame rates. Both are true, but for very different reasons in very different cases. You just can't simplify the data that much and at some point you have to trust that your audience is capable of understanding when and why the different numbers matter.

[@alexlowe2054]

Getting "um acktuallyed" in youtube comments by someone who repeated exactly what I said. Perfection. @@Renegade605 For reference, I said "the percent difference is exactly the same". That's a verbatim quote from my unedited post. Check yourself before you wreck yourself. We agree, so I'm not sure why you're telling me I'm wrong about the percent difference. On my other points, I'll clarify. When I mentioned "High FPS games always have a larger impact on average performance", I was specifically talking about average performance charts like the one at 8:37. Charts that take the FPS values from each game, and average the result. That's a graph where the inverse relationship skews the data. Specifically, it heavily weights the graph towards the result of the game with the highest FPS values, and reduces the weight of games with low FPS values. You can compare the graph at 8:34 that included CS GO and the graph at 11:41 which excluded CS GO, to see how much a single outlier game affected the entire average result. That inverse relationship definitely skews the results when you average the FPS values between games, as basically every reviewer does. That's a big problem, because an extra 3fps in CS GO mean basically nothing when it's already maxing out your monitor, but 3fps more in cyberpunk at 4k is a pretty big change. However, both those changes are averaged out on the same graph. That's why the inverse relationship is a problem. Rewatch the 10:43 "CS:GO is Wonky" segment to have Linus explain why games with framerates that high aren't valuable for benchmarking. There's some CS GO specific stuff there, but there's also a lot of statements that can be generalized to all high FPS games. If anyone is still confused, I can clarify further.

[@ZimmMaster]

I loved how much meme-age they worked into their BSOD in the intro. They even got the QR code go to the LTT store lol

[@BoraHorzaGobuchul]

Absolutely no surprise re marketing included.

[@2GunzPerformanceTuning]

"come on guys, we're not that young anymore" im dead lmaooo

[@greeshmaisave5410]

Ltt is quitting gpu reviews?

[@stefanheuve9670]

I like how GN is not included in the other media outlets part

[@vadicus]

Subtle shade

[@tiagobelo4965]

Neither was jerryrigeverything, nor dawid, nor a bajillion other (including many major) tech YT channels/media outlets. You might be over-reading between the lines

[@vadicus]

@tiagobelo4965 or its a valid observation of the absence of perhaps the most prominent hardware reviewers on the internet. He doesn't carry the moniker of 'tech jesus' for no reason.

[@Vidadebroadcast]

Let's get the pitchfork, guys! /s

[@thebestdamager7400]

He's mostly called tech Jesus because of his hair. Just saying...

[@Bugattiboy912]

Anyone notice how they didn't show Gamer's Nexus when mentioning "other outlets"?

[@etherealmistlc]

I noticed a bunch of channels that I sub to that they didn't put in there. Very angry right now.

[@joog79]

Not surprises though after the nutpunch GN delivered to LTT. Eventhough it made LTT force to do beter.

[@John120196]

I literally scrolled to find this comment as soon as I saw it :P

[@MarioGoatse]

Lmao. Yes. Very clearly on purpose. I knew he wouldn’t show it as soon as I what they were saying. Hilarious

[@spdcrzy]

TBF, GN also isn't doing the same exhaustive kind of testing in the same space. Their niche, and by definition the type of testing they do, is completely different.

[@coinboy_coin]

naming them pokemon is fire

[@feralb415]

What If we force to get about 8% CPU Usage with a software for that, so that all the CPUs are always on 8% Usage even if the background tasks are not that heavy atm and then run a benchmark

[@ikillyou96]

Throw in a benchmark running under a minimal Linux install for good measure, will give a good indication of raw performance without many background tasks.

[@thegeekno72]

How many of us game on Linux ? They run tests under windows because that's what the humongous majority of us use to play

[@rigf1997]

@@thegeekno72 But they remove a lot of the bloat anyway, they try to get the raw performance of the CPU's. Not the performance of the CPU on Windows. I'm not sure why they don't use Linux, maybe they've tried and something didn't work out?

[@thegeekno72]

@@rigf1997 Linux raw performance and Windows raw performance are not comparable things, they're software that have wildly different operation, it could be analog to comparing X86 and ARM, it's just not the same in the way it behaves

[@Apeeksiht]

​@@thegeekno72we are talking about cpu performance variance doesn't matter which platform. linux might have given better consistent results because of less bloatware. Regarding arm and x86 they are different cpu architecture you comment doesn't make sense.

[@adrian2ka46]

​@@rigf1997linus said that he wants to be relevant for the general user and that pretty much explains it all. Yes maybe using linux will give interesting results, not gonna lie but things change once you change use cases and os and the difference might become minimal for general users tasks.

[@iaxyz]

One of the best videos you've made in a long time. I really like the labs content and the transparancy that you're trying to give. There is a bunch to learn about testing to learn in this video. Really looking forward to more of this content.

[@BruceCrossan]

I love how when they recommend looking at multiple outlets for reviews, Gamers Nexus didn't show up in the footage.

[@stupidscholars9842]

I think I speak for everyone here that this is EXACTLY what we want. We love tests like these, in depth, practical and perfectly executed. Thank you so much.

[@perplexedon9834]

Yeah after you explained it I am not nearly as mad about this as I thought I'd be. If they design it and manufacture it to achieve a standard, but include features that may be able to get even MORE than advertise, then I don't really care about the variability of how much more it gets. It's kind of like rock climbing gear, where a carabiner will be rated to 22kN of force. 99.7%+ of the time it'll be stronger than this, and it could easily vary from 23kN to over 30kN. That's be like a CPU having a performance spread of nearly 40%. The thing is, the lowest number of 22kN is still 5x stronger than any realistic safety situation, and so maybe we should think about CPUs in terms of the demand we expect to place on the. Eg. "This CPU will have a 99.7% chance of running *game* at more than 60fps/144fps/250fps" For your testing, maybe Linux with Wine/Proton would be better in terms of minimisation of background tasks? The lack of oversight vs automotives and, indeed, rock climbing gear kind of checks out because if your CPU is 5% slower, you aren't going to find yourself in an accident where your brakes fail and you, your partner, your two kids and another random family all fucking die. It is definitely worth doing in a decentralized, nongovernmental fashion though. Maybe a consumer union could be made where manufacturers have to provide representative samples in order to get a trademarked seal of independent testing.

[@themisterx8660]

Yeah I think you are right. There are margins in probably every industry so I think the consumer has to be fine when the product reaches its rated performance. When we talk about cars for example, there you can often measure differences between the power of engines. But then the average consumer doesn’t care about fuel quality, tires and regular oil services that much, so probably those have more impact on the actual performance on the road than the engine itself. The same is probably true for pc part’s performance in different countries with different average temperatures. So in the end I guess you should buy a product by it’s confidential rated speed and be happy if you get more than you paid for but not be necessarily sad if you don’t gat that much of a benefit. Your friend might have ended up with a better chip itself but maybe he has a low quality power supply, which makes the cpu perform worse than yours. Or maybe there are also margins in psus?

[@Dr.Spatula]

​@themisterx8660 the margins are in EVERY industry. If it's not federally regulated, it's financially regulated or consumer regulated... excerpt vitamins and supplements...

[@memethief4113]

the problem with using Linux is that it's just not a realistic OS to use for benchmarks. Linux has many problems working with NVIDIA GPUs, but the main thing is that most people just don't run Linux so it's better to just use Windows, even if it's bloated and makes getting reliable test data harder

[@Winnetou17]

@@memethief4113 True, but it can be used for non gaming tests. As it stands out, Linux will have to be used in specific tests. Like you said, having Linux be usable for ALL tests, we might get there one day, but not today. Not this year either. Maybe next year, in the most optimistic timescale, but realistically I think it's more 5 years, IF it continues to grow like in the last 2 years.

[@perplexedon9834]

@@memethief4113 I guess the argument would be that it would be fore controlled testing. If one chip ran a game with proton on linux at 100fps and another at 105, then you could meaningfully say the second is 5% faster, even if on people's actual systems the real world framerate would be 45 or 145 or whatever. The problem with testing in realistic scenarios is that realistic scenarios have a lot of uncontrolled variables. If you tested on windows you could get 145fps or you could get 95 depending on what windows fuckery is happening in the background and you couldn't know without doing 100s of tests and averaging it. Also Linux gaming, even with NVIDIA, is my personal main use case :P.

[@jackalovski1]

As someone who used to do statistical process control for safety critical electrical components in the automotive industry, I approve of the comprehensiveness of this. Now that you have the process laid out for doing this, it will only get faster and more routine.

[@sirjman]

You know you are old when, you don't recognize 75% of those Pokémon.

[@coolcatjk30]

Really? Most of em where like gen 2 and 3

[@jpdj2715]

Thread-cores (shared I/O) next to core-cores (private I/O) may result in a random performance variance? Especially when the OS has zero "awareness" of the hardware architecture and how to best utilize that?

[@AveryChow]

8:31 honestly really disappointed to see that diabetes didn't win out over the other CPUs. I had my bets on this one :(

[@accordingone3598]

Well, it makes progress in the real world tests.

[@GiacomoDose]

Awesome video! Lab is gaining a lot of credibility after delivering such great and in depth analysis. Impressive! I must admitt however that is a bit sad to see at 17:24 no mention of gamer nexus.. I guess the community would love to see a reconciliation... I'll keep dreaming

[@pitmaster226]

The ball’s in Steve’s trunk. “If an article contains personal or serious allegations or claims against an individual, it may be appropriate and necessary to give that individual an opportunity to respond to these claims, or to deny them if they wish” - Independent Press Standards Organization But they’re UK based, so why should Steve care (:

[@ionutpogacean]

yeah, a bit of a shame. especially since GN, HUB and Der8auer did a video on this exact thing kzbin.info/www/bejne/hobIi4RppdCsY9k - GN kzbin.info/www/bejne/mnjFiGqlp7-GY6M - Der8auer

[@aaronmorrow4957]

I noticed the GN omission too. Really sad to see.

[@Xerpadon]

6:35 this is why we need a gaming windows that disables all that, back in the days of XP someone actually managed to do this strip entire windows and literally left barebones. And to be fair it did make quiet a difference. there is so much stuff that they can disable.

[@JimParshall]

As a guy that teaches data analysis and data science this video brought a big grin to my face it'll be great for my students to watch. Great job Linus and everyone there. I love seeing benchmarking and testing getting better and better as time goes by. I used to do this for gaming companies all the time and it's not easy getting proper quality benchmarks. You and gamers Nexus and level 1 techs and others are actually pushing this technology forward and bringing awareness to customers and others of the issues related to benchmark. Thanks a lot

[@Substant_]

Respect for the Vegemite shout out. Love, Australia

[@bruceanderson8588]

expect thats not vegemite ,real stuff is black not brown

[@Bacender]

17:30 haha GN conspicuously missing

[@TheVillainOfTheYear]

Noticed that. Also, GN just did this exact same video a few months ago. Sad to see they clearly haven't buried the hatchet, but I can't exactly blame them. Steve wasn't wrong with what he said, but he clearly took a huge swing at Linus' reputation without seeking comment. That comes across pretty malevolent to me. Linus' lab guy who trash talked GN appears to be the instigator, but Steve's response was not proportional, nor particularly professional IMO now that the dust has settled.

[@yutt]

@@TheVillainOfTheYear I agree about everything other than Steve not being wrong. He was absolutely wrong, and was stirring drama for both views, and to undermine his main competitor. He didn't want solutions, he wanted to gain market share. That's integrity in the same way as Apple criticizing Google.

[@Thep184]

I mean, it is known that companies like Nvidia, amd and intel produce chips, test them and decide after that which chips it will be ( 4090, 4090 ti etc. ) but that in these categories even there is an up to 12% change is wild. What i would love to see is cpus going through benchmarks before selling, and with an average you get a percentage of what you get. For example you buy a cpu that performes 12% less than average, which makes it cheaper. A cpu that on average performes 12% better is a bit more expensive

[@JanMagerl]

How about adding Error Bars to your results?

[@richardwiech]

Love the update on testing methodology! Very detailed and I do appreciate that! I did notice that Gamer's Nexus didn't get a spot in the B-roll alongside the other creators, I honestly hope you two make up and I strongly recommend looking to them every now again for sanity checks because they are really great and highly respected.

[@Martycrane]

I noticed that as well. The Steve's are my go to reviewers for almost all purchases on parts

[@the_undead]

I lost all respect for that pile of garbage when he made that hour-long video, as someone who knows a great deal about human psychology, it is obvious to me that that video was not made for any reason other than revenge. And more specifically revenge about an LTT labs employee calling him out specifically on his data testing procedures, more specifically calling into question how accurate gamers Nexus testing is. And seeing as Steve's response was that hour-long video instead of an hour-long video detailing his processes or the updates he's made to his processes, I'm going to suspect those same concerns are still valid today, and I feel this is partially confirmed by the lack of gamers Nexus in that b-roll shot, Yvonne and Luke would not allow Linus to hold a grudge for this long, also, linus isn't even the CEO anymore, so I suspect there was a reason other than the controversy for the lack of a gamers Nexus in that shot

[@Spawn-]

​@@the_undead "as someone knows a great deal about being a fanboi" If my firm would be slandered like it happened I'd do exactly the same. And oh btw to clear your delusional POV even more... Surely LLT just issued apologies and shut down testing because there wasn't any truth about the claims... surely! If you fire a gun you better be goddamn ready to get backfire. Back to School or where ever you gained that dubious knowledge about human psychology.

[@ramih43082]

Very useful video. Interesting that you didn't list Gamers Nexus despite them making an even bigger 68-CPU test back in August.

[@Nekudza]

Also noted that, too bad if they are not friends anymore

[@Shapershift]

Linus' ego is probably still bruised.

[@skak3000]

​@@ShapershiftIt's Steve's ego, he was insulted big time when an LTT employee on a tour criticism he's way of testing. That was started it all... 😂

[@soul_slayer7760]

@@skak3000 Dude, if someone criticises your work, would you do nothing? Well, he did, and he also proved with data and examples that actually his testing was better, and more reliable than the LTT ones, which were ridden with errors. Everyone knows Linus is a little narcisistic, he said so himself, and he apologised. That said, I watch them both and other outlets to make my decisions, I just don't understand why people side with anyone in cases like this, research everywhere and make your own decisions. Watching only one outlet and trying to protect them like they are your friend is just the stupidest parasocial shit. Fuck GN, fuck LTT, they are means to an end. Stop thinking emotionally and start thinking rationally.

[@yutt]

@@soul_slayer7760 People who say they are thinking rationally are cringingly hilarious - and peak irrational. Anyone who thinks they or other people "think rationally" are so ignorant and arrogant it isn't worth discussing.

[@zebkayne4253]

#1 - this is the labs content I am here for #2 - I hope you swapped the CPUs between the benches they were on, to ensure the issue is on the CPU and not something else on the bench.

[@vyor8837]

They didn't do the second, indeed they tested with EXPO which is an *extra* variable that AMD does not list as a stock configuration.

[@the_undead]

As this is not a team of data analysts so they made a couple mistakes in communicating their process, but if you pay attention throughout the whole video, The process they are using is as follows, The only component that changes from one CPU test to the next is the CPU they're using probably PTM 7950 for their thermal pad material so that you don't have to worry about the thermal paste application being a problem, and they're doing this in their thermal chamber with a very specific temperature and humidity setting so that even just humidity in the lab building isn't a concern

[@richarda7058]

As a research scientist, who looks at a lot of large dataset analyses, it’s pretty cool to see the common approaches. Choosing geometric mean, Euclidean data clustering etc. I expect to start seeing some PCA and heat maps soon!

[@superslash7254]

The problem is that all the methods in the world can't cover up bad data.

[@joao-o-santos]

Amazing work @LinusTechTips! Your discussion on sources of variability really got my brain going. As someone who works with psychological data, this is something we often discuss (inter-trial vs inter-participant). Have you looked at using linear mixed models (aka hierarchical models, aka multilevel models)? They could allow you to control for multiple sources of variability without having to aggregate data so much.

[@tbrooke3016]

This video feels like it would be great for teaching high school science students about isolating variables and the impact it can have on results! Such an interesting video!

[@Fiyaaaahh]

When you say you "scaled" your data (14:10) I really hope you actually mean you "normalized" it (e.g. by using the standard score (assuming your distribution is normal (you should test this too))). Otherwise the Euclidian distance metric is biased in certain dimensions. Judging from the post-scaling screenshot this is in fact _not_ what you did. So what kind of "scaling" formula was applied here? PS. You also assume all numbers scale linearly with performance. This might very well be the case, but since different tests use different units (opposed to e.g. "fps") and I have no clue how they work I feel compelled to point this out.

[@joshuam511]

Why not fix the CPU frequency to a frequency that all of the CPUs can hit and maintain? You'd drop performance, but gain consistency. This is a good bit of work. I'm a computer vision and machine learning engineer with a PhD in physics and I'm impressed.

[@LightningTechNL]

This is a good investigation! I've been waiting for someone on this investigation to come out because, as a system integrator for CAD pc's I thought I was losing my mind! Ever since the 13th gen intel and the X3D line that AMD brought out, I've been seeing these inconsistencies. And I was really worried that something in our testing wasn't right! Glad to see my sanity is in check... But I REALLY think you should also take a look at intel! The 13th gen was already a real 'silicon lottery' thing, but the 14th gen intels are... WAY out of spec now and then! PLEASE PLEASE check intel too!!!

[@ABaumstumpf]

", but the 14th gen intels are... WAY out of spec now and then!" But in general that is not the CPU but the motherboards. Sadly intel allows the boardpartners to run insanely bad automatic overclocks as the default settings -.-

[@LightningTechNL]

@@ABaumstumpf OK! Nice to know! But the thing is. If I swap the same cpu with the same cooler between different motherboards (of the same type that is, we use the b760m aorus elite a lot and the b760 tomahawk) I don't see as many differences as when I change the cpu's out... When I only swap mobo, I see a difference of about 2% max in cinebench. But between different cpu's on the same motherboard I can get up to 7% difference! So, what am I doing wrong in testing?

[@ABaumstumpf]

​@@LightningTechNL "So, what am I doing wrong in testing?" Pretty much nothing. Getting an identical CPU in the same motherboard with Intel normally stays within 1-2% of each other, at least according to the 3 companies i know the hardware off. If you see bigger differences then first thing to check would be temperatures and powerconsumption as many boards increase Voltage and thus powerdraw even without affecting clockspeeds which can lead to shorter boost-durations. This is done excessively with Z-chipsets but i had seen that with B boards too. Had built a 12600K for a relative (for broswing and looking at pictures :P ) and disabling all the "enchancements" and running it according to specs has dropped the powerdraw to sub 100W at full boost. there are always some differences in how efficient each CPU is and the boards increasing powerdraw for no reason will exacerbate those differences greatly.

[@madeintexas3d442]

This is some incredible research. Not many, if any review channels that do testing would go to this level of effort to obtain the results you have here. I think the amount of effort that it takes to obtain these different samples is probably a reason why. I have always heard of the silicon lottery but did not think about that the lottery may be based on the piece of silicone being used were the environments of the factory. As an afterthought it seems obvious but of course you would need to obtain units from different manufacturing runs to get samples that would perform differently. I appreciate the effort that it took to do this and this demonstrates the true value of The LTT lab.

[@remoteviewer8524]

Binning? One of the retailers in your list has been doing this for near two decades. I'd rather be subject to the silicon lottery than from an outlet that uses binning to cream off the better CPUs for, and let's say this quietly lest they sue, a higher revenue stream at the expense of regular buyers. Ouch!

[@MYG]

That actually also happens at grocery stores, it's called Shrinkflation.

[@Jeranhound]

Guys, when you're flashing through 10 different graphs comparing performance between 8 different chips, you need to keep the chips in the same order. The Productivity charts, in particular, have their orders shuffled between every slide. Makes it harder to track the changes.

[@MeshJedi]

In addition to the order being by perofmrance, it also doesn't matter. it's not about the perofmance of the individual chip, it's about the consistency.

[@VoxVenenatus]

⁠@@MeshJediNot sure if I misread, but: I think that's exactly the point. Keeping the CPUs in the same order over multiple graphs, should make it way easier to figure out if the fluctuation in performance is due to (or scales with) the different tests (aka. is consistent) or stems from (possibly faulty) hardware (aka. is inconsistent).

[@Jeranhound]

@@VoxVenenatus Exactly. If the chips stayed in the same place, you could easily see the bars jumping up and down as the slides changed.

[@MeshJedi]

@@VoxVenenatus that dosent really matter and only workshop when you see all the stats of a single cpu

[@coolgirladrian]

If you ever name something similar to like you did with the Pokemon, I would consider doing it alphabetically. So the first one starts with A, second starts with B, third starts with C and so on and so forth. It's just one of those nice-to-haves that doesn't change much but makes the presentation a bit easier to digest

[@RepChris]

7:45 youve said this before, but actually you can have both consistency and comparable results. Computers cant do random. They use whats called pseudo random number generators which give you a sequence of random looking numbers, based on a seed you give them. This uses modular arithmetic and prime numbers usually. However, a PRNG will *always* give you the same sequence of "random" numbers for the same seed. To avoid issues the seed is chosen to be something like the current unix timestamp which means youll never get the same sequence. But for the test you could just always choose the same seed. This would still simulate the AI and physics as usual, except the same way every time. Unless you choose a unusually bad seed this should still be entirely representative of a random seed, at least in the same way two runs of the random seed simulation should always be comparable. Except now each simulation run exactly the same between each run making comparison much easier

[@realfoodman]

I listened to an interview with a science journalist who was talking about how in any laboratory setting, there are the big-picture people who organize it and get everyone focused on the right direction, and there are this other set of people who know the tiny little peculiarities of how the machines that do the measuring actually work. I'm glad LTT is doing that latter kind of work on this, since it really ensures that we get data that's actually TRUE.

[@maxgarcia9829]

as someone who works in big data and analysis of it the way you guys described all your statistic analysis in a way easy to digest for regular people was amazing 10/10 vid

[@rafal5863]

That is what makes synthetic benchmarks more consistent and deterministic. But are less relevant to real life performance which is not deterministic.

[@cricketerrushi]

Possibly the most thorough and well-explained video on the effects of the Silicon Lottery I've ever seen. The only thing I found missing in the video was the thermal performance of these chips and how they maintain their clock speeds. I have been there when I compared my Ryzen 5 3600 with another R5 3600 of my friend with the same mobo, RAM, and Storage. The difference in thermal performance and the clock speeds was noticeable... that was the day I realized silicon lottery is a thing too.

[@low-key-gamer6117]

As a microelectronics student, larger the wafer larger the variation among chips in general. Larger the chip, larger the variation inside the chip, like variation between different blocks.

[@timduncan6750]

6:50 Smiled when I saw the only thing you add is Notepad++ because this is truly the only thing I miss since going from Windows to Mac on my work computer...

[@mjdevlog]

Notepad++ is great for it's autosave and multiple tabs, though now the regular notepad (WIndows 11) has the same feature🤔

[@Seed]

Same lol.. vscode isn’t nearly as good

[@K.Parth_Singh]

​@@Seedheh really?

[@rikuleinonen]

@@Seed sure wish np++ had support for practically any language.

[@KeinNiemand]

Vscode is way better the only thing notepad++ does better is beeing a bit faster@@Seed

[@joaopedrosa9060]

honestly, HUGE respect for doing this, first of all, the video is hella interesting and second of all knowing that you guys do this is extremely reassuring. great job dudes.

[@joaopedrosa9060]

@@boscotheman82 thank you! but I did not ask

[@elatedmaniac]

I know this was extremely time-consuming and detailed, but it gives me confidence in watching your channel and the results you obtained. Just want to stop and appreciate the hard work everyone is doing at LTT.

[@nicoeberlein6217]

12:20 why are 1% lows higher than avg fps? :(

[@ironic5790]

Those pokemon names I feel had to be decided by one of the engineers. we like to troll like that and it is totally something they would do.

[@the_undead]

Personally, I choose to interpret the other option Linus gave in this video as quite literal, I don't think someone actually suggested that specifically, but when it comes to something that would actually work in this video, I suspect pokémon was the only viable option

[@clayhudson7104]

Freaking amazing video, well done everyone. It would be cool to see if the results are still as varied if the clock speed was locked across all of the cpus. I assume this wasn’t looked at because that’s not as equivalent to the real world use of the chips, but still could have been an interesting add.

[@StarsForward]

A similar occurrence happens in the musical instrument industry. Guitar manufacturers (for example) will send their lower-quality instruments that may not live up to a luthier's quality expectations to Big Box/Discount Stores and Online Retailers, and save their "best of the best" for boutique stores and distributors that target professional musicians. These imperfections are minute, and often go unnoticed by most all beginner and amateur musicians. Are they still playable? Yes - very much so. I'm just saying that it happens.

[@user-ob2sp6de7y]

Tiny Linus hands.

[@brianranta5408]

I found this review of reviews and previews of reviews re-freshing. Whenever I've used reviews for purchases in the past, I haven't worried about the cherry-picking theory that was mentioned, but the random variance has been a concern of mine. I feel like it's obvious to know that LTT won't eliminate this completely, but knowing that they are keeping it in mind and checking from time to time is a good little piece of mind. Thanks LTT!

[@benwu7980]

Dunno exactly about that, the term of 'silicon lottery' has been around for a long time. What this video did do, was highlight quite well, how it actually applies if trying to make a standardized test. That it's LTT basically only, that has the resources to actually go and get get ?11 cpu's of the same sku, to do this , is pretty crazy. From what i remember of this test, said ages ago on WAN, the idea was to use the 3 most close to the centre for the test benches

[@Renegade605]

Couldn't help but notice that Gamers Nexus was missing from those "multiple outlets".

[@cartman20000]

I wonder what the average tempurature was for these cpu's, as that can have an effect on clockrate and voltages being applied to the cpu's.

[@BixbyConsequence]

My upgrades are far enough apart that I'm always pleasantly surprised by the improvements. 8% or even 12% spread really doesn't matter in that context. Those who absolutely must have the very fastest CPU du jour, can pay for pre-tested/binned CPUs to get the Unicorn chips.

[@_j_j]

0:05 🤔 there's something wrong with your Vegemite mate. I'd be having words with the bloke or sheila that sold it if I was you.

[@nrvinl]

Only controversy from this will be his pronunciation of corsola😭

[@Riyozsu]

And not including GN

[@The_Mister_E]

Which is funny considering he pronounced Corsola more or less correctly (basically a tomato tomato pronunciation), but royally messed up Raikou's pronunciation by falling for the classic "pronouncing the u" beginners trap.

[@Lampe2020]

Well, when the OS is doing random stuff you can either use minimized Window$ or even a highly-configurable Linux distro to prevent as many random factors.

[@BoraHorzaGobuchul]

On the other hand, that may provide "spherical x in vacuum" results which have little bearing on real-life where peculiarities of an os may result in skewing performance in a particular direction. A very complex issue, hard to see which approach is best

[@niklaskoskinen123]

This reminds me of buying prepackaged vegetables. What the manufacturer promises, should be the worst case scenario. If I buy a 100g pack of lettuce, it should always be at least 100g. But I've seen ones even 150g. People rarely complain if they get more than they were promised, but companies will definitely hear, if they deliver any less.

[@LiLBitsDK]

of course they will hear if they don't keep their promises...