Intel has a Pretty Big Problem

[@Level1Techs]

Intel PR posted an update to... reddit. So check that out, update your bios, and look for another update to your bios coming mid august 2024! www.reddit.com/r/intel/comments/1e9mf04/intel_core_13th14th_gen_desktop_processors/ tho they seem to have deleted some of the better questions about whats covered under rma, are folks that have delidded covered (guessing: lolnope), etc.

[@ZombieLincoln666]

@@Level1Techs they posted the update to the Intel website. But yes it’s clear most of you guys get your info from Reddit…

[@Level1Techs]

Intel left out the oxidation issue on the website, which is real, interestingly

[@MrSaywutnow]

​@@Level1Techs Sounds to me like they're trying to cover their asses. They're focusing on the voltage issue because that's something they can fix. The oxidation problem is a manufacturing defect, and that potentially means the dreaded R word - RECALL.

[@le0king69]

does this also apply to i7 14700 hx laptops ?

[@jasperboleyley9659]

@@Level1Techs sh8 i just ordered a laptop with a 19400hx on the acer helios 18...am i screwed? considering i live in the phil and my sister bought it for me there in the u.s...yikes

[@MIBxSpartan]

Quick we need to throw Wendell in witness protection before the Intel hitman finds him!!!

[@honeybadger6275]

good thing he isn't looking into boeing planes right now

[@RowdyDandy]

Dr Su needs to send some AMD Bodyguards XD

[@sirisaac6125]

At least he isn't talking bad about Boeing.

[@LegendaryPhenom]

He probably will have BSOD long before he reaches Wendell

[@acerx203]

Intel is not Boeing, yet.

[@cuteswan]

Intel saw Boeing's crash rate and said *_CHALLENGE ACCEPTED!_*

[@leorickpccenter]

hahaha

[@bobbastian760]

Hold my beer....

[@tinkerman5220]

Ba DUM!

[@Lurch-Bot]

As a former aviation maintenance tech and lifelong computer nerd, I approve this message. I won't fly commercial and I won't buy intel. Both Boeing and intel are a complete disaster.

[@CD3WD-Project]

Thanks cold LMAO

[@Zorro33313]

"AMD Is In the Rear-View Mirror" - Pat Gelsinger, Intel CEO "Intel CEO’s Comments of AMD ‘in the Rearview Mirror’ May Indicate Intel is Driving Backwards" - Areej Syed, Hardwaretimes

[@loz9324]

😂 lmao

[@EzrealLux]

Good one! Thanks for sharing.

[@catsspat]

I saw a funny meme cartoon of Pat driving towards a cliff and Lisa driving away from the cliff. Lisa's car is of course in the rearview mirror from Pat's perspective. v-cliff

[@Maverrick2140]

more like flipped over on the road :D

[@azo0ownz]

lool

[@damazywlodarczyk]

Actual investigative journalism instead of talking about cpu's for 20 minutes just like 99% of other tech channels. Quality content.

[@giomjava]

Heck yeah

[@NightRogue77]

@@damazywlodarczyk the rise of independent journalism bout the only good thing about the Internet anymore

[@abdullahzafar4401]

True, I hope this gets more appreciated and we get more

[@charleshines5700]

Could be worse, if I had a nickel for every time someone just had to add some sort of awful music to a video I would be rich!!

[@1blackice1]

Same avatar! :D

[@dgo4490]

Intel forgot how to make fast cpus, then they forgot how to make efficient cpus, and now they appear to be forgetting how to make properly operational cpus... Not a good character arc...

[@michelvanbriemen3459]

heh character "Arc"

[@DarioCastellarin]

@@countvonthizzle9623 nice flamebait, if only AMD's CEO wasn't a woman, and her tenure wasn't wildly successful...

[@afos88]

@@DarioCastellarin women arent diversity

[@rarinth]

oh boy i hope this aged well

[@kornar]

Typical too quickly developed too little tested, lack of QA because the new lineup has to be released and delivered to keep up with the competition and not upset the shareholders behaviour. I wish we had fewer stock corporations again. A world in which profitable companies are devalued because they don't constantly increase their profits is completely f'd up.

[@TheSickness]

Feels like Intel wants to drag out the problem till next gen launches and play the Spider-Man meme till then

[@seeibe]

Intel is hanging on for dear life trying to compete with AMD. They rather take big risks that can backfire on their customers, then try to silence it, than being honest about the limits of their CPUs and losing their high spot in the benchmark charts.

[@TayschrennSedai]

That feels way too real...

[@Lady_Zenith]

Exactly, there is no fix for what is plain old Silicon degradation caused by too high voltages, clocks, temps and Amps pulled trough the silicon long-term. There is no software fix, there is no bios, there is no microcode that will revert this once its already degraded. The only fix is "do not exceed 1,3v and clock them 500-600Mhz lower". And also, do not clock your CPUs so high that they have less than a 5% headroom left in them, so even the smallest degradation throws then to the "unstable on stock settings" category. Intel is just making smoke hoping the attention will go away so they do not have to do the thing they should - full scale recall.

[@Gen0cidePTB]

@@seeibe This. Consumers are used to seeing i7 at the top of the chart and Intel will break bones to keep it there, because they desperately need to keep it's brand power. I just had to leave a bad review for a "i7 Desktop" selling in my country's Amazon with no GPU for $900(!), because I noticed that it had a HD4500 and realised nobody else could work out that meant it was an 11 year old PC. They just see "i7" and buy. It's those idiots that Intel made the 14900 for. So Google still says "i7 is fastest" and the plebs carry on buying it.

[@kylexrex]

This is me when I was 14, I saw i7 and chose the crappy HP Pavilion HPE H8-1103NL with the i7 2600k haha

[@brawldude2656]

-Talks about a real problem -Explains what he found in true objectivism -Admits data is not big enough -Properly digs through the problem What a gigachad

[@gatowololo5629]

Terrachad

[@azo0ownz]

yea first video of saw of him and he's smart, sympathic and accurate. should be new president

[@Stinger913]

Yeah, noting limitations in data lends lots of credibility and we can still probably draw some conclusions as he did about it. Mega based.

[@CyranoLeRapier]

I work at a gaming marketing agency in the IT department, and I have been absolutely losing my mind over these issues. I have built ten systems over the last six months, and had three of them experience various failures, from One of the more consistent problems I've seen has been system corruptions, where I'm unable to even run simple Windows updates, SFC, DISM -- you name it. Recovery media fails, fresh installs on new drives fail, and the crashes on games have gotten utterly out of control. Thank you for this, it's definitely incredibly helpful for tracking our OWN sanity.

[@chrisx742]

Takes five minutes to downclock.

[@CyranoLeRapier]

@@chrisx742 Appreciate the thought, and while yes, it does, and we did -- used the Intel XTU, didn't go above 52x, dialed in Intel specified limits all across the board, and had little to no consistent improvement, but definite performance loss. Tried the Intel Processor Diagnostic Tool and failed consistently across all settings. RMA'd one 14900K already, popped in a spare 12900K while waiting for the replacement with none of these issues with all the rest of the same hardware. The point ultimately is that we shouldn't need to do this, and it's infuriating to have to fight with multi-billion dollar companies to get what we're paying for, and have it actually work consistently.

[@fancynin12]

@@chrisx742 the fact that you would have to do that is hilarious.

[@skyhawk21]

What motherboards used? And ram?

[@NUCLEARARMAMENT]

@@chrisx742The fact people are so lazy these days is proof of how incompetent most computer users these days really are.

[@yoboymoi]

I mad respect the person who kept playing regardless if they were crashing every 2 hours,I feel then on a spiritual level

[@SledgePainterArt]

This was me playing Remnant 2. I would play about 4-6 hours a day and would experience at least 2 or 3 crashes straight to desktop.

[@UniqPlays]

That was me lol. I got my replacement CPU 2 days ago

[@RayneAngelus]

I've had games with bad enough memory leaks that you pretty much need to reload them every 2-3 hours or the game will bog down and eventually crash. Common problem with specific operations in Unity. All you really need to fix that one is a cleanup subroutine - which I actually talked one developer into implementing during Early Access.

[@brawldude2656]

My old ass laptop crashes every 30 mins while playing Roblox :D

[@bogartwilley]

Player: I've got you now Pirate King! PC: (Crashes) Player: Ahhhh you thought a mere catastraphic failure could save you?! Well prepare to doom your DOOM! LMAO

[@FinsterDexter]

Just want to point out this is the kind of investigative work traditional tech journalism doesn't do anymore. Thank you so much for this work!

[@noxious89123]

Steve from Gamers Nexus: "Excuse me?"

[@Adowrath]

@@noxious89123 Are you saying GN is traditional tech journalism?

[@camplethargic8]

Frame Chasers 7/14/24 - problem described and solved

[@Jimmy-Vance]

You should publicly mention that you are not suicidal in case Intel approaches Boeing for advice and business cards.

[@matthewgaines10]

The mighty days of Intel supremacy have long been over. Now they are trying not to be mediocre.

[@WayStedYou]

The fact its happening in game servers is pretty crazy and an entire 1000 dollars extra for support.

[@OllieHamon]

That's capitalism for you

[@rezenclowd3]

@OllieHamon so? A company should charge more to work w products w more problems. I do....up to 10x more of my usual150 per hour rate.

[@andytroo]

the extra 1000 for supporting a cpu that is less than 1000 implies a very large support time sink and replacement cost ...

[@Dendodorion]

@@rezenclowd3 That's a really dumb way to do business, people talk to each other. It's like if I started pumping out car parts, let's say a water pump that kept failing prematurely and blamed everyone but themselves and tried charging for repairs but knew it was self inflicted the whole time... Oh wait, my Golf R had that issue. Oh, there was a class action lawsuit? Nice.

[@user-sd9dk5ir7v]

​@@Dendodorion I mean it does make sense if you aren't the company producing the faulty part in the first place. Charging more because something is worse to work on seems fair to me.

[@Jessassin]

15:05 "We already replaced a lot of customer's 13900k with 14900k and the issues don't seem fully resolved." This statement is extremely telling. 14:56 "$1,000 extra" for support is insane and really tells the whole story here. The part is not reliable. I am sure a lot of conversations are happening behind the scenes here, but based on what I've seen, Intel has not committed to fixing the issue. We can hope that it is due to incompetence but more likely they do not want to admit fault here due to the cost of "making it right" Thanks for sharing this Wendell, it will be interesting to see other news around this topic in the coming weeks.

[@monad_tcp]

oh yes, instead of issuing a recall, lets damage even more the brand name

[@GoldenSun3DS]

It's like how insurance companies are refusing to insure houses in certain states against extreme weather. When even the Capitalists charge more or refuse to cover something, that's a sign that there's a real problem there.

[@RurouTube]

@@monad_tcp The issue is that Intel itself doesn't really know what to fix and currently the most reliable way to make it run stable enough is to lower the multiplier like to 53 and running the RAM slower. If that is what it takes then the CPU wouldn't run as specified and also would invalidate Intel marketing around the CPU, thus even if Intel want to do a simple recall and giving the costumer the fixed product, it can't be done. Assuming actually running it slower does fix the problem, Intel probably need to replace the product, force it to run the CPU slower, and give some monetary compensation.

[@cemsengul16]

I think Intel owes every 13900k and 14900k owner a new 15th gen CPU since this is a defective chip.

[@coladict]

I don't think it's about being committed to solve the issue, as it is that they have probably realized there is no fix, and they can't afford to recall two whole generations of product. That would kill the company. Trying to mitigate it through microcode and power limits is probably the best they can do until getting a 15th gen out, and it might even be too late to fix the issues for that one, so they're really just holding on for their life for a 16th gen.

[@renehoyvik]

When server providers are starting to actively recommend AMD systems after YEARS of Intel being the "safe bet", this is the point where Intel should start sweating. Because mind-share is a huge factor in what gets bought in large volumes.

[@HyperScorpio8688]

The number of Intel fanboys trying to trash Wendell is hilarious. Never stop calling out anyone doing shady business stuff!

[@winnieid2727]

I don't think he cares. He/dr ian. said in lv1 & tech2 potato podcats.

[@Crazyneo2917]

I agree with most of your statements as I work for a small SI. I stated on initial videos a few months ago when the fingers were pointed at motherboard makers that the problem could be with Intel processors. We mostly provide 14900k to businesses on premium and entry-level B760 boards. So right off the bat, no O.C. Next, we disable motherboard enhancement and limit power to Intel stock. And some board are weak and can only do intel stock power. Failures have happened in all these scenarios. These systems can run cinebench all day long or occt stability test or aida64. But try to install the nvidia driver(compression/decompression). The installer fails. Another way to test this is tekken 8 demo game. No matter what bios we choose, those processors are gone. Replaced. There is no coming back from this. Has happened with ddr4 and ddr5 systems.

[@explosivemonkeys]

Exact same issues here. Waiting for my RMA CPU to arrive as I type this.

[@Crazyneo2917]

@explosivemonkeys I hope you get yours soon. We are not in the U.S. so our rma is a couple of weeks with Intel. But once the system is diagnosed with this problem at the service centre, we replace it with a new processor and return the system back to the customer. So, for a customer, it is basically 2 to 3 days down time. I am also hearing stories of Intel denying rma for these cases. That would make this worse.

[@jamesaspinall9248]

The NVidia installer was a big thing for me too. Would crash on install about 95% of the time. At the start I figured it was a corrupt Windows install or maybe a failing HD. Almost dropped a lot of extra money to start replacing components before I started to see all the other issues out there, which thankfully saved me.

[@anjelomichelangelo7726]

@@explosivemonkeys my i9 13900K is unstable if i set more than intel's baseline specs, Do RMA covers me?

[@noth606]

Just so it's clear - did replacing the CPU actually fix the issue? Was it properly verified, taking a system exhibiting the problem consistently, replacing only CPU and then testing again with the same exact test and no problem found? Asking because I was around in the past for the FDIV bug in P60/P66 by Intel, where you could actually even get it to calculate wrong in Excel using a specific formula to prove the bug. Initially Intel said there is no issue, then they made RMA's for all who complained trying to shut them up - or at least it looked like that is what went on, then finally they had to stop all the BS - and rework the CPU and announce that people who had affected chips could register to get it replaced with the new fixed chips shipping out as they were coming off the production line. I had one of the FDIV bug P60's for years as a memento of that cluckerf*ck but I traded it eventually for a DEC VAX 11/750 in 1997-8 something like that. Shoulda kept it. Point being with the FDIV bug you could not replace your way out of it until Intel made fixed chips, 100% of the P60/P66 initial production were affected. You could kernel patch your way out of it, but it was a software workaround for a hardware problem.

[@highiqwarrior]

25% is a ridiculous failure rate for the high volume OEMs. Wonder if intel is getting back the failed chips to do some failure analysis on them with test boards and thermal imaging or something.

[@pf100andahalf]

So far they're taking the failed cpu's and sweeping them under a very big rug.

[@highiqwarrior]

@@pf100andahalf maybe that rug will come up in 40 years on an asmr “intel rug cleaning” video 😂

[@justanothercomment416]

Sounds like it's a dynamic clocking bug getting stuck into a strange mode, causing crashes.

[@rich1051414]

My guess is it's simply that power/frequency limits are pushed beyond what the platform can reasonably support, which means this was an intentional choice intel made. It is a known workaround that underclocking and reducing power limits increases stability.... at first. But once it starts happening, it seems that the damage is done, and it's only going to keep getting worse.

[@pf100andahalf]

@@rich1051414 I think you're right.

[@JoPOW7]

My god this is so useful. I am on my third 13900K in 18 months. First one ‘lasted’ a year (3 months of that was me getting more frustrated trying to diagnose wtf was happening as more and more and more games failed). The pain was that it came out as IO errors, memory errors, everything except the CPU. In the end I found a forum thread where someone mentioned it. I got it swapped under warranty and the second one began to fail identically 5 months in. Third one is still quite new but if this goes as well I’ll be sharing all of this with the builders and telling them they can either refund the entire PC or put something else back in. I have no overclocking. The games that crashed were often games that a 13900K wasn’t even remotely needed for - like WoW - and it was absolutely infuriating

[@granityseis104]

goodluck with 4th rma.

[@WaterspoutsOfTheDeep]

Have you come across anything mentioning if this is happening to the laptop 14900hx cpu?

[@matyasselmek3673]

@@WaterspoutsOfTheDeep refund the laptop

[@WaterspoutsOfTheDeep]

@@matyasselmek3673 they are fine the laptops arent affected

[@WaterspoutsOfTheDeep]

@@matyasselmek3673 evidence so far shows laptops are not affected

[@klwq]

When AMD released its 7000 series, I went with a 7900x. I had buyer's remorse when I saw the better value of Intel 13th gen. But with this situation now, my mind is totally free from that feeling.

[@chubbysumo2230]

you can upgrade to a 7800x3d or a 9000x3d chip when they come out, without having to worry about the CPU killing itself.

[@klwq]

@@chubbysumo2230 Well, I am not upgrading unless my current CPU ceases to function 😅

[@nuunuu6943]

@@klwqwelp my intel 13th gen is dying… time to go team red

[@sandiaswara1940]

AMD x series always scam for price/performance. If you want to buy AMD better waiting for non-x version or 3D version

[@klwq]

@@sandiaswara1940 Yeah for my use case the 7900 would have been perfect actually

[@Momfasa]

Intel PR department: Lets talk about Lunar Lake instead. What a great CPU that is.

[@DKTD23]

Lol that's the Intel way

[@Crixer234]

Also Intel: Muh AI

[@Lead_Foot]

My 13900k was so unstable after about 1.5 years that I couldn't copy data off an SSD without getting a BSOD

[@cutterboard4144]

Did you try a live Linux USB stick? Just to rule out Windows is the culprit.

[@Ikxi]

lmao that's an amazing product

[@polpolaris]

It's cooked 💀💀

[@Need4FPS]

Almost 2 years with 13900KS , since day one.. power unlocked.. 300 plus watts.. no issue whatsoever.. your point is ?

[@bhume7535]

@@Need4FPS "I had no problems, therefore the problems don't exist!" This guys an idiot.

[@urielalbertodiazreynoso6309]

AMD: We need a fastest and better CPU Intel: we need more anti-competitive strategies

[@pjmorgan]

The absence of AMD data is also data

[@greebj]

kind of... a lot of statistics don't like zeroes e.g. MTBF

[@Ornithopter470]

​@@greebjthe lack of failure data points to AMD chips not failing in the same ways, in numbers sufficient to record.

[@PrismariLaura]

@@greebj but we don't have a zero. We have an insignificantly small number: iirc 4 out of ~1500 documented failures, or 0,25%, when 30% of the population use AMD, is a clear indicator that this is an intel-specific issue.

[@Dm3qXY]

he already mentioned survivorship bias, look it up if you missed it for not being aware of it

[@POVwithRC]

Good lord, the analysis and access is just swell. More, please. (Just don't get Boeing'd for your trouble)

[@thegutlessleadingthecluele7810]

"Boeing'd" 😄

[@SpoonHurler]

If WotC can send the Pinkertons then Intel should be *_looks over shoulder_* I gotta go, guys...

[@jairo8746]

I love that that is a verb now.

[@XIIchiron78]

We must not lose Wendell 😭🙏

[@raya4633]

Let's hope Boeing doesn't use K series CPU's in their aircraft.

[@nexusyang4832]

14:58 - HOLY BALLS!!!! That’s crazy!!! When the data center operators notice issues and you know they collect data…something is up for sure!!!

[@korhanaydemir8366]

I'm surprised they didn't leak this earlier.

[@aminorityofone]

This is going to drive companies to AMD much faster than before.

[@blindsniper35]

​@@aminorityofoneAbout 30 seconds later, we're pushing our customers towards the 7950X platform. Which is exactly what I would be doing in their position.

[@HanmaHeiro]

I got my 13900KF 4090 build last year thinking it would last 8 years. Not knowing it'd be having such issues within a year...😂😅🥹🥲😥😢😭

[@complexity5545]

Yeah -- this is somewhat surprising from intel. This is amateurishly like the late 2000s AMD graphic card performance. I bet its heat problems from bad caps or arrays or interdigitated caps. If its design then, that's really really going to be surprising. I've fixed my AMD processor from swapping out dry caps before.

[@InnerFury666]

as someone who is always paranoid, obsessively paranoid about stability over performance, this whole fiasco just made me paranoid of the entire brand of intel.

[@quademasters249]

My 10700k is as reliable as the day is long. I run it for months at a time with no reboots. It does make me leary of upgrading. I guess I could step up to the 12000's but is that much of an upgrade?

[@flintfrommother3gaming]

@@quademasters249 Thinking the 14th gen was an upgrade over 12th gen is a joke on it's own

[@damara2268]

​@@quademasters249go Ryzen instead. Much much better than anything Intel offers

[@quademasters249]

@@damara2268 I ordered an i9-11900 KF. It's the last hurrah for this motherboard and ram. It was a couple hundred. Should tide me over till intel pulls their head out. Not sure what my next motherboard upgrade will be. Maybe AMD. My last AMD experience wasn't great.

[@andreabriganti1226]

​@@quademasters249Unless you want to go with a high-end GPU of current gen, while being on a low resolution, the 10700k should still serve you well. I would wait, save more money and upgrade when there's some big deals. The 12th Gen as far as I know, isn't having those issues and it's a good gen but I don't consider that, as a great upgrade, not with the CPU you got. Just my opinion anyway.

[@serlancerlot315]

So glad I got a Ryzen back in 2021. I couldn't get a 5900x, but my 5800x has been running like a beast, couldn't be happier.

[@trparky]

Damn does it feel like I dodged a bullet by going with AMD. My 7700X-based system has been rock-solid stable.

[@connors3356]

real. I love my 7700x. Such a good chip. I can Craft so many Mines with it

I bought 2 7900X3D's for 100 GbE testing on Supermicro H13SAE-MF. I guess I dodged the Intel enshitification bullet. Meanwhile, my AliExpress craptastic and old Xeon rando recycled parts impulse buy laughs in ECC.

[@andrewdonohue1853]

Ryzen is an awesome cpu. I have a 3600 ryzen 5, I have had it a few years. Absolutely rock solid

[@LatvianVideo]

@@andrewdonohue1853 I have the 2600x, also rock solid

[@Frostie3672]

I remember a time when people would say to go with Intel, it's a much more stable platform, how times change eh lol. Ironically the only build I ever had so much hassle with was the one time I decided to jump ship over to Intel back in 2015 with a 6700k skylake cpu due to the crap ipc the fx8350 had that I was running. Always used amd other than that from the athlon days in the early 2000s through to a number of ryzen builds & I've literally never had any problems, all the amd builds have just worked, currently running a 7800x3d.

[@Heatranoveryou]

Intels decades of fuck you buy our shit has not changed

[@samwalker7567]

And they're likely about to get completely shafted by ARM with the mass migration of both datacenters and consumer computers to ARM chips (or other RISC designs) due to the huge drop in power consumption that ARM brings.

[@dschwartz783]

@@samwalker7567 Yep. I wouldn't buy any PC right now if you can hold off. Things are changing, and ARM seems to be poised to dethrone x86 in the coming years. Best to wait and see what the industry looks like once the dust settles.

[@andreabriganti1226]

​@@darthmelbiusFanatics I Guess.

[@prw56]

@@dschwartz783 Disagree, if the transition happens it will: a) take years to acquire significant share in the desktop market, and b) there will still be a large marketshare of x86 for years after that happens, so support will remain for quite some time after. Makes no sense at all to hold off buying a PC in anticipation of this platform.

[@BrendonGreenNZL]

​@@prw56legacy compatibility won't be a problem; the Apple M1 has proven that an appropriately designed ARM processor can competently emulate x86-64.

[@JohnWilliams-gy5yc]

Today we learn a bright side of telemetry.

[@tomtomkowski7653]

If 50% of CPUs are failing one way or another but the other 50% running perfectly fine I would closely look at the production starting even from the silicon provider and its quality.

[@SpunckyJew6969]

I just checked the latest batch, it sucks

[@jonathanoxlade4252]

Smaller you go well chip lottery that's why I don't see the point in clocking oh I can get 6ghz on water yea how long does the cpu last its equivalent to a top fuel dragster being pinned to its max breaking point intel has pushed the silicone far enough that it's getting bad yields some chips are good alot of chips are degrading faster due to the whole chip getting intense heat that's the problem 1nm chips are gonna have problem we need to move to a better Base substrate graphene or carbon substrates

[@caleb4254]

The crash rate is just absolutely unacceptable especially for OEM and datacenter.

[@andersjjensen]

Crash rates measured in weeks is not acceptable for consumer products. If it's measured in months you still have an iffy product. If you see game logs where crashe intervals is close to hourly... the whole house is on freaking fire!

[@hubertnnn]

Standard SLA in the industry is 99.9999%, so you can have about 30 seconds of downtime per year. With this high of a failure rate keeping that level of SLA is simply impossible. Even for the cheap providers that have just 99.99% SLA keeping the downtime to 5 minutes per month might be hard.

[@marcogenovesi8570]

@@hubertnnn that's the marketing and contracts. In actual practice it's often lower than that for pretty much all hosting providers, even big ones like Google/Amazon/Microsoft or OVH. Source: trust me bro. They refuse to acknowledge to us a bunch of small incidents and stuff during their own upgrades so they can keep claiming such high SLA wasn't breached

[@fleurdewin7958]

But to be fair, datacenter shouldn't even use consumer class CPU. 1 of the biggest difference between consumer class and enterprise class CPU is ECC RAM support. If you want 24/7 operation and stability is very critical, ECC RAM is a must have feature which only Xeons can provide .

[@Gen0cidePTB]

And it's a lot worse than this video says, because when he says "25% of systems are having errors" he's not counting all the ones that are already dead or RMA'd, because they can't produce bug reports... If we assumed even half of the buggy chips go back to Intel at some point, the math would put it at at least 33% dead, which is absolutely disgusting.

[@repatomonor21]

Heys, I'm one of those who were "tremendously suffering" from the instability, to the point where regular programs such as Discord, Chrome, Davinci Resolve, and games were crashing left and right. I just RMA-d my i9-13900KF. Intel already knows that shit's going down because customer support didn't put up too much of a fight and went straight to the point - RMA. Unfortunate, but it is what it is. Purchased a secondary, i7-12700K (LGA-1700 slot was needed so that I don't need a new MOBO) and it feels like a breeze of fresh air not having to put up with the constant instability.

[@MrXelaim]

Couldn't you also return the mobo, and go AMD instead?

[@repatomonor21]

​@@MrXelaim No. 1. I'm a contractor (work from home) and cannot afford to lose a single day of work for making such a switch. This was a carefully orchastrated RMA process where I timed my CPU replacement on the same day the courier would come for the RMA CPU. 2. The motherboard is not faulty. Buyers protection in my country only covers the first 3 days to return a product without a good reason and this is a year old system.

[@owlmostdead9492]

@@repatomonor21 That's terrible buyers protection

[@repatomonor21]

@@owlmostdead9492 Why though? You still have a time window to return a product if you changed your mind. Otherwise, wouldn't people ignore warranty and just return age old products left and right out of greed?

[@owlmostdead9492]

​@@repatomonor21 We have a 14 day return period, 3 days is not enough to evaluate a lot of products

[@methanbreather]

I am surprised people even buy intel. Even without this issue - you buy intel and 6 month later 50% of performance are gone to fix all the security issues.

[@Val-xi4we]

I nearly cried watching this. Someone else that has spent countless hours going over crash logs!!! Someone else has felt my pain!!! I'm not alone! Anyone considering a job in IT.... This is your possible future!!! and its a thankless job.

[@Mike80528]

I can't recall an issue like this since Sun accidentally shipped radioactive chips that were unstable.

[@benjaminlynch9958]

I mean, Intel had to do a massive recall back in the day (1990’s?) when their CPU’s literally couldn’t do simple math instructions consistently. They didn’t exactly fail, they just never worked in the first place because of a critical design flaw.

[@nessotrin]

I'd like to know more. Do you have a link ?

[@Mike80528]

@@nessotrin If you Google it you might find some information. This was back in the early 90's. The company I worked for had the issue with our SPARC system. SUN made customers sign an NDA if I recall. I don't recall how widespread it was. I did a search and only found limited mention of it from some old forum posts...

[@AgentAsteriski]

they WHAT

[@alexandermorrison9077]

@@nessotrin I think its this, en.wikipedia.org/wiki/Pentium_FDIV_bug , Its floating point math would mess up a small amount. But that error could compound and cause problems depending the calculations. I recall having a professor say he had to redo part of his thesis due to this problem.

[@jackoneill84]

This is why you make a statement. The speculation machines rolling now.

[@currentlykent4593]

*you’re* a statement!!

[@zodwraith5745]

I suspect Intel themselves probably don't know what's wrong.

[@kyu9649]

@@zodwraith5745 Or they do know, but it is a hardware problem that they can't patch with a microcode update.. So they just try to play the long game until their next gen comes out.

[@zodwraith5745]

@@kyu9649 I'm not saying that's not a possibility, but to automatically assume as much would be highly unfair. Intel is WAY too big a company to cover up the kind of conspiracy theory you're insinuating without a leak. They haven't done anything to try and cover it up and have acknowledged it publicly. They just haven't provided a solution which just circles back around to my previous comment. They have a hard enough time keeping a lid on leaks they DO want kept secret like future products. This would be way too juicy for a leaker with all the coverage it's had.

[@TheBaldOne]

​@@zodwraith5745they know, they're trying extremely hard to patch it via microcode so they can come out with a fix at the same time they acknowledge the failure. Welcome to the age of publicly traded companies, their shares will tank for sure, they're just trying to control how much they crash.

[@qwe14205]

Solution: AMD

[@ExHyperion]

Buyer’s regret in two words

[@smakfu1375]

Right smack in the middle of the Raptor Lake 8+16 die is the ring interconnect fabric, with a ring agent interconnect per P-core and an agent per E-core block. Given you’re seeing a mix of IO related errors, including the out of VRAM errors (which is likely triggered by a PCIe operation failure), my gut says this is the system agent logic in a bad-state, specifically due to failing ring signaling at one or more ring agents or in the system agent itself, blocking and going through some form of error / bad-state recovery. My guess would be that this is due to electromigration degradation of the ring interconnect logic due to the very large Raptor Lake 900MHz top end ring clock bump by Intel, versus Alder Lake. This is why we’re seeing these issues in 13th and 14th gen, but not 12th gen. What’s concerning is that if this is where the problem is occurring, than Intel appears to have not implemented internal telemetry, or not exposed it via CPU or other system driver, to drive WHEA reporting around ring interconnect and the system agent. Given the complex nature of the SOC, with its asymmetric architecture, many clock domains and sophisticated “system agent” IO block, that would be bad if true.

[@kevinzhu5591]

If the ring causes those issues, I wonder if raising ring voltage could solve those issues or not. For example, I had 2 i9 13th gen, one is 13900K that I sold for the current i9-13900KS, both runs stable at 253W, 400A IccMax with BIOS default settings (I set my power limit lower than 253W initially due to having a 240mm AIO). I did turn off MCE in BIOS and set my custom load line to undervolt the chip. I used the previous chip for about a year with no noticeable degradation. I run both chips with SA voltage of 1.35V for my DDR4 ram OC. Both chip had ran mining softwares, Prime95 and other stress tests. I suspect that for the degradation to happen. most people did not configure power limit, they let their chip runs with high voltage with high current, and that cause fast degradation. Also, their silicon quality is probably average or below average in the first place.

[@xthelord1668]

@@kevinzhu5591 it probably would further accelerate degradation of the ring bus and lead to even more crashes my bet is that intel has a design flaw regarding ring bus which got exposed by their lack of regulation regarding power profiles which slowly cooked ring bus on stock speeds and fried it when people did do some overclocking

[@Wahinies]

​@@xthelord1668that is tragic. I would hate to pay exorbitant prices for a faulty processor

[@ahs9674]

@@kevinzhu5591 Wouldn't it make more sense to lower ring voltage and clocks a bit? It shouldn't make a huge difference in terms of performance to reduce it by 300mhz or whatever.

[@Deniryer]

@@ahs9674Doubt it would help. 12th Gen have had (minor and not often but I've encounter them way to many times...) problems with asymmetric uarch, 13th & 14th not only suffer from that but you have to have in mind that they do bent, are made using Intel 7 process that took ages to be "mature" enough to be even considered viable to be used for manufacturing and so much more. There might be a reason (beyond being behind with their own foundries and R&D on advanced nodes) why Intel is hell bent on using TSMC instead of their own process for CPU tile.

[@JBrinx18]

I clicked because Wendell was on the left

[@v3xx3r]

He's on the right for me

[@haewymetal]

hes in the middle for me

[@Brutaltronics]

He's upside down to me

[@mihirsinghyadav]

He is on top of me

[@kickazzdrummer666]

I don't see him.

[@e.m.6581]

THE KEY ISSUE IS "INTEL INSIDE" ON THE WARNING LABEL ON THE FRONT OF THE COMPUTER CASE

[@theodentherenewed4785]

Given the fact that 13900K has been on the market since October 2022 and this issue hasn't been fixed, it must be a design flaw. It's shocking that i9 still die at W680 motherboards.

[@looncraz]

On the warranty repair side I have seen a major shift in the willingness to accept that an Intel CPU might be faulty. It went from a multi-hour process to get a processor RMA approved to a self-approved process. I just order the CPU replacement and I'm done. It's akin now to ordering a replacement memory module or SSD under the warranty process (this is at the OEM level, not with Intel). Talking to other techs, we have seen a HUGE increase in CPU failures - almost always seen as general instability with a longer running process (which is a really hard thing to verify during a warranty service, so we do typically take the customer's word for it). I have tracked down the problem on a few of them to the memory controller contact pads - often you can even see visual discoloring of the pads from prolonged heat. The instability in those cases seems to come from the CPU getting hot, changing shape, and the signal integrity to the memory modules shifting... then BOOM, you're unstable... until it cools a little, then you're okay. Basically, I think the problem is the shape of the CPU.

[@AaronALAI]

I think this comment needs more attention, it's the only hypothesis that accounts for all the observations. I'm convinced this is the case too.

[@WaterspoutsOfTheDeep]

Is this also affecting the 14900hx laptop cpu? I just got a laptop with one and could probably return but got a really good deal and has a good mini led screen and would get so much use out of it for the whole almost a year till next model laptops are out with the 50 series rtx so really don't want to return.

[@RowdyDandy]

@@WaterspoutsOfTheDeep The Laptop Cpu is soldered, not socketed

[@AdrianOkay]

back in the days the cpus were binned with a way looser margin of error, in some cases you could easily make the cpu run 25% faster with a bit more of cooling, but once they caught wind of overclockers pushing the cpus so hot, they started to push power into them themselves, to make these margins extremely tight and cpus that are edging being defective are still sold, most people don't stress test their new cpus so by the time they realize they are defective it's too late and they get away with it

[@MrMartinSchou]

One of the biggest take-aways from this video is the price difference on service agreements between a 7950X ($139) and a 14900k ($1,280) server - "3 years parts & labor, 24/5 - Next business day onsite repair - zone 1". For the 7950X, that isn't covering the cost of a single hour onsite repair technician. For the 14900k, that is slightly less than the cost of an Asus Pro WS W680-ACE ($330) + a new 14900k ($600) + 4 x 48 GB of G.Skill Ripjaws DDR5 5200 RAM (2x$190) (total of $1,310). You can basically do a complete hardware replacement, AT RETAIL COST, for the price of that service agreement.

[@yoyoma-z5l]

So this explains why AMD came out with the server board for the AM5 socket a couple of weeks ago.😊

[@NickCharles]

"The system just gets miserably slow for up to a minute before an actual crash" ...uh oh. I don't get crashes in games, but I *do* I get this kind of behavior from time to time just at my windows desktop on the 13900K if I leave it running for a long period of time (like, more than 1 day). I'd start getting weird things like programs hanging in the background, Explorer stops responding to inputs, even CTRL+ALT+DEL doesn't work. It never actually has a hard crash, but it never recovers either. Only fix is a full restart, or hard reset at the power button. Guess I can expect this to get worse?...

[@LL-dv2vf]

Crazy that this was happening for so long even the oodle devs chimed in.

[@Bobbias]

Honestly not surprising at all. RAD game tools are some of the most mature middleware in games (probably as close to bug free as you can realistically get without formally verifying your software). If their software is suddenly failing, they're going to want to know why.

[@hammerth1421]

And Oodle is deployed basically everywhere and very stable outside of this situation, otherwise all our games would be crashing constantly no matter the hardware.

[@lllXavierllll]

The 14900K chips are degrading insanely fast. My 14900K on day one could run at stock frequency with a -0.075 voltage offset on the last 2 steps of the V/F curve, without any WHEA errors either at full load or idle. Every 2 months or so, I would randomly find a couple of WHEA errors which were instantly fixed by increasing the previously mentioned voltage offset by +0.05v. This week WHEA errors began showing up again. Now the CPU needs to run at stock voltage as is unable to handle any negative voltage offset whatsoever.

[@douglasmurphy3266]

What kind of cooler do you have and what temps would the chip run at full load before it started to worsen?

[@CyberneticArgumentCreator]

Source: trust me bro

[@genejones7902]

Be careful going down that WHEA error rabbit hole. There are endless reasons why they occur which may or may not be related to your cpu. My pc with a 13900KS paired with a Z790 extreme mobo has sometimes thousands - literally tens of thousands of corrected errors in a given hour. And yet it is perfectly stable. Will it be stable if I ran it for a week continuously? Maybe - maybe not. In my case, not too long ago, I ran my cpu extremely out of spec (115°C, 450+W) for about an hour and more started to appear than usual afterwards. I'm now running the AI overclock and no more than usual errors are occurring and actually after about an hour they completely go away. It's almost as if there's a bad connection somewhere in my system and heating up is somehow expanding and allowing that connection to be more secure temporarily... I do not know. I've been following all these reports, and all I know is my particular ks has been the most stable CPU I've ever owned in history! It under volts, it over volts...even mimics core for core clock speeds of the 14900KS. Wait... same cpus. Suffice it to say, I deleted the 14900ks from my shopping cart hahaha. I do wish those with problems the best of luck and hopefully Intel will step up if there truly is a problem at the hardware level.

[@lllXavierllll]

@@douglasmurphy3266 I'm using an IceMan Direct Die waterblock with an external MORA3 420 radiator. My 14900K max core temp at full load is 70C. Temps are not an issue.

[@lllXavierllll]

@@genejones7902 There are specific WHEA error codes that are related to CPU errors. Whenever I get them every couple of months, increasing the voltage slightly is the only way to make them go away. So yeah, its a CPU issue.

[@fletcherchambers7175]

AMD watching this: "Kermit drinking tea" meme.

[@renevandenbosch9967]

If the CPU performance halves before crashing maybe it is the IMC and memory transfers stall causing bandwidth starvation. This would explain why most pure cpu metrics continue to look fine. I'm curious if the issue reported is exclusive to ddr5. Megekko (a big Dutch retailer) has an informal policy to push ddr5 dual dimm over quad dimm for stability reasons but they didn't specify this would be for Intel only.

[@cmdr_talikarni]

I work in commercial IT, we have had 30+ PCs where the intel CPU essentially died, this has been affecting them all from 10th gen to current. Recent one was a month old 12th gen 1255U. Affects desktop and laptop CPUs. Larger companies that deal with more PCs have had a lot more of the same.

[@alexanderzawydiwski9534]

I am starting to build up a pile of dead intel i5 10th generation CPUs from a bunch of friends and family computers that I’ve repaired most of them didn’t post. Some of them wouldn’t detect hard drives in certain sata slots on the motherboard there was also ones that wouldn’t detect ram sticks in certain slots or run very slowly for no reason and crash I thought it was weird why the CPUs were failing first pull them out of Dell and HP desktops and had to replace them the computers. Work fine now. so in total there are three dead i5 10th and 1 i5 12th the computers were roughly 2 to 3 years old

[@bulletpunch9317]

How many amd cpus do you have?

[@LupusAries]

I work in Labtop repair and our main customer is a Major IT firm that probably everybody knows (used to be very innovative in Hardware). And we've got Lenovo T14 Gen 3 (12000-series Intel CPU) and T14 Gen 4 (13000-series Intel CPU) dying like flies right now. Sometimes the replacement system bosrd dies with 2-3 months also. Could be the undersized cooling, could be the horrid Lenovo Thermal "Paste" (only at first*) or something else. *We replace the shitty Lenovo stuff with something a lot better as part of our service, especially when Systemboards/Mainboards have to be exchanged. Edit: typos, damn phone!

[@TheHighborn]

Where I work we use exclusively Intel CPUs. We're so fucked LMAO

[@alexturnbackthearmy1907]

@@alexanderzawydiwski9534 Its normal for some sata slots to not work, as well as ram not being detected in certain configurations. There is manual included with motherboard for a reason, some configs working well is a miracle (like 3 mismatched RAM sticks, sometimes it works, sometimes it doesnt, game of luck with ANY processor regardless of age, there is also good reason why mobo makers have a list of supported memory modules and configs for each one).

[@DustinShort]

Looking forward to the Gamers Nexus exposé 🤣

[@niyablake]

3 month later they pull out a 5,000 page reports complete with charts and 2 hour video

[@jorge69696]

And then for all the fanboys and bad faith actors to call it "drama" and dismiss everything they found.

[@protecteratr]

I like that they want to keep places honest. I just want a standard. Don't favor giving one company a chance to explain and make steps to fixing things then not go back to other companies that did the same. If said companies didn't change let us know too.

[@gazsoimi]

Intel execs be like: hey who is that long haired jesus looking guy doen there filming our hq?

[@Ikxi]

I'd watch it​@@niyablake

[@4RILDIGITAL]

I am also troubled by the lack of clear messaging and solutions from Intel. We do need more transparency to better understand and manage these hitches.

[@realsharpninja]

After watching the video and thinking about your findings, the thing that stands out to me is your observation of the tick rate on the game servers falling by 50% before the crashing. This tells me the CPU pipelines are being invalidated, which is what's impacting performance so badly. Whatever is happening to cause these pipeline invalidations is the real culprit, and I have a suspicion about what it is. I believe the CPUs have new microcode to detect attempted side-channel attacks, and when that is triggered the CPU intentionally invalidates the pipelines on all cores and additionally flushes the L1 cache. This would immediately flush any security keys from the CPU causing the side-channel attack to whiff on stealing something valuable. Next, the L1 cache needs to be reconstituted, including reloading keys. With NVMe drives, you can realistically expect 12+ CPU cores requesting the same key file simultaneously would work fine, but I suspect the IO errors being encountered are OS-level locks causing the cache reconstitution to fail, thus crashing the process, and possibly even the kernel. Another cause could be actual side-channel attacks, and the CPU trying to protect itself. If the defense strategy chosen by Intel is to crash the CPU rather than allow compromising data then they would not want to publicize that because now you have an instant mechanism to launch a DoS attack against a service or service provider. For end users, I think that certain users are affected more frequently because their system has been compromised and is under frequent side-channel attacks. So this begs the question, which is worse? Do we have less active side-channel protection, probably increasing risk of keys being stolen, or do we have active protections that ultimately choose to unalive the CPU rather than allow keys to be leaked? I would see if you can dig up any data on the occurrence of side-channel attacks on these CPUs compared to the older 12900k systems. Final thought here, the reason that slowing down the memory improves stability is that at the lower clock speeds, the mechanism in the CPU that monitors for side-channel attacks is able to better analyze the flow of data coming into the CPU and makes less frequent false positives that trigger the failsafe mechanism.

[@benjamin3044]

I always learn so much whenever Wendell just wants to do something like play Dwarf Fortress.

[@FennecTECH]

This isnt armchair science my man. This is absolutely real research. You’ve negotiated access to data not really available to the public and doing real research on the data instead of making educated guesses as to whats going on which is what armchair science is

[@johnsmith-gs4qf]

My work pc uses a 14900k. Great chip when it works (fast). Unfortunately, the PC crashes all the time. We were blaming the software, but it looks like it's been the hardware all along. I use AMD on my home computers, and I'm sticking to that (rarely crashes).

[@KeepIt1HuniTT]

I bought my 13900k in October 2022. Many 13900k and 14900k owners didn't experience any problems until this year. I find it very odd that so many of us experienced issues pretty much at the same time simultaneously. Most would say the power profile caused the wear and tear but practically all at the same time? Different motherboards, different profiles that are aggressive and non aggressive? Different levels of use? Many working PC gamers aren't on their computers for long periods. Really strange.

[@techwandoUS]

I made a video about this a year back, a long with Tech Yes City. A lot of people called me crazy. Stability issues, crashing, latency... List goes on. I appreciate you talking about this.

[@IntelArcTesting]

He has also made another video on it recently

[@techwandoUS]

@@IntelArcTesting This is true. Thanks man.

[@QLPJosh]

Wow, it is great to hear people talking about this. We deployed some i9 machines at work, and after awhile 2 of them started crashing for no reason. The, usually really good, Dell diagnostic software could never catch it. Reformatted, replaced RAM, SSD, etc, no luck. I saw deep on a Reddit thread to disable turbo boost and it resolved the issue. I gave this information to Dell and they ended up replacing the CPU on these machines. I've been helping desking and admin-ing for nearly 10 years in various sectors, and saw less than 5 CPUs die in that time, then I had 2 back to back. Was mind blowing

[@razgrizadler]

I had been an Intel user all my life up to my last one the Core i7 5930k It was perfect until it gave up on me after 7 good years of constant OC. I took the plunge and got AMD Ryzen 7800X3D for my new gamin rig instead of 14900k and now... I realized that I just narrowly avoided the path of suffering. I hope Intel gets back in the game soon. AMD CPU is so good now or even better in many aspects. My dad (R.I.P) who was a die hard fan of Intel would never believed AMD would be able to pull this.

[@_Ingcom]

I’m so relieved all my machines are AMD right now

[@oscarcharliezulu]

Hmm the data center support information - even tho it’s indirect is very interesting.

[@ActuallyHardcoreOverclocking]

Could you get the AC/DC LL settings of an unstable Supermicro board?

[@DamienRamirez]

I love how people ignore the goings on with the 12900K & KF's which are suffering the same issues be it not as bad (lagg/stalling but not hard crashing)

[@mrwolfe0812]

Your intro just hits right, and I don't know exactly what it is, I just really like it.

[@ThioJoe]

I’ve had so many bizarre issues lately and I’m now wondering if they’ll disappear if I swap back in my 12900k instead of my 13900ks 🤔 A couple of them highly repeatable but one that others can’t reproduce

[@Sekir80]

Good to see you here!

[@urielalbertodiazreynoso6309]

Not reported Internal high temperatures can lead to undefined behaviour. High Current leakage can be interpreted as a wrong bit.

[@RayneAngelus]

From the sound of it... yes. Put your not-just-a-refresh 12900k back into service.

[@RowdyDandy]

@@ThioJoe Just try it, for science! It wont be that much of a performance drop, and if you d hopefully get a more stable system out of it until this is resolved ;) Post your findings on the L1 Forum if you actually do it :D

[@UnrealOG137]

Try it and make a video about it!

[@garrettkajmowicz]

This is slightly terrifying. A problem that impacts perhaps 0.1% of enthusiast customers is a poor experience, but I can appreciate how difficult it would be for Intel to root-cause the issue. If it's 10%, that speaks to a product design issue which *should* have been caught much earlier. Intel's own internal testing should have caught this and resulted in mitigation before customers noticed, and preferably before the chips shipped.

[@ponocni1]

Thing is, if its getting worse over time, this smells like hardware defect in fabrication or material failure, not design flaw. I would not be surprised if we find out this down the line.

[@zodwraith5745]

@@ponocni1 But then you should see the same issue cropping up with i7s and i5s, which we don't. What's so different with the i9s? If it's that 8 e core cluster then why doesn't disabling e cores help?

[@bofast]

@@zodwraith5745 i9s are binned for higher max boost clocks and more power, so maybe whatever criteria that requires also makes them more vulnerable to this problem if it's an issue with them having pushed their manufacturing process to the breaking point.

[@LucasHolt]

@@zodwraith5745 It is happening with the 14700k. A number of people have reported it and I've seen it with mine. Bios updates helped some and I have run it with Asus MCE disabled the whole time. I've got a custom loop so it's not a cooler issue. (420mm + 120mm + 280mm rads too)

[@zodwraith5745]

@@LucasHolt But what is the problem specifically? I've heard it seems to occur mostly with specific games. I haven't had a hiccup from my 14700k but I'm on MSI. Keep in mind most of these issues were on ASUS boards, a few with Gigabyte, and I've never heard of an MSI or Asrock occurrence. Although I think the narrative changed fast because Intel is such a juicier target. But what Wendell is talking about in this video isn't Z790, he's talking about W680. *_ASUS_* W680. A SERVER platform that uses consumer CPUs. That means 24/7 full tilt all cores running. That's why the error rates are so much higher. I haven't had an issue but I wouldn't be shocked to see my system throw up an error every few hours under _that_ kind of abuse.

[@mcwolfbeast]

This kind of gradual failure rate feels like an electrostatic issue. ESD damage will do something VERY similar over weeks or months. Maybe the silicon design has electrostatic buildup somewhere in use, especially at higher power limits or speeds?

[@jinxterx]

I'm never buying Intel again.

[@4brigger]

I'm glad they got the boiler snake a sunning lamp.

[@phyro4143]

**chrome crashes continuously while watching this video with my never overclocked 13900k**

[@someperson1829]

You never overclocked it, but your motherboard did. And that's the problem.

[@backupplan6058]

@@someperson1829no the problem is Intel and goes well beyond overclocking when it’s seen on boards that actively avoid overclocking. What we where seeing is those systems failed sooner due to it but failures are starting to be seen across the board on systems that were well within safe limits.

[@someperson1829]

@@backupplan6058 Wow. There are LGA 1700 boards that don't activilly shove overclocking. How interesting.

[@killswitch8493]

@@someperson1829 standard specs for these chips seem to be already quite far off the good envelope and inside a territory of power consumption, stability and longevity that would normally not be accepted from a chip at factory specs.

[@SaHaRaSquad]

@@someperson1829 As the video mentions this problem also occurs with server motherboards and no overclocking.

[@JureRepinc]

Love that KDE Plasma wallpaper on the screen in the background :) And I am so glad that i went with Ryzen this time. I missed the AM4 era but I sure hope AM5 lasts as long as AM4 did.

[@redsnow846]

Its not just 13th and 14th gen. I had this exact same problem on a 12700k. Just erratic gremlin type behavior that i was never able to resolve until i replaced the cpu. I litterally replaced every other component.

[@DingleBerryschnapps]

Nope. Must have been something else.

[@okesik]

The sadest part about this is that I have been telling users it's their CPU ever since it started poping up en mass with UE5 game releases, so over a year now. Hundreds of cases accross multitude of games even outside UE5. And even if I link them the articles now that mainstream media picked up the story they will not even bother reading it and simply reply "but it's only in THIS game".

[@maltalent]

they're NPCs, there's no saving them !

[@iyaramonk]

Yeah I get similar shrugs

[@BrendonGreenNZL]

It's only in THIS game ... for now.

[@Nolano386]

I gotta say, the intro was a solid hook. You had my interest and now you have my attention.

[@b-ranthatway8066]

I've made the mistake of getting an i9 13900k and an Asus Rog Strix Z690-E. I had zero problems when using my EVGA 1070, but as soon as I got a 7900xtx and upgraded from 1080p to 1440p - 180hz, I've been getting green screen / black screen crashes when just applying a gpu fan speed profile and loading into games. I just updated my mobo bios from 2300-ish to 3600-something and so far no crashes, but I know I wont avoid the inevitable. 😅 Ready to switch to AMD.

[@wsck694]

MSI recently released the 129 microcode, which I hoped would address the voltage issue. Initially, I thought I'd still need to set my P-core ratio between 54-57. However, I'm pleased to report that after installing the 129 microcode and keeping default settings, I've experienced no problems, even when using Edge and Chrome browsers. Surprisingly, I didn't need to adjust the P-core ratio at all. I can't believe how long it took them to get to the bottom of this... I guess it's better late than never, but the delay was frustrating for users dealing with the issue. After running for a day status break point error appeared in edge again back to running pcores at 57 again.

[@otaking71]

Based off my experience overclocking these things......I'm pretty damn confident in saying the problem lies in either the ring bus or the system agent is just at it's damn limits. I would more than be willing to bet on the VAST majority of the true raptor lake dies that are failing are all going to be larger 16e core dies. It never sat right with me how wildly the power consumption on these chips was affected by the ringbus and how minor bumps in speed took absolutely shocking amounts of power and tons of system agent voltage fiddling. In retrospect, this explanation falls in line with how cagey intel has been about the SA voltage in the non-z chipsets. Basically we're seeing the limits of one of these two things and that it's so close to the limit that natural degradation of the silicon will cause faults.

[@otaking71]

The craziest example I can remember is an alder lake cpu that would do 5ghz ring.....at about 400watts. Drop it down to 4.8?...300 watts. Drop it to 4.6...250...and 4.4 only took 220-230ish. E cores were all disabled and the p core settings were the same on all tests...the only thing that changed was the ringbus/cache.

[@LiveType]

Seconded. I high suspect uncore interconnect degradation being a proximate root cause. Linpack won't catch it failing until things catostrophically fail later down the line. Harder to validate. "Just shove more voltage into it and it'll be fine. How long do you expect users to run these chips? 15 years? They'll be obsolete in 15 years." Try less than 1 year past intel execs. I run my 13900k with turbo disabled so basically sitting pretty at 0.8v vcore about 95% of the time. Bought it second hand for $300 about 5 months after launch due to "issues hitting ddr5-8000" pre-dellided with a custom copper ihs. Person I bought it from got a second chip direct die cooled it and from what I can tell that new chip worked just fine. Likely doesn't anymore. Lol. Probably swapped it out for a 14900ks. My chip still shows signs of degradation. SA was at 1.3V and vdd at 1.35V for 96gb dual rank ddr5-6800 for over a year. Now experiencing random latency spikes in the past 6 weeks. Even reinstalled windows as it's been over year but no change so it's not software related. Memory test passes at 32 hours as I don't have time to test for longer. Here's to hoping whatever releases this year is better than the hot garbage intel released or I'm switching back to AMD. Intel cpus failing were unheard of before really 11th gen. Started going downhill quick. 11th gen also had SA/IMC issues compared to 10th gen. It's telling intel is being hush hush about this as it's a huge blow to their QC reputation.

[@quanlethienminh6002]

Could it be that this is due to the AVX offset was not triggered probably in light tasks such as gaming? I mean we will never see this in all these stability test then, because the load is heavy enough in these benchmarks for AVX offset to kick in

[@otaking71]

@@quanlethienminh6002 It's not likely to be an avx offset issue. The way to understand what i've pointed out here (to the best i can oversimplify it)..is the ring bus is like a road/traffic network between all the cpu parts while the system agent is the traffic cop/monitor/system. The pcie controller and memory controller all sit on the ring bus. In the scenario where the SSD corrupts....the data can't make it from the cpu core down to the pcie controller. Add this explanation to what wendell is saying and it should make things much clearer as to wtf exactly is going on. Disabling/slowing down stuff doesn't change the fact underlying infrastructure is just pushed to it's limits the way it is.

[@otaking71]

Iirc, the 1700 socketed Xeons... have avx512 by virtue of having no e cores......if the xeons are ecoreless dies and no failures exist..we might having something of a smoking gun. I havent touched a 1700 xeon...and havent seen a delidded die shot

[@BReal-10EC]

This is unfortunately not uncommon when large companies deals with a big design/quality issue. When you have a reputation for quality... the last thing you want to do is openly admit you made bad things (and this also can open you up for lawsuits- like these server support companies looking to recoup all the extra service costs, or the company that lost 100k in their player base due to instability). Say vague things to sound like you care while claiming some type of user error. See how Toyota handled the 3.0 V6 oil sludge issues (but that was more a Toyota cultural thing), and how Apple handles EvErYtHiNg. Intel has been playing catchup to AMD Ryzen for years now.. after making fun of the chiplets being "glued together". All they really have left is their reliable reputation... so they are intentionally keeping tight lipped and hoping the media coverage of the issue just "goes away" after they release their next CPU generation - one that actually will be new and not just the old ones with more voltage.

[@damara2268]

That's exactly why this Intel instability issue should be headline on all big PC news websites

[@BReal-10EC]

@@damara2268 I bet Intel is really glad CrowdStrike pushed out a global Windows crash update filee. That gives the media something ay meatier to talk about versus Intel making bad CPUs that degrade quickly even when not OC'd.

[@TurandSand]

had my 13900k super unstable and returned it. Next one burned through. Next 14900k after that was bad too and now im running a 14700k stable finally.

[@zag1productions]

We have a couple Dell Precision 3660 Tower CTO in our Datacenter with 13700 and 13700K - they crash consistent when operated longer than 7 days with IO-Errors on the NVME PCIe bus. Equipped with 128GB ECC memory and Samsung Enterprise SSDs it is not possible to run a storage space or consistent VMs under Hyper-V.

[@la009895]

This is how you do investigative reporting!

[@josuad6890]

Intel's is counting on their reputation and brand name to carry them through since the 11th gen.

[@Odin3v]

They have been doing that since 9th gen........

[@Disnamesucksass]

You mean 7/8th Gen?

[@TheUniverse915]

If not longer

[@LethargicSquirrel]

I'm curious if you've found much data involving non-i9 CPUs, or rather, specifically involving the i7 CPUs. Here's why: TL;DR: I have two i7 series, a 13th and 14th gen. The 14th gen is currently experiencing the problems mentioned. I believe it is cooked, and will be RMAing it shortly. Long Version: I've got a 13700(non-K) and a 14700K. The 13 ran so hot with an air cooler that I decided to switch to watercooling for the first time ever. I figured since I had to redo that much, I might as well upgrade and bought the 14 (this was before I found out that the 14s were just rebadged 13s). Got lazy, and kept using the 13+air for a couple more months before I finally put in the work and did the swap. That was back in late December, early January. Before I continue, I should note that even though I bought a K CPU, I did not do any overclocking besides what the default settings on my Gigabyte Z790 Aorus Elite AX mobo was doing. I even had XMP disabled on the RAM because testing with Cinebench showed no difference between XMP Off (4800MHz) and XMP On (6400MHz). I left everything else untouched, including Gigabyte's default "Optimized" settings, which I assumed meant 'optimized for performance + efficiency/power', but turned out it meant 'automatic overclocking'. A couple weeks ago, things started getting funky. The game I've been playing (Sons of the Forest / SOTF) was running fine, but other applications around Windows were running slower than usual. Then I went to host the game one night, and the machine BSOD'd... then began boot-looping. Eventually, I got it back into Windows... briefly. It BSOD'd just loading Discord. It began bootlooping well before getting into Windows, and even got to the point where it was experiencing hard crashes just sitting in the BIOS. I did some BIOS flashing to bring it up to date, and the machine started booting a little further again, but still resetting before loading into Windows. I had just made a Ventoy USB stick a day or two before the issues began, which included Linux Mint, which includes Memtest, so I ran that. Tens of thousands of errors began accumulating within 10 seconds of starting the tests. A bunch of troubleshooting and configuration changes later, I was convinced it had to be the CPU, so I dropped the 13700 back in. Worked flawlessly, and faster than the 14700K had been working in at least a couple months. I was ready to call it and see about sending the 14700K into Intel when an annoying little thought popped into my head: What if it's just a pin-contact issue? What if it works if I pop it back in? So to avoid Intel receiving it and telling me there was nothing wrong with it, I tried it. It worked! No BSODs through several Memtest runs, several Cinebench runs, or playing SOTF. So I assumed it was just a bad pin contact or something to that effect. That was a week ago. It has slowly been degrading ever since, to the point where it took 20 seconds to load the Windows calculator. It now fails to boot and the BIOS interrupts and tries to get me to revert to "Optimized Defaults", but it turns out I can choose "Enter the BIOS" and "Save & Exit" and it will boot fine. Choosing "Exit without saving" results in another boot failure, though. Other desktop applications were running obscenely slow, too, but oddly enough, SOTF ran fine. I found Gamers Nexus' video on this topic and watched it. Went into the BIOS, disabled the iGPU (being used for my second, non-gaming screen), disabled TurboBoost, disabled Gigabyte's overclocking, and turned off every other performance-enhancer related to the CPU, along with turning off the E-cores. I even dropped my RAM from 4800MHz down to 4000MHz. Now, instead of sitting around 60ºC while playing SOTF, it sits around 39ºC. According to HWMonitor, the core voltage doesn't go over 0.925v. But it still fails to boot without pulling the trick above. So I think my 14700K is just as cooked as the 14900K's you mentioned. I don't think it's limited to the i9 series.

[@ZVexxPlayer]

Interesting I had two board swaps and a cpu swap, still same issues, thought It was just crazy bad luck.

[@nuttinspecial88]

Maybe you should do something similar to what Steve/GN do sometimes and set up some form of communication so devs/data center admins can share data to hopefully get closer to a real answer? Heck, maybe even get physical examinations of problem CPUs if possible

[@CMDR_bravoMike]

A Wendell x Steve investigative piece on this issue would be amazing

[@arthurmoore9488]

@@CMDR_bravoMike That would be great, but I don't want to bankrupt GN. They always spend "yes" money on their investigative journalism.

[@LucasHolt]

I'm seeing a lot of people confused. It's not just i9 chips! The 14700k is also a POS. I've seen others with issues as well. People talk about the 14900k more but it's still a problem for the rest of us with 14700k chips too! I have not tried to overclock mine. it's running with DDR5 5600 48GB corsair modules. Custom loop with 420mm + 120mm + 280mm radiator. 6900XT GPU. It crashes in two different operating systems! I dual boot Windows and MidnightBSD. In the latter, LLVM clang will start crashing during long compiles (10+ min). On windows, I mostly game and that is also unstable. I made all the recommended changes that intel proposed. I turned off asus mce on the first day of ownership last november. Worst CPU I've ever owned. It's slower than the 3950x i had before for compiling and with all the nerfs to make it stable, it's starting to lose the gaming advantage. Huge waste of money.

[@quademasters249]

@@christopherlindloff8331 25% failure rate means 75% with no issue. There's no guarantee you're going to keep being this lucky in the future though. For many the problem starts small and progresses.

[@WaterspoutsOfTheDeep]

Hear anything about this also affecting the 14900hx laptop cpu? Or they safe?

[@LucasHolt]

@@WaterspoutsOfTheDeep I've not seen anything about laptop CPUs either way.

[@charonme]

you don't have to "try" overclocking it, turbo boost is usually turned on by default so it automatically overclocks for you. If you want to avoid overclocking you'd have to turn off turbo boost.

[@kylehutto8044]

It honestly seems like intel is trying to quietly smooth this over with patchwork "solutions", in hopes of avoiding the backlash that would come from a more unified, public announced one. Because if they basically come out and conclude what seems to be the obvious takeaway: that 13900/14900 were running in a capacity that caused long-term silicon degradation, therefore the 13900/14900 that was released was not a stable for long term use version and was done so in order to achieve benchmarks and reviews that would allow it to take unwarranted leads over competing products, that would mean the entire product line need to be re-reviewed with new BIOSes and updates that would put it solidly behind even 7000 series non-X3D processors in gaming and well behind them in productivity. It would be a scandal that would shake their user base in both business and the enthusiast class, both segments that intel has desperately tried to hang onto.

[@nian60]

Here from the GN video. Thanks for the information.

[@tk72231968]

Thanks for this breakdown of your hard work. It reinforces my decision to change from Intel to AMD for my most recent build. I migrated from a 10900k to a 7800x3d, and so far I am very satisfied.

[@BeeWhere]

My first two computers used Intel. And once I saw AMD really supporting their sockets for longer, I decided to make the jump and couldn't be happier. More powerful for what I need and less power hungry than Intel.

[@greg8909]

@@BeeWhere Yes, less power = less heat = less noise too.

[@JarrodsTech]

My 13900K (running at stock) has been rock solid for about a year, hopefully I don't have a ticking time bomb 😬

[@urielalbertodiazreynoso6309]

I hope high temperatures and very low size processing technology don't degrade the chip faster than you should expect.

[@RodrigoLobosChile]

Is this @Intel PR speaking through @JarrodsTech account? 🤑💸💰

[@Born_Stellar]

mine is also fine, however I noticed a problem in a single core if I overclock. solution was to just overclock the other cores. but mine is direct die so the temp is like 40c under load.

[@josejorgegranados661]

i have a 13900kf at stock settings with intel power limits rock solid since january 2023

[@SVT_LIGHTNING]

My 13700k / 4090 build has been pretty much flawless for almost 2 years now. Even started out of a ddr4 board for about 5 months until msi released the z790 tomahawk ddr5 board. But 13700k oc to 5.6ghz on all 8 p cores, All 8 E core oc to 4.4ghz, Ring oc to 5.0ghz at 1.34v. Even the mem controller has been very solid. 32g.b Corsair rgb vengeance 6400 cl32 hynix A-Die oc to 7200 34-41-41-83 at 1.45v. On a msi z790 tomahawk. I can't complain. I've also been using a contact frame since day one for both boards I've had this 13700k in. I've heard a lot of the problems could be related to the bend they get with the stock mounting bracket. But until intel releases a statement / fix we really don't know what the issue is. I've used mine hard for over 8 hours a day every single day for work and gaming for almost 2 years now and it runs perfectly fine. I haven't babied it and I won't. I bought a k series for a reason like all the k series intel cpu's I've had.

[@hammerth1421]

Could the slowdown before the hard crash be a self-catching mechanism? I think Linus said at some point that Nvidia GPUs seem to have the ability to detect almost-crashes and pull back the clock speed for a few dozen cycles (very different time scale, I know) which seems to be part of the reason why clocking the GPU higher can actually result in lower benchmark scores before it then becomes completely unstable if clocked even higher.

[@Dimon12321]

I'm surprised there are no sue cases yet, knowing that there are countless cases when Intel refused refunds

[@JohnnyGomezSn]

"I'm not really sure that I got good Intel from those companies" Yeah, me neither

[@chieftron]

From my extensive overclocking on my 13600k I've had this issue as well. What I've found from my testing is that it seems related to heat/power spikes. What I've noticed is that when at the edge of the thermal envelope +80c, that there could be times where it spikes over 100c but not record it because of how fast it can spike before the temp is reported. But ultimately, the instability comes down to voltage fluctuations and temp fluctuations. When you lock the voltage and keep the temps below 80c on max prime95 loads you'll have a pretty stable system.

[@elfferich1212]

Then maybe that’s the way to go for now on those chips? (At the cost of some performance)

[@chieftron]

@@elfferich1212 I am currently in my "summer clocks" Which is standard boost clocks on both p and e cores, but massively undervolted. By default my chip wants to run at 1.37v to achieve stock boost clocks. However, I can achieve this with 1.16v on load and 1.08v on idle. Yes, I have my load line calibration set where it adds slightly bit more voltage when under heavy loads so it doesn't have vdroop. But with this settings I always have a temp 6-7c above ambient on idle and temps not exceeding 75c on full load. This is even with ambient temps of up to 38c. Yes, ambient temps of 38c, that is not a typo, and yes it sucks badly when it's that hot in here. Specially with 70+% humidity.

[@giornikitop5373]

@@chieftron i feel ya. high temp + high humidity is my pain also, it just hits differently. intel's solution to faster cpus seems to be just more voltage on the cores lately. i've seen some crazy voltages in boosting cores on some 12th gen. always thought it was a misreport from the tool, but now? not so sure anymore.

[@TankEnMate]

Sounds like the memory controller / cache interface. What would be interesting is snooping the cache bus and tracking locking issues; in particular seeing if there are any illegal MESI protocol transitions. Of course this might be getting bad over time, i.e. silicon degradation that wasn't picked up during the early days, A silicon etc. The reason I say this is that decompression code really stresses this part of the CPU. Also branch prediction but memory timing seems to be a trigger here and the branch prediction unit is wholly within the CPU (although it can lead to non deterministic memory fetches).

[@Kronosxviii1]

From what I've heard they're having issues due the size of things for this next generation. See the smallest part of the transistor is the gate (control mechanism) and it's been in the 10nm-5nm range for a while now. Even though according moore's law we're supposed to continuously shrink our technology so we can keep on increasing the transistor count every few years. But rather than shrinking things we've been using complex geometric structures inside the chips to increase component density instead, in an attempt to stay on track with moore's law. Now onto the size thing, while 5nm was fairly stable and with what we're seeing now is that as we go below that range we're seeing issues pop up all over the place because of the increased fragility of the structures within the chips. Overall we're fast approaching hard limits when it comes to current technologies on the mass fabrication side of things. For example, electron beams with a minimum 2nm diameter doesn't leave a lot of wiggle room when we're operating in a 1nm-3nm range for the critical dimension.