Why CUDA "Cores" Aren't Actually Cores, ft. David Kanter

We talk about NVIDIA CUDA Cores vs. AMD Stream Processors and why neither is actually a "core," featuring David Kanter of Real World Tech.
Ad: Buy the Corsair Dark Core SE on Amazon (goo.gl/3psBno)
Check out Real World Tech here: www.realworldtech.com/
Read David's article on TBR: www.realworldtech.com/tile-ba...
David Kanter of Real World Tech (formerly Microprocessor Report) joins us to discuss why CUDA "Cores" aren't actually cores, later explaining the differences between CPU & GPU cores, stream processors vs. CUDA cores, and more. The discussion follows GPU architecture and explains the building blocks of a GPU, with details on streaming multiprocessors (SMs), multiply-add operations, texture map units (TMUs), and so forth.
This is a great opportunity to learn from an expert about GPU architecture basics, and to help demystify some of the marketing language used in the industry.
We have a new GN store: store.gamersnexus.net/
Like our content? Please consider becoming our Patron to support us: / gamersnexus
** Please like, comment, and subscribe for more! **
Follow us in these locations for more gaming and hardware updates:
t: / gamersnexus
f: / gamersnexus
w: www.gamersnexus.net/
Host: Steve Burke
Expert: David Kanter
Video: Andrew Coleman
Links to Amazon and Newegg are typically monetized on our channel (affiliate links) and may return a commission of sales to us from the retailer. This is unrelated to the product manufacturer. Any advertisements or sponsorships are disclosed within the video ("this video is brought to you by") and above the fold in the description. We do not ever produce paid content or "sponsored content" (meaning that the content is our idea and is not funded externally aside from whatever ad placement is in the beginning) and we do not ever charge manufacturers for coverage.

[@GamersNexus]

Our thanks to David Kanter for joining us. It's old now, but we think you might really like his article explaining tile-based rasterization on NVIDIA GPUs: www.realworldtech.com/tile-based-rasterization-nvidia-gpus/ Check out our DDR5 memory news: kzbin.info/www/bejne/m6elfHycd8p6gM0 Back-order a GN Modmat here for the next round! We sold out already. Again. store.gamersnexus.net/

[@d3c0deFPV]

Thank you, David Kanter! I'm very interested in machine learning and have been messing with Tensorflow recently. It's amazing to see stuff like random forests, which were invented way ahead of their time, finally being used for things like neural networks. Good stuff, would love to see more videos like this.

[@TiagoMorbusSa]

And the GPU is throttling for sure, because the draw distance is dynamically reducing. The FPS are rock solid though, good job.

[@error262]

Draw distance is a bit low, did you forget to set it to high in settings? Tho hair physics are spot on, are you using TressFX?

[@ObsoleteVodka]

Error989 Is the exaggerated levels of depth of field.

[@michelvanbriemen3459]

Rendered using N64

[@Markel_A]

They had to turn the view distance down to reach 60FPS at 4K.

[@JosephCaramico]

They had to reduce draw distance to get better out of the box thermals.

[@mhoang_tran]

Steve HairWorks

[@ggchbgigghb7699]

David sounds like the kind of guy who could take a few empty soda cans, some plastic spoons + duck tape and produce a 1080ti in the garage.

[@abdulsadiq8873]

someone get that man a cape!

[@AwesomeBlackDude]

OR" rebuilt Radeon AMD Vega to... R.A.V.2.

[@TanTan-ni4mg]

Ggchb Gig ghb David needs help wiping his own ass. Show me something he has done in the last year? 2 years? 5 years? Ok, lets make it easy, EVER?

[@josh223]

Tan Tan hes done more than you

[@TanTan-ni4mg]

joshcogaming proof?

[@Jelle987]

So that could imply that the Corsair Dark Core SE does not contain an actual core either? /mindblown

[@GamersNexus]

Brought to you by the Corsair Dark Lane on a Vector Unit SE! Doesn't quite have the same ring to it.

[@DoctorWho14615]

It really has stream processors.

[@charlesballiet7074]

So if bob owns a orchard with apple trees and its over 400 apples per tree.....(does math) omg bob has over a million cores on his farm, hope its a crypto farm

[@samuelj.rivard]

no its a apples farm... xD

[@thomasanderson8330]

some computer mice have actually an embedded microcontroller. these run their firmware what is a tiny bit of software. that does probably some stuff to smoothen your mouse movements and to detect when you lift up your mouse. this mouse could actually have a working processor in it. nothing powerful but it executes commands and processes sensor data

[@sliceofmymind]

THIS! Steve, this is the type of content I love. Pretty please, keep this kind of content coming!

[@markp8295]

Frank R If you're smart enough to keep up with this, you're smart enough to know why this cannot be the regular content. Sorry to say but it's views that pay the bills and thus making it accessible to the majority is financially the best option. Leaning either way too often kills revenue.

[@sliceofmymind]

I discovered David Kanter through Arstechnica (one of my favorite tech/science websites) and learned a lot about chip engineering. While that website tends to cater to a rather technical demographic, Gamers Nexus has come close to matching their attention to detail. There is a lot of science behind Steve's content and his audience (based off his current subscription total) is far from the majority that visits, say LTT.

[@Atilolzz]

The inner machinations of my mind are an enigma! -Patrick Star

[@joeykeilholz925]

Atilolzz poetic

[@DarkLinkAD]

**Bill Fagerbakke

[@yledoos]

Really loving the technical topics GN has been presenting lately

[@carniestyle6454]

Steve's hair flowing majestically in the wind.

[@nO_d3N1AL]

As a Computer Science PhD student I find this channel to be an amazing source of technical information that, at least for bleeding edge hardware, can't be found anywhere else. Keep up the great work!

[@MultiMrAsd]

The most interesting thing about CUDA cores and SMs (at least in my opinion) is that one SM can render multiple pixels at once, by having it's different CUDA cores calculating with different values. But since there is only one instruction fetch and decode unit all CUDA cores must be state locked (meaning calculating the same instructions). This may result in wired behavior when running shaders with many branches, since all CUDA cores may run all branches, but discard the results of the calculation. This is why it can be faster to calculate everything the shader could need and then multiplying it by one or zero instead of using an if, since both parts of the if statement may get exicuted if the different pixels need to take different branches. This is something every graphics programmer should know, but many don't. Also that's why it's actually not that bad to market with the amount of CUDA cores, since that directly corelates to the amount of pixels that can be drawn simultaneously. (Please note that this is simplified you don't need to tell me that some details are not perfectly correct :p )

[@yumri4]

i think you left out to much detail but yeah

[@SerBallister]

A lot of graphics programmers are familiar with branchless programming. It's some what relevant to CPU optimisation as well were missed-branch predictions cost so much performance.

[@sjh7132]

How is always calculating what each branch needs first more efficient than letting one branch calculate (while the others are on hold), and then the others doing their thing? It seems equal to me, except that in the case where an entire warp takes a single branch, using the second method, you save the time of doing the calculations for the other branch. Given that a warp is only 32 ''threads', the odds of everyone taking the same branch sometimes aren't that bad.

[@metatechnocrat]

listen man, you left out some details. I want to speak to your manager.

[@npip99]

​​​@@sjh7132 Because splitting the work across cores isn't free. When the GPU is executing code inside of an if statement, it needs to enable and disable its execution flag for the cores that aren't going to execute this cycle. Enabling/disabling the execution flag takes time. Additionally, doing so breaks the pipelining process, giving you less than 1 instruction per cycle before and after the execution flag is changed. (Instructions usually take several cycles to execute, the reason why cores achieve 1 instruction per cycle is because they're pipelined) If you have short if statements, the enable/disable issues are a significant fraction of the time spent executing the whole if statement. If the if and the else statement both have very long bodies but are doing something similar, branchless programming can often mean combining them into one body. For a CPU it doesn't matter, for a GPU it's twice as fast in warps that execute both sides of the if. Of course, if you're just wrapping a texture access with an if-statement, it's better to keep it an if than access the texture every single time (Unless you expect the if to happen a very high percentage of the time, but that's up to you)

[@trazac]

Everyone likes to forget about the GeForce 8 Series where 'CUDA Cores' had yet to receive their name and they were just called Streaming Processors. In the same way that Hyperthreading is often called an Intel thing but SMT is an AMD thing, yet they are one-and-the-same. Hyperthreading is just marketing.

[@SteelSkin667]

This was immensely interesting. I love the series of interviews you have been doing lately, it's great.

[@EmpadaoPerfecto]

Need to watch it again, I'm damn lost

[@S3rial85]

i think i am to stupid for this type of content......will watch anyway :)

[@AwesomeBlackDude]

Sebastian wait until the discussion about CGI real time DXR gaming technology.

[@oak8728]

it's two

[@Dracossaint]

Same, but it excites me to have a lane to further my understanding of computers and a direction to look for it also.

[@Advection357]

Basically a core is a self sufficient & independent structure that can access memory and other things on it's own. What the guy is saying is that he doesn't consider CUDA cores as true cores because they are not independent from the rest of the structure. A vector execution unit (which is what a CUDA "core" really is) is the part of the chip that traces 3D vectors, angles, magnitudes etc... it does the math to move & transform 3D objects. 3D graphics is mostly vector math. A vector is essentially a position in 3D space with a direction & an amplitude. Think of it as an arrow pointing in some direction. It's the fundamental structure of all 3D graphics.

[@AlienFreak69]

In shader programming, in order to create the illusion of lighting (lit/unlit objects), you have to multiply or in simpler terms; join two or more colors together. These colors are stored as floating point numbers or decimal numbers. Once all the math is done and the final color is defined, the pixel is drawn on the screen. Now before it actually becomes a pixel, during the math process it's actually a vector stored inside a matrix, not a pixel. That's about as simple as I can explain his multiplication process.

[@Multimeter1]

Steve can you do technical analysis just like his some time in the future? It’s always great to get sources from multiple people. As shown at the beginning. I love in depth info of how computers work transistor level.

[@Cafe_TTV]

This guy knows what the hell he's talking about!

[@SCrowley91]

Videos like this are why you guys are my favorite tech channel! Keep them up! Educating the community is so valuable, I'm glad you guys take the time to get into the real details!

[@Gahet]

Awesome video, I love this discussion format. This is a really interesting topic, explained simply enough to make sense to me, but just far enough over my head to get me to do some additional reading. I love it! I would really enjoy more videos like this. o/

[@danshannon1001]

You really Hammered that answer out !!

[@danshannon1001]

Sorry but had to it . !! lol !!

[@MentalCrusader]

I really like all those interviews you are putting out lately

[@BillyMizrahi]

This is good content. These technical videos are a great change of pace compared to the usual content GN produces. Love it.

[@PJosh23]

Thank you! So many questions answered! And then some! I appreciate videos like this Steve, keep up the great work, love your channel!

[@OGBhyve]

Love these highly detailed videos!

[@RamB0T]

I loved this talk, so much great information packed into 17min!

[@chaoticblankness]

More David plz, always a treat. I really miss the old TechReport podcast days.

[@LandoCalrissiano]

Love this this kind of technical discussion. Keep them coming.

[@orcite]

Thanks for this one! Learned alot

[@jacoavo7875]

I like the use of "knocking on heavens door" as your backtrack throughout the video.

[@gravitationalpull1941]

Great and informative as usual. Thanks for the knowledge!

[@VithorLeal]

Great video! Loved the depth of it! Want more!

[@osgrov]

Interesting chat. I love these little talks you have with people now and then, please keep that up! No matter if it's random cool people or industry insiders, it's always interesting to hear people who know what they're talking about, well, talk! About stuff they know! :)

[@landwolf00]

I really like this type of content; in depth technical descriptions of how GPUs are excellent at linear algebra!

[@doppelkloppe]

This video aged really well. Thanks for making these Videos, Steve.

[@issaciams]

I love these intelligent discussions/interviews/explainations. Please continue doing these GN. 😁

[@MarkJay]

great info in this video. Thanks!

[@rowanjugernauth5519]

Very informative. Thanks!!

[@Jdmorris143]

Videos like these are the reason I come to you. Also because you are honest about your reviews. All hail tech Jesus.

[@janhrobar782]

very nice, I really enjoy it, I wish more talks with this guy

[@spinshot6454]

Thank you for a great piece of content. This is exactly why I subscribed in the first place, keep it up!

[@jadedhealer7367]

love these kinds of videos

[@cainzjussYT]

Thanks! This exactly the video i needed. This CUDA core thing was driving me nuts.

[@JordanService]

Wow this video is so excellent, David is amazing!

[@TJCCBR47]

Easy to fallow, great interview!

[@Najstefaniji]

hahahahaahah

[@ghosty1233]

Super interesting conversation!

[@BUDA20]

HAMMER TIME !, awesome video btw, keep it coming!.

[@theDanMicWebshow]

Loved the video, very informative

[@Xenonuke]

Someone's trying real hard to tell us a *knock knock* joke

[@lukefilewalker9454]

Very cool, knowledge is power. This video opens up a whole different perspective on how GPU's and CPU's differ but yet the same. Very interesting. Thumbs up!

[@mrnix1001]

I like this (minus the hammering in the background). Let's have more expert-interviews where they explain something in detail!

[@evansmith9586]

Best content GN has ever made! Thank you so much! Need more of this.

[@kaihekoareichwein9392]

Cool stuff, thanks for the content!

[@l33tbastard]

This is some good background noise. Thank you for that! :)

[@llanzas95]

Ivo Silva ambient noise lvl set to 100%

[@justinlynn]

Huge thanks for publishing technical material like this! It's really nice to get behind all the marketing nonsense to get a better understanding of the underlying technology.

[@Slavolko]

Please do more technical content like this.

[@Disobeyedtoast]

I want more in depth content like this.

[@ebwrightjr]

Great video!

[@TehPlanetz]

Bold content. Much appreciated. 10/10.

[@Advection357]

Great topic!

[@roax206]

just found this video and for someone who has a bachelor degree in computer science this makes a hell of a lot more sense than the usual jargon that is usually thrown around. In short a "core" is usually made up of: 1. A set of circuits that handle integer instructions (full number arithmetic/logic) or "Arithmetic Logic Unit". 2. A set of circuits that handle floating point instructions (decimal place arithmetic) or "Floating Point Unit". 3. An "Instruction Decoder" which directs the instruction to the right set of circuits (ALU or FPU) and converts the instruction into the electrical signals that go into that set of circuits. 4. lastly we have the "clock" which turns "on" and "off" at a set speed to differentiate between one instruction and the next. In this video he says the GPUs use a core setup where there are still only one "Instruction Decoder" and "clock" but a large number of FPUs designed for specific instructions. Thus what Nvidia calls the shader core is not the whole package but just the FPU that actually computes the decoded instructions. By limiting the core to only handle a small set of instructions and only use floating point numbers they can make these FPUs much smaller and cram many more of them into the same area.

[@haozheng278]

roax206 you forgot the fetch unit and the store unit, also modern cores have other units like branch prediction

[@AchwaqKhalid]

This is the exact piece of information that i was looking for

[@sobhan704]

Love such content!

[@BlitzvogelMobius]

I'd like Steve and GN to do a multi-episode detailed analysis of graphics architectures since the beginning of PCs. I've been PC gaming since the final days of vertex and pixel shaders (2005 & 2006), and the switch to unified shaders rightfully took the PC gaming world by storm. Interestingly enough, the first unified shader GPU - Xenos in the 360 - more or less were directly related to the pixel shaders found in the Radeon X1000 series.

[@meladath5181]

As a C++ Programmer (dabbled in GPGPU) I found this video very interesting!

[@MazeFrame]

Finally a video that goes into detail. Would have loved some more insight into Bulldozer.

[@Peter_Cordes]

Kanter's deep-dive article on Bulldozer is on his site: www.realworldtech.com/bulldozer/. Lots of block diagrams and detailed analysis comparing BD to previous AMD and Intel microarchitectures. Bulldozer is an interesting design, but several of its features were mistakes in hindsight that they "fixed" for Zen. e.g. Zen has normal write-back L1d cache vs. Bulldozer's weird 16k write-through L1d with a 4k write-combining buffer. Having 2 weak integer cores means that single-threaded integer performance is not great on Bulldozer-family. Zen uses SMT (the generic name for what Hyperthreading is) to let two threads share one wide core, or let one thread have all of it if there isn't another thread. Zen also fixed the higher latency of instructions like `movd xmm, eax` and the other direction, which is like 10 cycles on Bulldozer (agner.org/optimize/ ) vs. about 3 on Intel and Zen. Steamroller improved this some but it's still not great. Kinda makes sense that 2 cores sharing an FP/SIMD unit would have higher latency to talk to it, and it's not something that normal code has to do very often. Although general-purpose integer XMM does happen for integer FP conversions if the integer part doesn't vectorize.

[@NeimEchsemm]

Niceeeee :D Also some point that were not mentioned: Programming on a gpu is much harder than multi threading (use every core of a cpu) due to their physical properties And even for multi cpu, most of the time parralelising a workload is not so easy

[@gravelman5789]

Good interview technique!!! i wish everyone listened to their guests as good!!! 👏👏😁😁🖒🇺🇸☯♾

[@RadoVod]

I like this type of content. Same as the one with primitive shaders. Keep it up!

[@dimitris.p.kepenos]

Best. Channel! This contend is amazing!

[@FrumpyPumpkin]

The legendary Kanter! Whooooooo!

[@ADR69]

This dude's awesome. Thank you

[@JasperSkallow]

Thanks David

[@Tubaditor]

Please makes these interviews much longer. When you are already interviewing someone who knows about that low level stuff that you should try to get even more information out of them. Good interview. Keep going!

[@Valfaun]

i always thought it was strange that a GPU could have thousands of cores while CPUs are lucky to reach double digits also, knock knock

[@theDanMicWebshow]

Valfaun who's there?

[@Sliced_cheese98]

Daniel Bryant Me a nugger

[@Valfaun]

Daniel Bryant someone who wants them nerds off their roof, perhaps

[@grandsome1]

Best analogy I heard is that a CPU is a room with a few geniuses that solve complex problems while GPUs are like gymnasium filled with teens trying to solve for x.

[@joesterling4299]

More like a gym-ful of teens told to go paint one picket each on a fence.

[@blazbohinc5735]

A very interesting piece of content. Goof stuff, Steve. Keep em coming :)

[@rockboy36]

sick no-scope @ 8:04

[@il2xbox]

Recently I was thinking I wanted to know about real time raytracing and you made a video on it. I also wanted to know more about GPU architecture since I've studied CPU architecture but never GPUs. And here you are again making a video on it. Are you reading my mind, Steve? XD

[@nutterztube]

Someone was hammering for sure.

[@Mac_Daffy]

Nice! I suggest the next video should be about FPGA‘s and why they perform better for ray tracing.

[@silkmonkey]

Neat. I still don't get it, but I'm not a microprocessor engineer so what are you gonna do.

[@yumri4]

in short what GPUs do most of is a Fuse-Multiply-aggregate instruction the take 2 values in binary put them together using the "ADD" instruction then multiplies them together then adds it to the value in the holding place that it got the data from until the FMA stream is done and does it very fast as it does not need anything but the FMA instruction for them to use the other instructions done are done with less units or more that nVidia and AMD do not tell the public how many are in that block of their GPU's block diagram

[@jordanrodrigues1279]

@@yumri4 AMD is *very* forthcoming with architectural details. There's even an assembly language optimization manual for Vega that's about as detailed as the similar document for Zen/Zen+ CPUs. The part that's hard for anyone to wrap their head around is that a group of threads is forced to follow instructions together. It's like a three-legged race, except it's sixty-four people wide instead of just two per team and it's not just a race but each one is actually playing hopscotch and the hopscotch diagram is your shader program. AMD actually explains in moderately useful detail the rules of this game, which is miles ahead of nVidia. (nVidia is better at providing high-level APIs, at least if you don't mind pledging allegiance to nVidia hardware and their closed-source software; why, they'd *never* exploit a strong market position.) ((Full disclosure: I'm an AMD fangirl. But I'm really hoping that Intel enters the market in a meaningful way: they're pretty good friends of open source, almost as friendly as AMD is and they can afford to donate a *lot* of time and information. Better open source middleware will force nVidia to compete on raw hardware merits and/or price instead of the "we bought out the new and upcoming GPU ideas, all your gaming is belong to us." That said, nVidia hasn't been slacking the past several generations, it's just that they've but built a little too good of a money printing machine and it's making them *lazy*.))

[@yumri4]

@@jordanrodrigues1279 how does that relate to the fuse-multiply-aggregate instruction on the instruction set of the GPU? II agree they are more open about what they are doing but you most have misread or not understood what i meant. Both nVidia, AMD, ARM and Intel use some version of this instruction. The programmer never gets to see the binary most times as programming using APIs is a lot quicker and a lot more compatible with more hardware than programming using binary code. Now how they do it is different in APIs from it on a GPU to it on a CPU due to what is being "called" but it is the same operation being done. It is all math so you only have add, subtract, and store. That is all all microprocessor can do when the instructions and data is on the microprocessor.

[@dwindeyer]

Are the FPU lanes in each SM individually addressable or asynchronous? Or does each SM/CU have to run all the lanes in a single pass?

[@dteitleb]

I think a little clearer explanation is to point out that all the threads in a warp / workgroup share a single program counter. If two sides of a branch are taken by different threads then you wind up with branch divergence and the threads have to serially execute the two branches individually. Following that serial execution the thread scheduler can then reconverge the threads so they all run in parallel again. But basically if you're not branching then the performance of all the cuda "cores" behaves the same as if they were all full-fledged "cores"

[@Imhotep397]

So, is there a memory controller that’s doing the fetching and delivering for all of the clusters, some of the clusters or is that part of the CPUs job? (Not likely but I had to ask)

[@Michaelwentsomewhere]

Anyone know why the corsair mouse linked in the description os sold out everywhere? This video isnt that old

[@aplacefaraway]

During filming a construction worker is performing many hammer to nail operations

[@kevin-jm3qb]

Steve u gotta make a video on the break down of a cpu architecture

[@thevortexATM]

More vids like this!

[@MatrixJockey]

Dope video

[@disinlungkamei2869]

Nice talk

[@TheWheelerj5000]

I love the guy hammering in the back ground

[@mr.needmoremhz4148]

Good content, love the more in-depth stuff. This really separates your channel from the other tech channels!! Everyone talking about the same stuff in short videos is kinda boring. While i don't understand everything this triggers me to do some learning and read more.

[@callumturner9101]

Lovely weather.

[@unvergebeneid]

Is that talk he gave at UCS Davis online somewhere?

[@raredreamfootage]

Can you talk about how the fundamental "core" design changed when going from Fermi to Maxwell? I believe this was one of the paradigm shifts for CUDA.

[@ventisca89]

1. So that's THE David Kanter, amazing 2. As far as I know (before watching this video) the CUDA "cores" are more like ALU than complete core. Nice deeper explanation here.

[@hrvojeherceg2636]

That guy with hammer is really annoying 😠

[@AlienFreak69]

Steve looking over the edge to check if the guy is within spitting range before realizing he's on camera and proceeding to nod in agreement.

[@joer8854]

Omg. I thought maybe there was something wrong with my receiver. I know it's not their fault but holy crap is that distracting.

[@joer8854]

At least he had parents who gave a damn unlike you who acts like an asshole for no reason.

[@PSNGormond]

That's not a hammer, it's the audience banging their heads against the wall trying to follow the explanation!

[@nswitch1083]

We are at the brink of advancement in computing technology. The only problem is making the computing technology smaller without affecting temps and transfer. If we ever start using gallium nitride as a core component and replacement of silicon then we might be looking at the next jump in hardware evolution. So exciting.

[@samdovakin2977]

can you use hbm as cache in a cpu ?

[@kn00tcn]

2:47 steve mesmerized by his dream coming true

[@reinerheiner1148]

Was that smog in the background or just fog?

[@Kokinkun]

Great informative video! It's easy to tell who has lived in California when they give statements but make it sound like a question. Is this prominent in other regions?