What are Tensor Cores?

Here's the Nvidia blog explaining Tensor Cores in great detail: devblogs.nvidia.com/programmi...
Support us on Amazon! geni.us/RJ1nymB
Newegg link for cool kids: bit.ly/2AkBuRt
Join our public Discord server with Tech YES City! / discord
▶ Our studio gear on Amazon:
Sony A7 III (Primary): geni.us/VNQk
Sony A7S II (Secondary): geni.us/nGAGx
2x Pixio PX275h IPS Monitors: geni.us/MJISRP0
Neewer LED Lighting Kit: geni.us/9VTXr0
Audio-Technica AT2035: geni.us/xvQc
Yamaha MG10XU Mixer: geni.us/d4VvGM
Neewer Microphone Arms: geni.us/1w5Sx2Y
Tribesigns L-Shaped Computer Desks: geni.us/ZMHUcU
Arrowzoom Sound-Damping Foam Panels: geni.us/EKFxZc
TWITCH: / shiftystudioyt
FACEBOOK: / sciencestudioyt
TWITTER: / gregsalazaryt
INSTAGRAM: / gregsalazaryt
Subscribe to our channel!
MUSIC:
'Orion' by Sundriver
Provided by Silk Music
/ silkmusic
DISCLOSURES: All Genius links are tied to our Amazon Associate account, from which we earn a small sales commission. Links containing a 'bit.ly' reference forwarding to Newegg are tied to our CJ account, from which we earn a small sales commission. All sponsored links and comments will contain the word "SPONSOR" or "AD." Any additional revenue stream will be disclosed with similar verbiage.

[@TechLevelUpOfficial]

4:38 owh that didn't age well, we have Tensor cores in all Geforce RTX cards now lol

[@tomthb8439]

Don't count on it 😂😂

[@Arloh2222]

Seems you made an error, around 2:33... You completely confused flops and FP32/FP16 FP16 and FP32 stands for the Floating (point) Precison, basically how many bit's the value is stored in, not - flops, the theoretical max floating operations per seconds

[@anandsaha01]

+Science Studio, you should put a message in the video with the correction.

[@njr940]

Also consumer grade graphics cards do come with tensor cores now :P

[@JB-ym4up]

This+1. worse yet he made something up to cover.

[@MrJonojenkins]

He makes lot of these type of mistakes in his videos.

[@tuck1726]

I noticed that too he doesn’t know what single and double precision floating points are. Aren’t graphics cards measured in giga or tera-flops? 16 floating point operations per second would be awful.

[@KnifeChampion]

"Should we expect to see tensor cores in consumer grade graphics cards? Dont count on it." Who else is watching this after RTX reveal? xd

[@tylerdurden3722]

Lol...I guess no one saw this coming.

[@dans.8198]

@Ashwin Mouton: No one saw this coming ? AI is the next big thing, and the iPhone has got a similar hardware “Neural Engine” since the iPhone X released at the end of 2017.

[@wildkev1010]

The RTX 2060 has 240 Tensor cores

[@hemanthkumarHere]

and they find a potential application to boost gaming performance with it as well. DLSS 2.0

[@ClarkPotter]

@@dans.8198 another way to say "has got" is "has."

[@zamalek4079]

2:35. FP16 and FP32 refer to 16bit Floating Point number and 32bit Floating Point number. They are often called "half" and "single" respectively (there is also "double" which is 64bit, but is not really useful in AI). The reason for the inverted acronym is because nearly all programming languages typically require that a name start with a letter (but can contain numbers), so 32FP would be an error in most languages. It is not a measure of operations/FLOP because that would measure nothing interesting, it is a measure of precision. They determine the number of unique values that can be represented. FP32 is actually *slower,* because more transistors are involved in the calculation. You'd typically use FP16 because of the improved performance, only using FP32 if you needed the precision (which is extremely rare). This would be the exact opposite of what you'd expect if they stood for 16 FLOP and 32 FLOP, the latter would be more operations (ignoring that there is no time unit, again, a strange unit of measurement in this context). en.wikipedia.org/wiki/IEEE_754

[@Myvoetisseer]

Google is really going too far. Usually when I start googling something new to learn about it, I would have dozens of adds or suggestions thrown my way for the next couple of days. But this time, Google actually commissioned a guy on KZbin I've been watching to make a VIDEO explaining the concept I've been trying to understand. Wow, that's just freaky.

[@ChandlerUSMC]

Google then followed up with a call to Hugo's cell phone leaving a message asking him why he stepped away from his computer after watching the video. He didn't call them back so they reached out to his parents followed by his best friend from elementary school.

[@hercion]

Let's talk about cross device tracking. They really go too far. If I watch bikinis ( I am human, after all) those videos shall not show up in my other device where I Focus on science and tech .. Well KZbin is a monopolistic service. What's else? Vimeo? Google is a monster.

[@paranoyd1]

I don’t think it’s because you searched, or that he was even commissioned in the first place to make the video, but yeah we never know xD

[@Chemy.]

So it's mostly a different way to solve problems using better resources for a specific kind of operations, I watched the video two times to see if I got it right, great one Greg. Video liked as always

[@SSS-mp8th]

I love the minute science videos. It's one of the main aspects of differentiation you have compared to other tech tubers. I love the reviews and builds as much as the next guy but learning about what goes into the hardware or software is really interesting and not many do it. You video on nano-meter in regards to CPUs was really informative.

[@zeke7515]

They need to start making relaxor cores. They might chill out at Nvidia and drop some new GPU's

[@rtv190]

that's a joke worthy of a laugh track

[@React2Quick]

rtv190 are we talking about a laugh track that's used in moderation like in old TV shows? Or an overused laugh track in current TV shows?

[@mattgraves3709]

You sir are a worthy adversary. I absolutely love watching your videos thank you so much for explaining this term

[@josephbaker9673]

Greg: Tensor cores are not likely to be in consumer grade GPU's any time soon NVidia: Hold my drink

[@faezlimpbizkit]

The only youtuber that can takes him time to actually explain the engineering behind these stuff. Thanks Greg :)

[@CPMEDIACREATION]

not gonna lie this video was loooking spot on as hell, the camera is just making everything look so clean and crisp

[@GregSalazar]

Thanks mah dude.

[@CPMEDIACREATION]

Science Studio your welcome my man

[@SALSN]

A few corrections/clarifications, the matrices, as the text says, are 4x4x4 i.e. 3 dimensional. (not 4x4 i.e. 2 dimensional). And without knowing for sure, I will bet that FP16 and FP32 are referring to 16 bit (two bytes) and 32 bit (4 bytes) precision rather than anything to do with speed.

[@hantzleyaudate7697]

Simon Als Nielsen you are correct

[@udust1]

A matrix is 2D by definition. There is no such thing as a 3 dimensional matrix. Tensors can be 3d. But tensor cores are dealing with 2d matrices, and should really be called matrix cores not tensor cores, but sounds better for marketing. The FP part is indeed a mistake. tbh i dont really understand where how they can say that the "processing array" is 4x4x4. Multiplying two 4x4 matrices produces a 4x4 matrix, and adding two 4x4 matrices produces a 4x4 matrix. Where does the extra dimension come in?

[@udust1]

Actually when I think about it, each node in the 4x4x4 space probably represents one multiplication result each. So for each row*col you need 4x multiplication-accumulate to complete the calculation. Still we are only dealing with matrices (2d vectors), its just that you need 4x the matrix dimensions in muliply-accumulate operations. Imo visualizing the multiplication in 3d is just confusing, as its all happening in 2d. Atleast it confused me.

[@JP_Stone]

Really like the Techie videos. Found this one exceptionally interesting. Would love to see a video on why GPUs would not make good CPUs and Visa-Versa and the differences between them on an architectural level.

[@johnf.354]

I really appreciate the way you’re able to reduce these ridiculously complex descriptions into moderately complex examples. I still have no idea what a Tensor Core is/ does because I was just staring at the RGB in the PC behind you...

[@brockfg]

I use my 1080 ti for tensorflow ML on a daily basis so I was excited to see you had a video related to ML. Great video ! -UL alum

[@danbodine7754]

Hearing dot product got me triggered about cross products.

[@grantwilliams630]

Hey thanks for the video! I work in ML so its kind of fun to see videos like these pop up on my feed.

[@funbucket09]

Manatee Licking?!?

[@therealnmg]

Man I miss your channel so much!

[@xXDarthBagginsXx]

Great explanation, love these informative videos man 👍

[@GregSalazar]

Thanks!

[@dorryoku919]

You must be reading my mind because I've been looking for videos on tensor cores lately

[@tejasnaik14]

So what cores are likely to be used in the future for handling ray tracing in games?

[@spokova]

Thanks for Teaching us..! Good Job Sir

[@Maisonier]

So there is any software (no games) where I can use this tensor cores? some IA that organizes my files, desk, answer my emails and more?

[@Epiphany-cl7fe]

I Love to learn...more qna if you get time..I listen to it while im at work..keeps me motivated..sounds crazy... but it really does

[@GregSalazar]

Thanks for the support, Paul.

[@davidjenkins852]

Science Studio you're one of very few tech tubers who keep me motivated to learn too

[@Vizeroy9]

FLOP is derived from FP, not the other way round. FP16 is a 16 bit number, where the decimal point can be at any position of the number - hence floating point. FLOP/s is just the measurement of how many operations with this kind of numbers can be done per second - but the single precision FLOP count refers to FP32 numbers - 32bit floating point numbers.

[@joechevy2035]

Tensor cores sound like this saying from Bruce Lee. "I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times."

[@phantom1199]

but what about the new ray tracing that nvidia unveiled? isnt that meant to run on tensor cores?

[@DavOlek_ua]

but for rendering and modelind tensor cores should be beter? is it some kind of ACIS?

[@megalopath]

Excellent video! I watched the Google I/O on Tensor Cores and didn't quite get it (or at least recall it) beyond that it's good for machine learning. This video made it super easy to understand! Excellent job!!!

[@iceefrags8770]

+Science Studio so if an application that was previously written in cuda had functions in it that would be more efficiently run on tensor cores, would the driver automatically run those operations on the tensor cores, or would the developer need to rewrite their application to take advantage of them?

[@EMANUELE42]

Say a certain application is written in CUDA 7 SDK (tensor cores 100% not supported, as there was no NVIDIA GPU utilizing them) and it uses tensors and related (mathematical) functions, even if you would upgrade the driver to the latest versions, it would not be able to take advantage of tensor cores running on such a tensor cores-equipped GPU. That would be because the latest CUDA SDK (with tensor core support) might be backwards compatible with (programming) functions called in older SDKs, but newer (programming) functions which take advantage of both the driver's tensor core support and GPU's tensor cores are never called. If the driver could automatically detect that certain (mathematical) functions can be accelerated using tensor cores instead of traditional CUDA cores, that would be the case were an older application can get an automatic performance boost using a tensor core-equipped GPU, but I doubt it's the way NVIDIA did this, as it would make the driver too large and complex (edit: humble opinion from a programmer, definitely not a CUDA engineer, but still handled similar things between different SDK versions)

[@abvmoose87]

Can the api/code libraries for CUDA be used for tensor cores?

[@AzVidsPro]

What about Raytracimg for that sweet real-time dynamic. Lighting?

[@larryteslaspacexboringlawr739]

thank you for tensor core video, could you do a benchmark video of tensorflow on a rtx 2080 or rtx 2080ti or 2070?

[@webgpu]

"larry" :-)

[@patohernandez1637]

Great video, Greg!

[@michaelvanrheede7225]

So did we get tensor cores in consumer gpus?

[@MohdAkmalZakiIO]

Tell us about CUDA Cores and how it differs from Tensor Cores. As far as I know, CUDA Cores also used for parallel processing in MLDL works.

[@TheBigBawsss]

Damn this is beautiful. Should do more on processing micro architectures. Also it'll be so cool if you could discuss the RTL implementations. Thanks. 💪🏿🦍🐅

[@GregSalazar]

Thanks for watching.

[@TheBigBawsss]

Love your work man !

[@Pastor_virtual_Robson]

time line 2:38 , The FP16 and 32 are not Flop related. They are regarding the Floating Point bits representation, half precision = 16 bits and single precison =32 bits.

[@Alfer230]

One correction, at 2:35 you refer to the "FP16 or FP32" as being an abbreviation for FLOPS - this is incorrect. In the document you are referencing, the FP in FP16 and FP32 stands for "Floating Point" and the 16 and 32 correspond to bits of precision.

[@TechDunk]

Thanks! Wanted to know this :P

[@jupitersailing6911]

This is a very interesting video, I wonder if NVIDIA or AMD will put R&D into a new kind of core for gaming?

[@AlexiGVS]

Thank you, great video!

[@HokgiartoSaliem]

I hope Nvidia has tensor core on GT1130 or 2030 or lower segment but newer product althought it is not as fast as hi end one. Btw if CUDA was so good, why there are no program that can do simple HEVC endoding using CUDA? Mainly for older GPU than doesn't have NVENC that support HEVC encoding?

[@rjsantiago4740]

Great video! One thing though. If the next gen cards (gtx 11 or 20 whatever they will call it) are supposed to support rtx/ray tracing, doesnt that tech use the tensor cores in order to function in real time (AI calculates most of the light rays and where they go as I understood it, though I could be wrong)? So wouldn't that mean they'd probably have some tensor cores? I know we know little about the tech and the next gen cards, but if I understood those rtx demos correctly I believe these new cards would actually have to have tensor cores in order to do it. Your thoughts? Just wanna know if I am understanding the rtx correctly or not.

[@wing0zero]

RJ Santiago you can use CUDA for ray traced effects just tensor is better.

[@rishabh8766]

there will be tensor cores in gaming gpus i can bet on that. remember that real time ray tracing demo? nvidia developed these technologies to take advantage of tensor cores in games. cuda cores alone just cant handle ray tracing. why would they invest there money in something no one can use?

[@LiLBitsDK]

because the world is more than gaming, for example AI as I mentioned

[@Vamavid]

You were right on the money! 👍🏿

[@pradeepraghuraman4430]

Yep. There are now.

[@diegomireles]

I'm an electrical engineer and Tensor is a term we never use, that term is for civil and mechanical engineering. But it seems similar to a State-space representation matrix used in Dynamical Systems and Control.

[@mirmohammadsaadati6526]

FP is not short for FLOPS, it's short for Floating Point. FP16 means Half Precision floating point number. FP32 means Single Precision floating point number. FP64 means Double Precision floating point number. The number 16, 32, and 64 refers to the number of bits that it takes in the memory to store floating point numbers with respect to their precision !!!

[@user-um9sl1kj6u]

This is why each AI matrix models is 8 x 8 x 8 (512 cuda cores) and why Nvidia includes accuracy for32 bit, 16, and 8 (being the simplest and fastest, especially for high speed language models, vision, , or audio) In which case, it can do multiple models and correct for itself. This is also why the hopper architecture has over 16,000 GPU cores and 72 CPU cores. You have 132 matrix instances, with 2 CPU cores per matrix model, and 6 left over for overhead. Besides AI, this is also an excellent Data center CPU and GPU combined. Along with the memory. It greatly simplifies data center deployment

[@bikerbones4615]

What about Ray Tracing? From what I've heard it can only be done with tensor cores.

[@gorepuppy]

Matrix math was super easy for me, and I have number dyslexia oddly. I had no idea a Tensor Core was pretty much the same as Googles 8 bit matrix single instruction ASIC processerror. Good video. Short and informative.

[@jerryh559]

4:39 well..... rtx 2000 and rtx 3000 series have them. Used for DLSS and other AI

[@nathanas64]

good explanation

[@Doudoudelamotte1902]

so a normal graphics card with a driver/firmware modification can create Tensore core? or is it physically different ?

[@kevinbroderick3779]

Physically different. The transistors that forms it's logic are optimized for certain types of calculations.

[@norbertnagy4468]

More vids like this please

[@1nsaint]

@sciencestudio I was hoping you were gonna mention real time ray tracing in game, that is supposedly being handled by the tensor cores, thats the reason i and sure many others were interresed in them for a gaming aspect. it was shown in a video called: The state of Ureal, during the developer conference

[@wing0zero]

INSAINT the new Metro game has ray traced lighting in, maybe shadows not sure, but that will be the first game to use RTX. Not on consoles though PC only.

[@arvindraghavan403]

Thanks for this video and explaining things dude now i can brag about these things with my friends 😂😂😂

[@thefullsetup]

Thanks for making this video getting a bit fed up of all the rumor channels keep banging on about tensor cores being in next gen gaming cards.

[@Vladek16]

excellent video :D

[@Natewu188]

The Engineering Explained of tech.

[@tomislavcelic8806]

Nate wu Love the Senna at the profile pic mate

[@nurbsenvi]

Nate wu Are they related?

[@marvelv212]

So you can replace tensor cores with cuda cores when needed although not as efficient.

[@bitscorpion4687]

yep, there is a major difference in speed www.nvidia.com/en-us/data-center/tensorcore/

[@sapphie132]

This makes me want a titan V. Not because I'd utilise its full potential with my dumb cuda programs, just because it's cool af

[@Special_Sharpie]

Ooh just got on my break!!

[@dhruvmini01]

Did the mathematics behind your petroleum engineering degree help you in understanding concepts like these within computer hardware technologies?

[@ivailomanolov6908]

FP 16 and FP 32 is not a flop . It is a floating point that tells you how precise your calculation will be . What are you talking about ?

[@JD_Mortal]

So much for that "prediction"... Tensor cores are in consumer cards. 20xx-RTX, which was oddly out before this video was made. Though, honestly, Titan-V was a consumer card too. The non-consumer cards being Quadro cards. (Or whatever the Quadro-variant is actually named, to identify it as the VOLTA/TENSOR version.)

[@cappuccino-1721]

what about Ray Tracing? doesn't that need tensor core?

[@potatomo9609]

It's amazing how RTX is just becoming normal today now while only 2 years ago when this video came out, it still seemed like something totally unreachable.

[@space302]

3:21 : " Tensor cores can handle 64 floating-point mixed precision operations per second ". How do you not realize how that's clearly false because it would be incredibly slow ?

[@leonardbrkanac9150]

You could mention that neural networks make extensive use of matrices

[@B4dD0GGy]

32GB Titan V CEO Edition please, so I can open DOS prompt and type dir

[@springbok4015]

You need to add a correction for your mistake at 2:33 +Science Studio

[@KryoTronic]

Love these videos please do moarrrrrrrrrr

[@GregSalazar]

Yessir!

[@brentgreeff1115]

For someone starting to learn ML would you recommend a 2060 or 1070. - this seems to indicate that for ML, the tensor cores really count- towardsdatascience.com/rtx-2060-vs-gtx-1080ti-in-deep-learning-gpu-benchmarks-cheapest-rtx-vs-most-expensive-gtx-card-cd47cd9931d2

[@LimitIsIllusion]

Great video... I guess. Let's just say I remember why I dropped out out of engineering school. I'm sure this made sense to someone. Wooooosh

[@DinoTheThird]

That’s a clean ass shirt bro 😍🙏🏽

[@92trdman]

The processing power never get enough (Because a multi-million market bussiness pushing it) But for how long ? We may never know....And BTW, the picture remind me of "Terminator 2" the movie !

[@hansbehrends438]

Very well. And now over to the Gamer's Nexus to learn about memory subtimings.

[@4.0.4]

I believe 1180 will have Tensor cores, for the same reason CUDA cores were introduced before all games used them. Machine learning is starting to become quite mainstream, maybe not in games but who knows, that could also happen!

[@Rafael96xD]

Aaaah yeah, the good old 2000 series, a lot has changed and its amazing.

[@woolfel]

The newer tensor cores are different now and supports 8bit and 4bit numbers.

[@FarrellMcGovern]

Nice video! You might find it interesting that about a fifth of the Top 500 supercomputers in the world use Nvidia based Tesla GPUs, as I guess you would call them "math co-processors" for the supercomputers. Two of the systems use the new Volta cards. What *I* find really interesting is that they have Tesla Voltas running on IBM POWER CPU based systems, not Intel or AMD. Except for 26 systems, all of today's top 500 supercomputers run on Intel Xeon CPUs. Of the non-Intel CPUS, 22 are based on IBM POWER CPUs, 6 on the Sparc64 CPUs, two AMD Opterons, and two weird ones, named ShenWei, made by China, and said to originally be based upon the DEC Alpha CPU. I love digging around the Top 500 Supercomputer list's spreadsheet each time they are released...and the most recent one is notable for the total dominance of Linux as the operating system of Supercomputing. All of the Top 500 run Linux....No AIX, No Solaris, no HP-UX, and certainly no Windows!

[@allenqueen]

Farrell McGovern interesting digging. Anything else which you found?

[@FarrellMcGovern]

Well, there is another co-processor that is starting to be used, it's the Intel Xeon Phi. It is basically a version of the Xeon CPU that has added around it math units that were designed for GPU use, giving it superior specialized math capability, around an x86 core which makes it easier (in theory) to program. Only 7 of the Top 500 list use Xeon Phi co-processors, while 97 use Nvidia Tesla, 2 use Nvidia Volta, and 2 use a combined Tesla/Xeon Phi setup. There is one other co-processor, a proprietary accelerator made by a Japanese company PEZY Computing / Exascaler Inc., called the PEZY-SC2 and PEZY-SCnp, which I can find very little about in a language I can read.

[@rudysal1429]

Farrell McGovern gotta love those corrupt companies

[@darthtahguy]

How about the new snapdragon 845 it has tensor cores

[@texasdeeslinglead2401]

But can it djent

[@STRAGGLER36]

So, you mean efficiency of software can impact the efficiency of hardware? What are unique and interesting idea. Also I guess what you're saying is read nvidia's blog on what a tensor core actually is. Thanks

[@PaulAllenPiano]

Thanks for simplifying all this for us "Dummies"!

[@tiagooliveira95]

i want to game, but i also want to train my models, my CNNs and RNNs will appreciate every tensor core, soo, yep, bring more tensor cores to regular Gpus, not only to 2k+ gpus

[@ananthu6]

hmm... when you pointed that finger multiple times , it reminded me of supreme leader allladin talking about enriched uranium... sorry i dunno why.. lol k will subscribe now... even though i dont understand much of it... a vid put together well... n u know yiur thing.. looking forward learn more from you.. :)

[@vfn4i83]

Titan got 640 tensor cores, Vega 64 has 1024 tensor cores????

[@funbucket09]

Nice shirt Greg. It suits you. Lovely Colour :)

[@texasdeeslinglead2401]

Would love to see more graphical demonstrations on micro architecture.

[@Mr.VaSimple]

We need games with a great AI that will utilize Tensor Cores, so this cores may appear in consumer GTX videocards later.

[@junakkibria6380]

Only if Greg knew Nvidia's going to drop a bomb (DLSS) with this tensor cores a year later then he would be more emphasizing on this for sure XD

[@calvint3419]

So all in all, cuda cores can do what tensor cores do, just slower.

[@jim1550]

0:47 mmm, Veals...

[@captainconcernedsr.5360]

Doesn't stop me from wanting tensor cores to enhance AI's for shooters and Sim's while the reduced workload on the CPU can provide more room to do other things.

[@GregSalazar]

Don't count on mainstream devs jumping on board any time soon.

[@wing0zero]

The new Metro game is the first to use ray traced lighting, maybe they are using tensor cores as they are perfect for it.

[@IVI00101]

I thought, they were planning to use tensor cores to axelerate ray tracing.

[@rickgrimezz3942]

got curious... and looked this up... came across this video... and then I heard him say @4:37 .... Seemed silly since all we get now are Nvidia cards with Tensor cores. Queue the SUPER SERIES.... the follow up to the 20 series cards all mainstream cards with tensor Cores... hehe! Found this comment silly!

[@mcericx]

5 minutes video translated into one sentence: they are fixed function units

[@jrno93]

4:39 oh how wrong you are

[@IgordasTunfisch]

Battery is filling up