NVIDIA Jetson AGX Orin Full Review - 2048 GPU Cores

The Jetson AGX Orin is a high-end machine learning module from NVIDIA. The developer unit, which contains the module and the carrier board, has a 12-core Cortex-A78 CPU, a 2048 core GPU, 32GB of RAM, 10GbE, PCIe, and much more! Here is my review.
Unboxing, overview, and teardown: • NVIDIA Jetson AGX Orin...
Machine Learning on a Raspberry Pi 4: • Use Machine Learning o...
Let Me Explain T-shirt: teespring.com/gary-explains-l...
Twitter: / garyexplains
Instagram: / garyexplains
#garyexplains

[@fabriciochamorro2985]

Really enjoy your Arm content

[@tytrvd]

Gary this amazing stuff thanks so much

[@itaco8066]

Wow so much power 💪🏽

[@TheProductCritiques]

Hey Gary, Thanks for the detailed breakdown of the Jetson AGX Orin and its impressive capabilities. It's clear that this device packs a serious punch when it comes to performance, especially in the machine learning domain. The comparison with other boards like the Raspberry Pi and Jetson Nano really helps put its power into perspective. I'm particularly interested in the potential applications and use cases for the AGX Orin. You mentioned factory automation, self-driving cars, and robots as some of the target areas. Could you elaborate a bit more on specific real-world scenarios where this board shines? I'd love to hear about some practical examples of how it's being utilized in the field. Also, you mentioned some unexplored features like the PCI Express slot and hardware-accelerated encoding. It would be great to see some follow-up videos or content on those topics if you decide to delve into them. Keep up the fantastic work on your channel, Gary Explains! Looking forward to more informative content from you.

[@NexGen-3D]

The text recognition is pretty impressive.

[@thaernejem7317]

You are brilliant!

[@theWIZireland]

But the internet needs to know, can it play Crysis ?

[@LisaSamaritan]

All of the above (except AV1 encoding), would be interesting. :)

[@leledumbo]

I'm gonna need that voice recognizer code, hopefully even if it's hardware accelerated, it's well abstracted to work across multiple platforms.

[@adfjasjhf]

I would be interested in running proxmox on this and perhaps home assistant

[@miguelagueda3928]

I'll be really interesting to see some general benchmarks to see how good of a "desktop replacement" this device makes.

[@COMATRON.]

4:3 gary even more for president!!!

[@user-ti5ce4hg1o]

If you can (have time / energy) please see how well it does with uncensored Large Language Models like gpt4-x-alpaca. You can use the lovely oobabooga UI to run and chat with text models locally. I have the RTX 4090 and would love to see the difference giving the Jetson Orin has more VRAM (64GB) compared to my 4090's 24GB, which means it can fit far more intelligent language models than what 4090 can. Also, if you can, see how well it does in stable diffusion in text to image generation (use stable diffusion UI) P.S. Just search youtube for gpt4-x-alpaca and you should find lots of tutorials on how to install oobabooga web UI and gpt4-x-alpaca AI model (or any other more sophisticated models) to chat with.

[@sammcbride2464]

It would be nice to do an update to this and include the Apple M1

[@azw409]

Not a great comparison but building a large rust project takes 9m24s on an M1 and 27m on the AGX. Single core perf is about 3x slower, but the AGX has 12 compared with 8. Probably it compares favourably with previous gen intel macs but not M1 for CPU.

[@Tugedhel]

Do you know if NVIDIA has a ready to learn AI software load available for the Jetson AGX Orin like they do their high end AI boxes?

[@mrtech2259]

What happened to speedtest G?

[@muddyexport5639]

Good show. For other explorations >>>===> Any and all!

[@CommodoreFan64]

I'm not a big fan of Nvidia with how they have treated the Linux community as a whole over the years, among other reasons, but I'll admit that this is an impressive piece of hardware.

[@ubacow7109]

Imo it's more that the mentality of the Linux community alienates others from them.

[@joao333hd6]

Hi, Is this like a normal computer? I can train the NN in here as well, or just used already trained ones? Thank you

[@itsmuhammad2305]

Hello professor. If you will excuse me I have an unrelated question, can you say when are the speed test G results of new smartphones coming out? Thanks.

[@abhishekjain7389]

Hi im interested in understanding the GPIO performance of the Orin Dev kit

[@carnby24]

Hey Gary, you are bringing NSA to our homes....jolly good!

[@techandgames9160]

Could you please check out the AV1 encoder feature

[@tonysheerness2427]

How things have changed. I remember when voice recognition first came out in the 90's and it took half a day to train the computer to recognise your voice, had to repeat certain key words over and over again. They thought this would replace typists, it was very unreliable. I noticed in the face recognition clip people were wearing masks especially person number 0. It knew there were 6 people there but if one left and came back would it know it as the same person?

[@andrijaandjelkovic5751]

Damn and today my phone can recognize 45 words/s

[@shadow7037932]

Depends on the implementation. The facial recognition can use other features like ear/eye position to recognize even with a mask covering nose/mouth. Accuracy/reliability will of course be lower.

[@NexGen-3D]

And you had to speak like an American with some words or it would not work at all....Dragon Natural Speaking comes to mind.

[@user-um9sl1kj6u]

How fast is storage on the Orin? Have you tried running a server or cluster? I think it would be amazing to see an all flash server cluster run on one or a few of these tested individually or in a seven module board (I know it’s used for other things, but it could be essentially a rack of seven modules on a board, times 10) with all flash. I wonder how it would perform at 15 W, or 45 watts.

[@ReubenHorner]

The storage is insanely fast from what I can remember.

[@neomcl]

Hi man! Did you test gaming or vg emulation on it? XD Is it running LLMs well? Stable DIffusion?

[@terryphan7451]

Can we get a benchmark comparison to Xavier?

[@nico96gmail]

I am looking at something like that for a quality control station. Where I need several high resolution cameras (4K o more) but slow framrate 0,5fps. Can te Orin also train the models to later do inference. Or should I train on a pc.

[@AI-xi4jk]

It can train models but it will be slow. Ideally you want to train on a big gpu like 3090, then convert to tensorrt for inference on a jetson.

[@ubacow7109]

We just want this in the next sheild pro....

[@martin777xyz]

I'm interested in locally run LLMs

[@user-um9sl1kj6u]

How does the performance scale at different TDP’s? Is it better to have an Orin Nano 8GB at 8 or 15 watts?

[@user-um9sl1kj6u]

What’s the performance at 15 Watts? Is it really that much of a loss? I’m wondering how this would perform in a four cluster board or a seven cluster board. At 15 watts a piece, do you think this would do fine in a cluster board?

[@user-um9sl1kj6u]

I'm curious: I was looking at some of the benchmarks of the Cortex A78AE and I saw some benchmarks showing it basically was a little better than a Core i5750 (just with 12 cores) How would you rate it vs. modern x-86 CPU like Ryzen's first generation 15 watt U series processors? Does that even matter? I wasn't sure if some of the benchmarks on the net were legit

[@bhuvankiran3922]

What happened to Speedtest G videos on New chipsets?

[@johnkost2514]

It would be interesting to see how this Orin SoC compares to the Apple M1.

[@Katzelle3]

Orin has the more powerful GPU compared to base M1

[@denvera1g1]

Of note, if you have equal FP32 and INT compute needs, you actually halve your CUDA cores to 1024. This is just like the Geforce line where the INT cores can also run FP32 when not running INT. This was the main reason we didnt see the " more than 2x" performance claims come to fruition on the RTX 3000 series. This is because in most games where this performance would matter, the game uses both FP32 and INT On a somewhat related note, the Ampere (Tesla) A100 does not fall under this rule, its CUDA count is the full CUDA count of the card, with an equal number of dedicated INT cores. Nvidia probabably wanted to avoid lawsuites from supercomputer manufacturers if they ever found that the performance halved for their customerswhen the GPU ran INT at the same time as FP32

[@WorldPeace-pi1uq]

How GPU comparison between the NVIDIA 3000 series and the Jetson AGX Orin? You compared the Ampere architecture, but speeds and numbers of GPU cores might be different. This is important because you can buy a decent desktop (with enough power) a NVIDIA GeForece 3080 or even a 4090 for about the same price as the Jetson. But the inclusion of a full-AI stack that is only intended for the Jetson has advantages IMO. You can install a full image (minus the Linux OS) and re-install it if you mess it up. This would be perfect for educational use by multiple students.

[@PAOLOHAON]

Can I ran Nvidia's Isaac Sim, IsaacGym, and Orbit simulations directly on this processing unit ?

[@anga6275]

booting a vanilla linux os compatible with arm? try that, that would be fantastic!

[@dr.mikeybee]

You might look at the base M1 or M2 Mac Mini. As it turns out, it's an amazing system for machine learning, and it's much less expensive than Nvidia devices.

[@GaryExplains]

If only I had videos about my MacBook M1! 🤦‍♂️

[@user-um9sl1kj6u]

How does this perform alongside a Ryzen 7 7840U, as far as CPU and GPU are concerned?

[@EyesOfByes]

3:33 SpeedtTestG? :)

[@dhruvkumar7802]

Can it run windows?

[@kevin_delaney]

This is fascinating and my finger is hovering over the buy button, $2k price tag is enough where I'd like to see more of what it does, could you dig into this a bit more? :)

[@GaryExplains]

I would recommend that you start with a Jetson Nano. When you have learnt about it but need more performance then upgrade to the Xavier or the Orin. I have videos about the Nano and the Xavier on this channel.

[@kevin_delaney]

@@GaryExplains Haha I bought a Nano when it came out, I was thinking of an NX when it came out, then it for hard to find and went up in price and I didn't think about jetson for a year or so, now I'm coming back to play with them and stumbled across this...this thing is a BEAST. $2k is just a hard pill to swallow but it looks so worth it

[@AI-xi4jk]

Some kind of gpu benchmark comparing this to rtx cards would be nice. Nvidia has some graphs and numbers but it’s good to see real life ml benchmarks from independent source.

[@adamrak7560]

The problem is that this is a very heterogeneous arch. So it would be relatively slow unless the NN was optimized for the HW.

[@macrobionic]

Can you use it to run text editing software on it like Microsoft Word?

[@GaryExplains]

Like all Linux distros, you can install LibreOffice. You can also use Web based productivity services like Google Docs and Office 360.

[@lashlarue59]

When I see devices like this I'm always curious how they compare in terms of performance with a PC with a 2080ti or 3080ti installed running these same texts.

[@hassanlabyad4082]

A laptop 95w rtx 3050 in terms of gpu power, CPU tho I don't know

[@adamrak7560]

"Pass the butter" robot would probably also need this compute power (so he can be depressed about his fate)

[@user-um9sl1kj6u]

I'm curious how this would do compared to an AMD powered NUC, like a Ryzen 7840U. I know Nvidia will Slaughter it in terms of it's GPU, tensor cores, and optimizations over time, but I'm curious just How the two would hold up in an industrial environment (there's a reason they bought Transmeta decades ago - you are seeing it now)

[@Epion0485]

All of them especially gaming and leave out AV1

[@69MrUsername69]

Gary, can you also compare with AGX Xavier vs Orin ?

[@GaryExplains]

Unfortunately I don't have an AGX Xavier.

[@slax32]

What CPU benchmark did you perform?

[@69MrUsername69]

@@slax32 7zip benchmark it’s cross platform

[@marcusk7855]

Will it run windows? And how many bitcoins can it mine?

[@whothefoxcares]

is NVIDIA Jetson AGX Orin better than the new Mac Studio?

[@GaryExplains]

Better at what? The Orin can't run Premiere Pro or Garage Band, etc. So if you need Mac software then it can't be "better". As for ML tasks, that would be an interesting test to use Core ML etc and test equivalent tasks on the Orin and on Apple products. Having siad that. the Orin is an embedded module, designed to be out into robots, cars etc. I don't think connecting up a Mac in the trunk of a self driving car is a good idea!

[@Simoss13]

Dam. If Nintendo can put in the 8 or 6 core model in their next console (hoping for the 8 core model), take away alot of unnecessary components; this would be amazing!

[@hassanlabyad4082]

If they manage to put this chip on a handheld for 499$ it will smoke the Xbox series s Heck Nintendo would kill the market with such a chip But they wouldn't

[@Tau-qr7f]

Can you please compare Agx orin with a pc or an m1 mac?

[@GaryExplains]

That is an interesting request. What would be the point? It can't run Windows or macOS so it would turn into a Win vs macOS vs Linux comparison. Or have I misunderstood?

[@andrewwhaley7639]

I've done a comparison: M1 single core CPU is abour 3x faster than AGX (based on large Rust project build times). Multicore is about 2x faster since AGX has 12 cores compared to 8. For an ML training task (training MinGPT), AGX is about 4.25x faster than M1 (CUDA vs MPS).

[@Tau-qr7f]

@@andrewwhaley7639 wow! thank you :)

[@NexGen-3D]

Every time you say its name, my ageing ears hears "Agent Orange"

[@GaryExplains]

LOL, yeah, I can understand how that could happen!

[@wingchong68]

We have a new SOC that's twice as fast as NVidia Shield TV box , the Rockchip RK3588. It can plays 8k videos at 60 fps and puts out over 400k in Antutu scores. This chip going to dethrone the king of TV boxes, the Nvidia Shield TV console.

[@GaryExplains]

It is hardly difficult to dethrone a Cortex-A57 processor 🤦‍♂️

[@bigdog7419]

the rockchip is tuna

[@user-um9sl1kj6u]

How many instances of linux can this handle with the GPU? Considering it has 3 Quad-core clusters, is it possible to have 4 instances of 2 or 3 cores/8GB Ram/512 GPUs? Having 4 instances with 512GPUs and 8GB ram Really Makes or Breaks the Orin. You Should be able to have 4 AI instances and it's hypervisor @15 to 35 watts, and minimal cooling. Imagine a 4 or 7 cluster board with 15, 25 to 35watt Orins on all of them. That's a LOT of horsepower for essentially a laptop's worth of power! Between being able to audibly and visually reading a room (like self driving) read and identity faces and speech (face net, besides speech) and track an environment’s activity and model behavior (like we do all the time) you need a lot of processing power and instanced AI networks. To be able to work like us, that’s a lot of processing power. That’s a lot of instanced networked AI’s like processes in your brain Whenever you think of AI, Think of neural anatomy and structure In the human brain. Think of being blind and reading a room, picking out conversation and then understanding it, especially if they are Addressing you, and then planning what you are going to say based on that and what’s going on, and then speaking. Besides reacting to everything else in the room (not Just audibly, but visually, while Walking and balancing yourself in space. Now imagine a robot playing baseball, listening to a crowd, listening to it being addressed in baseball codes, and then playing and reacting to the crowd. Besides tracking, executing, and planning it’s next move, everyone else’s movements, and what It will do in the next few minutes, hour, days, weeks, months, or years. We have a long way to go. And even Boston Dynamics and Tesla know that. AI is still in the baby phase.

[@HenrikoMagnifico]

Nintendo Switch 2, perhaps?

[@GaryExplains]

No, way too power hungry.

[@Ashiodyne]

@@GaryExplains it’s a misconception because that thing is gonna use the “T239” so some people assume it’s a cut down “T234” (AGX Orin) when it’s a completely different chip under the same GPU architecture

[@anilchandra617]

cool and costly

[@user-um9sl1kj6u]

See

[@snapo1750]

just looked at the price.... nope thanks

[@foxsux6000]

Bit of a bummer ARM won't really matter for Windows for a very long time (mass adoption) It's just a waste of hardware really, since nearly nobody will be using it. The idea that windows will magically cause an arm switch has turned out to be nothing more than a delusion, and it will remain that way for years to come... (Can't compete with the more open/free to do as you will "even if x64 is closed" platform.)

[@GaryExplains]

I don't think doing ML on Windows is really what the industry is trying to achieve.

[@foxsux6000]

@@GaryExplains I don't mean that, i just mean general compute. It's just a bummer Windows support for ARM sucks and isn't really usable as a desktop (it's like linux in the 90s.)

[@GaryExplains]

Ah, I see sorry. OK. With regards to Windows on Arm, I have found it very usable. What exactly do you find "sucks"?

[@foxsux6000]

@@GaryExplains Just the performance isn't all there, X86 support is meh, it's just a little sub-optimal. Yeah you can browse and watch videos, but heavy tasks (arm or x86) and applications can make the whole thing stall (like Linux in the late 90s.) Maybe my expectations are too high?

[@GaryExplains]

So, I think what you are describing is underpowered hardware, not a problem with the software (other than the normal Windows is bloat kind of problems with exist on x86 as well).

[@stalinvlad]

Theology. Imagine Elon Musk puts the engineering skills of spaceX into religion! Robot theologians doing high precision monotheism. Fuck grading fruit, sorting plastic waste. No give the BibleBelt robo-priest. Yeah real cool until India catches on...Blockchain? THERE IS NO BLOCKCHAIN!

[@mentalplayground]

Can we have so more Please. ;)