I built a MONSTER AI Pi with 8 Neural Processors!

[@jonathantribble7013]

It's nice to know that you could run multiple neural networks on independent NPUs! Like one for facial expressions, another for voice recognition, and another for text-to-speech!

[@electrofreak0]

Can't wait for a decade from now when they're packing 1024 TOPS into "entry-level" devices claiming "you definitely need all this power for current models"

[@heblushabus]

1 BOPS?

[@Level2Jeff]

640 TOPS ought to be good enough for anyone

[@tuqe]

@@heblushabusT = trillion, so B would be billion. Next step is Pflops for Petaflops

[@heblushabus]

@@tuqe oh, right. bops sounded funny tho. so, POPS?

[@Draggeta]

​@@tuqeisn't T tera in this case?

[@Rostol]

seriously tho, 26 TOPS on that small thing is impressive by itself. combining a card with a few of those 26ers with something with PCIE-4x16 would make for impressive AI processing in a small package.

[@Level2Jeff]

Yeah, Hailo makes a 'Century' card that does just that-hopefully as multi-NPU becomes more popular, programming for it also becomes more popular!

[@digibluh]

a Snapdragon 8 processor has 45 TOPS NPU and is in android phones, it's also got a CPU and GPU with a total of 75 TOPS of compute power across the whole chip.

[@tinkerscustom9568]

i would love a video on the home assistant power consumption!

[@OmarMekkawy]

+1 Yes, me too

[@bluesquadron593]

Well that would be a 1 minute video 🤣

[@fridje]

🤗 Accelerate might be able to manage using all the NPUs at once. It's meant for using multiple GPUs for a single inference or training task but it might support NPUs too

[@RaaynML]

It feels like a lot of those libraries are just magic at this point. They seem to be able to shove any network into any hardware and just kinda run with just a bit of finagling, not what one would expect from such a new way of doing `software`

[@Rushil69420]

Oh thank god, I’ve been itching ever since you showed that b-roll

[@realandrewhatfield]

OMG!!! Who are you going to be from now until the next video?!?! End of an era...

[@Level2Jeff]

Ha!

[@TheJonathanc82]

Never stop doing what you do Jeff. Love the content, love the experimentation.

[@Petra-yd1fi]

This reminds me of the Friends meme where Chandler gets a fancy expensive computer, and when asked what he is going to use it for he just says "Idk, games and stuff". The infra is still fun even if you don't actually need the compute lol.

[@the_hetman]

It’s always fun to see you pushing the limits on single board computers. I’m sure it helps the hardware makers consider unusual use cases. Jeff, have you seen the stories on the new research into IBD which suggests a genetic link and that certain cancer drugs might be effective? I saw the story on the BBC news site. Obviously I can’t post the link here, but I thought you might be interested.

[@Level2Jeff]

Yes, I've been following that research-still 5+ years out from being practical but it's promising. The key is to find drugs that have fewer side effects than current TNF-blockers (which have similar mechanisms but target the immune system more broadly).

[@BrianMaddox]

I appreciate that there are so many different TPUs/NPUs on the market, I’m just frustrated that we’re all beholden to nVidia when it comes to actually training models and running a lot of things.

[@Level2Jeff]

Ditto. Wish at least AMD could offer something that would take the bottom out on either price or efficiency, but right now it is what it is :(

[@BrianMaddox]

@@Level2Jeff I’ve got a used Tesla P40 with a water cooler in addition to another rtx 2060. Now that Intel works with Tensorflow and PyTorch, I’ve seriously considered just getting a 16 gig Arc 770 and paying for cloud computing if I need to train a model that needs more memory.

[@RaineyPeng]

My favorite part of this video is definitely when the box gets identified as a cell phone and he holds it up to his ear 😅

[@higon99]

Oh, man. That's a visually menace pi XD Yesterday, you tried and SPAGHETTIBLY FAILED chaining NPU and now this. lol Thank you for the attempts otherwise I would have tried myself. I think 2 NPUs can easily work with running 1 neural network on each NPU. This kind of configuration can realize many real world applications I have been dreaming of so many years. Thank you again.

[@Level2Jeff]

Definitely! I think on the Pi 5 at least, that would probably be the ideal number of NPUs. You could stretch it to 4 okay too, but at that point the cost/build could point you to something a bit beefier like Jetson Orin.

[@MaxHeadroomGPT]

Jeff, I too would love to see a video on how to setup a Home Assistant Dashboard for Power Monitoring. You sir, are a Wizard!

[@DiamondMaster115]

I had no idea this channel existed, this is awesome!

[@Level2Jeff]

and now you do, ha! this is the channel where things get crazy

[@Biedropegaz]

jeff sometimes is running low on iq because from time to time he post videos about making a fatality on a pc

[@ShannonJosephGlomb]

This is perfect for having multiple models working on different tasks then feeding all there results to the AI that monitors them 😊

[@LeeZhiWei8219]

Hey man! Wow. Saw this on Instagram and thought this would be on the main channel 😂. But awesome job man! This is so great 😂

[@jeremybarber2837]

I would love to see a video going over your choice of power metering smart plugs & the integration into HomeAssistant.

[@zepesh]

That was fast

[@inferno14142]

I was about to say that

[@Level2Jeff]

My AI predicted it

[@neb_setabed]

that's what she said!

[@harriet-x.x]

​@@Level2Jeffu okey? u look a little red :p

[@ewasteredux]

Bravo Jeff! That was alot of work on your part. Again, congrats and thanks for all the hard work you do for us!

[@Argent_99]

I’m amazed by the things you are doing with the humble RPi. Not as amazed as that fractal north case for the pi5 they are showing off at computex, but still amazed.

[@markaphillips14]

Jeff!!! The increase in content per week has been amazing. Don’t over due it but man I’m loving it

[@oasismike2]

Thanks for trying AND explaining the problems! Can't wait for someone to write a "Kitchen Sink Aggregator" patch that'll normalize or make modules generically access -- ahh, I just wanted to use kitchen sink in a sentence.

[@stevanastardust8487]

I actually met the alftel guy and he is a GENIUS with RF stuff.

[@Level2Jeff]

I don't doubt it. The contraptions he makes...

[@awetmore]

The trick to using those NPUs in parallel will be building a pipeline across them, where each NPU is supporting a subset of the layers in a model. This is a common technique in both training and inference, though I'm not sure if Tensorflow Lite supports it. A pipeline would allow you to partition the model weights and compute across all of the NPUs, giving you a chance to run larger models then you could do on a single NPU. Your PCIe setup is very low bandwidth, but that is less of a concern here because pipeline parallel is only sending the activations (relatively small input tensors in inference) between the NPUs, not the larger weights. Based on the limited information that I can find about Hailo and their sample hardware (a few of them have many Hailo chips on a single PCIe card) it looks like their software may support this.

[@Jaabaa_Prime]

I love the crazy/nuts thing that is this set up! And, even if the "Coral Dual TPU" only shows up as one of them, I'll bet that it is still cheaper and faster than the USB version. Now, where was that AM power meter that your Dad has . . . ? ;-)

[@omersalem73]

Hi Jeff, just wanted to let you know that you CAN use multiple hailo cores together, using the VDevice API - it automatically identifies the cores (granted it will only work with hailo chips).

[@Level2Jeff]

HailoRT seems to have some multi-core configuration too... definitely some fun to be had here!

[@TheMostOrdinaryPersonOnEarth]

Please make a power outlet video, I'm just getting into HASS now and power monitoring is next on my list - Also great video cheers!

[@AerialWaviator]

Being able to run two Halo8's would be a cool project as in theory would be (2x 26, or up to 54 TOPs in theory). Combined with dual Pi cameras, would offer high frame rate stereo depth of field, or other fun video processing. BTW: the Home Assistant monitoring is fascinating. Would be interesting in hearing more details.

[@waynefilkins8394]

That's kind of like the pi supercomputers where they were hooking like 25 50 pi's together and calling it a super computer, but you have to make custom software to support the thing and take advantage of the whole cluster or w/e they call it.

[@LockonKubi]

"Why?" well to quote The Doctor from The Waters of Mars, "Fun"

[@gu9838]

hehe a pi mad scientist cobbling together contraptions no one thought of. love it! great scott!

[@prince3121]

Jeff the mad RPi guy! Love these builds when you push the envelope! 😎🤣

[@packerbackermk1]

For anyone with these dual Google coral chips, if you would like to use both corals you can with adapters. For example the ones for sale on MBS-Shop.

[@stratos7755]

If you want 200 TOPS, just buy the Hailo-8 Century. It has 208 TOPS and is a single PCIE card. Although normal person probably can't buy it...

[@Level2Jeff]

I'll just do it myself! Haha, so far I haven't found a reliable way to buy Hailo-8 outside of being an OEM partner. Hopefully they open up more individual sales at some point.

[@stillblazinkush]

Level 2 Jeff is truly on another level.

[@Level2Jeff]

The second one!

[@glabifrons]

At first i was thinking the tribute was to Major Hardware, but the intonation and rhythm is completely wrong... it's closer to Explaining Computers, but the line doesn't match. Ya got me on that one! @Level2Jeff !

[@egrinant2]

I also recognized the tone as Explaining Computers, but the message doesn't match.

[@Level2Jeff]

Check the description ;) I've started to put the tribute into the bottom of the description to make it a little easier :)

[@glabifrons]

@@Level2Jeff I skimmed and totally missed it! Guess I should've ^F-d instead! 😄

[@tonysheerness2427]

As price drops for NPU's people will use more then software will write the software for it. That is what the raspberry pi was designed for learning in a fun way.

[@txga2005]

Extremely helpful video for beginners

[@farzadb82]

@level2Jeff Where did you buy the Hailo-8 card? On their website they only provide an option for product enquiry, not for purchase.

[@catsupchutney]

Jeff, you have to write a book with a chapter on each of these types of Pi mods.

[@MystikIncarnate]

Hey Jeff! I'm still pretty curious about everything "AI". I'm just not sure how to take advantage of any of it right now. I'd love to see a video going over a bunch of different AI projects that these can be used for, either here or on the main channel. Obviously frigate is one, I've also seen some self hosted AI chat bots, though I'm not sure how well any of them would fare on a pi. I know you're the "pi" KZbinr, but I'm also curious about other applications of such accelerators. I wonder if an AI chat bot would work decently well on a 1L PC (or some similar micro x86 system) using something like the Hailo for processing rather than trying to cram a GPU in a small system like that. If you know of a KZbinr who is doing that sort of thing, I'm happy to check them out, just let me know. Keep up all the cool videos. Cheers!

[@tuttocrafting]

We need cheap PLX chips that take a newer standard bus in uplink and can provide many downlink at lower speed. Also on consumer mainboard it is starting to be annoing. 20 or plus pcie gen 5 lanes where almost all hardware is gen 3 and 4. Lot of throughput lost.

[@Level2Jeff]

Couldn't agree more. Though those of us needing all that PCIe goodness are a slightly rare breed... and the answer till now is usually buy a big server CPU that gobbles up like 120W idle :D

[@tuttocrafting]

@@Level2Jeff Even if we are the minority I think that the approcha that apple did with the MAC pro using the PLX and many lanes/slot is something that other HEDT/Workestantion OEM should start to consider. New AM5 EPYC has been annoucned but lot of bandwidth is lost if you just plug a single HBA card on today mainboards x16 slot.

[@afaulconbridge]

This would be useful to run Frigate AI detection from multiple cameras (e.g. PoE RTSP streams). Even this many devices is less power hungry than a 4080 to run 24x7, and compared to the cost of a good IP camera its not unreasonable.

[@dfgdfg_]

"I've created a monster! No one wants to see Marshall no more, They want Jeff. I'm like chopped liver"

[@Brahvim]

"And I will see you, in the next one!" AAAAAAAAHHHHH WHO SAYS THAT?! WHO SAYS THAAAAT!??!?! It JUST escaped my head! WAIT, Chris Titus Tech does?! I just read the description! _Well then!_ ...I think _this exact one_ is a bit common, is it not?

[@TheGamesZilla492]

Needs a compatible case to keep it all safe or maybe 3d printed case perhaps ,Sound like there needs to be a Part 2 to this video.

[@Flare1107]

How do GPU bit miners support the core splits? Some setups are looking at spreading a single calculation over thousands of cores. Maybe there's a way to port a miner task divider to NPU tasks. But I also wonder if we are still limited to running each TPUs individual Floating Point rating? Or could we run full 32-bit models?

[@saiyantwan]

think you might be able to look over the nanopi boards since most have built in NPU? I know the Nanopi R5S has one. Not that much of one but still there

[@GameDesignerJDG]

I wonder how hard it would be for a Pi to orchestrate all these TPUs together. Could it offload some orchestration onto another TPU? I kind of want to write some code for this thing to see how it performs with multiple processors.

[@ThereWillBeCake]

This isn't Level 2 Jeff. This is Level over 9000 Jeff.

[@ws01212]

I really like your videos they always bring novel information Do you know when it will support running NVMe SSD and Hailo AI Kit simultaneously? (with the NVMe SSD used as the system boot disk)

[@yorkan213swd6]

Great video, does this also works with Pi4 ? Please also mention in future if the projects also works with Pi4 possible since not everybody has the latest and greatest :-)

[@Level2Jeff]

It works with CM4, but not Pi 4 directly since the Pi 4 doesn't expose PCIe without a pretty serious hardware hack.

[@vladimir.smirnov]

You can try upgrading that thing with replacing those TPUs with Intel NNPI-1150. Those can be found for ~30$ per board, it is M.2 board with special version of 11th Gen Intel CPU with 12 ICE (Inference Compute Engines) that is SRAM with DSP processing. It claims to provide 50 TOPS per M.2 card.

[@thegreeneyej]

Yes, a dual m2 hat with a Boot and storage SSD, and that Hailo card for friggit, object detection , write to a db that is searchable (events) in plain text…😅

[@jobasti]

​ @Level2Jeff - Power Monitor Dashboard all the things - YES PLEASE

[@ur1friend437]

Love these kind of videos so keep em coming

[@JTB_Computers]

Here before 20k subscribers! Keep up the great work Jeff

[@jacquesdupontd]

Hey, thanks a lot for your videos. I'm wondering something that i'm sure has been adressed but i can't find a definitive answer on this. Does TPUs accelerate local LLM answer generations (Ollama for example) ? Thanks

[@avibank]

Might be interesting for scientific computing. Can you send instructions via MPI or something?

[@TT-it9gg]

Nice! So, other SBC with PCIE Gen3x4 can leverage the full power of all AI chips installed?

[@NoMore12345-z]

Great information in the video. Is there a PCI-e board like this that would work in a desktop, without buying the Nvidia RTX 40 series?

[@Level2Jeff]

Hailo makes a 200+ TOPS 'Century' card that straps a bunch together and would fit inside a desktop case (full height PCIe card).

[@NoMore12345-z]

@@Level2Jeff Thank you so much. 🙂

[@redactedofficial]

Hell yeah, finally you dont just tease😂 i was waiting soo badd, am i rpi addicted🤨

[@w13rdguy]

I know you won't tribute aVe, but, that would be hilarious!😂

[@OriNachum]

Can you show us actual runs? Also would love to see the 10H version when it comes to your hands.

[@sirkingjamz101]

You choose to switch "geerlings" and it seems it paid off for the better :)

[@worfrozhenko4032]

0:04 Just flappin' in the breeze

[@gendalfgray7889]

Can those neural chips be used for something custom other than video recognition? Hi, do you have an idea on how to utilize 2 TPUs of that double coral chips? I want to make similar board for deskyop PC, how to do that?

[@RickySupriyadi]

so Google TPU might be the LPU arranged in certain way? doesn't seems so, because LPU have huge memory chips... well the tops seems matched I wonder where this experiment will go into this is really interesting. Really interesting! indeed!

[@turnkit]

before the WHAT it's compelling to know the WHY. waited for the payoff but I guess I don't get enough why I'd want to use all these TOPs I'm excited about using new tech but using it is the key. The excitement about the build comes after the excitement about the functional power. So I guess I don't get this video too much.

[@tinkerscustom9568]

thanks for the video!

[@nekomakhea9440]

I wonder how feasible it is to put an abstraction layer in front of the 25 coral TPUs so that the software only sees one big TPU? That's more-or-less what RAID drivers for ZFS and BTRFS do so clusters of 25+ drives appear as one block device. And what CPUs do to make scalar code think it's still being executed in-order as the only process on a single-threaded CPU in a virtual memory space, despite out-of-order execution, superscalar execution, hyperthreading, multithreading, branch prediction, and more, all happening in the background.

[@BaldyMacbeard]

It's cool to see that working _technically_, but in terms of inference, that's kinda the equivalent of having a 1000 CPU cores running on a single, shared stick of DDR4 memory. Do you hypothetically have enough cores to run a ton of applications at the same time? Yeah. But how much you'll be ably to do with the system will depend on the throughput of your memory. And it's the same with these little compute sticks. They rely on your Pi's memory so while you - on paper - have a lot of TOPS, running any task that operates on a reasonable amount of data or live streaming data will quickly saturate your onboard memory and pcie bus and prevent you from doing anything useful with multiple NPUs

[@phischtv4497]

What's the best lowcost solution today for running local image-detection CNNs on a Pi4 or Pi5? Those USB-TPUs?

[@asteriondaedalus6859]

8:20! Google doesn't support the software for the Coral?! Is that at all? Or are you meaning on PiOS? Or?

[@jameswarnock5655]

I had trouble getting my coral to work with codeproject ai long-term. It would work for a few hours but then stop responding. I don't know where the issue was on that. I mostly just gave up on it.

[@Level2Jeff]

I had a similar issue in my testing, though didn't take too much time to debug it. One time it seemed to lock up the frontend of the Pi, had to force-poweroff!

[@jeremybarber2837]

Oh man… this makes me think of how great a CM5 board akin to the CM3588 NAS board from FriendlyElec but for NPUs would be. Wait… could you just use that board for NPUs as is?

[@guy_autordie]

You need that 12xm.2 card with a 16x connector and put it in your ampere workstation.

[@SnapWireOnlyOne]

You don't need to make software for them to all work together just create something in python and back off it. you know what have you using the python multithreading library to trick it into multithreading ?

[@azertyQ]

There a Qualcomm AI accelerator that use Dual M.2 slots, that board could be perfect for it (I've never been able to find the spec for dual m.2, so the spacing might be off), just one of those could get 200TOPS/25W Good luck finding one though...

[@noth606]

Well, regarding early multiprocessor things on PC, there was quite a long while, several years in fact, when nothing supported SMP on DOS/Windows even though there was hardware like it out in the wild, it got supported by things like SCO, NetWare, and then to some degree unix flavors pre-linux (forget which - Solaris probably?) and then badaboombadabing linux grew up real fast on that front and started going down the list of obscure weird poop SMP platforms adding support for one multiprocessor 486 here, early Pentium there until it ran in some form on almost anything that had some server or desktop CPU like architecture. This happened in tandem initially with Windows NT, or well, NT was first but was then overtaken by Linux at some point. And I mean NT 3.51 here folks, 4 was still a twinkle in BillyG's eye at that point. A lot of 'lateral expansion' going on then, I still vividly remember booting NT 3.51 on Alpha the first time and being mindblown at something non-x86 running something I had until then considered the kindergartner/mainstream type OS for PC's. I ran it on all kinds of strange stuff with PowerPC and all manner of previously unix only systems within a short period of time, not because I specifically wanted to try to run NT on everything, but because it was hard to track down install media for obscure things, and I needed to first test and then figure out how to use some of these things. Point is these NPU's are getting to critical mass, and when they do there will be an explosion of new stuff - both hardware which has started and software, that will likely coalesce around a couple of main players or standards. For those who want to be in on the ground floor and be part of the process shaping how this goes forward, now you may still have time - soon it will be too late as some main players reach critical mass and everything gravitates towards a few choices only. So grip your NPU's firmly and go fight for your vision of the AI ecosystem, the time to act is now!

[@c0d3warrior]

Wow, FOMO is a sure way to make me lose interest. 😂

[@Sintrania]

We have level 1 tech,level 2 jeff what’s next level 3 steve 😂

[@sysfried]

We love you for your tinkering!

[@chrisbiddle3674]

Great job Jeff, you achieved 55 flops. Also today, NVIDIA released GB200 NVL72 platform achieving 1.4 exaflops/tops 🎉

[@Level2Jeff]

Heh, Nvidia costs a wee bit more too :) But the more you buy, the more you save!

[@brownb2vid]

How do you work around the PIs memory limitations, can this thing run Ollama too?

[@andre-le-bone-aparte]

Question: Oobabooga (text-generation) Web GUI supports multiple GPUs and TensorFlow (TPU) with mixing + matching - would that work for your setup?

[@HamguyBacon]

can this be used for llm's and stable diffusion or is it only useful for video tracking. one day graphics cards will be replaced by AI accelerator cards, and all you'll need is a low power gpu.

[@knifekitty_ls]

time to find some splitters for converting m.2 x4 into 4 m.2 x1 plugs for those dual-edge tpus

[@iammahabub]

can we add this PCIe HAILO AI chips (i.e. 50+ TOPS) in our Mini PC? I meant, If we want add it to NUC or Dell Opt. Mini PCs instead of Raspberry PI

[@Xamz_pok]

Can these boards be plugged into a full-fledged PC?

[@mrpocock]

Is something like this a possible way to upgrade my Linux pc? Or am i still better off buying a second gfx card?

[@Megaflare47]

I think the name is probably "Micro PC city" and the u is used as a substitute for mu because it wouldn't be recognizable at that size.

[@justinknash]

Do these NPUs / TPUs work with ollama and the llama3 model? Currenting running ollama and llama3 on my gaming PC which has a 4060ti.

[@isaacyonemoto]

Arent they all sharing memory bandwidth? Are there any AI m.2s with onboard memory?

[@Information_Seeker]

software support for parallel processing huh, something similar to Bend programming language?

[@novantha1]

So like, objectively I know that the Hailo isn't really meant for inference of production grade LLMs or anything. ...But like, I still want to see if a person couldn't do something silly like a bespoke MoE architecture with 128M active parameters and still get okay quality and speed.

[@hyperverbal]

I wonder if Bend would work with this config ❤

[@SLLabsKamilion]

Marco Reps at the end?

[@tacolover619]

Orange PI 5 Plus + Hailo 8

[@ssteele1812]

Hello Jeff. Odd question here. I have a Lenovo X230 that I will repurposing to "play" with local AI, specifically LLMs. It is going to struggle with the onboard GPU and I will eventually be moving everything to a different machine that will let me add a proper GPU card. Until then, is there a way that I could use a PCIE slot to NVME/M.2 adapter to put one of these little M.2 AI chips in the laptop? Since the machine has USB3 ports on it already, there isn't really anything useful I can put in the slot. If the adapter actually worked, would the extra AI board do me any good without custom software to utilize it?

[@Level2Jeff]

Yes, at least under Linux. Not sure about Windows support for these things.