Why would you build a Raspberry Pi Cluster?

[@freelancer42]

"Because it's fun" and "Because you'll learn a lot" are honestly two of the best reasons to do anything ever

[@chabad360]

I'm literally making over-engineered pi cluster because of that

[@JrockProject1]

That’s why I built my pc. It was a fun pandemic project

[@extantpedant1481]

But using a bunch of 10- year-old cheapo machines could be almost as fun, probably more performant, and teach you about the more widely deployed x86 ecosystem. I'd be interested to see where the line is, where a raspberry pi 4 becomes more performant per watt for server loads compared to older, low end pc cpus. Are we talking early i3, i5, or could a core 2 duo beat it?

[@codemakeshare]

Depending on which country you're in, the cost of a pi cluster is still a looooot cheaper than tuition fees to have a formal degree or course in the topics you'll learn :) and you get to keep the cluster too.

[@sounds252]

Exactly. Those are the top reasons. The third for myself is cost. Currently building / testing out a Kubernetes cluster and learning about the differences of between the virtual and physical hardware. Its a total blast. :D

[@KarlMathiasMoberg]

"Running flightsimulator on a giant cluster isn't going to make it faster" Actually, Flightsimulator is one of the only games that actually can utilize clusters of computers and there are "many" people out there doing it - home cockpit builders. You dedicate a single computer to render the graphics for your view outside, then dedicate other computers to render for instance the panels and how the airplane actually flies, weather, and a whole bunch of other things on different, less powerful computers! Other than that - very informative! :)

[@goodchoice4410]

Do you have a source of this?

[@JeffGeerling]

Dangit! I should've done more research on it... I guess there are a few games where the community will go to great lengths to make the game exceed initial expectations.

[@knightoftheoldcode1391]

@@JeffGeerling Crysis, Jeff, you always go with Crysis. ;)

[@RudyBleeker]

I believe it was possible to run Doom 2 (or some other fps game from back in the day) on 3 different machines at once, one rendering your frontal view and the other 2 rendering your side views. Technically that's clustered.

[@goodchoice4410]

@@KarlMathiasMoberg thanks!

[@miguelagueda3928]

On-Die ECC is basically error correction within the memory module, while as standard ECC is error correction on memory transfers better cpu and memory, so to say. This was a source on confusion for many people when DDR5 was announced, as some people assume the standard mandates ECC, but it's actually only on-die ECC, meaning we'll still see both kinds of modules.

[@liquidluck711]

Ok say this again, but like for people with an IQ of a 5 year old. Asking for a friend :]

[@miguelagueda3928]

@@liquidluck711 AFAIK, On-Die ECC corrects error inside the memory chips, while as standard ECC corrects errors when data is sent from the memory to the cpu. On-Die ECC is a newer technology that is being implemented on new memory (like LPDDR4 or DDR5) as protection because new memory chips have higher capacities and higher chances of data corruption, but it doesn't substitute traditional ECC technology used in servers and industrial devices

[@georhodiumgeo9827]

Literally what I was wondering thx.

[@0x8badf00d]

So, it'll still have the SEC but not the DED?

[@eusebiusthunked5259]

@@0x8badf00d the issue isnt about SEC and DED... The issue is that the traditional server ECC covers not just the internal RAM, but the RAM bus. This ECC will detect and correct errors only with the memory module, but doesnt guarantee that protection throughout the full memory subsystem back to the CPU.

[@PrajjalakChattopadhyay]

I am from Physics background and I feel RPi clusters can be used for scientific computing. Yeah, we have big CPU, GPU, FPGA banks in our Institute - but small RPi clusters can be used for learning the techniques, or even solving smaller problems that doesn't require heavy computational load.

[@cdl0]

Good comment: you saved me the trouble of writing the same. Parallel code development for physics-based simulations is not easy, so it is convenient to have a small, cheap system for this purpose before submitting production jobs to a very expensive supercomputer, which also likely has a long queue.

[@BuriedAudioUK]

I'm considering this for a compute node for BEM calculations in acouetic modeling, since solving for millions of frequencys to draw responses, can just be split into a set of frequencies for each CPU. Issue is they run on windows

[@JustDux]

RPi clusters will save you a lot of time on debugging berfore you occupy big expensive CPU, GPU, FPGA banks

[@BlurryBit]

4:26 Ahhh that LTT hardware failure joke!!! Priceless :D

[@syrus3k]

I'd like a link to the original video please

[@kaede15]

The episode where Linus lost all his video archive hahaha

[@stevietech]

@@syrus3k kzbin.info/www/bejne/nYTVn4uddtJ-bs0

[@s.i.m.c.a]

on rpi cluster you would lose everything in case of some chip or memory card failure, and it is even harder to create a proper raid using rpi due to weak cpu and need to offload it with hardware raid (which is a pain to rescue)....thus nothing laughable there

[@BlurryBit]

@@s.i.m.c.a That could be true. This is why I think it's absolutely necessary to have a service always backing stuffs up to another machine that's not on the same physical setup. This is given that the data you have is very important to you. For LTT's case, I can see why it would be a problem. They edit 8k footages, and those arr huge files. For my case, all I need is my site(s), bots, and dbs to be backed up; which is only a few gbs max. So, I always prefer having that backup service running. Plus I use git for the "script" files, so I am always prepared for the doom day.

[@TheProjectHelpDesk]

"Because it's fun" I can totally understand this concept. Most of my projects fall off the radar and get covered in dust once I get them up and running. That is because the real fun is the experimenting and learning that it takes to get them going. Once they are up and running I get bored and move onto the next project.

[@CptPatch]

I'm a network guy but I wanted to upgrade my linux/server/virtualization skills and I learned a TON by building a home Proxmox cluster. I opted for the x86 route with a couple Ryzen servers that I built with ebay hardware and a couple refurbished micro form factor PCs, though I would have gone with refurbished servers if noise wasn't a concern. It's so convenient to have a bunch of raw computing power that you can just throw at a problem when needed, and every time is a learning experience.

[@KiraSlith]

Which Micro PCs did you end up settling on? I've been dipping my toes into the idea but I haven't settled on if I want Optiplex 9020s (i5-4590S) or EliteDesk 705 G4s (PRO 2200G) since the Pros and Cons with the current market balance each other out.

[@MisterRorschach90]

Here’s the truth about life Just because something is dumb or pointless, doesn’t make it any less cool.

[@JeffGeerling]

Red Shirt Jeff hearted this comment.

[@DaPanda19]

@@JeffGeerling is red shirt Jeff represented by your lawyer? I couldn't imagine wanting that liability on my company 😬 That being said red shift Jeff ftw! 🤙

[@tormentum]

Very much this. Cool, fun, and I'd argue not pointless if you're learning from the experience of working with the underlying tech.

[@oscarromeu2129]

For me the primary reason to build a cluster is to learn and to explore what can be done. As a secondary gain I improve the skills and the knowledge that I use on a regularly daily basis on my job. Currently I'm working to spin up a monitoring stack with k3s, Ansible and Helm. Thank you for all of your content it is awesome. Regards from Barcelona! 🐧👨‍💻☺️

[@BrunodeSouzaLino]

I feel like this is the Achilles heel of pretty much every single cluster video I've seen on KZbin so far. They seem to be made for cluster users by cluster users. Many of the information is glossed over, with the assumption the viewer already knows what's being talked about, little is talked about the specific hardware under the same guise and so on.

[@PraktikoolSinik]

Yes, he mentions various software used for and applications of clustering. Here's a simpler answer... The Pi and similar inexpensive hardware gives people without access to a data center an affordable means of learning what all that stuff is.

[@cdl0]

@@PraktikoolSinik Good answer. It is also useful for developing parallel code, which is not easy.

[@doc_sav]

What, you are sick of endless tutorials of setting up nginx on a K8s cluster? Well, have you tried the vast amount of other learning options, which are exclusively Kubernetes applications that address Kubernetes shortcomings? You didn't like reference books < 2 years old that are already full of deprecated commands and demos that no longer work? WELL YOU MUST HATE LEARNING THEN.

[@xybersurfer]

@@emeraldbonsai it depends on what they are about. web development probably moves the fastest

[@HainjeDAF]

I love the way you make computing concepts understandable. And I really like the way you brag about your mistakes in the outtros.

[@____________________________.x]

When I used to do IT stuff, I realised that the overall resilience of a IT infrastructure was better when servers just did one thing each. Every task you added to a server just made it more likely that it glitched in some way. This was before virtualisation though.

[@perwestermark8920]

In many situations, two multicore PC with Docker/Kubernetes can do quite well for running massive amounts of small services. And the services can be bounced between the machines when doing maintenance .

[@MikeStavola]

I was really excited to see your blade video. I was just talking about this concept for use in my home automation design. Fully redundant controllers for all the systems in the house, data backup, home security management, network management, web hosting, and so on.

[@michaelprezioso7368]

Hey Jeff, say for what it’s worth I run an 8 node K3s cluster at home as a personal lab. It’s been amazing for POC/testing/and working with Kubernetes and all of the numerous cni’s, containers, devops tools, and etc. This has helped reduce my time to market, while moving and developing terraform, ansible, python, helm charts, and may other things all while migrating an on prem data center into the cloud. So for the investment/RTO I say it’s a no brainer! Also love your work and content; cheers keep up the amazing work!

[@tailkinker1972]

I once built a cluster out of the six computers in our house - Three being mine, two being my son's, one being the HTPC I built for our television - for purposes of rendering video (using Povray). Didn't get to use it often, because I'd have to get my son to switch both of his computers to Linux. But boy, did renders happen quick! :)

[@nexxusty]

This sounds fun.

[@binarycat1237]

I should look into this for blender...

[@scurvofpcp]

@@binarycat1237 Depending on your renderload for blender, something that can work very nice is bitcoin mining boards and just a crap tonne of 760- 1080 gtx video cards (because they look like their about to bottom out in price with nvidias latest news) . Keep in mind it will be a pcie 1 connection that they all have which in this application can be just fine. I'm using a Biostar TB360-BTC PRO and some hawked up server powersupplies,

[@linuxstreamer8910]

i subbed to this channel not because I'm gonna do anything that this channel showed but because I'm in awe of what the raspberry pi can do

[@mikestechtalks6781]

A pi cluster is a neat experiment to teach you about building out clusters. However, 4 core cpus and limited ram and storage, severely limit what you can do with a cluster. Where it can make sense is on the edge where you combine a pi cluster w IoT elements that would feed into the cluster to curate data before sending it to your main cluster.

[@mcdermg]

Another great video, as always informative and interesting with the right level of detail. The pi is and always will be a learning tool, without it I wouldn't know next to anything about Linux, networking, security or kubernetes. Following Jeff's lead I build a little cluster out of Pi's for a home lab that makes it easier to upskill and has so many positive benefits on my skill set to the point where I'm certain the jobs I landed are in no small part due to this experimentation.

[@ScenegraphStudios]

Thanks for using our render for the International Space Station @6.12. We have been watching your videos for a long time now, and this was cool to see!!!

[@JeffGeerling]

It's a beautiful render!

[@ScenegraphStudios]

@@JeffGeerling If you want any other renders doing, please do reach out :)

[@ebrewste]

I use mine to record several cameras and stitch the result into a panorama video at a remote site. The compute is pretty heavy the Pi, so splitting compute up across the different Pis helps a lot. Someday, I'll get this on a Jetson with GPU...

[@Mark-gl2fz]

I did create a cluster using 3 raspberry Pis, May of 2020. I chose 3 to save money and still learn. I wanted to learn because I thought it would give me some insight into machine learning or AI and to teach myself something new. When I finished I couldn’t find anything to put on it, along those lines, so I reused the Pis elsewhere. I want to create another one because I am learning more about possible applications on your KZbin channel and I miss the challenge. There is a shortage of just about everything right now, so I am in the read and learn stage. I am hoping that the supply shortages will ease next Spring. In the meantime, please continue with your work and I look forward to every one of your presentations.

[@kinslayermds]

I really don't have the patience to do any of this, but I love that this can exist in a semi affordable manner.

[@nmstoker]

Jeff: sometimes I feel like you read my mind! I'm keen to try a cluster setup but I also wondered about the performance comparison and suitability for my needs. You have that covered here perfectly! Thanks 👍

[@tommytigerpants]

Hey Jeff, I am running 4x RBpi 4s for a K3s cluster for several applications and will use a 5th RBpi 4 for a baremetal install of Ubuntu to the run the GUI for cluster management. This has been a huge learning curve as I only have interest in this topic but no experience!

[@ymemag9861]

Like I posted in your other video to people asking questions about a cluster, they are also great for learning. Students practicing setting up and administering clusters need something to use, these are far cheaper and have more access time for students than a typical cluster at school or work, being used for production work.

[@stevedonkers9087]

Any software that can be split up into small pieces will greatly benefit from a cluster. Many large universities have cluster computers. Often scientific analyses can be broken up into small chunks. Many thousands of datapoints requiring calculations could be split up among many processors using an MPI-based system. Depending on the budget, it could be more useful to have a cluster of any machine type over one really expensive computer, especially if you have many users that have different requirements.

[@magnuslindgren9460]

I've been running a kafka cluster on 5 pi's for a couple of years now, use it for processing rsyslog messages, mostly for logs from my pfsense's but also from other machines. The cluster holds data for about a week so I can tolerate a single point of failure after that. Also gives me the ability to extract data for development without interfering with the main flow.

[@BrianWebbN]

Great video, and I agree with most of what you said, but, having been using/building/programming cluster computers since the late 80's, I have to agree with your first assertion that they're not really the best choice for the vast majority of users. You compare the 64 cores in the pi cluster to a high-end 64 core server CPU, but I believe you could easily build a more capable system using lower-end hardware that might not have the same number of cores, but could beat the pi cluster in pretty much every comparable stat. You do have a point with the benefits of being able to more easily restore services in the case of failure with redundant hardware, but that's also the case with docker/kubernetes, and my experience with SBC clusters is that they go down much more often than a single PC, and you know when your server goes down, but it's more common for silent failures when you have a bunch of SBCs running different applications that you might not be able to monitor as easily. That said, I do recommend to anyone that is interested to build a cluster. It is a very good learning experience, and it's relatively inexpensive.

[@rhekman]

4:19 "You might even name them..." LOL I'm dying.

[@kebien6020]

Just like that server

[@SenpaiSerene]

Whonnock literally knocked

[@thegardenofeatin5965]

@@SenpaiSerene So that's how you spell "Juaniq"

[@Felice_Enellen]

Heh, I laughed at the subtle jab at LTT for the Whannock incident.

[@joir2000]

I made my first cluster on various VM's.... just for learning Building a PI-cluster is still on my wishlist. Thanks for the interesting video!

[@johngeverett]

This takes me back to when I had to solder together the 8" diskette drives for my Appke ][+, back when 'personal computers' were truly personal!

[@circuitsandcigars1278]

My poor Vic20 and C64 were full of hardware mods and looked like porcupines with multi color quills from the wires sticking out

[@jacoboldfield88]

I'm currently running a 3 node Kubernetes cluster using Raspberry Pi's, planning to expand it out to 12 nodes once my 2U bracket arrives from UCTRONICS. The best use I've found for this is as a learning tool. I'm a software developer, so for me it's about having a platform to experiment with Kubernetes and containerization in general. To build a cluster of similar scale using cloud services would cost upwards of $1000 AUD a month, so a Raspberry Pi cluster is cheaper option with an upfront cost of $2000-3000 AUD up front, considering I plan to run this 24/7. I also see the performance bottlenecks inherent to a low powered ARM device as an opportunity to explore methods to optimize my code and split the workloads across several different threads, compute limitations force me to get creative and ultimately make me a better developer.

[@JeffGeerling]

I didn't even touch on the bottleneck topic-but I found the same. The Pi can expose code paths that are _usually_ not a problem on fancier hardware, but are glaringly bad on Pi (e.g. disk accesses where you don't need them, high cyclomatic complexity in functions that get really slow on the Pi...). Running code on slower hardware exposes those flaws and takes a 'decent' app on fast hardware and can make it _fly_.

[@jacoboldfield88]

@@JeffGeerling One area where the compute bottlenecks really show is in database applications and cryptographic functions. I’ve had to experiment with using in memory caching mechanisms, currently playing around with Redis. I’m looking into options for offloading cryptographic functions to a USB module, but so far haven’t had much luck.

[@BowenMarmot]

I have a 13 node CEPH cluster all running on Pi4 8GB and hosting 44TB raw storage. It used to reside on two older server class machines that have slowly been dying. Since I couldn't easily replace the hardware with comparable I decided to go all in on the Pi cluster so I could trivially replace any component if it dies. The up side is way more redundancy, higher performance and lower power consumption!

[@JeffGeerling]

Wow, nice! Have you done any benchmarks on the cluster to see what kind of throughput you could get for sequential access (and random / IOPS too)? I've been planning on testing Ceph again sometime this year, would be great to see how your cluster does so I can get a point of comparison!

[@BowenMarmot]

@@JeffGeerling Thanks. Surprisingly, no I haven't done any benchmarking aside from the seat of the pants "good enough". It runs all my VMs and other services without any notable slowdown and the Client IO metric on the Ceph Manager dashboard shows up to 200MB/s which is about double what I could get from the two server machines. The drives are all 4TB WD Blacks (HDDs) connected via USB3 so not slow but nothing like SSD drives and NOTHING like the performance you have gotten with some of the PCI connected devices recently. Now, of course, I am going to go do some benchmarking. I guess the Prometheus install is going to have to wait a bit longer. LOL Edit: GB -> MB I wish it were 200GB/s

[@danielbarnes3406]

The Odroid HC2 is great for this. I have eight of them running LizardFS. I'm planning to get more.

[@BowenMarmot]

@@JeffGeerling First round of benchmarking done. I used the commands available from ceph. "rados bench ...". My performance for any one device is network limited as it will saturate the gigabit link on any read or write. Same with two devices and almost with three. Running the benchmark on four machines ends up with approximately 300-320 MB/s throughput on writes. There is something I am missing with coordinating the read benchmarks using "rados bench..." across multiple clients so don't have good numbers for that yet. Issue sorted: Sequential reads across 4 devices tops out at 406MB/s, random reads is about the same. Makes me think it may actually be a little quicker but I don't have another physical wired device I can use for benchmarking.

[@JeffGeerling]

@@BowenMarmot That's great to hear! I'm glad to hear the numbers are so good; was thinking it might not be able to saturate the Pi's network connection.

[@atreusduvelll600]

Jeff, please never stop appending outtakes to your videos. They are fantastic. Great video, as always! Thanks. I have a Pi sitting next to me as we speak and I've been trying to think about what to do with it. Sounds like it needs some brothers and sisters next :)

[@demusicfan]

When you re-build your Kubernetes cluster, would you mind doing an updated tutorial that would include ways to deploy without using ansible? Or maybe just explain how to set up ansible on a pi and deploy from there? Anyway, I always appreciate the effort you put into your content. You and others on YT have finally convinced me to set up my own homelab and not be intimidated by the thought that it's too complicated. Failures really are part of the process and you never know till you try. Thanks!

[@JeffGeerling]

I will likely do another series once I can source enough parts for a new CM4-based cluster :) - just been hard getting everything this year :(

[@tezgale]

Well done Jeff on trying to explain this. I remember many a conversation trying to explain the topics you cover. This was 20 years ago working on IBM RS6000 SP clusters and following this working on pentium 3 clusters running RedHat 6.1( yes the RedHat before RHEL), Beowulf, XCAT on 1000s of nodes. Got myself a Turing Pi a few months back and plan to go back through your videos as soon as I have some free time. Also really interested in Ivan Kuleshov's work with the blades. This has got a lot of potential moving forward. Keep up the good work.

[@etiennebruines467]

I like the advice: treat your servers like cattle and not like pets

[@isaackvasager9957]

drink their milk and eat their dead bodies? if you say so.

[@voxelfusion9894]

@@isaackvasager9957 a little extra iron and copper never hurt a diet.

[@Rekhan4242]

Poke with a shocky stick?

[@yorickmeulenbelt4all]

Redshirt Jeff holding the Turing Pi v1? Does blue or black shirt Jeff know about that?

[@FreedomAirguns]

There have been attempts at making virtualization software running on multiple machines and be seen as a single huge machine by the guest OS. One such software is ScaleMP but since they've been acquired, it's like they have vanished. They offered demos to companies before and there still are some youtube videos about it. Theoretically speaking, one can rebuild QEMU to run parallelized and do "the same". Also theoretically, one can even try to make a "Frankenstein" where two or more boards are controlled by one single bios chip with custom firmware that should be responsible for the initialization of all the boards, sharing the same ram and non volatile memory from a "master" board. To me though, this looks like a nice learning experience, nevertheless, price-performance isn't there yet. I mean, if it's about learning, one might as well learn properly if the same price gives better results with different hardware, like the N series from Intel (example).

[@KarstenJohansson]

I was experimenting with MPI on the RPI's. It's pretty interesting. A good entry-point for learning is the ClusterHAT where you can use a Raspberry Pi 3 or 4 with four Raspberry Pi Zeros, using only a single power supply. It is kinda fragile to power spikes, but for learning inexpensively, it's hard to beat. Kubernetes is great for web stuff, but MPI gets down to the application layer. K8s seems like a compact version of a data center, whereas MPI is more like a massive (but more sluggish) multi-core CPU. They would probably play well together.

[@cluerip]

Would also be great for data analysis. Would love to have had one of these when I was going through very large files.

[@aer185]

Hey I haven't finished the video yet, but I'm a computer engineer studying distributed systems, and a cheap Pi Cluster was the ideal way for me to implement some of my research without waiting for time on a commercial Beowulf Cluster. Ok I just finished the video and this is a great explanation for folks, love it.

[@pelegsap]

You could also run parallelized numerical simulations on the cluster. Probably not the most efficient to do at home, but hey - hobbies are hobbies ;)

[@mustafabayzid]

I was thinking why people are making pie cluster and you already made a video about that!!! Thanks Jeff.

[@ny4i]

It appears you are more talking about having multiple worker systems versus a classic cluster for parallelism. For example, having a cluster where I share state between devices in an HA environment is difference than having 8 Pis in a pool of workers behind a web server load balancer. Your blade Pi "cluster" appears to be more a question of ease of form factor (which is great).

[@JuicyJonesHQ]

Yeah you get it. The educational value of this stuff can't be exaggerated. Keep in mind that everyone of a certain generation grew up and invented every whiz bang thing you see around you today only after growing up during the dawn of personal computing. And the computers we used were sinple low power devices that we completely digested and took on one piece at a time, added to one generation at a time, and more or less fully understood just from using. That's not easy on an Android phone but it's a blast with a raspberry pi. Love these videos man.

[@nothere3042]

This is the question I need answers to. Thank you for this. Can't wait

[@soulife8383]

That Gary explains guy did a pretty cool demo with a jetson nano cluster, using the added GPU to brute force passwords. You're getting more power with the added GPU but it's a good visualization of what clustering can do.

[@adubs.]

I'm still hung up on price/performance of a pi cluster. I get the idea of it being fun, and its certainly neat, but I would challenge the need for comparing to epyc. I think even lower end Threadripper is probably overkill. I've got a little 3800x in my home server that I think would probably give you a run for your money... but theres no reason to think it would when we can just compare compile times. Pick the source code and lets benchmark and find out. What do you say?

[@JonathanSwiftUK]

I think there is a community version of Hadoop you can run - clustering where you take a job and split it up to process parts of it in parallel to get the results faster. You could run clustered sql/database. If you have boards all running separate software it's not a cluster, just a convenient way of physically connecting up a group of Pi boards.

[@grumpy-badger]

We are using a cluster of four pis in docker swarm. It serves a tomcat webapp very well.

[@CurrywurstMafia]

Depends on what you are willing to do. Since the ARM architecture doesn't support everything, it really depends on its purpose. But a big plus is the small power consumption. How about a comparison between a PI cluster and an equivalent server? What can the server do that the cluster can't? How high are the power differences at idle and under load?

[@grubbyga7601]

I use my Pi as a Wifi USB-Hub for all PC's. Its easy to install and runs allog with other things. Its called VirtualHere.

[@airy_co]

interesting

[@lucaspelegrino1]

What that means? You use it as a router?

[@afriedli]

I'm sorry if this has been covered in a video I've missed, but the area of clustering I'd be most interested in concerns video processing. I'd imagine this is a good application because the comms bandwidth requirements probably aren't excessive, but the gains from having more grunt (CPUs & GPUs) could be significant. For example, if there was cluster support in something like FFMPEG, that would be great because, say, converting from one video format another can be painfully slow even on a high end machine! It would be even better if I could build a Pi cluster, attach it to my Mac via LAN or USB and render videos faster in iMovie and other Apple video apps by 'farming out' video processing to the cluster. Anyone who cracked this would make a lot of Apple users very happy!

[@kaitlynk2145]

When you finally set up Kubernetes on your pi rack would love a video on that! Keep up the great work love all your videos

[@YeOldeTraveller]

I have a 4-node bramble of RPi 3s that I use for core services in my home network. I initially used this cluster to experiment with the overhead involved in spreading work over multiple nodes for different types of tasks. $250 for 4x 4-core nodes on a dedicated network switch seemed a good investment at the time. The performance was not as important as the node count for my use case. Having 4 identical systems with identical cores was a significant benefit. I could directly compare the performance of 1 core on each of 4 nodes working on a specific task to the performance of 4 cores on one node doing the same thing. Sometimes, the only way to understand how scaling works is to work with under-powered hardware. This allows you see what different types of bottlenecks look like as you ramp up the parallel threads. Professionally, I have found issues with software that was written on the fastest computers running on the fastest network that impacted real world installations in ways the developers could not see due to the power of their development environment.

[@davidg5898]

Excellent explanations. For myself, it's all about uptime of lightweight applications. The fun, and frustration, of the learning process is just icing on the cake. Have you seen Techno Tim's video on High Availability Pi-Hole? The people saying "that's what virtualization is for" obviously didn't pay attention to the section on power consumption. For my use case, virtualizing on a "real" computer would result in 2-5x the power consumption (and heat generation) with no performance benefit.

[@JeffGeerling]

I haven't seen that video, but I am planning on getting my Pi-hole setup to be a bit more robust later this year. I've had that Pi lock up twice since I switched my whole home to it, and it's annoying to have to reboot the thing once every couple months!

[@spvillano]

I've mostly did application pool clustering, distributed compilation and some distributed transcoding over the years. When using disparate hardware, scheduling can get... Interesting. But, that's half of the fun!

[@SirDragonClaw]

To be fair, you can run pretty much everything you are running + home assistant + hosting for multiple other services + plex + a bunch more dockers, all on a single i3 pc. So while I love the idea I still don't see much of a use for pi clusters. Maybe for a distributed ray tracing system, but then memory becomes a non trivial issue.

[@LiLBitsDK]

and that one PC dies and he still got 5+ pi's running while you are offline ;-)

[@maplenerd22]

He mentioned it in the video. Availability and up time. When one server dies, all the the other servers will still perform the task.

[@LiLBitsDK]

@@maplenerd22 and a new one can be spun up automatically as well for the one that crashed

[@Winnetou17]

To be fair, unless you're only actually using only 1-2 services at a time and not have an actual public website, then that i3 PC comparison is bullcrap. I mean, it cannot cope. Good luck watching a movie from plex when the backup service is running and you have several visits on the website. Though I have to agree, if you can somehow use past unused CPU cycles when needed, then an i3 running at 100% should be able to power most things effectively. Anyway, having a cluser also improves the bandwidth and IOPS that the storage and RAM effectively have per process. If there's a spike and 10 processes really do need to do stuff at the exact same time, a cluster of 4 Pi with modest RAM/storage would be better than one i3 with good (not best) RAM&storage.

[@Santor-]

Just because it "can" be done, doesn't necessarily make it the best option. Plenty of use cases where having stable functions working independently from others can easily be imagined. Gathering data from external IOT devices, assembled, posted onto an internal local website for monitoring, or for student matlab projects, redundant micro server, or each PI zero as a powerbank/fitbit clone, which can play music on you outing, and when back to recharge by imserting into cluster, heart rates, oxygen bloid lwvel etc can be uploaded for your own private health check, that bypasses the vendors of such systems that sell your data. I ran a professional CAD service some years ago, and while I mostly was the only person doing it all, save for hiring in people for some projects, I still had all the tasks spread out on many different machines. 1 was used for new software installs, testing, pushing the envelope etc, 2 other machines was "Stable" CAD workstations, with absolute minimum installed to avoid software breaking when new versions came along, then a specific machine for customer visits, to avoid crisis if theft, incompatabilities etc, some accounting only machine, with temote access for CPA, standard office (word/excel ect) machine, and finally redundant hot swappable file server with client backups, connected to remote storage ++. Run all that on 1 machine? Sure, i guess you could make it work. Would be slow and need lots of reinstalls and reboots though. And good luck if you broke the HD. While not a cluster, a cluster could been used for a set of thin clients instead, which was a technology a little above my cost range.

[@RecklessRowly]

I too decided to build a Pi cluster during the Pandemic. It stopped me going crazy. I've used mine for: 1) trying to crack a hash algorithm we use at work to determine how safe it is. 2) Calculating magic squares. 3) Trying to get WRF (Weather Research and Forecasting) to run sensibly on it, but have not had luck yet, but still persevering. 4) Trying to network render Blender. The first 2 projects I wrote the code myself in C so learnt quite a bit on how to break a problem down into seperate managible tasks that each core can handle. I learnt how to Network Boot the Cluster and suddenly I had a whole load of redundant sd cards. The latest project I have had to use Kali Linux to run Blender 2.8.3 as the Raspberry Pi Os only supports Blender 2.79. Anyway this project is still on going, I've just managed to Network boot Kali. Still got quite a bit more to do. Many thanks to Jeff Geerling for his videos. They very well made and very informative. I often laugh at his outakes. I have made a couple of videos myself and know how difficult it is. I often get tongue tied. so I know how you feel.

[@JeffGeerling]

Weather stuff is so hard to get running well on _any_ cluster. I really want to go tour the local NWS offices someday and talk to them a bit about it. We don't have the compute resources here but the weather people have a lot of deep knowledge in specific forecasting areas!

[@WillMakesTV]

I would love to build a Pi cluster to experiment with Kubernetes. Thanks for another great video!

[@tramcrazy]

What a great explainer Jeff! The production quality on your videos just keeps getting better too - I loved the graphics and b roll.

[@ZaquMan]

Well I'm a little surprised that Jeff did not mention parallel computing, that's always the first thing I think of when the topic of Pi clusters comes up.

[@gounthar]

Super nice job of explaining it all, Jeff, well done!

[@World_Theory]

Can't really think of anything useful I could do with a Pi cluster, unless you count something like Folding@Home. If you could utilize a Pi cluster to do computer vision tasks at a higher frame rate, with no latency cost, that would be cool. It could make some interesting things possible for cheap.

[@harrin1982]

I loved the included snippet with Linus from LTT!

[@FitForMyPurpose]

Just weeping at this cat that said “but can it run crysis?” 😂 Old memes are old.

[@jacekruzyczka3058]

I've had the idea of building a cluster (mostly for storing my photo archive, which is becoming too large for my 5-year-old monolithic home server, and for running a PostgreSQL database) for a bunch of years now, but after your presentation of the CM4 + M.2 SSD blade, I've been thinking of such a cluster (which would then be a SAN + multi-node database). A Turing Pi 2 would then be the "head" of the cluster and the interface to the outside world. I could attach FPGAs to its two mini-PCIe connectors for payloads like encryption. But I am not sure about the fairly slow network connection: 1 GBit / sec is nothing special any more. The Mixtile Blade 3, on the other hand, features 2× 2.5 GBit / sec plus a PCIe daisy chain.

[@zambonidriver42]

I’m sorta lucky, I have access to racks of servers. I can spin up stuff quickly. But it’s at the office, tied to work. I really want that new Turing Pi Cluster 2 for my home. I have a half dozen containers I’d like to run. Seems like it’s taking forever. I put my email address in for updates, and ….nothing. I used to check their blog weekly, now it’s monthly. Edit: I should also add that I take servers out of production, and dump them into a sandbox, and I frequently let people (at the office) experiment. I own the routers, switches, I throw a subnets, VLANs, and a dhcp/dns server, PXE, http, tftp, I give them access and email them a couple of howtos, and let them have at it. I check back on them in every few days. I only ask that they publish what they learn and throw what they did on our company git site.

[@nodidog]

I'm also impatiently waiting for the Turing Pi 2, I've already bought 4x CM4s in preparation... I hope they don't just give up on it! It's been radio silence for months now

[@sugoruyo]

2:00 - I'd like to address this to avoid confusion: a bunch of Pis (similar computers) running different workloads (Prometheus, Docker, Web services) *is not* a cluster. Unless you're running all of those under something like Docker Swarm or K8s where you have a bunch of machines and software that decides what to run where. The hallmark of a cluster is not the heterogeneity or homogeneity of nodes or workloads but the fact that workload and nodes are managed in an automated way by specialised clustering middleware (K8s, Mesos, Beowulf etc.). What you're describing in that section is more like a server farm.

[@KienTran]

"Everything fails all the time" - Dr. Werner Vogels

[@JeffGeerling]

I still have an HBA from you on my test bench :D

[@danmanx2]

Jeff, these are very good reasons to build RPi clusters. Thanks for the video!

[@Dygear]

Because it's ... AWESOME!

[@TheGamesZilla492]

always a reasons to do it,i myself learned on pi clusters, and now have a job working for a company maintaining clusters in a data centre.

[@nnm35]

Oh, I thought I was a subscriber, well now I am ;-).(you turn up all the time on my Roku un-logged-in feed), but hey, what a Tease, you're better than that, aren't you!? Will check back tomorrow!

[@zrodger2296]

I specced out and built a 32- and then 64- node Beowulf cluster in the late 90s / early 2000s at school (high performance computing research). (Think of 2 rows of shelves holding 64 desktop towers -- rackmount was too expensive.) We used MPICH for the communication. When solving a single problem (in our applications) speedup was never great across a fair number of nodes. But sometimes you needed more memory than a single node could handle, or just wanted to solve a problem fast, even if it was a little inefficient.

[@alliejr]

Back in the day, we called it “scale OUT” vs “scale UP”.

[@Tea-Spin]

I think we still call them that, maybe Jeff just doesn't want to confuse more people that aren't aware of terms like vertical and horizontal scaling

[@startiger2]

I used to work for a large aerospace company. Besides the servers, the company would occasionally need to do massive calculations for airflow or structural analysis of systems that were being developed. They built a system similar to SETI@home and installed it on all the company's office desktops. IT wanted employees to leave their computers on over night for update purposes anyway, so this made use of their processing power during idle times. This system would run the various calculations for the R&D work by splitting it across the hundred of computers on site in their idle time. It was a lot cheaper to use existing computers to build our own in-house "supercomputer" that to actually buy a supercompuer to do these calculations that were only needed a few times a month and due to various contract requirements, we were not allowed to do this work by outside groups.

[@artucuno_6491]

Hey Jeff, you should create a big Folding@Home Pi cluster. That would be awesome to see.

[@jaqque9633]

I assembled a Turing Pi cluster. It uses seven CM3 boards. It's a nice little board, but the limitations of the Pi 3 compute module means it can't run modern kubernetes (of any sort, including k3s and microk8s). Instead it runs Docker Swarm and I've had a "fun" time setting up GlusterFS to aggregate the available storage. Of course, except for the initial imaging, everything is set up via Ansible.

[@ShotgunKlaus]

that quake 3 hitsounds ...

[@chadcordero1618]

Pretty much sums up why I built my Pi cluster. I wanted to learn how to manage Kubernetes and Docker, though I've since replaced Docker with Containerd in my Pi cluster. Great little machines for testing.

[@CheapHomeTech]

Interesting topic. Considering virtualization to save on money, energy, and arguably complexity has been popular for the last 30 years. So yeah, Pi clusters in most scenarios are not optimal...

[@siegfriedkettlitz6529]

...but in a few scenarios they are the cheapest solution to get a stack of real processors.

[@darkfire2703]

You can get 5 RPi4Bs with 2GB Ram for 200€ + a few bucks for other accessories and a power supply with enough powerto run all of them together. This makes it less expensive and significantly more power efficient than any other option on the market. You get 20 Cores and 10GB Ram in total for this. Low power NAS options are less powerful and are much more expensive. And if you try to build something yourself it will get even more expensive if you want comparable performance and efficiency. So if you want to run something that can be run on a pi cluster, this is actually not a bad option

[@autohmae]

Funny fact is: I was running Linux (Linux-vserver) containers in production before VMs were popular, most people had no idea what was possible.

[@wiel-spin]

I use older Supermicro motherboards with Intel Xeon’s and ECC memory in my homelab. Because Supermicro still use ATX formfactor which is familiar and helps with finding parts and fitting it into non-Supermicro cases. Plus since its also x86 you can run anything and has all the enterprise features you don’t find on consumer oriented stuff. Only downside is the power-consumption because at idle it still uses a lot in comparison with a PI.

[@LazyBunnyKiera]

I'd be happy with a single pi4 for Home Assistant. I just can't justify dropping the cash on even a single pi4 when i have enough PC parts to build an old but functional x86/x64 pc.

[@thegardenofeatin5965]

Before watching the video: I've seen several Pi cluster projects posted to KZbin over the years. "Hey, check out my Raspberry Pi Beowulf cluster! They're hooked together over Ethernet and the one on the bottom is controlling the others. See how the lights on them are blinking in coordinated patterns?" And the commenters are like "What do you run on it?" And I've NEVER seen a straight answer to that question. Of course some jackass always asks "can it run Crysis" and that basically derails any actual conversation, but the best I've gotten is "No, you don't understand, it's not about running software, it's a cluster that I built for school." Running several instances of smaller tasks like web hosting in parallel for increased throughput and failure tolerance makes sense, and for educational or small scale home usage yeah raspberry pis make sense. You initially described a bunch of individual Pis doing unrelated tasks, which I guess doesn't count?

[@johncnorris]

I always had trouble getting the people who controlled finance to understand why we needed reliable stand-alone development equipment to use as test and experiment devices. I usually scraped together used equipment and at times transferred it to act as temporary production devices to solve problems we needed at the time. Knowing how to setup and manage a cluster is no small effort as well. I think the people who resist these projects are either too cheap to understand why they are valuable or just not very curious.

[@Cynyr]

or IT is a hobby for us and even $3000 isn't going to be approved by the budget troll. I completely agree if you are supporting a production environment a test environment is very useful, as is a place to practice managing a cluster.

[@jonasbergkvist6381]

Great video Jeff, just so I understand it correctly - lets say you're running a webserver in the cluster, the service is distributed between all the clients in the cluster? So one Pi can break and the webserver keeps working? At 1:45 you say that running prometheus and grafana on the first Pi, so if that specific Pi dies then you lose prometheus and grafana?

[@JeffGeerling]

Yes, and yes-so you'd need a system like Kubernetes to run on top of the cluster to provide better fault tolerance (so you could manage multiple copies and distribute HTTP requests among the nodes). And the way I'm currently running Prometheus and Grafana (attached to one Pi), they are *not* fault tolerant, so it would be better if I could get them into Kubernetes. There will likely still be a single point of failure in most clusters, though, whether it's a load balancer in front, a single DNS server, one DNS provider, etc. - but the key is to try to get high availability into all the parts of the stack most prone to failure-usually that's in the webserver instances, for that type of application.

[@gerald4027]

All of my pies are a cluster in the fridge.

[@TurkeyDev]

I've built the 2U version of the Server rack mount system you have and currently have 2 pi's in it running off POE. 1 Runs Home Assistant and the other is a build node for my Jenkins server. I'm hoping over time to get more pi's to add to the rack and move more services out of the cloud to be hosted locally. I am also hoping to eventually network boot them all to avoid using the microSD cards since I know it's only a matter of time before one goes bad and I lose everything on that pi. I definitely love using and building things with the pi's! Especially the Pi Zero

[@TurkeyDev]

@AstroCat Having space for the SATA device is what it really comes down to xD but also network boot is easy enough since it only needs an ethernet cable and that's already there for internet and poe. I did stumble upon some 3D print rack designs though that include some sorta of external storage option.

[@TheWebstaff]

I use my cluster of pi's as a dust collection system.

[@JeffGeerling]

Kill two birds with one stone!

[@TheWebstaff]

@@JeffGeerling sorry misspelt that's.. Should have been clutter.. :)

[@Blu702]

You're my favorite tech nerd Jeff. I'm a nerd myself. So I use that as a term of endearment .

[@janklas7079]

BAsically when you describe what your cluster is doing, you're proving clustering them didn't bring you anything. Those pi's would have done those task faster and more efficiently outside of a cluster.

[@SeeJayPlayGames]

Maybe not in terms of performance, but clustering at least brought redundancy in case one failed. You just proved you weren't paying attention.

[@janklas7079]

@@SeeJayPlayGames You can have failover without a cluster. Clusters are NOT used to bring redundancy. You don't know your stuff at all.

[@SeeJayPlayGames]

@@janklas7079 I never claimed I did. I was just saying that you missed that point in his video. So you're saying he's wrong? OK, whatever. I don't know anything about clustering; I was trying to learn. That's why I clicked on this video. But to say I don't know my stuff *at all* is a bit generalized and quite presumptuous on your part. You come off like a total jerk.

[@lucaspelegrino1]

He did give a valid example of loadbalancing. How would you do a failover without the cluster? And whats the pros/cons?

[@janklas7079]

@@lucaspelegrino1 The same way; with a load balancer. or in simpler fashion DNS round-robin. Options are plentyfull. The HUGE pro is the missing overhead of a cluster. Also if for some reason your cluster breaks, everything is broken, so redundancy is a moot point.

[@raulcorreia232]

Loved the old reference to the Quake 3 sound, keep up the good content!

[@Miafunfactory]

Becuz it looks cyberpunk AF

[@Jonix2000]

Thank you for inspiration and your educational and enjoyable videos I just may be one of the 10% that would benefit from a Pi cluster Mostly for educational purposes and the wow factor to have a cluster at home There is so many interesting and fun networking infrastructure projects nowadays

[@stevepope6084]

O well, there goes my idea of playing cyber punk at 8k at 1000fps...

[@FoX84tac022]

I was having this conversation with a friend of mine whom I found out also watches your videos and we were specifically discussing the video you are here. While I agree simply _performance_ isn't the end-all-be-all; I think it's more performance per $. What I mean is, why would I spend $60+ per pi (x4=$240), when I could recycle some thin-clients like the dell wyse variants for $20-30 ea. (or 4x=$80-120) and slap whatever flavor of linux I'm feeling. Not to discredit your hard work and there definitely was a cool factor. I just think pi's themself cost *too* much for what they are. My tc's have 4x USB 2.0, 2x USB 3.0, Display Port, Wifi (w/ antenna), sata port for storage, oh and a gigabit ethernet jack; a pi would have to be extremely custom to have anything close to that.