Why would you build a Raspberry Pi Cluster?

[@freelancer42]

"Because it's fun" and "Because you'll learn a lot" are honestly two of the best reasons to do anything ever

[@chabad360]

I'm literally making over-engineered pi cluster because of that

[@JrockProject1]

That’s why I built my pc. It was a fun pandemic project

[@extantpedant1481]

But using a bunch of 10- year-old cheapo machines could be almost as fun, probably more performant, and teach you about the more widely deployed x86 ecosystem. I'd be interested to see where the line is, where a raspberry pi 4 becomes more performant per watt for server loads compared to older, low end pc cpus. Are we talking early i3, i5, or could a core 2 duo beat it?

[@codemakeshare]

Depending on which country you're in, the cost of a pi cluster is still a looooot cheaper than tuition fees to have a formal degree or course in the topics you'll learn :) and you get to keep the cluster too.

[@sounds252]

Exactly. Those are the top reasons. The third for myself is cost. Currently building / testing out a Kubernetes cluster and learning about the differences of between the virtual and physical hardware. Its a total blast. :D

[@KarlMathiasMoberg]

"Running flightsimulator on a giant cluster isn't going to make it faster" Actually, Flightsimulator is one of the only games that actually can utilize clusters of computers and there are "many" people out there doing it - home cockpit builders. You dedicate a single computer to render the graphics for your view outside, then dedicate other computers to render for instance the panels and how the airplane actually flies, weather, and a whole bunch of other things on different, less powerful computers! Other than that - very informative! :)

[@goodchoice4410]

Do you have a source of this?

[@JeffGeerling]

Dangit! I should've done more research on it... I guess there are a few games where the community will go to great lengths to make the game exceed initial expectations.

[@knightoftheoldcode1391]

@@JeffGeerling Crysis, Jeff, you always go with Crysis. ;)

[@RudyBleeker]

I believe it was possible to run Doom 2 (or some other fps game from back in the day) on 3 different machines at once, one rendering your frontal view and the other 2 rendering your side views. Technically that's clustered.

[@goodchoice4410]

@@KarlMathiasMoberg thanks!

[@miguelagueda3928]

On-Die ECC is basically error correction within the memory module, while as standard ECC is error correction on memory transfers better cpu and memory, so to say. This was a source on confusion for many people when DDR5 was announced, as some people assume the standard mandates ECC, but it's actually only on-die ECC, meaning we'll still see both kinds of modules.

[@liquidluck711]

Ok say this again, but like for people with an IQ of a 5 year old. Asking for a friend :]

[@miguelagueda3928]

@@liquidluck711 AFAIK, On-Die ECC corrects error inside the memory chips, while as standard ECC corrects errors when data is sent from the memory to the cpu. On-Die ECC is a newer technology that is being implemented on new memory (like LPDDR4 or DDR5) as protection because new memory chips have higher capacities and higher chances of data corruption, but it doesn't substitute traditional ECC technology used in servers and industrial devices

[@georhodiumgeo9827]

Literally what I was wondering thx.

[@0x8badf00d]

So, it'll still have the SEC but not the DED?

[@eusebiusthunked5259]

@@0x8badf00d the issue isnt about SEC and DED... The issue is that the traditional server ECC covers not just the internal RAM, but the RAM bus. This ECC will detect and correct errors only with the memory module, but doesnt guarantee that protection throughout the full memory subsystem back to the CPU.

[@PrajjalakChattopadhyay]

I am from Physics background and I feel RPi clusters can be used for scientific computing. Yeah, we have big CPU, GPU, FPGA banks in our Institute - but small RPi clusters can be used for learning the techniques, or even solving smaller problems that doesn't require heavy computational load.

[@cdl0]

Good comment: you saved me the trouble of writing the same. Parallel code development for physics-based simulations is not easy, so it is convenient to have a small, cheap system for this purpose before submitting production jobs to a very expensive supercomputer, which also likely has a long queue.

[@BuriedAudioUK]

I'm considering this for a compute node for BEM calculations in acouetic modeling, since solving for millions of frequencys to draw responses, can just be split into a set of frequencies for each CPU. Issue is they run on windows

[@BlurryBit]

4:26 Ahhh that LTT hardware failure joke!!! Priceless :D

[@syrus3k]

I'd like a link to the original video please

[@kaede15]

The episode where Linus lost all his video archive hahaha

[@stevietech]

@@syrus3k kzbin.info/www/bejne/nYTVn4uddtJ-bs0

[@s.i.m.c.a]

on rpi cluster you would lose everything in case of some chip or memory card failure, and it is even harder to create a proper raid using rpi due to weak cpu and need to offload it with hardware raid (which is a pain to rescue)....thus nothing laughable there

[@BlurryBit]

@@s.i.m.c.a That could be true. This is why I think it's absolutely necessary to have a service always backing stuffs up to another machine that's not on the same physical setup. This is given that the data you have is very important to you. For LTT's case, I can see why it would be a problem. They edit 8k footages, and those arr huge files. For my case, all I need is my site(s), bots, and dbs to be backed up; which is only a few gbs max. So, I always prefer having that backup service running. Plus I use git for the "script" files, so I am always prepared for the doom day.

[@CptPatch]

I'm a network guy but I wanted to upgrade my linux/server/virtualization skills and I learned a TON by building a home Proxmox cluster. I opted for the x86 route with a couple Ryzen servers that I built with ebay hardware and a couple refurbished micro form factor PCs, though I would have gone with refurbished servers if noise wasn't a concern. It's so convenient to have a bunch of raw computing power that you can just throw at a problem when needed, and every time is a learning experience.

[@KiraSlith]

Which Micro PCs did you end up settling on? I've been dipping my toes into the idea but I haven't settled on if I want Optiplex 9020s (i5-4590S) or EliteDesk 705 G4s (PRO 2200G) since the Pros and Cons with the current market balance each other out.

[@TheProjectHelpDesk]

"Because it's fun" I can totally understand this concept. Most of my projects fall off the radar and get covered in dust once I get them up and running. That is because the real fun is the experimenting and learning that it takes to get them going. Once they are up and running I get bored and move onto the next project.

[@BrunodeSouzaLino]

I feel like this is the Achilles heel of pretty much every single cluster video I've seen on KZbin so far. They seem to be made for cluster users by cluster users. Many of the information is glossed over, with the assumption the viewer already knows what's being talked about, little is talked about the specific hardware under the same guise and so on.

[@PraktikoolSinik]

Yes, he mentions various software used for and applications of clustering. Here's a simpler answer... The Pi and similar inexpensive hardware gives people without access to a data center an affordable means of learning what all that stuff is.

[@cdl0]

@@PraktikoolSinik Good answer. It is also useful for developing parallel code, which is not easy.

[@doc_sav]

What, you are sick of endless tutorials of setting up nginx on a K8s cluster? Well, have you tried the vast amount of other learning options, which are exclusively Kubernetes applications that address Kubernetes shortcomings? You didn't like reference books < 2 years old that are already full of deprecated commands and demos that no longer work? WELL YOU MUST HATE LEARNING THEN.

[@xybersurfer]

@@emeraldbonsai it depends on what they are about. web development probably moves the fastest

[@oscarromeu2129]

For me the primary reason to build a cluster is to learn and to explore what can be done. As a secondary gain I improve the skills and the knowledge that I use on a regularly daily basis on my job. Currently I'm working to spin up a monitoring stack with k3s, Ansible and Helm. Thank you for all of your content it is awesome. Regards from Barcelona! 🐧👨‍💻☺️

[@HainjeDAF]

I love the way you make computing concepts understandable. And I really like the way you brag about your mistakes in the outtros.

[@MisterRorschach90]

Here’s the truth about life Just because something is dumb or pointless, doesn’t make it any less cool.

[@JeffGeerling]

Red Shirt Jeff hearted this comment.

[@DaPanda19]

@@JeffGeerling is red shirt Jeff represented by your lawyer? I couldn't imagine wanting that liability on my company 😬 That being said red shift Jeff ftw! 🤙

[@tormentum]

Very much this. Cool, fun, and I'd argue not pointless if you're learning from the experience of working with the underlying tech.

[@____________________________.x]

When I used to do IT stuff, I realised that the overall resilience of a IT infrastructure was better when servers just did one thing each. Every task you added to a server just made it more likely that it glitched in some way. This was before virtualisation though.

[@MikeStavola]

I was really excited to see your blade video. I was just talking about this concept for use in my home automation design. Fully redundant controllers for all the systems in the house, data backup, home security management, network management, web hosting, and so on.

[@tailkinker1972]

I once built a cluster out of the six computers in our house - Three being mine, two being my son's, one being the HTPC I built for our television - for purposes of rendering video (using Povray). Didn't get to use it often, because I'd have to get my son to switch both of his computers to Linux. But boy, did renders happen quick! :)

[@nexxusty]

This sounds fun.

[@binarycat1237]

I should look into this for blender...

[@scurvofpcp]

@@binarycat1237 Depending on your renderload for blender, something that can work very nice is bitcoin mining boards and just a crap tonne of 760- 1080 gtx video cards (because they look like their about to bottom out in price with nvidias latest news) . Keep in mind it will be a pcie 1 connection that they all have which in this application can be just fine. I'm using a Biostar TB360-BTC PRO and some hawked up server powersupplies,

[@mcdermg]

Another great video, as always informative and interesting with the right level of detail. The pi is and always will be a learning tool, without it I wouldn't know next to anything about Linux, networking, security or kubernetes. Following Jeff's lead I build a little cluster out of Pi's for a home lab that makes it easier to upskill and has so many positive benefits on my skill set to the point where I'm certain the jobs I landed are in no small part due to this experimentation.

[@perwestermark8920]

In many situations, two multicore PC with Docker/Kubernetes can do quite well for running massive amounts of small services. And the services can be bounced between the machines when doing maintenance .

[@nmstoker]

Jeff: sometimes I feel like you read my mind! I'm keen to try a cluster setup but I also wondered about the performance comparison and suitability for my needs. You have that covered here perfectly! Thanks 👍

[@ScenegraphStudios]

Thanks for using our render for the International Space Station @6.12. We have been watching your videos for a long time now, and this was cool to see!!!

[@JeffGeerling]

It's a beautiful render!

[@ScenegraphStudios]

@@JeffGeerling If you want any other renders doing, please do reach out :)

[@mikestechtalks6781]

A pi cluster is a neat experiment to teach you about building out clusters. However, 4 core cpus and limited ram and storage, severely limit what you can do with a cluster. Where it can make sense is on the edge where you combine a pi cluster w IoT elements that would feed into the cluster to curate data before sending it to your main cluster.

[@michaelprezioso7368]

Hey Jeff, say for what it’s worth I run an 8 node K3s cluster at home as a personal lab. It’s been amazing for POC/testing/and working with Kubernetes and all of the numerous cni’s, containers, devops tools, and etc. This has helped reduce my time to market, while moving and developing terraform, ansible, python, helm charts, and may other things all while migrating an on prem data center into the cloud. So for the investment/RTO I say it’s a no brainer! Also love your work and content; cheers keep up the amazing work!

[@ebrewste]

I use mine to record several cameras and stitch the result into a panorama video at a remote site. The compute is pretty heavy the Pi, so splitting compute up across the different Pis helps a lot. Someday, I'll get this on a Jetson with GPU...

[@joir2000]

I made my first cluster on various VM's.... just for learning Building a PI-cluster is still on my wishlist. Thanks for the interesting video!

[@johngeverett]

This takes me back to when I had to solder together the 8" diskette drives for my Appke ][+, back when 'personal computers' were truly personal!

[@circuitsandcigars1278]

My poor Vic20 and C64 were full of hardware mods and looked like porcupines with multi color quills from the wires sticking out

[@BowenMarmot]

I have a 13 node CEPH cluster all running on Pi4 8GB and hosting 44TB raw storage. It used to reside on two older server class machines that have slowly been dying. Since I couldn't easily replace the hardware with comparable I decided to go all in on the Pi cluster so I could trivially replace any component if it dies. The up side is way more redundancy, higher performance and lower power consumption!

[@JeffGeerling]

Wow, nice! Have you done any benchmarks on the cluster to see what kind of throughput you could get for sequential access (and random / IOPS too)? I've been planning on testing Ceph again sometime this year, would be great to see how your cluster does so I can get a point of comparison!

[@BowenMarmot]

@@JeffGeerling Thanks. Surprisingly, no I haven't done any benchmarking aside from the seat of the pants "good enough". It runs all my VMs and other services without any notable slowdown and the Client IO metric on the Ceph Manager dashboard shows up to 200MB/s which is about double what I could get from the two server machines. The drives are all 4TB WD Blacks (HDDs) connected via USB3 so not slow but nothing like SSD drives and NOTHING like the performance you have gotten with some of the PCI connected devices recently. Now, of course, I am going to go do some benchmarking. I guess the Prometheus install is going to have to wait a bit longer. LOL Edit: GB -> MB I wish it were 200GB/s

[@danielbarnes3406]

The Odroid HC2 is great for this. I have eight of them running LizardFS. I'm planning to get more.

[@BowenMarmot]

@@JeffGeerling First round of benchmarking done. I used the commands available from ceph. "rados bench ...". My performance for any one device is network limited as it will saturate the gigabit link on any read or write. Same with two devices and almost with three. Running the benchmark on four machines ends up with approximately 300-320 MB/s throughput on writes. There is something I am missing with coordinating the read benchmarks using "rados bench..." across multiple clients so don't have good numbers for that yet. Issue sorted: Sequential reads across 4 devices tops out at 406MB/s, random reads is about the same. Makes me think it may actually be a little quicker but I don't have another physical wired device I can use for benchmarking.

[@JeffGeerling]

@@BowenMarmot That's great to hear! I'm glad to hear the numbers are so good; was thinking it might not be able to saturate the Pi's network connection.

[@kinslayermds]

I really don't have the patience to do any of this, but I love that this can exist in a semi affordable manner.

[@ymemag9861]

Like I posted in your other video to people asking questions about a cluster, they are also great for learning. Students practicing setting up and administering clusters need something to use, these are far cheaper and have more access time for students than a typical cluster at school or work, being used for production work.

[@rhekman]

4:19 "You might even name them..." LOL I'm dying.

[@kebien6020]

Just like that server

[@SenpaiSerene]

Whonnock literally knocked

[@thegardenofeatin5965]

@@SenpaiSerene So that's how you spell "Juaniq"

[@Mark-gl2fz]

I did create a cluster using 3 raspberry Pis, May of 2020. I chose 3 to save money and still learn. I wanted to learn because I thought it would give me some insight into machine learning or AI and to teach myself something new. When I finished I couldn’t find anything to put on it, along those lines, so I reused the Pis elsewhere. I want to create another one because I am learning more about possible applications on your KZbin channel and I miss the challenge. There is a shortage of just about everything right now, so I am in the read and learn stage. I am hoping that the supply shortages will ease next Spring. In the meantime, please continue with your work and I look forward to every one of your presentations.

[@linuxstreamer8910]

i subbed to this channel not because I'm gonna do anything that this channel showed but because I'm in awe of what the raspberry pi can do

[@demusicfan]

When you re-build your Kubernetes cluster, would you mind doing an updated tutorial that would include ways to deploy without using ansible? Or maybe just explain how to set up ansible on a pi and deploy from there? Anyway, I always appreciate the effort you put into your content. You and others on YT have finally convinced me to set up my own homelab and not be intimidated by the thought that it's too complicated. Failures really are part of the process and you never know till you try. Thanks!

[@JeffGeerling]

I will likely do another series once I can source enough parts for a new CM4-based cluster :) - just been hard getting everything this year :(

[@stevedonkers9087]

Any software that can be split up into small pieces will greatly benefit from a cluster. Many large universities have cluster computers. Often scientific analyses can be broken up into small chunks. Many thousands of datapoints requiring calculations could be split up among many processors using an MPI-based system. Depending on the budget, it could be more useful to have a cluster of any machine type over one really expensive computer, especially if you have many users that have different requirements.

[@tezgale]

Well done Jeff on trying to explain this. I remember many a conversation trying to explain the topics you cover. This was 20 years ago working on IBM RS6000 SP clusters and following this working on pentium 3 clusters running RedHat 6.1( yes the RedHat before RHEL), Beowulf, XCAT on 1000s of nodes. Got myself a Turing Pi a few months back and plan to go back through your videos as soon as I have some free time. Also really interested in Ivan Kuleshov's work with the blades. This has got a lot of potential moving forward. Keep up the good work.

[@Felice_Enellen]

Heh, I laughed at the subtle jab at LTT for the Whannock incident.

[@atreusduvelll600]

Jeff, please never stop appending outtakes to your videos. They are fantastic. Great video, as always! Thanks. I have a Pi sitting next to me as we speak and I've been trying to think about what to do with it. Sounds like it needs some brothers and sisters next :)

[@tommytigerpants]

Hey Jeff, I am running 4x RBpi 4s for a K3s cluster for several applications and will use a 5th RBpi 4 for a baremetal install of Ubuntu to the run the GUI for cluster management. This has been a huge learning curve as I only have interest in this topic but no experience!

[@jacoboldfield88]

I'm currently running a 3 node Kubernetes cluster using Raspberry Pi's, planning to expand it out to 12 nodes once my 2U bracket arrives from UCTRONICS. The best use I've found for this is as a learning tool. I'm a software developer, so for me it's about having a platform to experiment with Kubernetes and containerization in general. To build a cluster of similar scale using cloud services would cost upwards of $1000 AUD a month, so a Raspberry Pi cluster is cheaper option with an upfront cost of $2000-3000 AUD up front, considering I plan to run this 24/7. I also see the performance bottlenecks inherent to a low powered ARM device as an opportunity to explore methods to optimize my code and split the workloads across several different threads, compute limitations force me to get creative and ultimately make me a better developer.

[@JeffGeerling]

I didn't even touch on the bottleneck topic-but I found the same. The Pi can expose code paths that are _usually_ not a problem on fancier hardware, but are glaringly bad on Pi (e.g. disk accesses where you don't need them, high cyclomatic complexity in functions that get really slow on the Pi...). Running code on slower hardware exposes those flaws and takes a 'decent' app on fast hardware and can make it _fly_.

[@jacoboldfield88]

@@JeffGeerling One area where the compute bottlenecks really show is in database applications and cryptographic functions. I’ve had to experiment with using in memory caching mechanisms, currently playing around with Redis. I’m looking into options for offloading cryptographic functions to a USB module, but so far haven’t had much luck.

[@kaitlynk2145]

When you finally set up Kubernetes on your pi rack would love a video on that! Keep up the great work love all your videos

[@soulife8383]

That Gary explains guy did a pretty cool demo with a jetson nano cluster, using the added GPU to brute force passwords. You're getting more power with the added GPU but it's a good visualization of what clustering can do.

[@BrianWebbN]

Great video, and I agree with most of what you said, but, having been using/building/programming cluster computers since the late 80's, I have to agree with your first assertion that they're not really the best choice for the vast majority of users. You compare the 64 cores in the pi cluster to a high-end 64 core server CPU, but I believe you could easily build a more capable system using lower-end hardware that might not have the same number of cores, but could beat the pi cluster in pretty much every comparable stat. You do have a point with the benefits of being able to more easily restore services in the case of failure with redundant hardware, but that's also the case with docker/kubernetes, and my experience with SBC clusters is that they go down much more often than a single PC, and you know when your server goes down, but it's more common for silent failures when you have a bunch of SBCs running different applications that you might not be able to monitor as easily. That said, I do recommend to anyone that is interested to build a cluster. It is a very good learning experience, and it's relatively inexpensive.

[@cluerip]

Would also be great for data analysis. Would love to have had one of these when I was going through very large files.

[@magnuslindgren9460]

I've been running a kafka cluster on 5 pi's for a couple of years now, use it for processing rsyslog messages, mostly for logs from my pfsense's but also from other machines. The cluster holds data for about a week so I can tolerate a single point of failure after that. Also gives me the ability to extract data for development without interfering with the main flow.

[@FreedomAirguns]

There have been attempts at making virtualization software running on multiple machines and be seen as a single huge machine by the guest OS. One such software is ScaleMP but since they've been acquired, it's like they have vanished. They offered demos to companies before and there still are some youtube videos about it. Theoretically speaking, one can rebuild QEMU to run parallelized and do "the same". Also theoretically, one can even try to make a "Frankenstein" where two or more boards are controlled by one single bios chip with custom firmware that should be responsible for the initialization of all the boards, sharing the same ram and non volatile memory from a "master" board. To me though, this looks like a nice learning experience, nevertheless, price-performance isn't there yet. I mean, if it's about learning, one might as well learn properly if the same price gives better results with different hardware, like the N series from Intel (example).

[@pelegsap]

You could also run parallelized numerical simulations on the cluster. Probably not the most efficient to do at home, but hey - hobbies are hobbies ;)

[@KarstenJohansson]

I was experimenting with MPI on the RPI's. It's pretty interesting. A good entry-point for learning is the ClusterHAT where you can use a Raspberry Pi 3 or 4 with four Raspberry Pi Zeros, using only a single power supply. It is kinda fragile to power spikes, but for learning inexpensively, it's hard to beat. Kubernetes is great for web stuff, but MPI gets down to the application layer. K8s seems like a compact version of a data center, whereas MPI is more like a massive (but more sluggish) multi-core CPU. They would probably play well together.

[@yorickmeulenbelt4all]

Redshirt Jeff holding the Turing Pi v1? Does blue or black shirt Jeff know about that?

[@spvillano]

I've mostly did application pool clustering, distributed compilation and some distributed transcoding over the years. When using disparate hardware, scheduling can get... Interesting. But, that's half of the fun!

[@ZaquMan]

Well I'm a little surprised that Jeff did not mention parallel computing, that's always the first thing I think of when the topic of Pi clusters comes up.

[@JonathanSwiftUK]

I think there is a community version of Hadoop you can run - clustering where you take a job and split it up to process parts of it in parallel to get the results faster. You could run clustered sql/database. If you have boards all running separate software it's not a cluster, just a convenient way of physically connecting up a group of Pi boards.

[@etiennebruines467]

I like the advice: treat your servers like cattle and not like pets

[@isaackvasager9957]

drink their milk and eat their dead bodies? if you say so.

[@voxelfusion9894]

@@isaackvasager9957 a little extra iron and copper never hurt a diet.

[@Rekhan4242]

Poke with a shocky stick?

[@aer185]

Hey I haven't finished the video yet, but I'm a computer engineer studying distributed systems, and a cheap Pi Cluster was the ideal way for me to implement some of my research without waiting for time on a commercial Beowulf Cluster. Ok I just finished the video and this is a great explanation for folks, love it.

[@davidg5898]

Excellent explanations. For myself, it's all about uptime of lightweight applications. The fun, and frustration, of the learning process is just icing on the cake. Have you seen Techno Tim's video on High Availability Pi-Hole? The people saying "that's what virtualization is for" obviously didn't pay attention to the section on power consumption. For my use case, virtualizing on a "real" computer would result in 2-5x the power consumption (and heat generation) with no performance benefit.

[@JeffGeerling]

I haven't seen that video, but I am planning on getting my Pi-hole setup to be a bit more robust later this year. I've had that Pi lock up twice since I switched my whole home to it, and it's annoying to have to reboot the thing once every couple months!

[@TheGamesZilla492]

always a reasons to do it,i myself learned on pi clusters, and now have a job working for a company maintaining clusters in a data centre.

[@WillMakesTV]

I would love to build a Pi cluster to experiment with Kubernetes. Thanks for another great video!

[@mustafabayzid]

I was thinking why people are making pie cluster and you already made a video about that!!! Thanks Jeff.

[@SirDragonClaw]

To be fair, you can run pretty much everything you are running + home assistant + hosting for multiple other services + plex + a bunch more dockers, all on a single i3 pc. So while I love the idea I still don't see much of a use for pi clusters. Maybe for a distributed ray tracing system, but then memory becomes a non trivial issue.

[@LiLBitsDK]

and that one PC dies and he still got 5+ pi's running while you are offline ;-)

[@maplenerd22]

He mentioned it in the video. Availability and up time. When one server dies, all the the other servers will still perform the task.

[@LiLBitsDK]

@@maplenerd22 and a new one can be spun up automatically as well for the one that crashed

[@Winnetou17]

To be fair, unless you're only actually using only 1-2 services at a time and not have an actual public website, then that i3 PC comparison is bullcrap. I mean, it cannot cope. Good luck watching a movie from plex when the backup service is running and you have several visits on the website. Though I have to agree, if you can somehow use past unused CPU cycles when needed, then an i3 running at 100% should be able to power most things effectively. Anyway, having a cluser also improves the bandwidth and IOPS that the storage and RAM effectively have per process. If there's a spike and 10 processes really do need to do stuff at the exact same time, a cluster of 4 Pi with modest RAM/storage would be better than one i3 with good (not best) RAM&storage.

[@Santor-]

Just because it "can" be done, doesn't necessarily make it the best option. Plenty of use cases where having stable functions working independently from others can easily be imagined. Gathering data from external IOT devices, assembled, posted onto an internal local website for monitoring, or for student matlab projects, redundant micro server, or each PI zero as a powerbank/fitbit clone, which can play music on you outing, and when back to recharge by imserting into cluster, heart rates, oxygen bloid lwvel etc can be uploaded for your own private health check, that bypasses the vendors of such systems that sell your data. I ran a professional CAD service some years ago, and while I mostly was the only person doing it all, save for hiring in people for some projects, I still had all the tasks spread out on many different machines. 1 was used for new software installs, testing, pushing the envelope etc, 2 other machines was "Stable" CAD workstations, with absolute minimum installed to avoid software breaking when new versions came along, then a specific machine for customer visits, to avoid crisis if theft, incompatabilities etc, some accounting only machine, with temote access for CPA, standard office (word/excel ect) machine, and finally redundant hot swappable file server with client backups, connected to remote storage ++. Run all that on 1 machine? Sure, i guess you could make it work. Would be slow and need lots of reinstalls and reboots though. And good luck if you broke the HD. While not a cluster, a cluster could been used for a set of thin clients instead, which was a technology a little above my cost range.

[@harrin1982]

I loved the included snippet with Linus from LTT!

[@grubbyga7601]

I use my Pi as a Wifi USB-Hub for all PC's. Its easy to install and runs allog with other things. Its called VirtualHere.

[@airy_co]

interesting

[@lucaspelegrino1]

What that means? You use it as a router?

[@tramcrazy]

What a great explainer Jeff! The production quality on your videos just keeps getting better too - I loved the graphics and b roll.

[@nothere3042]

This is the question I need answers to. Thank you for this. Can't wait

[@YeOldeTraveller]

I have a 4-node bramble of RPi 3s that I use for core services in my home network. I initially used this cluster to experiment with the overhead involved in spreading work over multiple nodes for different types of tasks. $250 for 4x 4-core nodes on a dedicated network switch seemed a good investment at the time. The performance was not as important as the node count for my use case. Having 4 identical systems with identical cores was a significant benefit. I could directly compare the performance of 1 core on each of 4 nodes working on a specific task to the performance of 4 cores on one node doing the same thing. Sometimes, the only way to understand how scaling works is to work with under-powered hardware. This allows you see what different types of bottlenecks look like as you ramp up the parallel threads. Professionally, I have found issues with software that was written on the fastest computers running on the fastest network that impacted real world installations in ways the developers could not see due to the power of their development environment.

[@nnm35]

Oh, I thought I was a subscriber, well now I am ;-).(you turn up all the time on my Roku un-logged-in feed), but hey, what a Tease, you're better than that, aren't you!? Will check back tomorrow!

[@raulcorreia232]

Loved the old reference to the Quake 3 sound, keep up the good content!

[@World_Theory]

Can't really think of anything useful I could do with a Pi cluster, unless you count something like Folding@Home. If you could utilize a Pi cluster to do computer vision tasks at a higher frame rate, with no latency cost, that would be cool. It could make some interesting things possible for cheap.

[@startiger2]

I used to work for a large aerospace company. Besides the servers, the company would occasionally need to do massive calculations for airflow or structural analysis of systems that were being developed. They built a system similar to SETI@home and installed it on all the company's office desktops. IT wanted employees to leave their computers on over night for update purposes anyway, so this made use of their processing power during idle times. This system would run the various calculations for the R&D work by splitting it across the hundred of computers on site in their idle time. It was a lot cheaper to use existing computers to build our own in-house "supercomputer" that to actually buy a supercompuer to do these calculations that were only needed a few times a month and due to various contract requirements, we were not allowed to do this work by outside groups.

[@FitForMyPurpose]

Just weeping at this cat that said “but can it run crysis?” 😂 Old memes are old.

[@zrodger2296]

I specced out and built a 32- and then 64- node Beowulf cluster in the late 90s / early 2000s at school (high performance computing research). (Think of 2 rows of shelves holding 64 desktop towers -- rackmount was too expensive.) We used MPICH for the communication. When solving a single problem (in our applications) speedup was never great across a fair number of nodes. But sometimes you needed more memory than a single node could handle, or just wanted to solve a problem fast, even if it was a little inefficient.

[@ny4i]

It appears you are more talking about having multiple worker systems versus a classic cluster for parallelism. For example, having a cluster where I share state between devices in an HA environment is difference than having 8 Pis in a pool of workers behind a web server load balancer. Your blade Pi "cluster" appears to be more a question of ease of form factor (which is great).

[@sugoruyo]

2:00 - I'd like to address this to avoid confusion: a bunch of Pis (similar computers) running different workloads (Prometheus, Docker, Web services) *is not* a cluster. Unless you're running all of those under something like Docker Swarm or K8s where you have a bunch of machines and software that decides what to run where. The hallmark of a cluster is not the heterogeneity or homogeneity of nodes or workloads but the fact that workload and nodes are managed in an automated way by specialised clustering middleware (K8s, Mesos, Beowulf etc.). What you're describing in that section is more like a server farm.

[@zambonidriver42]

I’m sorta lucky, I have access to racks of servers. I can spin up stuff quickly. But it’s at the office, tied to work. I really want that new Turing Pi Cluster 2 for my home. I have a half dozen containers I’d like to run. Seems like it’s taking forever. I put my email address in for updates, and ….nothing. I used to check their blog weekly, now it’s monthly. Edit: I should also add that I take servers out of production, and dump them into a sandbox, and I frequently let people (at the office) experiment. I own the routers, switches, I throw a subnets, VLANs, and a dhcp/dns server, PXE, http, tftp, I give them access and email them a couple of howtos, and let them have at it. I check back on them in every few days. I only ask that they publish what they learn and throw what they did on our company git site.

[@nodidog]

I'm also impatiently waiting for the Turing Pi 2, I've already bought 4x CM4s in preparation... I hope they don't just give up on it! It's been radio silence for months now

[@danmanx2]

Jeff, these are very good reasons to build RPi clusters. Thanks for the video!

[@adubs.]

I'm still hung up on price/performance of a pi cluster. I get the idea of it being fun, and its certainly neat, but I would challenge the need for comparing to epyc. I think even lower end Threadripper is probably overkill. I've got a little 3800x in my home server that I think would probably give you a run for your money... but theres no reason to think it would when we can just compare compile times. Pick the source code and lets benchmark and find out. What do you say?

[@chadcordero1618]

Pretty much sums up why I built my Pi cluster. I wanted to learn how to manage Kubernetes and Docker, though I've since replaced Docker with Containerd in my Pi cluster. Great little machines for testing.

[@Dygear]

Because it's ... AWESOME!

[@toddbeedy]

LOVE the outtakes Jeff!!!!

[@KienTran]

"Everything fails all the time" - Dr. Werner Vogels

[@JeffGeerling]

I still have an HBA from you on my test bench :D

[@gounthar]

Super nice job of explaining it all, Jeff, well done!

[@gerald4027]

All of my pies are a cluster in the fridge.

[@jaqque9633]

I assembled a Turing Pi cluster. It uses seven CM3 boards. It's a nice little board, but the limitations of the Pi 3 compute module means it can't run modern kubernetes (of any sort, including k3s and microk8s). Instead it runs Docker Swarm and I've had a "fun" time setting up GlusterFS to aggregate the available storage. Of course, except for the initial imaging, everything is set up via Ansible.

[@ShotgunKlaus]

that quake 3 hitsounds ...

[@RecklessRowly]

I too decided to build a Pi cluster during the Pandemic. It stopped me going crazy. I've used mine for: 1) trying to crack a hash algorithm we use at work to determine how safe it is. 2) Calculating magic squares. 3) Trying to get WRF (Weather Research and Forecasting) to run sensibly on it, but have not had luck yet, but still persevering. 4) Trying to network render Blender. The first 2 projects I wrote the code myself in C so learnt quite a bit on how to break a problem down into seperate managible tasks that each core can handle. I learnt how to Network Boot the Cluster and suddenly I had a whole load of redundant sd cards. The latest project I have had to use Kali Linux to run Blender 2.8.3 as the Raspberry Pi Os only supports Blender 2.79. Anyway this project is still on going, I've just managed to Network boot Kali. Still got quite a bit more to do. Many thanks to Jeff Geerling for his videos. They very well made and very informative. I often laugh at his outakes. I have made a couple of videos myself and know how difficult it is. I often get tongue tied. so I know how you feel.

[@JeffGeerling]

Weather stuff is so hard to get running well on _any_ cluster. I really want to go tour the local NWS offices someday and talk to them a bit about it. We don't have the compute resources here but the weather people have a lot of deep knowledge in specific forecasting areas!

[@artucuno_6491]

Hey Jeff, you should create a big Folding@Home Pi cluster. That would be awesome to see.

[@afriedli]

I'm sorry if this has been covered in a video I've missed, but the area of clustering I'd be most interested in concerns video processing. I'd imagine this is a good application because the comms bandwidth requirements probably aren't excessive, but the gains from having more grunt (CPUs & GPUs) could be significant. For example, if there was cluster support in something like FFMPEG, that would be great because, say, converting from one video format another can be painfully slow even on a high end machine! It would be even better if I could build a Pi cluster, attach it to my Mac via LAN or USB and render videos faster in iMovie and other Apple video apps by 'farming out' video processing to the cluster. Anyone who cracked this would make a lot of Apple users very happy!

[@CheapHomeTech]

Interesting topic. Considering virtualization to save on money, energy, and arguably complexity has been popular for the last 30 years. So yeah, Pi clusters in most scenarios are not optimal...

[@siegfriedkettlitz6529]

...but in a few scenarios they are the cheapest solution to get a stack of real processors.

[@darkfire2703]

You can get 5 RPi4Bs with 2GB Ram for 200€ + a few bucks for other accessories and a power supply with enough powerto run all of them together. This makes it less expensive and significantly more power efficient than any other option on the market. You get 20 Cores and 10GB Ram in total for this. Low power NAS options are less powerful and are much more expensive. And if you try to build something yourself it will get even more expensive if you want comparable performance and efficiency. So if you want to run something that can be run on a pi cluster, this is actually not a bad option

[@autohmae]

Funny fact is: I was running Linux (Linux-vserver) containers in production before VMs were popular, most people had no idea what was possible.

[@rybec]

Something critical you missed is collective memory bandwidth. If you have several Pis, even if they are running different software, they may be able to perform better than a single higher power machine, for applications that read and write memory frequently, because the applications aren't constantly competing for the memory bus. (Note that this isn't about how much memory is being used. It's about how much _traffic_ the memory is seeing. So, for example, a web server might not need to use much memory, but if web site traffic is high, it's going to have a lot of memory traffic, especially if it is serving a lot of files.) And it's also worth noting that you can't just directly compare collective memory bandwidth directly against a single system. More applications on a single system also means more memory traffic for the OS, for context switches between applications and process scheduling. That means the single system running lots of applications has more memory bandwidth _overhead_ than a handful of Pis, each running only one or two of the applications. (Sadly, this makes it hard to quantify the benefits of using the cluster...) I've used a Raspberry Pi cluster for prime number finding. I didn't find any million dollar primes, but I wasn't actually trying to. Perhaps eventually though! I've got a few other distributed computing uses for Pi clusters as well. In most cases, the performance benefits aren't great, but there are some where the memory bandwidth thing will likely make a pretty significant difference.

[@alliejr]

Back in the day, we called it “scale OUT” vs “scale UP”.

[@Tea-Spin]

I think we still call them that, maybe Jeff just doesn't want to confuse more people that aren't aware of terms like vertical and horizontal scaling

[@stalevolsen]

Loving the quake hitsounds at 6 sec in :D

[@LazyBunnyKiera]

I'd be happy with a single pi4 for Home Assistant. I just can't justify dropping the cash on even a single pi4 when i have enough PC parts to build an old but functional x86/x64 pc.

[@jasnix]

Clustering is something that I have had to scratch the surface of for my job, and the appeal of rolling basic cots hardware (pi, nucs) via a single unified and automated work flow is very attractive. especially as I have less time then ever to manage snowflake vm/server. Other times I find the containers may not have the needed functionality (router/switch feature testing makes for weird needs beyond basic x task)

[@johncnorris]

I always had trouble getting the people who controlled finance to understand why we needed reliable stand-alone development equipment to use as test and experiment devices. I usually scraped together used equipment and at times transferred it to act as temporary production devices to solve problems we needed at the time. Knowing how to setup and manage a cluster is no small effort as well. I think the people who resist these projects are either too cheap to understand why they are valuable or just not very curious.

[@Cynyr]

or IT is a hobby for us and even $3000 isn't going to be approved by the budget troll. I completely agree if you are supporting a production environment a test environment is very useful, as is a place to practice managing a cluster.

[@mousermind]

The fact that a Pi Cluster isn't called a Pi Shop disappoints me.

[@jinmong5417]

Why "shop" ? you don’t buy anything at it

[@TheWebstaff]

I use my cluster of pi's as a dust collection system.

[@JeffGeerling]

Kill two birds with one stone!

[@TheWebstaff]

@@JeffGeerling sorry misspelt that's.. Should have been clutter.. :)

[@TapeOperator]

I tuned in about a month ago, when the algo decided to feature you, chuck and a guy named chris. Just finished putting my first cluster (4 nodes, Raspberry Pi) together today.

[@TapeOperator]

Oh, yeah. I want to set up NextCloud or OpenStack and a CI/CD pipeline.

[@JeffGeerling]

Nice! Congrats and I hope you learn a ton!

[@TapeOperator]

@@JeffGeerling - Me too! Pi clusters contain the answers that I didn't have in some of the interviews I did last year. I hope that the job I got lasts until I tire of it, but in the meantime, the cluster gives me plenty (and motivation) to learn.

[@Miafunfactory]

Becuz it looks cyberpunk AF

[@foca2002]

Being a sysadmin for the last 10 years I built many clusters. Running Windows, Linux and BSD's, everything you said you could learn using vms. But you wouldn't have the fun of messing with it. So because is fun is a good enough answer.

[@janklas7079]

BAsically when you describe what your cluster is doing, you're proving clustering them didn't bring you anything. Those pi's would have done those task faster and more efficiently outside of a cluster.

[@SeeJayPlayGames]

Maybe not in terms of performance, but clustering at least brought redundancy in case one failed. You just proved you weren't paying attention.

[@janklas7079]

@@SeeJayPlayGames You can have failover without a cluster. Clusters are NOT used to bring redundancy. You don't know your stuff at all.

[@SeeJayPlayGames]

@@janklas7079 I never claimed I did. I was just saying that you missed that point in his video. So you're saying he's wrong? OK, whatever. I don't know anything about clustering; I was trying to learn. That's why I clicked on this video. But to say I don't know my stuff *at all* is a bit generalized and quite presumptuous on your part. You come off like a total jerk.

[@lucaspelegrino1]

He did give a valid example of loadbalancing. How would you do a failover without the cluster? And whats the pros/cons?

[@janklas7079]

@@lucaspelegrino1 The same way; with a load balancer. or in simpler fashion DNS round-robin. Options are plentyfull. The HUGE pro is the missing overhead of a cluster. Also if for some reason your cluster breaks, everything is broken, so redundancy is a moot point.

[@TimothyCizadlo]

The shirt was a good one for clusters. The Space Shuttle ran on a set of 5 computers (IBM AP-101s, which were in the System 4/pi family!) with four of them running the primary flight software in a voting cluster and the fifth running independently coded (different team from a different company) backup flight software.

[@stevepope6084]

O well, there goes my idea of playing cyber punk at 8k at 1000fps...

[@terrygilligan790]

Very nice video. It inspired me to build my own cluster. With the help of the your other videos. Thanks again.

[@graealex]

Yes, Pi Clusters are dumb and there are zero use cases for it - at least none that wouldn't be better suited to virtualization. But I am really interested in seeing your ideas on the topic, because up until now, every time you published a Pi cluster video, I just rolled my eyes and skipped the video entirely.

[@Lorthiz]

There are a lot of use cases. If you are a developer working on your own projects you can have your own development/integration/production enviroments. Going that route you may need to install more raspberry pi's to host frontend/backend(or just different app versions), database. backup tools or even your own CI/CD server/app. Managing everything using a docker swarm/kubernetes is a lot easier, it can be configured to start automaticaly after power outrages and with that you are gaining a lot of knowledge about maintaning your own personal "cloud" server. Ofc it's not as cheap o/effective/easy as just renting a cloud server, but being a 'maker' was never about costs was it? :) And even after all that let's say you want to add pi hole, some IoT devices, RAIDs and a few other apps if you run out of resources you can just connect another RPI to already existing cluster and you are ready to go. At least that is how I use it and I am pretty happy with the results I have.

[@graealex]

@@Lorthiz As I said, not a single use case that wouldn't be better suited to virtualization. I like how you are talking about Docker, neglecting the fact that ARM-support is still very limited there. In general, any serious development is going to be x64, and might also involve some GPU offloading, an option that doesn't really exist with Pis, no matter how many you bring to the party.

[@Lorthiz]

​@@graealex I did not neglect anything. I pointed out that just renting a server would be cheaper more effective and easier to set up. The question is what you need and if you don't mind some more googling and building your own images sometimes. You are talking about it as it is on/off switch - good/bad - zero use cases/all the use cases while it all depends. For personal use, collecting some in house data, experimenting with pesonal small projects(you can just basicaly skip implementing security if it's in your house isolated network) I would choose it always over paid services. If you need to do a lot of computing(GPU offlading), handle a lot more load in general, build another facebook/google/netflix or just scaling up projects you already have then that is a moment I would consider a more powerfull equipment/paid services. The point stands that just because you can't find a use case does not mean there are none.

[@graealex]

@@Lorthiz There simply aren't any use cases. And I am not sure why you are talking about renting a server? The PC you'd use to manage and develop for the Pis would be the right machine - it probably has a CPU with 8+ cores, a reasonable amount of RAM, and some sort of GPU. Install Linux, virtualize how many machines you need or want, create container compositions, basically do whatever you might do with a Pi cluster, just with far more flexibility, and the possibility to start and stop many dissimilar setups/scenarios with just a single command. There is a very reasonable use case for Pis, and that is edge computing - whenever you need far more computing power than what a microcontroller could provide, yet with similar form factor and power constraints. A job server for your 3D printer, a signage terminal, a network gateway, wireless robotics, camera AI applications, the like. But that is the exact opposite of a cluster, where you centralize multiple devices in one place. And if ARM is a requirement, there are a few far more powerful alternatives out there. The M1 would be the low hanging fruit, but there are some boards and even workstations with powerful ARM processors (LX2160A, 8180, A8040) out there. Still the better choice to put a lot of processing power into a single place.

[@Lorthiz]

​ @Alexander Gräf >The PC you'd use to manage and develop for the Pis < I thought that point would be to not add ANY additional load to my personal pc and make it run 24/7 especialy if I am a windows(yeah because it has better docker support than RPI :) ) user with applications I may or may not need in the next day/days. While I don't care for additional few wats from a pi cluster. It's small. it's silent and at any point I can deploy another small apps to it and I know they will be working good enought for my needs. And as I pointed out in my first comment: I have and use it as a personal development server/playground. That makes it a 'valid use case' in my opinion. It may be inefficient, it may not be cheaper but it serves a purpouse. In the end when I am done with a cluster I can still move RPI's to other tasks you mentioned without having another boards/workstations or apple bricks just lying around gathering dust. But let's just assume you are right. There are no use cases, RPI clusters are dumb and I would rather just do something else than to try to convince you otherwise. Thanks for a quick chat and have a nice day. :)

[@federicogalloni8665]

I'd like to build a Pi Cluster to transcode videos from h264 to h265 codec, with Handbrake or other softwares. I know that the computing power of Raspberries is lower than an average CPU, but why not? It would be really fun and also useful to see a video about this. Thanks for your contents Jeff. Amazing job! Saluti dall'Italia! 🇮🇹💻

[@spiff-FPV]

I would totally get in on this.