Raspberry Pi Supercomputer for Quants | How to build a Raspberry Pi Cluster | SLURM Cluster Config

[@PallidWhite]

Are you blurring out private IP space on your home network?

[@Determinator21]

People who know cyber security are really paranoid

[@ftate]

Great information. Thanks a lot for the step-by-step runthrough.

[@statyoutube]

Hi, thanks for the nice tutorial, I am a novice and trying to learn about raspberry py configurations. A few things I would like to know. When you set up the IP addresses, the tpt link webpage you accessed on your Windows computer, or after installing raspberry os on all the systems. Second at what stage did you connect the SLURM device.

[@erenyeager4452]

just stumbled on your channel. subbed (duh!) could you do some sort of a ranking of Quant programs. MFE etc

[@ayoubrayanemesbah8845]

this person is the reason why we can't find Pi's (just joking)

[@stupidanimal5739]

Good fun project. I like your definition of "supercomputer" xD. For anything serious, I'd look at doing the same with proper HPC cloud instances though.

[@kayakMike1000]

That shit costs money, pi clones are cheap. You can get rock64 nodes for 20 bucks a piece.

[@josephciriza7783]

Could the master node also be used as a compute node? Like when creating the partitions, I want to include my master node in the workload.

[@nod32ia]

Thanks for another great video! If we were about to scale it down to like one Raspberry Pi to play around with as a dedicated quant simulator/server, would you consider that a worth the time and effort compared to just using your regular computer? I already own a Raspberry Pi but haven't really used it properly until now, but I would consider myself a fairly experienced Linux user.

[@QuantPy]

Nice one, sure great to learn how to install slurm on Linux servers in any case. In my opinion the best use case for raspberry pi’s in general is for learning. Learning how to deploy software, websites, algorithms ect.

[@scuzzyhoulihan9853]

whats the difference between getting a high end processor on your pc?

[@joaobp7ify]

Great! Thanks for sharing

[@rooster9517]

Firstly, I love your content as always... and I can see you are having fun doing this. Just curious as to the cost effectiveness of Raspberry Pi. Passmark CPU benchmark site indicates what you have built is about 30% of the processing power of the low-end AMD Ryzen Mini PCs selling for around $450.

[@QuantPy]

Thanks Frank! Yes I agree it of course depends on scale, the raspberry pi is ~$90 AUD for quad core 4GB ram. Whereas the $700 AUD AMD Ryzen 5500 processor with 6 cores and 2.1-4GHz capability will outrun the raspberry pi’s everyday of the week. However, the cluster is scalable with raspberry pi’s in a way that if you are completing many many scenarios that don’t take immense computing power, from my research distributed low compute low cost nodes (like the raspberry pi) is the cost effective way to go. I agree with you though, this build here with only 3 compute nodes is border line from one of those entry level mini computer’s capability. Hopefully we can scale up and add more nodes as the channel gets bigger!

[@rooster9517]

@@QuantPy I meant your entire cluster is a fraction of processing power. Cortex A72 listed as CPU Mark of 686. The $450 mini PC has the Ryzen 4300U with a listed CPU Mark of 7635. You would need 11 nodes to match the processing power of the $450 mini pc. PS: I am considering running algos on a dedicated Linux server, hence my curiosity.

[@QuantPy]

Now I’m with you. Sure, have you tested running the algorithms on a Linux server already? You could spin up some instances with varying resources to see what works best for you, before buying any physical gear. The AMD Ryzen Mini PCs do look like good value though 👍

[@sjwbond2]

@@rooster9517 I dont think you can compare like for like in this way, passmark is not really broadly representative, and its really going to depend on how you write your code. heavily optimised c for the specific arm instruction set on certain programs could potentially run faster than code just compiled O1 on x86. I think its horses for courses.... if you're doing generalist stuff, or interpreted ( like python) you're probably much better off just sticking with x86. you also need to consider the performance per watt though, that could change things considerably if you live somewhere where power is not cheap...

[@MrEo89]

Frankly it would be easier to just set up a cloud instance (aws ec2, Google/azure xyz) or at least, if the goal is computer per watt, then going with a small box w/ 2-3 linked nvidia cards with CUDA cores would be a good idea. Granted idk how much they would cost given the current chip inflation.

[@ravakanyaina9650]

can we create it with raspberry pi 3

[@motocrosser784]

or just right click in the boot drive, create new txt doc and name it ssh. done.....and why are you taking the time to blur an internal IP....

[@oa_math]

I really wish it went this smooth for me.

[@QuantPy]

Did you build one for yourself?

[@oa_math]

@@QuantPy Yes, I have 4 pi's and I followed the exact same instructions. I had issues with directory name conflicts in the slurm configuration file. This manual does not tell yo u how to set a fixed IP address for your raspberry pi's which you absolutely have to do. And on the final check at the end of the Part I tutorial, slurm fails. Ugh, idk if slurm is messing with their repos but my goodness the inconsistency has made me exhausted.

[@QuantPy]

Yes, that sounds annoying. Can you assign fixed IP addresses from your home routers website interface?

[@oa_math]

@@QuantPy I mean that really wasn't too hard, I did that no problem. I expanded the subnet in my router and fixed each pi's IP address manually. It's just with slurm mostly. Slurm starts on everything but everytime I try to run the last line of code to test the whole system, it tells me that the state of each node is *UNKNOWN (with the asterisk which apparently means that it cannot currently connect to the node)

[@nil0bject]

@@oa_math firewall issue?

[@screaminggoat4224]

My guy really be blurring private IP Adresses huh. Otherwise good video though

[@nil0bject]

you 're suppose to use the little acrylic tab to peel the paper off, not your nails.....

[@saurodeep09]

Followed your tutorial but facing some problems.. I would like to receive some help.. Can you share your mail id..

[@anispinner]

I think we should stop calling clusters a supercomputer. Any budget laptop will outperform this thing tenfolds yet you don't call them supercomputers.

[@123flowey_yt]

Because a cluster is a supercomputer oldest supercomputers are worse than fake iPhones also pi clusters have 32 cores sounds pretty hpc to me

[@colombelle]

lame