Best AI/ML/DL Rig For 2024 - Most Compute For Your Money!

[@MJOLNIR242]

I actually just purchased a Dell Precision 7820 workstation off eBay for $300. It came with 2 xeon silver 4114's, 32 gb ram, a radeon wx pro 2100 and 500 gb ssd. I upgraded the CPU's to 2x Xeon Gold 6138 (20 Cores ea) for total of 40 core 80 threads. Cost me $70 for those two and its a workhorse. Its practically a server in a workstation that could easily run AI/ML tasks all for under $500. Also, I think your E5-2670v2 is a 10 core cpu so it would be 20 cores total, not 40.

[@TheDataDaddi]

Awesome! That is gonna be a great setup for you. Yes, I think I misspoke in the video. I probably meant 20 logical cores not physical. Sorry for that.

[@tyswenson7542]

Dude, stellar video. I love the pragmatic approach to budget deep learning. Please make more videos.

[@TheDataDaddi]

Really appreciate the feed back! Thanks so much for your kind words and glad this video helped you!

[@dustintunget4177]

This is still a solid video 5 months later. I ended up with a 4U ATX case and one of the epyc/supermicro combos from ebay, then 2xP40 + a 3060 12gb. Upside is that a 4U ATX rack case can use 120mm fans that can spin slower, and quieter...and you can fit more GPUs, and use standard PSU's. I'd like to see this kind of breakdown in the context of a EATX case (rack or desktop)...those results could be interesting. I think mine was in the realm of $1700...but that's not with much storage....I bought it over time so I don't have solid numbers. Great video, earned a sub!

[@TheDataDaddi]

Hi there. Thank you so much for the comment and the sub! Really really appreciate it! Also, really appreciate you sharing your build. It is always great to see how other people are doing things. Definitely sounds like a great setup with the 4U ATX case, Epyc/Supermicro combo, and those GPUs. The 4U ATX rack case with 120mm fans definitely helps with keeping things quieter and cooler, which is a big plus. And the ability to fit more GPUs and use standard PSUs adds a lot of flexibility. It would indeed be interesting to see a breakdown like this for an EATX case, whether in a rack or desktop configuration. The potential for enhanced airflow and additional expansion options could make a significant difference, especially for those looking to maximize their build's capabilities. Maybe one of the these days I can do a build like this! I think it would be super interesting. Your total cost of around $1700 seems quite reasonable for such a robust setup, even without much storage. It's great to hear you managed to spread the cost over time, making it more manageable. Glad you enjoyed the video, and thanks for subscribing! If you have any more insights or updates on your build, feel free to share. It's always great to hear about different setups and configurations.

[@TrueHelpTV]

id love a parts breakdown, such as what cable/psu you're using to power two p40

[@silence5623]

Hi, Thank you for the great video! I am just getting into machine learning/AI and learning on the way. I would love to see a video of how you put all this hardware together and also how you would run LLMs on it. Great job bud!!!

[@TheDataDaddi]

Hi there. Thank you so much for the positive feedback! Really glad this video was helpful for you! I think you are the second person to ask for a full video on putting all the hardware together so I can certainly do one on that. There has also been a lot of interest recently on how to run LLMs so I will try to make a video in the near future on how to set that up a machine just like this. Thanks again for the feedback. Really appreciate it! For now though, I have several videos on putting the GPUs in the server and I also have a series on server setup for home lab use. Here is the link for both below: Dual P40 GPUs install kzbin.info/www/bejne/g2HGZnxpd7R3eq8 Home Lab Setup Series kzbin.info/www/bejne/eaiYq5RvedmUjNE

[@suryateja3640]

Hi, Thanks for the great video. If possible can you make a video for $2000 or $3000 budget for AI deep learning especially LLMs inference and fine tuning. Keep up the good work.

[@TheDataDaddi]

Hi there. Thanks so much for your feedback. Yep, this is top on my list. I am going to try to make some video addressing this in the next month or so. Just keep an eye on my channel and you should see a video on this soon. Thanks again for the kind words!

[@tsclly2377]

Thanks!

[@TheDataDaddi]

Hi there! Thank you so much. This is amazing. Really really appreciate your generosity. Can't tell you how much this helps the channel.

[@ericgolub8589]

Also I'd love if you made a best performing per dollar CPU list in the future, would definitely watch that

[@TheDataDaddi]

This is an interesting idea. I will looking into making a video on this as well! Thanks for the comment.

[@wasifmasood969]

Hey, great video, you really boost my confidence in P40. One quick question though, this dell server does not have PCIe 4.0 x16 slots should you choose may another HP or Dell server for that? and if so what would you recommend to use?🙂

[@TheDataDaddi]

Hey there. Thanks so much for the comment! Yeah unfortunately all servers I have and work with have pcie 3.0 slots. If you want pcie 4.0, you are likely going to have to buy a much newer (and more expensive server). I just looked it up actually, and it seems that pcie 4.0 in servers has really only been a thing since 2019 ish with AMD's EPYC 7002 series. For Dell servers I know that the PowerEdge R750 and R750xa, have pcie 4.0 slots. I am not 100% which others might. I would say though its a safe bet you will really only be looking at the latest generation of servers from each major manufacturer that comes will pcie 4.0. Hope this helps!

[@TazzSmk]

for AI/ML performance, Tensor cores really help speed things up, so recent nVidia cards are much faster, that said, speed is less concern than sufficient amount of VRAM, so 24GB Pascal P40 can yield better value than let's say 11GB GTX1080Ti from similar age

[@TheDataDaddi]

Hi there. Thanks so much for the comment. Yep, these are my thoughts exactly. Tensor cores are certainly wonderful to have, but if you don't have enough VRAM to load the model then the number of tensor cores you have is irrelevant.

[@EcomGraduates]

I just acquired an r720 this video is perfect! Thank you so much

[@TheDataDaddi]

So glad this video helped you!

[@EcomGraduates]

@@TheDataDaddimy question mainly after doing some research the p40s require a cooling fan to be added but the server itself seems to have great cooling. Is it necessary to add cooling?

[@TheDataDaddi]

I have found it is not. Checkout my video on this topic. It is for the P100s instead of the P40s, but on my testing it actually does a better job of cooling the p40s rather than the p100s. kzbin.info/www/bejne/iIa6ZHSvatemebs @@EcomGraduates

[@Demoxx1]

When you calculate the power costs. How many hours per day is it? Thanks

[@TheDataDaddi]

Hi there. Thanks so much for you question! In the video, I calculated the power costs based on my typical usage over a month, which includes all 24 hours of each day. This calculation takes into account both idle times and periods when the server is under significant load. However, I did not specifically analyze the power costs for situations where the server is either at full load or completely idle. Instead, the power consumption reflects a mix of both usage scenarios.

[@min-yishen9324]

Ahh, nice content. I am trying to do similar thing with a HP Z workstation and P40s for under... 500

[@TheDataDaddi]

Hi there. So glad this helped you! Yep, I think this will be a great way to go to get a good amount of compute and stay under your $500 target.

[@erockvaughn2190]

I did this as well. I like the Hp Z840 becasue I can get 512mb ram and a dual XEON for under 600 USD on Ebay.

[@TheDataDaddi]

Yep, the HPs are good options as well. I just never found as good of deals on them when I was looking for whatever reason.@@erockvaughn2190

[@lanceharvie6933]

Prices are way different now Tesla P40 is now $479.95 on eBay and the server 3 times the price if you can get this spec.

[@TheDataDaddi]

Yep, you are absolutely right. I am working right now on making a website that will keep all this going in real time to fix this issue. Stay tuned for that!

[@lanceharvie6933]

@@TheDataDaddi looking forward to this since I want to build a server for AI on the cheap.

[@TheDataDaddi]

@@lanceharvie6933 I will make a video here on it, and post it to the channel when I can finally get it put together!

[@lanceharvie6933]

@@TheDataDaddiawesome

[@cchow]

😢

[@erockvaughn2190]

Great job on this. I was looking to buy anew rig and it was upwaards of 10 grand. I am going to go off your suggestion because I think it is all I need. Plus if I need more I can just buy one more rig like this. You are amazing. Thanks for the research and work put into this.

[@TheDataDaddi]

Hi there. Thank you so much for the kind words! Those were my thoughts almost exact when I decided to go this route. I am so glad that this video was helpful to you. Cheers and good luck on your journey!

[@AndyDavis007]

Speaking of, "... buy one more rig like this...," is it worth looking at what network switch to consider best performance to connect multiple BIY rigs for DIY AI/ML/DL as your lab grows?

[@erockvaughn2190]

@@AndyDavis007 Well I guess if you were going to do this. most Ten-gigabit network switche would be enough.

[@TheDataDaddi]

@@AndyDavis007 I actually just made a video on 10G networking (link below). It might help you make a decision on if you really need it or not. I would say if you plan on growing your home lab beyond a couple of servers it is definitely something to consider. My advice here is look on EBAY for some used enterprise grade switches. They will be cheap and allow you to scale. For example, I am using a Juniper EX3300 POE+ in my home lab with 4 SFP+ ports. I bought it for $72 dollars on EBAY. I have 4 servers at the moment and it handle everything with ease. kzbin.info/www/bejne/jZO4g3aIqteWbtU

[@RimlaAri]

Just a question, do you guys make money with this or is it just a tech hobby or for studying for college students?

[@TheDataDaddi]

Hi there. Thanks for the question! I do not make any money with this currently other than what I get from KZbin ADs. Really just do it because I love it at this point, but it helps a ton with my PhD research and fuels my pet projects at this point. Eventually, I would like to start formally consulting, but I need to get a business setup first.

[@Keeeeeeeeeeev]

I would recommend arctic p12max insteat of lian li fan They cost ~10 -12 €...much cheaper really good performance

[@TheDataDaddi]

Hi there. Thanks so much for the comment! Okay. I will definitely check those out for the next build. Thanks so much for the recommendation! Always great to find good cheaper alternatives.

[@乾淨核能]

do you get 48 GB VRAM when using 2X P40? thank you

[@TheDataDaddi]

Hi there. This a great question. Unfortunately, when you use 2X P40 the memory pool is not shared so you just have 2 separate 24GB memory pools. This is still useful because it allows you to parallelize data processing across multiple GPUs. You can also split part of a particular model across GPUs and process data sequentially across GPUs. However, you would not be able to load say a 40GB model directly onto the GPUs.

[@乾淨核能]

​@@TheDataDaddithank you

[@reinsein518]

see that no where to buy for ur prices

[@TheDataDaddi]

Hi there. Thanks so much for the comment. Unfortunately, since I made the video GPU prices and hardware prices in general have increased significantly. Also, I normally buy the majority of my components from EBAY so sometimes once the deal is gone prices may be more expensive. If you are patient on EBAY you may find some deals that are close to what I found.

[@wasifmasood969]

Hi, I am facing a different challenge now. I have this machine at home and it creates a lot of noise. I have checked the fan setting, even on the "Optimal" mode it is loud but when I start some training, it goes really unbearable. I have checked various fan configurations, it unfortunately has none stating "Quite mode". What else can I do to make it some reasonable to sit next to, are there any high quality fans which can make a difference. Many thanks in advance?

[@TheDataDaddi]

Hi there. So I have a video actually related to this. kzbin.info/www/bejne/iIa6ZHSvatemebs Check this out. In the video, I should how to adjust the fans manually. You could put yours manually on the lowest setting and you may be able to turn them off completely. However, please be carefully when doing this. It is never good to run your GPUs or the rest of you system too hot for too long. Another alternative might be to get some snail blower fans and install them that make less noise then lower the naive fans to the lowest setting. Something like the following may work for you. www.ebay.com/itm/186459922797?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=2ucjrgqosnm&sssrc=2047675&ssuid=xv3fo9_stiq&widget_ver=artemis&media=COPY

[@noodlz3660]

Thanks! Can this same build work with 2x 3090’s instead of the P40’s?

[@TheDataDaddi]

This is a great question. I am actually about to try this very soon. I am planning on using an external rig though because many of the 3090 form factors will not fit in the r720. I am planning to use 2 Zotacs, and I know those will not fit. The founders edition GPUs might fit. However, I am really not sure about that though. As far as if the r720 will recognize them. I am thinking it should, but I really wont know for sure until I give it a try. Hopefully soon I can tell you more exactly. Been on the look out for a video on this soon!

[@dev9619]

Excellent Videos!! I was looking at the P40 and then saw your video covering the best GPU price to performance video. It gave me the confidence I needed to pull the trigger on getting them for my companies project. We built a proxmox server with 2 P40s that are passed through to 2 terminal servers VMs that will support about 30 users! In LM studio the P40 is hard to beat for the price. I was getting 42 tokens per second!! Only thing I would change about this build is the cpu. The Xeon E5-xxxxV2 CPU family only support AVX. A lot of stuff like LM studio require AVX2 support. The E5-xxxxV3 line does support it. It’s a newer socket so it will also require a new MB. Again great video and thank you!!

[@TheDataDaddi]

Hi there. Thanks so much for the feedback! So glad this video was able to help you! Yeah, might be better to go with a 730 or something with the correct mobo then. To be honest with you, I just bought that particular server because I got a great deal on it.

[@mateuslima788]

Thank you for the video!

[@TheDataDaddi]

Hi there! Of course. So glad that you enjoyed the content!

[@ahsanmasood1459]

By the way, that 3060 variant has 13 TFLOPS (12.95 to be precise).

[@TheDataDaddi]

You are correct. Thank you for the correction. I think I accidentally copied in the wrong number. I even thought to myself "seems odd the performance is lower with this newer GPU." Turns that was odd because it was wrong. lol. Anyway, thanks again!

[@scosee2u]

It would be cool if a program could make it easier to fix a mistake in a video using llm prompting and image generation

[@TheDataDaddi]

That would be awesome for sure. Certainly save me a lot of time. Lol@@scosee2u

[@cdrbvgewvplxsghjuytunurqwfgxvc]

Great research!

[@TheDataDaddi]

Thanks so much for your feedback!

[@SomeStuffShin]

did you use Clore ai servers for rent?

[@TheDataDaddi]

I have not. I actually just heard about this project from your comment. I love the idea. This is something I knew had to exist. In my opinion, there is a natural opportunity for P2P hardware renting via blockchain technology especially as hardware prices and the cost of the cloud keeps increasing. Nice to see this already exists. Thanks so much for the comment! I will be doing a deep dive into this soon.

[@rlarose134]

Seems the P40 is double the power consumption of newer GPU, would it make more sense to get a setup with an overall lower TDP? Like a RTX 1660 super maybe? I'm a complete beginner herr however willing to learn.

[@rlarose134]

NM, just found your GPU guide video. great thanks btw!

[@TheDataDaddi]

Of course! So glad it was useful for you!@@rlarose134

[@InstaKane]

Can you run a mix of the P40 say with 3090’s for training and inference? Would I run into compatibility issues?

[@TheDataDaddi]

Sure you can run a mix. Nothing will break. However, you may run into weird batch issues due to the performance differences between the 2 GPUs. I actually doubt this would happen though. Try it and let me know how it goes. Lol

[@InstaKane]

@@TheDataDaddi yes do, this would be very informative for my next purchase. I just started buying up hardware for my home AI server and am looking at starting with one 3090 and M40 GPUs. Cheers 👍

[@MatiasHegoburu]

This is exactly what i am trying to build, except i have no idea how :D thanks! any way you could give me an idea how many tokens per second you would get running something like ollama and llama 3.1 7b for inference? Also, this is dumb but, what OS do you need for this? can ubuntu server run on it?

[@TheDataDaddi]

Hi there. Thanks so much for the comment! Hopefully this video helped you with that! I also have other videos that explain how to setup a server if you need help in that area as well. As far as tokens per second goes I would estimate somewhere around 12. I have not actually tried that exact combination on that machine and GPU pairing so I cannot say for sure, but 12 is a probably a good ball park estimate. You can really pick any OS you like. I prefer working with Linux and Ubuntu is the most standard of the distros with the largest community around it so I normally stick with that. However, there are many others out there I really like. You could also go the Windows server route as well. I believe there is also a macOS for servers, however, I have never used it. Hope this helps!

I would like to see what performance you are getting out of the P40s, especially in Stable Diffusion benchmarks, if you have them, otherwise in AI tasks where I can compare them to consumer cards. I'm really interested in the cards but performance looks bad compared to cheap consumer cards.

[@TheDataDaddi]

Hi there. Thanks so much for the feedback! I have been wanting to make a video on this exact topic for awhile now. I just have not had the time. I will be making one in the next couple of weeks. Also, out of curiosity, which consumer cards are you comparing with?

@@TheDataDaddi It will be great to see that video. For comparison, I'd look at cards that people could get on the second hand market for no more than $300. Something like the RTX 2080 super is a relatively inexpensive card to get started with Stable Diffusion. People seem to get very different results between Windows and Linux, particularly with AMD cards that have better drivers on Linux. I'm a Linux guy, so it suits me that Linux seems to get better results.

[@TheDataDaddi]

I am also a linux guy as well. I run the large majority of all my projects in ubuntu 22.04. I also like the RTX 3060 for that price range, and I already have one so I could start there. Thanks again for the feedback. Its really appreciated!

[@ericgolub8589]

I cannot find a E5-2670v2 online with 20 cores, they all have 10

[@TheDataDaddi]

I think I meant 20 logical cores not physical because I am not able to find one either!

[@tsclly2377]

hey.. what do you think about the AMD Radeon Instinct Mi line of GPUs???? I'm starting out with P40 also, but as you have said in your last video, one has to know or search out the programs (versions) that don't use FP16 for the P40, or convert all those code lines and operations to FP32 (and hope for the best!!?!!) I have an aversion for TLC NVME and always look at the Petabyte write levels that the NVME give.. settling on using the S35XX-P37XX X4-X8 (and Optane) PCIe for the GPU dumps and fast loads Me going, ML350 G9 (still PCIe 3). my SSDs are slower, SLCs Raid5 and adding Infiband40Gb for the second machine (the winter heaters) .. I estimate my cost is three times of yours, because I bought the units two-three years ago with 128GB memory (installed).

[@TheDataDaddi]

Hi there. Thanks so much for the comment! A lot to unpack here. 1) I want to start by saying from my experience my DL models tend to use FP32 by default. Maybe with some of the LLMs this is not the case. I am honestly not sure here. Your point though still stands the P40 is generally bad at FP16 or Mixed Precision (MP) operations, or put another way, you do not get any speed up from using FP16 or MP with this GPU. Check out my video here for the actual numbers: kzbin.info/www/bejne/hXTbY6uoar2UoJo This point I am trying to make here is with the P40 while you don't get any speed up for FP16 or MP you do not lose any either. So it is not nearly as bad as the theoretical specs indicate. Also, for many programs you would not need to search out specific version that are FP32 compatible it would actually be the opposite. You would need to find version that use FP16 or MP by default. Additionally, even if the program was set to FP16 or MP by default the P40 (based on my testing) would run it just as as well as if it was FP32. 2) If you are going to go the AMD route, the MI series is probably the best way to go. AMD GPUs in general offer a ton of compute for the money and look really attractive on paper. However, based on the feed back I have gotten from viewer that have gone this route it is a pain in the ass to get things set up correctly. I will paste below the exact conversation I had with a viewer recently. """ > What are your thoughts on AMD GPUs in general? There is so much to say here I don't even know where to start. AMD is not worth it if you value your time, but once it's working it is fairly decent and a good alternative to Nvidia. I built my PC around a year ago and back then I would not have recommended it due to the lack of online resources. But since then, backends and troubleshooting support have gotten significantly better. The hardware is great! But it's the software, optimization and lack of general AMD support that make it incredibly time consuming to deal with. The documentation for Instincts is so poor that I had to use Nvidia's P100 flowrate recommendations to get even an idea on how to build my system haha. MLC LLM is a promising project that is optimizing LLM inference though, and it is looking to be an alternative to current CUDA wrappers. > How buggy is ROCm? Is it still bad as of now? ROCm has improved a lot since then. In fact ROCm itself is generally working very well! (Non-driver portion) The problem comes from their closed source proprietary drivers, most infamously amdgpu-dkms. You need very specific kernels and very specific distributions to successfully build the kernel module. Not only that, AMD refuses to fix the infamous GPU reset bug which makes stopping and starting VMs incredibly frustrating. For modern AMD cards, you *should run into fewer problems, but for Instinct cards you will need to do a lot of troubleshooting. Motherboard support is also very poor for Instinct cards, you will run into problems with the GPU reset bug, trouble initiating the GPU, detecting the GPU, and so much more. It took me weeks of non-stop work to get x2 MI100s even detected last year. It took countless hours of headache and trial and error without any help from AMD. I tried messaging their custom service and only got this response: "Your motherboard is unsupported. I can't help you." Unfortunately, the bugs keep coming. To get amdgpu-dkms working on Debian 12 I need to troubleshoot a driver build problem. AMD shipped with corrupted build packages. Pain lol. I'm currently dealing with a problem where the amdgpu driver keeps failing to get user pages, and it is extremely inconsistent to get llama.cpp running. I could keep going but I think you get the point. haha """ Personally, I have not gone down the AMD rabbit hole at this point for many of the reasons mentioned above. However, when I get more time I would like to. I think if you could ever get things running AMD might be a much more cost effective way to go. It just a question of how much time is it worth to work through all the bugs. 3) Damn that is awesome. Love your setup. Are you running a cluster or do you have them a two separate machines? Yeah I bet you paid quite a bit more a couple years back. Interestingly, I feel like a lot of servers and adjacent hardware has come down in price significantly in recently years, but has started to get more expensive again recently with the AI craze.

[@xxriverbeetxx1.065]

Hi Iam really into servers and homela. which usecases does such a ai server have. I run stable diffusion on my pc but I don’t use it regularly. What are some ais you ran. I can’t imagine one which it is worth buying a server?

[@TheDataDaddi]

Hi there. Thanks so much for the comment! This server setup specifically is really meant to be a good general purpose setup for a wide variety of AI/ML/DL tasks. It should be suitable for most common DL applications. Some of the most common deep learning models and their use cases include GPT (Generative Pre-trained Transformer) for text generation, language translation, and conversational AI, with examples like GPT-3 and GPT-4. BERT (Bidirectional Encoder Representations from Transformers) is widely used for natural language understanding, question answering, and text classification, including variants such as BERT, RoBERTa, and DistilBERT. In the realm of image processing, models like VGG (Visual Geometry Group) and ResNet (Residual Networks) are popular for image classification and object detection, with examples like VGG16, VGG19, ResNet50, and ResNet101. YOLO (You Only Look Once) is known for real-time object detection, with versions like YOLOv3, YOLOv4, and YOLOv5. GANs (Generative Adversarial Networks) are used for image generation, style transfer, and data augmentation, including models like DCGAN, StyleGAN, and CycleGAN. For image segmentation and medical imaging, UNet and V-Net are commonly used. Transformer models, such as the original Transformer and T5 (Text-to-Text Transfer Transformer), excel in sequence-to-sequence tasks and language translation. RNNs (Recurrent Neural Networks), including LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit), are applied in time series prediction, text generation, and speech recognition. MobileNet models, like MobileNetV1, V2, and V3, are designed for efficient image classification on mobile and edge devices, while EfficientNet models (B0 to B7) offer improved efficiency in image classification and object detection. DALL-E models are used for generating images from textual descriptions, with notable examples being DALL-E and DALL-E 2. CLIP (Contrastive Language-Image Pre-Training) connects images and text, enabling zero-shot learning. Lastly, AlexNet is a foundational model for image classification. This should also allow you to get into stable diffusion and work with some of the smaller open source LLMs locally like LLaMa-7B, Falcon-7B, Mistral-7B, etc. Hope this helps!

[@testales]

How many tokens/sec do you get out of this r720 based setup when loading say the biggest Mixtral LLM version that fits into memory?

[@TheDataDaddi]

Hey there! Thanks so much for this question. I actually have not used this setup to run any of the open source LLMs yet. I am planning to do this as soon as I get a chance. Life has just been crazy recently so I haven't been able to get to it. I will try to get something out there by the end of next month.

[@testales]

@@TheDataDaddi Thanks, I'm currently in the process of deciding whether to go all-in and get a car-prized GPU for like 4 times its actual worth or to go for a cheap server with old 24 GB server GPUs. If I can run 70b models in decent manner with this it might suffice - for now at least. ;-)

[@TheDataDaddi]

You might want to check out this: www.ebay.com/itm/126021050668 This server can take 8 2 slot GPU. I actually just bought one. I think this might allow you to use cheaper GPUs just more of them to meet you VRAM requirements. Assuming you can split computation or load different parts of the model across multiple GPUs. This is the route I would probably take tbh. Good balance of price to performance. @@testales

[@bitcode_]

AFAIK you cannot run Cuda version higher than v10 on these, could someone confirm?

[@TheDataDaddi]

Hi there. Thanks so much for the comment! I have successfully run CUDA 12.2 with no issues. I have not tried any newer versions though as of yet.

[@Marioz773199]

Without tensor cores, do you think the P40 will still be usable in the future?

[@TheDataDaddi]

Hi there. Thanks for the question. I am sure there will come a point in the future when the p40 is no longer viable. However, I think it is still quite relevant. I use it almost everyday. There are couple things to note here. 1. Tensor cores have only been around since 2017. While that my seem like an eternity in AI time, it is still a relatively new innovation. Just because tensor cores are more efficient does not mean that older technology is obsolete. This video is really geared toward getting as much AI/ML/DL compute for as cheap as possible. With that goal in mind you will not be able to access the latest GPU technology. 2. It depends a lot on your application. If you want a good amount of compute to tackle most ML/DL problems, this will work well. If you are looking to train the latest LLMs at their largest sizes, then this would likely not be suitable. 3. AI/ML/DL at this stage is very cost prohibitive for many hobbyist and enthusiasts. In the end, this translates into doing the best you can with what you can afford.

[@Marioz773199]

@@TheDataDaddi Those are very good points. I want to be able to fine-tune some of the larger LLMs in a 2U PowerEdge R730. It has space for 2 2-slot GPUs, so I can't fit a giant 3090 or 4090 if I wanted to. Thought about going for an RTX 4070 ti, but the bus problem. It's a struggle trying to build an AI 2U server without shelling out loads of money. I'm thinking of maybe just building a PC full tower with a 3090 in it and connecting that to my 2U server. At least then I can fit giant consumer-grade GPUs.

[@TheDataDaddi]

Another route to go is building an external rig with PCIE extenders. That is the route I am going to try to go. I will be making a video on it as soon as the extenders come in. If you keep an eye on my channel soon you should see it. I would recommend going that route personally.@@Marioz773199

[@voltage952]

Can we pair 3090 and p40? dual gpu but better performance

[@TheDataDaddi]

Hi there. Thanks so much for you question! In general, when making use of data parallelism (copying the same model across both GPUs and splitting batches across GPUs) the slower GPU (the p40 in this case) will not create a bottleneck. It should still improve the throughput overall. However, when trying to use 2 GPUs to host larger models, specifically models that cannot fit on single GPUs and must be sharded, the p40 would be a bottleneck. Throughput would be constrained by the slowest GPU. Hope this helps!

[@lkvantaliani]

PowerEdge R720 Servers are sold out (probably after this video) any replacement?

[@TheDataDaddi]

Hi there. I really appreciate the question. Unfortunately, I just checked the deals and there aren't any i'd recommend at this point. They have new deals each week though. I would check again next week.

[@lkvantaliani]

@@TheDataDaddi Don't spend your time on this. You are doing enough by teaching and giving us some good ideas. thanks. Just was wondering, what is special about R720. is it size? anything else? It's u2 right? if anyone considers co-location to calculate price

[@TheDataDaddi]

I appreciate the kind words man! So nothing really is special about the R720. It just has all the right qualities and price to be a great base for a deep learning rig. It is 2U so it can fit 2 GPUs comfortably. Plenty of drive bays for storage. Can support up to 1.5 TB of RAM. Typically comes with 20 physical cores and 40 logical cores which is a good about for an work that involves parallelization. You can generally get all of this computing power for less than $300. So for people that are looking for a solid cheap way to get in to AI/ML/DL I think it just fits the bill really nicely. @@lkvantaliani

[@Jjitpimolmard]

Hello, your system look very interesting. im a complete noob on this area. I plan to build a data science system ( DNA sequencing) . The recommendation from the seequencer company is that they recommend A6000 ada or higher. However, I'm trying to save some budget. I wonder if is it possible to built a computer base on old server with 4-6 2nd P100 ? is it going to have similar compute power? or give a try on 4090

[@TheDataDaddi]

Hi there! Thank you so much for the interest and the feedback. Wow. Yeah if they recommended the RTX 6000 Ada thats like a $7k or $8k GPU. I think this really depends on your budget. If you have a max budget it would help me better understand what to recommend. Personally, though I am a fan of the p100 and p40 gpus for less than $500. Then for less than $1000 I am a fan of the RTX 3090s. These are the best price to performance GPUs I have found. If you wanted to get the closest performance without paying obscene amounts of money, you could try 2 RTX 3090s connected with NVLink. I like the 3090s specifically because they are the last generation that suppports NVLink. I do not believe the 4090s do. Overall, this would cost you about $2K and get you similar amount of total VRAM albeit on different GPUs (however NVLink will allow the gpus to act as one in some cases to my knowledge). Also you would be getting ~35 TFLOPS per gpu vs ~91 TFLOPS. So, if your use case can be effectively split across 2 GPUs you can get most of the processing power of the RTX 6000 with a fraction of the cost.

[@Jjitpimolmard]

@@TheDataDaddi thank for your kind reply. My perfect budget would be 10000 USD .

[@TheDataDaddi]

Ah okay. If that is the case, you could probably afford the A6000 Ada along with everything else you would need. Might be better off to go with 2 4090s though. Still leaves plenty of room for a really nice server. Not personally a huge fan of the 4090s because they promised pcie gen 5 are still gen 4 and they did away with NVLink for them. However, for your budget it would be a nice balance of performance vs price. One thing I would check is how easy it would be in your use case to split work across multiple GPUs. If that is not feasible for you then the single RTX 6000 ada may make more sense for your budget. @@Jjitpimolmard

[@Jjitpimolmard]

@@TheDataDaddi thank you very much ❤️. I'm working in Thailand, there is only 1 a6000 ada currently in stock for the whole country -_-. Hope it still there until my project starts 🤞. Thank for your suggestions. I will ask my bioinformatics engineer, if they like like to split or put the hole shank in. But for energy consumption, a6000 would considerably need less energy right?

[@TheDataDaddi]

Oh wow! I really hope you get it. Yeah definitely confirm with him first before buying anything. Yes, it would use considerably less. 300W with the A6000 and 900W with dual 4090s. This could be may be a major consideration as well depending on electricity cost in your country/region. @@Jjitpimolmard

[@PreparelikeJoseph]

Is using a server build better than a PC build or were parts just cheaper ?

[@TheDataDaddi]

Hi there. Thanks so much for the question! The reasons I prefer servers over OC builds: 1) Price for compute is almost always better (main reason) 2) Ability to support more cores and higher CPU RAM 3) Remote management tools like IPMI or IDRAC 4) Generally more stable and built to run forever with out being turned off 5) Built in redundant power supplies and failovers That said. A custom build is always going to be more flexible and likely give you the ability to have the latest and greatest. Overall though, I like refurbed servers because I find they provide the price to performance.

[@emanuelsavage-op5mm]

good video, what do you think about the x299 platform, seems a lot more usable for a desktop: faster cpu's, ddr4 memory, nvme support, decent performance on AAA games, even a good upgrade path. i'm thinking to buy my first desktop in the next few days. mostly for LLM's, stable diffusions and computer vision. with a i9 7900x, 48 gb ram ddr4 and asus x299 A motherboard, rtx 3060 12gb and a corsair rm 850w psu. in the future I would be able to double the ram and also to add another rtx 3060 12gb

[@TheDataDaddi]

Hi there. Thank you so much for the feedback! Sure this would definitely be a good path to get you started. It is just more expensive than going the server route.

[@emanuelsavage-op5mm]

@@TheDataDaddi thank you for your time, I'm flexible with what I want for the time being I want a desktop that can train models and ram intensive programs. in your experience a chinese x99 build with 64 gb of ram a p100 and a gt 610 for video output, is some what usable as a desktop?

[@TheDataDaddi]

So I am assuming by X99 build you just mean a mother board that is based on the Intel X99 chipset. If that is the case, I think everything you mentioned above would be great for a desktop build that can train most models and run ram intensive programs. @@emanuelsavage-op5mm

[@ThetHan-i9z]

​​I have to bulid my budget pc (1)Should i go cpu i5 14500 or i7 13700kf ?/ With RTX 4060ti 16gb / (2) Which ram should I go example 32 or 64 (3)Which psu should I use example750W 80+ or 850W 80+@@TheDataDaddi

[@TheDataDaddi]

So this will likely depend on your total budget, but I would do the following. 1) In AI/ML/DL the GPU is the most important thing. So I would allocate the biggest part of your budget here. I think the 4060ti is a great choice for a beginner build. 2) RAM is the next most important thing imo. I would go for 64 GB from a quality manufacturer if your budget allows. G-Skill is the past has always been a good value. 3) In you case, I would allocate more money to buy a bigger PSU more than what you think you will need. You may add component in the future (like another GPU) and the PSU is pain (or can be depending on the kind) to swap out. 4) The thing I normally look at last is the CPU. Just about any relatively new CPU is going to be just fine for more general computing tasks and be beefy enough to supply your GPU(s) with plenty of data. Plus the CPU is a relatively easy component to upgrade in the future if need be. So in your case I would probably save money here and go with the I5 4500. Hope this helps! Good luck with build. @@ThetHan-i9z

[@huanhe4084]

Great video!I am looking for upgrade my HPR620+1070ti now. I checked your google doc and found that there is no R720 available now. Do you know where else i can buy now?

[@TheDataDaddi]

Hi there! Thanks so much for the feedback. I do not know another reliable place off the top of my of head, but after some research. My two recommendations would be: 1) Try searching "Dell Power Edge R720" on EBAY then finding a reliable seller that refurbishes the machines and runs QA/QC 2) Try this website as well. It is seems similar to savemyserver.com. However, I cannot vouch for them as I have never used them: pcserverandparts.com/servers/dell-servers/poweredge-r720-server/

[@huanhe4084]

@@TheDataDaddi Thanks!! Just wondering why you didn't choose P100 which I heard it have better half presion performance than P40 BUT Similar price point?

[@TheDataDaddi]

It really depends on you use case. For example, the p100 has better half precision and double precision than the p40, but the p40 has more VRAM and greater single precision. I have found so far for my work mainly in computer vision atm. I prefer having more VRAM and greater single precision performance. However, I have not benchmarked either GPU yet so I cannot say with certainty which is better or worse. I will put out a video on this very soon. @@huanhe4084

[@huanhe4084]

@@TheDataDaddi Have you heard RTX 2080 ti 22g before? why not that one?

[@TheDataDaddi]

So I have never heard of a variant of the RTX 2080 ti that comes standard with 22 GB of memory. I have only every seen it with 11 GB VRAM. You may be able to modify or add memory for this purpose I am not sure. It is also twice the price as my check just now on EBAY with only a marginal performance increase. So, I do not really see a better value here over the p40 or p100. @@huanhe4084

[@bedirhan_tas]

Hello thanks for the video. I need to learn about machine learning for the bioinformatics area and also i have 2 options for gpu which are 3060 (12gb) and 4060ti (16gb). I have limited budget 4060ti (16gb) is %50 expensive then the 3060 which one should i choose for the CNN RNN and Bayesian network ? i can buy arc a770 is it another good option for the machine learning?

[@bedirhan_tas]

Additionally, Is the 8 gb vram enough for me i just want a buy for once. I am a long term customer in every elecetronical parts. I have that options this gpus: ARC 580 or 750, 4060 or 4060 ti 8 gb, amd 7600 gpus.

[@TheDataDaddi]

So if machine learning applications are your primary goal with these gpus I would stick with NVIDIA. The have a much more mature ecosystem in this area than the other GPU manufacturers. Given your options I would go with the 4060 with 16 GB of VRAM. This will be a good compromise of performance and memory to load models and data. I would say though in this age VRAM is more important than anything else. @@bedirhan_tas

[@bedirhan_tas]

@@TheDataDaddi First of all thanksgiving for your response. I have ryzen 5 5600g and 16 gb ram memory. I understand your answer clearly. I have another option which is AMD 7600 xt 16gb vram, that one is %80 cheaper than 4060 ti 16gb in my country. For machine learning things, what about the AMD ecosystem, especially ROCm 6.0 features ? If I buy this AMD GPU will I be able to work on machine learning in windows 11 ?

[@TheDataDaddi]

You could certainly try to go this route. Especially if it is 80% cheaper. However, ROCm has known bugs and is overall less mature than the NVIDIA ecosystem so there may be a reason that it is 80% cheaper. I would recommend doing your own research before you go the AMD route to make sure that ROCm will super that GPU and both will work whatever you eventual use case is. Reddit is usually a good resource here. @@bedirhan_tas

[@bedirhan_tas]

@@TheDataDaddi I am a beginner in the software area and thank youu bro if u have any questions about biotechnology feel free to ask. Warm Regards

[@teddyfulk]

R720 sold out? Can I just do the R630 instead you think?

[@TheDataDaddi]

Hi there! Thanks so much for the question. Are you talking about from SaveMyServer.com? If you are they do deals every week and they frequently have r720s at great prices. I would wait for this. As for the R630, I have no problem with the server itself. It is actually newer than the r720. However, if you are buying specifically to user for machine learning I would not recommend it. It is a 1U server and you will not be able to fit in many if any GPUs (unless you build an external rig of some kind). Now that said, I have never tried so I am not 100% sure how much space there is inside of it. You may be able to make some 1 slot GPUs work. Overall, though I would pass on it if this is going to be your primary rig for deep learning applications. As a side note, 1U server tend to be extremely loud so I normally avoid then for this reason. Hope this helps!

[@teddyfulk]

Ah thank you, this is great insight. Would you say wait for the deals for these specific items if you can to get these deals when they arise if you’re balling on a budget?

[@TheDataDaddi]

Absolutely. I would also say don't limit yourself to just the R720 or to Dell. There are other model and manufactures as well that would work. I would suggest sticking with 2U servers and above though. You could also check out similar HP models. In my experience though, dell r720 is gonna be your best bang for your buck. You can also check out other websites like savemyserver although I honestly cant personally recommend any because I have never used any others. If you find a any other good places to buy refurbed servers though, please do let me know.@@teddyfulk

[@BadStoryDan]

Have you looked at the HP DL580 at all? Im not doing ML (yet 😅) but have similar requirements (lots of cores, lots of gpu power and a bunch of RAM and I am building out a DL580 G9 with quad 8890v3s, 3TB of RAM and four gpus (I'm hoping k40s, but the power requirements are a bit much). Basically I'm thinking maybe you could get better value in a 4u server. But like I said, my applciation isn't exactly the same so maybe that's not true.

[@TheDataDaddi]

Hi there. Thanks so much for the feedback! Yeah this would definitely be a good way to go. For AI/ML/DL applications the main focus typically is number and quality of GPUs available. This is definitely a good choice. I like also that it is relatively cheap used from a quick look on EBAY. Its funny you recommended this server because I actually just bought a 4U server in the last month and am in the process of setting it up. I bought a Super Micro 4028GR-TRT 4U that is capable of fitting 8 2 slot GPUs. I will have a video out on that soon. For comparison, the HP DL580 does not look like it has as many 2 slot GPU slots available. However, it has 4 CPUs and the possibility for many more cores. All in all I think this a fine way to go for the 4U route, but I think there are better options out there specifically for AI/ML/DL. Thanks again!

[@BadStoryDan]

@TheDataDaddi Do you mean that the DL580 can't take more per chassis or can't take as wide a variety?

[@BadStoryDan]

@TheDataDaddi If it's the former, I've seen them with 5 GPUs - and beyond that I think you're looking at a huge jump in cost to something like an H3 Falcon 4210, no?

[@TheDataDaddi]

I meant it just won't be able to take as many 2 slot GPUs as other servers I am aware of. It might have just been the chasis configuration that I was looking at though. It seems like there are several different ones. @@BadStoryDan

[@TheDataDaddi]

Upon a second look. You are absolutely right. I counted 5 GPUs that could fit. Only thing I am not sure about is if it would support all GPUs at the full x16 lane bandwidth. My initial guess is that it would because it has 4 CPUs. Either way though, I am seeing some of these in the $300-400 range. For that price, I definitely agree this would be a great way to go! @@BadStoryDan

[@tianwen2000]

It should be 3.4KW per day not per hour. Or 0.141KWh.

[@TheDataDaddi]

Hi there. Thanks for the feedback. Yep, that makes sense. Is this a mistake in the video? If so could, you please let me know the timestamp so I can fix it?

[@ResIpsa-pk4ih]

Math looks correct to me - 3.4kwh per day * 30 days = 102 kWh * $0.12 per kWh = $12.24 total which is roughly what you got.

[@fosatech]

Hey man, do you have a Twitter/IG or an email I could reach out to? We're gonna be building out some ML infrastructure in the coming months, and your videos have been extremely helpful as I'm doing research into all of this stuff. - Vlad

[@TheDataDaddi]

Hey there. So glad these video have been helpful for you. Yeah absolutely. You can reach out to me at [e-mail]. I'm happy to help.

[@IntenseGrid]

@@TheDataDaddi no response to that email address. Maybe my email is in your spambox?

[@TheDataDaddi]

I just checked, and I found like 4 emails in there. Really sorry about that. Please let me know if you do not get a response from me. @@IntenseGrid

[@peterdamaris7112]

I wanted to propose a challange. Best value for money AMD machine, (both CPU and GPU), with min 16GB of VRAM (prefer 24 or 32). Then compare to your rig performance/$

[@TheDataDaddi]

Hi there! Thanks so much for the feedback. So this would be like a raw build? If so, I have made a similar video of a custom build geared towards those beginning in AI/ML/DL. Here is the video link: kzbin.info/www/bejne/bnuqlmarbMSYp7c Let me know if this is similar to what you were talking about!

[@mikeyjohnson5888]

Its 20 cores not 40 cores.

[@TheDataDaddi]

Hi there. Thanks for the feedback. Sorry for the confusion. I believe what I meant was 20 physical and 40 logical cores. Thanks for pointing that out though.

[@Fordtruck4sale]

Find two people desperate to unload used 3090s and you’re flyiiiiing for not a lot of money. Undervolt them and they won’t use THAT much juice

[@TheDataDaddi]

This would definitely be a good way to go. Problem is just finding those people. lol

[@sa_med]

What's the deal you got on those

[@TheDataDaddi]

On the GPUs from the video?@@sa_med

[@ragoonsgg589]

ahaahhhaaaAAAHAHHhh......I just found it...nvm!!

[@imbretzy]

90% of everything you are saying is going right over my head but im trying to learn. That said, I'm mainly following because im into your voice hahah 😅😙

[@TheDataDaddi]

Hi there! I appreciate the kind words. Please let me know if you have any questions I can help with. Also, glad to know my voice is a selling point for the videos! lol 😆

[@3ool0ne]

Hi. This is a fascinating video but I'm still unclear what type of inference you could run on this. Do you have X (Twitter) I could get in touch with you at? I'm asking because I'm working on a pretty big project but I'm not clear on several things.

[@TheDataDaddi]

Hi there! Thanks so much for the feedback. I have had quite a few people ask me so far for some performance benchmarks. I am trying to put together something specifically for AI/ML/DL. In the meantime, please feel free to reach out to me at [e-mail] (sorry I unfortunately do not have X (Twitter)).

[@anthonyhoward1296]

I have the same server but I can’t find the 20 core cpu E5 2670v2 you mention. Can you provide some links ? Thanks for your time and info !

[@TheDataDaddi]

Hey there! Thanks for the comment Yeah, I have gotten a couple of comment on this, and I think I misspoke in the video. I think I meant 20 logical cores not physical. Sorry for the confusion here.

[@chrismay2298]

10 Physical Cores, 20 logic threads each.

[@JvP000]

Hi, How did you get the R720 server with all those specs for just $200? I can't even see any options available on the linked site with 256GB RAM, let alone anything near $200. What system would be an alternative option if it wasn't on sale / special time discount? Thanks

[@TheDataDaddi]

Hi there. Thanks for the comment! Unfortunately, the price of hardware continues to go up pretty much across the board so my best advice here would be patient and try to find deals from certified refurbishers like the following: savemyserver.com/deal-of-the-week/ pcserverandparts.com/refurbished-servers/ www.servermonkey.com/default www.theserverstore.com/ EBAY is also a great resource for quality refurbed and used hardware Any 2U+ server that has at least 2 x16 lane PCIE slots should work. You will need to double check that the server will support the GPU form factor you are interested in. Some other ones you could consider: Dell PowerEdge R730/R740/R750 HP ProLiant DL380p Gen8/Gen9/Gen10 Supermicro SuperServer 6029GP-TRT ASUS ESC4000 G4 Gigabyte MG50-G20 2U Here is a link to a build I just speced out for another viewer. Hopefully this will help you get an idea of what you can expect more recently. docs.google.com/spreadsheets/d/11HJaPbho8wVKwholTBx-rka8hO5l-1Uh/edit?usp=sharing&ouid=105510157555294255385&rtpof=true&sd=true

[@DragonsR4Ever2]

I acquired this for about $250 on eBay. Dell PowerEdge R720xd - CPUs:2 x Intel Xeon E5-2697 - Memory:384GB DDR3 1866MHz - Storage:7.2TB HDD (7.2k RPM) - RAID Controller: Integrated PERC H710 - Power Supply:Dual PSUs X2 1100w

[@tyswenson7542]

I really love the idea of hardware ownership over cloud for the simple reason that I don’t want to be considering cost as a factor when I am tuning my models… I want to have autonomy to retrain at my convenience or even to test a small hyper parameter tweak. That is not practical with cloud when you need to manage your spend.

[@TheDataDaddi]

Exactly! I have work in both environments, and using your own managed hardware makes things so much easier. It is cheaper, trouble shooting is much easier, and you can pay for exactly the performance you need by swapping out your own hardware. Unless you are a small business that is not tech focus or a large business that doesn't want the aggravation/cost of managing a large infrastructure of on prem hardware personal hardware is the way to go imo.

[@RandomShart]

Not just spend either, it's about overall control as well. The amount of times I would be disconnected mid training, or have my allocated hardware given to higher paying customers just became so frustrating (Colab for example). If you can afford the hardware and the running costs for homebrew or small business stuff then definitely better than cloud.

[@TheDataDaddi]

Hi there. Thanks for the feedback! I absolutely agree. I suppose Colab and similar solutions are cheaper. However, they are really unreliable in my experience. Plus even Colab now you have to pay for compute units which don't go very far and expire if not used. Its really just for people who are getting their feet wet. @@RandomShart

[@ultraplexplextor]

Hi, thank you for the great information. Is it possible to witness some Homelap action? Heat, (P40) Power consumption, and Bensmark. I would like to upgrade from two Dell R720s with four Grid K2 GPUs to four Tesla P40s. :)

[@TheDataDaddi]

Hi there! So glad the information was helpful! It funny you asked about that. That is literally the next video topic I am going to cover. Great minds think alike they say. Lol. I have had that video idea on my radar for awhile. Unfortunately, I just have not yet had the time or enough data to post it yet. That will be the first video I put out in the New Year. Thanks so much for the feedback and Happy New Year to you as well!

[@redgenAI]

If I was looking to get an 'out of the box' solution - as in a useable workstation that could train decent-sized LLM's like mixtral, what would your thoughts on the apple m3 max be? Best value for the money since it can come with up to 128gb vram?

[@TheDataDaddi]

Hi there. Thanks so much for you question. Check out this Reddit thread it has a lot of interesting insight here. www.reddit.com/r/MachineLearning/comments/17lw3gu/m3_pro_for_machine_learning_deep_learning_d/ For my personal recommendation, I would advise against it. While the M3 Max is a beast of a machine. It seems much better suited for video editing, graphics rendering, or maybe even heavy gaming (although I have heard macs are not great in this area granted I am not much of a gamer). Also, I have the M1 and have had issues with several ML libraries in the past. There are much better and cheaper options out there for a true AI/ML/DL setup out there than this. To expand further, the M3 is over $3,000 at the moment. For this amount of money you can buy a prebuilt workstation from a boutique manufacturer specifically designed for AI/ML/DL. Now with that said, this is also a relatively expensive way to go. However, if you do not want to deal with specing out or building the rig yourself this might be worth the extra cost. One more well known companies that does this that I have heard of is called Lambda. Here is their link below: lambdalabs.com/gpu-workstations/vector From a quick google search. Here is another that looks decent: www.digitalstorm.com/workstations-data-science.asp I just want to be clear though I am not recommending these companies as I have never worked with them or used their products so I cannot vouch for them or their quality. Please do your own due diligence with any of these companies. If you go this route, I would also be sure to check on warranties and shipping info. It would really suck to pay top dollar for a machine with a broken component you cannot get fixed or replaced. Another option you might consider is looking for beefy gaming (or maybe even crypto mining rigs) with NVIDIA GPUs that people are looking to get rid of on Facebook market place or EBAY. This would be by far a cheaper option if you are patient. You will likely get something of equal or greater quality as a one of the websites above for less than half the cost. I have actually seen some pretty good deals lately on Facebook market place around my area. Anyway, whatever option you decide to go with I would be very interested to hear how it turns out for you. I have never bought a prebuilt AI Workstation nor do I have direct experience with the Mac M3 series. However, both have always been interesting to me. Please keep me in the loop on your journey. Hope this helps and best of luck!

[@gileneusz]

great video! any ideas for self managing 4xP40? I'm asking because AI inference is my use case and doubling VRAM to 24x4 would be best solution that would fit all current open sourced models. 48GB is okay, but a little too small...

[@TheDataDaddi]

Best idea I have seen so far is the following: www.ebay.com/itm/155634089875 This can house up to 8 p40s. Otherwise you are looking at a custom build which could get super expensive really fast. One idea you might explore as well is if you have 2 pcie x16 slot you might be able to split them and use x8 lanes for 4 GPUs. Never tried anything like this but in theory it could work your GPUs would just run with half bandwidth.

[@gileneusz]

@@TheDataDaddi this looks even better, you can stack even more of them. I actually was looking got SMX3 and 8xV100 32GB server, but can't find any in my location (Europe).... I need to explore your idea 😀

[@TheDataDaddi]

Yep. Like you I am finding myself in need of more a more VRAM these days. I think one of these will be my next purchase as I have the funds. Anyway, so glad this was helpful to you!@@gileneusz

[@andrewgarberXYZ]

that site doesn't include the r720 anymore?

[@TheDataDaddi]

Hi there. What site are your referring to?

[@KrnlKlink1]

Great Video. What power supply do I need? I've got dual 750W in my r720. Could that run 1 P40?

[@TheDataDaddi]

Hey there! Really appreciate the positive feedback. That should be more than enough. I run 2 on mine and my server consumes like max 850W.

[@KrnlKlink1]

@@TheDataDaddi That's good news... I figured it would be fine and I thought I would just wing and see what happens. I really love my R720 and am looking for a good price on a replacement for my R620. The only feature I wish it had was PCIe bifurcation. Might hold out for the R730.

[@TheDataDaddi]

Yeah me too man. I do agree with you there are several features I like better in the 730. However, they are generally less cost effective than the R720s. However, if you don't paying a bit more it would probably be worth it (would be about double or more what you get an R720 for with similar specs). Just saw this on EBAY actually. Wouldn't be a bad one to pick up: www.ebay.com/itm/196206475914?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=h6xfJfvISUG&sssrc=2047675&ssuid=xv3fo9_stiq&widget_ver=artemis&media=COPY @@KrnlKlink1

[@MattJonesYT]

I did a similar build with dual E5-2697 v3 and it's CPU bound with just one P40 when using aphrodite. I'm not sure if switching to llama-cpp will fix that but so far I'm thinking a more modern CPU would have been better.

[@TheDataDaddi]

Hi there! Thanks so much for the comment. I have not used the setup much yet for LLMs so I will report back as I get a chance to test in that area. Unfortunately, to my knowledge the V4s are the newest CPUs that this server will support so if you wanted something newer you would likely have to go a different route. So far though in my applications I have not experienced CPU bounding. Even when using all my GPUs.

[@SaveThatMoney411]

Excellent.

[@reinsein518]

what about the P100 or is the P40 better?

[@erockvaughn2190]

From what I can tell, the P100 performs far better at half precision (16 bit) and double precision (64 bit) floating point operations but only has 16 GB of vRAM while the P40 is slightly faster at 32 bit operations and has 24 GB of vRAM.

[@TheDataDaddi]

This is a pretty fair assessment. I am still working on a set of comprehensive benchmarks, but empirically I will say that the p100 seems to be a bit faster overall so far on the same experiments. However, the p40 can hold larger models. So, I think it really depends on your use case. @@erockvaughn2190

[@Keeeeeeeeeeev]

love your videos

[@TheDataDaddi]

Thank you so much for the positive feedback. So glad to know that these videos are useful for you!

[@tech6546]

Awesome, thanks! But isn't the lack of raw flops a big deal for training? I think a lot of modern consumer cards have much better fp16 per dollar... Also, how would that scale? Since kaggle gives 2xT4 for free which is 32gb ram, it doesn't seem like this rig is a big step up, unless you are training for more 25 hrs at a time... Can you suggest the best price/performance setups for $5-10k budget? Thanks

[@tech6546]

Oh nvm I saw your new video, its exactly what I've been looking for!

[@TheDataDaddi]

Awesome! So glad the new video answered your questions. Please feel free to reach out if you have any others.

[@redgenAI]

If I was looking to get an 'out of the box' solution - as in a useable workstation that could train decent-sized LLM's like mixtral, what would your thoughts on the apple m3 max be? Best value for the money since it can come with up to 128gb vram?

[@TheDataDaddi]

Hi there! Thanks so much for your question. So here is a Reddit thread that is pretty helpful in answering your question: www.reddit.com/r/MachineLearning/comments/17lw3gu/m3_pro_for_machine_learning_deep_learning_d/ For my personal recommendation, I think it has great specs, and it is an interesting idea, but for the price I don't think its worth it. It would be an excellent machine for graphics work, video editing, and maybe even heavy gaming (but I hear macs are generally not great in this area). I would rather have a cheap laptop or daily driver to SSH into a separate workstation or server. I have a Mac M1, and I have always had issues with various machine learning libraries. These may be fixed in the M3, but I would be warry of it. Also, in the reddit thread above most people agree that it is okay at handling small to mid size models, but it is significantly less performant than even lower end setups with NVIDIA GPUs. So for the 3000+ price tag I would get a daily driver mac book air or you could get something even cheaper going the window route. Then take the remaining money and invest it in a quality prebuilt workstation. This is an extremely expensive way to go, but if you do not want to do your own research or build it yourself it could be a good way to go. One I have heard of before is called Lambda. Here is the link below: lambdalabs.com/gpu-workstations/vector Also, from a quick google search here is another that might work: www.digitalstorm.com/ I do just want to say I have never worked with either of these companies so I cannot vouch for their products so please do your own due diligence here. I would also look into warranties, and how it will be shipped because you would hate to pay that much for a system then not get your moneys worth because of bad or broken components. Also, another way to go might be look for a gamer on Facebook market place that is looking to sell his rig and buy a full setup that way. Much cheaper option and might get similar or better components. I have seen some deals out there recently. Anyway, please keep me in the loop. Would be very curious to see how a prebuild workstation or the M3 max route turn out. Hope this helps!

[@TheJmac82]

No upgrade path. Thats a very expensive hardware that may be useless when next gen models come out. I would buy p40's once the next b100's come out people will be selling RTX6000's like they do p40 now. My 4x V100s's will smoke a $6k apple 2 to 1 on larger models and outtrain 100-1. I have less than that in my entire setup and I am running Threadripper 7k that can also be used as a Moonlight gaming rig for friends who cant afford good hardware.

[@דניאלכהן-ג6ג]

Is the new RTX 4070 SUPER now the best option? It's much better than RTX 4070 and at the same price and only a little bit worse than RTX 4070 ti

[@TheDataDaddi]

Hi there. Thanks for the question! Yeah if you are comparing the RTX 4070 and RTX 4070 SUPER I agree it does likely make more sense to get the SUPER because they are almost the same price used in many cases. However, if you are asking my if it is the best option overall I don't think so. I think there is better value in other GPUs for the almost $700-800ish price point.

[@דניאלכהן-ג6ג]

@@TheDataDaddi What GPUs for example? The RTX 4070 SUPER is cheaper than 700-800$? You mean more expensive GPUs? Or GPUs at the same price buy better?

[@דניאלכהן-ג6ג]

@@TheDataDaddi And another question, does 32 or 64 GB ram matters? Also how important is the CPU?

[@TheDataDaddi]

So for me I would recommend the RTX 3060 or Tesla p40 or p100 for the ~$250 price range. For the $500-$1000, price range I like the RTX 3090 best. These prices are what I can find on EBAY from a reliable seller I wouldn't mind buying from myself. The 4070 imo is kind of in no mans land. Cheapest I can find it for on EBAY is about ~$650. To me this is too expensive for a 12 GB GPU and so close in price to the 3090 it doesn't make sense. At the end of the day though, it comes down to personal preference and price. If you can find it where you are for less than ~$500, I'd probably that over my other recommendations. Also, if you care more about performance and are not planning on working with larger models or are going to use it for gaming as well, then the 4070 would probably be a better way to go. Anyway, I hope this helps explain my logic. Good luck! @@דניאלכהן-ג6ג

[@דניאלכהן-ג6ג]

@@TheDataDaddi Where did you find the RTX 3090 at these prices? I searched and the price is about 1500$

[@MartinGaertner]

that is a joke! when i look in the shop for the best selling for 40 cores no server under 1.200 dollar!!! 200 dollar tell me who that price you found this! all other server is coast 5x or 6x more money! 200 dollar are you joking?

[@TheDataDaddi]

I know the price seems almost too good to be true, but that is the exact price I bought it for. There is a local electronics refurbishing business I buy from frequently specifically because they have amazing prices. I bought this particular one for that price about 2 years ago as a flash deal there were having. So yes this was an incredible price for that machine. That was one of the major reasons why I bought it. The name of the seller is called Save My Server and I always look at their Deals of The Week. Here is the link if you would like to check them out! I highly recommend them. I have purchased multiple servers from them. savemyserver.com/deal-of-the-week/