Can you describe the scenario in which it would make sense for a job to use multiple MIGs like that? Our main motive with MIGs is to address the extremely common problem of jobs that underutilize a whole GPU. Maybe the short answer is: we expect to have whole GPUs available as well as MIGs. Whole GPUs make sense for jobs that can use them effectively, of course - including across nodes. (I am Sharcnet/DRAC staff.)

I suspect MPS is the answer, but this needs to be tested.

@markhahn0 Well, that's the point. There are virtually no reasonable scenarios in which it would make sense for an MPI job to use multiple MIGs, unless the code is extremely and embarrassingly parallel with little to no communication across the GPUs. That's why most of the users, who knows what they are doing, won't use the MIG-enabled GPUs for their MPI/Multi-GPU jobs, making the race to acquire non-MIG GPUs even more competitive than before. This results in longer waiting times than what you already expect. I get your motivation and it's absolutely important to take action. However, I suspect using MIG for a big chunk of nodes in an HPC cluster is an efficient solution to this problem, and it may backfire. There are some considerations/suggestions that come to my mind about this: 1. Enable MIG only on single GPU nodes (a portion of them, if there are any) 2. Enable dynamic MIG configuration - preferably by slurm itself, and ambitiously, depending on the jobs and their requirements 3. With number two, then maybe you can enable MIG on multi-GPU nodes, if there are enough requests for them. 4. Do what you are doing: instruct users when and why to use MIG, and help them write better code. @SHARCNET_HPC Yeah, MPS is a better choice I guess, and users should be encouraged to use it. I wrote a couple personal posts about MIG and MPS-on-top-of-MIG that might be helpful. amirsojoodi.github.io/posts/Enabling-MPS/ amirsojoodi.github.io/posts/MIG/ amirsojoodi.github.io/posts/MPS+MIG/ P.s. Regarding MIG and its related issues, NVIDIA said they might relax the constraints in the future, but who knows when. Read here: docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html#application-considerations and here: docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html#cuda-device-enumeration

You are right: "MPS enables cooperative multi-process CUDA applications, typically MPI jobs, to utilize Hyper-Q capabilities on the NVIDIA GPUs."

what do you mean? The compiler? yes

@@muhammadfaridkhandaq4379 thanks

did you try?

Glad you liked it!

Glad it was helpful!

We plan to replace graham with a new cluster in 2024 (if funding comes through), and this new cluster may contain H100 GPUs. Before then we may get a small number of H100 GPUs for testing purposes.

@@pawelpomorski4025 I'm interested in testing H100 GPUs from a researcher's perspective when they are available. 😁

@@dakezhang2845 There is a lot of interest. When we get some H100 cards for testing, we will advertise it, probably in our monthly newsletter which all users receive.

@@pawelpomorski4025 What happens to the old cluster? Can research groups purchase the old cores or GPUs?

@@ameerracle The cluster hardware is owned by the university, which decides how to dispose of it if we (SHARCNET) retire it. Anyway, it looks like graham will be running with the current hardware at least until early 2025.

i need the steps to take to get this done

RocM uses HIP which is very similar to CUDA, so any program written in CUDA can be ported to AMD (assuming it does not use any advanced features which are exclusive to NVIDIA GPUs). As for data centres, given the high demand and expense of NVIDIA GPUs, more AMD data center GPUs may be adopted.

My pleasure!

Glad it helped!

Glad it helped

same error bro, did u rectified it?

If you are interested in trying our experimental MIG setup on narval, please submit a ticket with a request. We might be able to give you some access in January.

u are correct

Can you tell exactly how to use or provide syntax to utilize multi gpu