CUDA Crash Course: Cache Tiled Matrix Multiplication

[@yuhangliu45]

You are a great teacher! Your videos help me learn CUDA much faster than reading the official guide.

[@NotesByNick]

Glad to hear that you found it useful!

[@user-kj8cb6ch4w]

I just started studying this course in one of the finest universities in India, and let me tell you that I've learned more from you than my so-called highly qualified professor!

[@NotesByNick]

Glad you've found them useful!

[@user-kj8cb6ch4w]

@@NotesByNick hey, which book do you suggest to learn more about CUDA?

[@NotesByNick]

@@user-kj8cb6ch4w Programming Massively Parallel Processors: A Hands-on Approach is the book most people use

[@bluewhale37]

Yeah.

[@jereziah]

Thanks for the tutorials mate, please keep 'em coming!

[@NotesByNick]

Will do! Thanks for the feedback fella!

[@chadmcintire4128]

Hey your content is great, I haven't taken any of the nvidia courses, but this is pretty great, and free. Thank you! Also this looks like a convolution.

[@yassermorsy8481]

I feel like I am not grasphing this. Doesn't this mean every thread running in parallel has to go through the process of loading the row and column tiles into the shared memory. Why is that not more expensive?

[@muneshchauhan]

Thanks for the wonderful video. Just one query on the shared memory size. Since you stated that it will be a 16 x 16 tile, so we need to remove the further multiplication by 4 in SHMEM_SIZE

[@NotesByNick]

Yes, that's correct! It's a mistake in the original video. The code on GitHub does not contain that error.

[@arsalananwari4377]

I think he got confused between size of elements and size of bytes thus why he multiplies by 4 like you do in cudaMalloc

[@drewb2053]

I think theres a mistake in the SHMEM_SIZE calculation. You multiply 16*16*4 with the 4 to account for integer size but wouldn't that be include for when the compiler computes an integer array for A and B? Curious as to whether I was misunderstanding something about shared memory declaration.

[@NotesByNick]

Yes, it's bug in the video - consult the code on GitHub for the most up-to-date code

[@drewb2053]

@@NotesByNick thank you so much for your expedient reply. Your video was incredibly helpful

[@olivercopleston]

Thanks for the video, really helped with one of my Computer Science modules.

[@NotesByNick]

Glad you found it useful!

[@Chiarettoo]

Thanks for the video !! I'm new to cuda programming and I don't know if I'm missing a concept, so I was trying to change the size of the matrices to see performance in larger multiplications, but when I tried to multiply 4096 x 4096 (n

[@pranavshankar1972]

Great video. If I am trying to implement kronecker product all the change is that I need to make is the logic from tmp += result of c to tmp = result of c right?

[@_jasneetsingh_]

on the code on github, you are defining shared memory size to be 1

[@金安迪-g5q]

great video, finally understood this mechanism!

[@_lilkm]

Hi CoffeeBeforeArch, i'm wondering why we don't free up the memory allocated in the error check function pointed by (verify_c) variable

[@NotesByNick]

It should have been (it's a memory leak, although the program does immediately end afterward so it is automatically). I've updated the github repo with a verification function that doesn't allocate any new memory as it should have been from the start.

[@_lilkm]

Thank you@@NotesByNick

[@EduardoValdez-f4j]

Question: when loading the columns to the shared memory, couldn't we transpose these values to have the be sequentially accessed during the execution?

[@unknownentity5354]

I was thinking the same thing. I would suppose its fine considering that's what we do when speeding up CPU usage

[@aneesmd7837]

Any tutorial on how to use extern shared memory iam getting assertion error when i use extern shared mem?? anyone please give some leads

[@TYGR2115]

ohhh how i love you for this video absolute life saver

[@NotesByNick]

Glad you found it helpful!

[@joseluismatabuenasedano6881]

Thank, this is really good material :) One quick question, where is the temp sum allocated?

[@sinaasadiyan]

what about nonsquare tiles? for example instear of 16*16, we use 8*16 or 16*8 (in first matrix and appropriate to second )

[@NotesByNick]

I'm not quite sure what your question is. If you are asking if it's possible to use non-square tiles, then yes, it is absolutely possible. It just requires a small change to the indexing to address the fact that the X and Y dimensions are no longer equal.

[@krisdobrev3546]

Thanks for the video! By the way, I have that noticed that when working with smaller matrices ( 16x16, 32x,32, 64x64) the normal( using global memory) matrix multiplication seems to perform slightly better than the tiled one. However, once the matrices become bigger than 128x128, tiled matrix mutplication perfroms significantly better. Do you have an idea why this could be hapenning?

[@NotesByNick]

Smaller matrices like that aren't a great target for the GPU (unless many of them are batched together). Just copying the data back and forth is likely slower than just running on the CPU. A likely reason why the non shared memory version is faster is that the extra instructions for managing the shared memory (extra loads and stores into shared memory) outweighs the tiny amount of re-use you are getting. Your data is likely also small enough to just fit in your caches without any manual tiling using shared memory. Hope this helps! --Nick

[@krisdobrev3546]

@@NotesByNick Oh wow! I was not expecting a reply so quickly. It did actually help a lot. Thank you.

[@NotesByNick]

Christian Dobrev no problem! Always happy to help!

[@yuanchen9602]

can anyone tells the skills of debugging cuda code in visual studio2017/2019

[@shuaiwang3901]

Thanks for the video. It helped a lot!

[@NotesByNick]

No problem! Glad you found it useful!

[@bobhut8613]

Thanks this really helped!

[@NotesByNick]

No problem!

[@tunaalatan35]

Great video!

[@williamgomez6226]

Amazing channel. Thank You sooo much¡

[@khizaranjum9296]

Hi, so I think the SHMEM _SIZE definition should be just #define SHMEM_SIZE 16*16 because you are using it to initialize a shared memory array, and it needs the number of elements, not the number of bytes.

[@NotesByNick]

That’s correct! If you look at the code on GitHub, that issue (and others in the series) has been fixed.

[@khizaranjum9296]

@@NotesByNick Thank you for the timely reply. This series is great! There is just one more thing. I do not understand why you sometimes do: int GRID_SIZE = (n + BLOCK_SIZE - 1) / BLOCK_SIZE; because you are converting to an int, the -1 in there gives you the same answer as n / BLOCK_SIZE due to floor when converting from float to int.

[@NotesByNick]

@@khizaranjum9296 so in all the updated code on Github, floating point numbers are never used (or converted from) in the block and grid dimension calculations. That formula is just a simple way of rounding n/BLOCK_SIZE up to the nearest whole number using only integers.

[@grantgasser136]

what if the matrix is not square

[@NotesByNick]

Great question! It just requires a small change to the indexing. Here's a quick modification I just wrote to show that indexing off - github.com/CoffeeBeforeArch/cuda_programming/blob/master/matrixMul/matrixMul_tiled_rectangular.cu . Handling a matrix dim that is not a multiple of the block size is another interesting problem that you can approach in a number of ways. The easiest way is to just pad the matrix with zeros until it becomes a multiple, and the extra computation does not impact the final results. Cheers! -- Nick

[@rafinryan4218]

@@NotesByNick Hey Nick, great content as always. I am struggling a bit when it comes to multiplication of non square matrices. The above link does not work. Can you please give an update on that?

[@shawnli1256]

Thanks for the series, they are really helpful. But I find them not perfectly for real fresh learners of cuda, better watch these after reading the book "cuda by example".

[@SuperChrzaszcz]

The `int GRID_SIZE = (int)ceil(n / BLOCK_SIZE);` survived here.

[@NotesByNick]

All the code on Github should be updated as of this morning.

[@armagaan009]

👏🏼👏🏼👏🏼