7 PyTorch Tips You Should Know

GitHub link: gist.github.co...
Here are 7 tips for improving your PyTorch skills. These are all things that I thought of because I use on a normal basis. PyTorch has a lot of need things you can do with modeling to distributions, let me know other tips you have in the comments below!
Tips:
1. Create tensors directly on the target device
2. Use Sequential layers when possible
3. Don't make lists of layers
4. Make use of distributions
5. Use the detach method when the gradient is not needed
6. How to delete a model from the GPU
7. Call the eval method before testing

[@kuretaxyz]

You can also initialize Sequential layer with an OrderedDict if you want to access the layers with names instead of list indices.

[@snippletrap]

Converting a list of layers to a Sequential module is not always best. For example, suppose you want to add positional embeddings to each intermediate output. Can't do that with Sequential. But you can use a ModuleList as if it's a vanilla Python list, and the parameters will be visible to both the optimizer and the GPU.

[@talkingbirb2808]

3rd tip: use torch.nn.ModuleList instead of list

[@scottmiller2591]

I would run into the need to use gc and clear the cache when running pytorch lightning. If allowed to stop on its own, lightning would clean up after itself, but at least in preliminary runs, you'd see things were going south or want to stop iterating for some other reason, and breaking the iteration loop in lightning before it completed its iterations would leave all kinds of garbage (the still-loaded models, etc.) on the GPUs, as you could easily test by looking at nvidia-smi - the memory would still be in use, even though lightning was supposed to release it. I can't remember if I used your exactly your lines, but I ran something to clean up the GPUs. It convinced me (along with the relative inflexibility of lightning) to stop using lightning. I don't know if the lightning devs have cleaned this up in the intervening 4 years, but it's something to keep in mind.

[@pullrequest1296]

The test time of the first case is obviously not correct since you do not synchronize before measuring time.

[@1331death]

Why deploying in PyTorch is not the best option?

[@jargolauda2584]

please use a bigger font in the video. code is taking oly 1/3 of the screen width, so there is empty space for bigger font, for us TV tubers and couch programmers

[@pritamkarmokar3674]

Thank you for the video. It could have been better if the display were zoomed in a bit more.

[@zenchiassassin283]

Another tip: use torch.no_grad() or torch.inference_mode() at evaluation time

[@timangar9771]

Very cool, thanks :)

[@fabiolaespinoza778]

Thanks for the video ! For the 6th trick, do you know if replacing the model's instance frees up memory or does it accumulate? For example, if I first train a model using the instance name 'example_model' and then train another model with the same name, in terms of memory will I be accumulating the 2 models' space? Or just the last? Thanks!

[@EdanMeyer]

Not 100% sure, but I believe that if you reassign the value of the model, then that should delete the model according to Python's standard garbage collection implementation. However, if you have any separate references to it, those would also need to be deleted. Once there are no references left, it should be put in the garbage collection queue, and will then be deleted at some point in the future when the space is needed.

[@fabiolaespinoza778]

Ok, thanks for the reply!

[@MadlipzMarathi]

Use pytorch lighting you will avoid 70% of your mistakes

[@PritishMishra]

Amazing video! I didn't knew adding device='cuda' can make such difference!

[@ssshukla26]

Subscribed...

[@user-wr4yl7tx3w]

Is Pytorch as good as TF for high performance?

[@mohamedmesto3644]

nice

[@edgostyn]

Thank you very much, Edan. Hugs from Brazil!

[@haritoshpatel4216]

I am a PhD student focusing on machine learning and I can assure you these are amazing starter tips - sometimes I even forget occasionally the .eval one xD

[@mariusfacktor3597]

Great video! I like to find out what other PyTorch users think about, and these are some helpful "best practices"

[@henryhsu9517]

5:24 essential issue! Thank you for sharing.

[@hassanhamidi7079]

Great video. Keep making videos like this, please

[@stracci_5698]

Tip number one and already got my like 🤯

[@736939]

Actually instead of the list it's better to use nn.ModuleList().

[@wretchedmoose5139]

i dont get coding but this make me want to understand it

[@nintishia]

Great tips. Thanks.

[@caiyu538]

Great lectures.

[@mamotivated]

Rock solid

[@mingleiyin2154]

super useful