PyTorch Tutorial 11 - Softmax and Cross Entropy

[@manuelsilveriof]

This tutorial is one of the best out there. Thank you so much for making this. It is really appreciated.

[@patloeber]

thanks! Glad it's helpful

[@СергейЯкушев-ъ3д]

​@@patloeber Thanx a lot! It is brilliant

[@bpmoran89]

nice video, one nit: it's important to remember that confidence scores != probabilities. they may HIGHLY correlate with probability, they may LOOK like probabilistic outputs but they are NOT empirically derived probabilities in the strict sense. a model can have a confidence of 0.99, but that is not the probability that label is correct. if a research scientist would like, they can correlate confidence scores to probabilities empirically using testsets.

[@Yoyo-sy5kl]

Awesome tutorial. I do find some of the concepts a bit tough to grasp, but when I review them it helps a ton. I also like to review them by rewatching your videos. Keep up the great work, and I'm looking forward to checking out other tutorials on your channel.

[@aminaleali7161]

Your videos not only are educational and informative, but they are also very enjoyable. Thank you!

[@tcidude]

Very nice, clear, and detailed PyTorch tutorial!!! I haven't been able to find anything as good so far! Please keep up the good work and continue to make more tutorials!

[@patloeber]

Thanks a lot :) Glad you like it!

[@gordonlim2322]

The slide at 6:50 was very helpful. Thank you.

[@patloeber]

You are welcome :)

[@MinhLe-xk5rm]

Sir your video was amazing! THank you for showing how softmax and cross entropy is implemented in python!

[@patloeber]

Glad you like it :)

[@annalavrenova5631]

Thank you very much for the valuable content! Very helpful tutorials Pytorch!

[@patloeber]

Glad you like it :)

[@Ftur-57-fetr]

Lovely approach, clear, structured

[@patloeber]

Glad you liked it!

[@xingfenyizhen]

It's very friendly to beginners like me,very awsome video and author!

[@donfeto7636]

def softmax(x): return np.exp(x) / np.sum(np.exp(x), axis=1, keepdims=True) x = np.array([[2, 1, 0.1]], dtype=np.float32) print(softmax(x)) this should be the softmax for handling multiple example in batch

[@grigorijdudnik575]

Great video series, man!

[@patloeber]

Glad you like them!

[@gabbebelz7522]

RuntimeError: size mismatch (got input: [3], target: [1]). If you get something like this, you havent put double brackets when declaring good/bad prediction array. [[2.0, 1.0, 0.1]] from my understanding it has to do with the outer/ inner dimension

[@heathenfire]

very nice, clear explanation

[@vatsal_gamit]

Can anyone please help me why there's Y = torch.tensor([0]) , there should be 3 values inside ???

[@rubenguerrero3207]

simply awesome!

[@haroldsu1696]

Thank you, Sir. You are doing a great job.

[@koraykara6270]

Is the cross-entropy loss that you mentioned the negative log-likelihood (NLL)?

[@mysteriousXsecret]

Does it make sense to apply the softmax function in tandem of the sum-of-squares loss function, instead of the cross-entropy?

[@samas69420]

no avresti sempre uno perché la softmax prende un vettore mentre la mse é uno scalare

[@asheeshmathur]

Very Good, explanation of Multi class vs Binary classification

[@patloeber]

Thanks :)

[@egeerdem8272]

3:31 what does N represent? Also at 7:49, how do we actually represent and classify the data (as Y = torch.tensor([0])) I am confused? Changing 0 to 1 and 2 produces results so I thought they represented positions as in a list (0: [1 0 0], 1: [0 1 0], 2: [0 0 1]); however, that doesn't appear to be the case since they yield different answers compared to the numpy method (I used the same Y'^).

[@albertma1]

Thanks for the tutorial!

[@waqasahmed8973]

Thank you for the video, may i ask where did you learn pytorch? Did you study in your bachelor/master studies?

[@patloeber]

I have a master's degree where I learned programming and the concepts of ML/DL. I learned PyTorch on my own in my day job. It's not that hard when you understand the underlying concepts and know how to code.

[@erfanshayegani3693]

Great brother! Greaaaat!

[@mtkoli]

Very good video, thank you

[@andrewliao1140]

Hi, great video! Could you explain why you passed in the name of the class and "self" into the super() function?

[@patloeber]

that was the old syntax, but it's no longer required in the latest Python versions...

[@fullychen300]

Thx very much! VERY CLEAR explanation!

[@patloeber]

thanks!

[@saifulislamsajol9377]

The last 4 minutes of this video is very important. Could you please explain what to do when I am using MSELoss for autoencoder based networks? For cross-entropy loss it's working (although that's incorrect) but for MSELoss it's not working.

[@SuperOnlyP]

at 13: 00 , Could you please explain more the number you change for multi outputs for good prediction : [ [0.1, 1.0 , 2.1], [2.0, 1.0 , 0.1] , [0.1, 3.0 , 0.1] ]) if the output respects to ([2,0,1]) : label 2 is predicted : [0.1, 1.0 , 2.1] label 0 is predicted : [2.0, 1.0 , 0.1] label 1 is predicted: [0.1, 3.0 , 0.1] How can It turn to be good? --- why it is not something like : label 2 is predicted : [2.1, 1.0 , 0.1] - As I understand: 2 will be one-hot encoded : [1,0,0]. I guess idx [0] the prediction [2.1, 1.0 , 0.1] is highest number to be considered a good. thanks !!

[@Небудьбараном-к1м]

I also got confused at first. If you index out the values of Y_pred_good then the index of the highest value will match the ground truth label

[@patloeber]

exactly this! Also, class 2 would be [0 0 1] as one hot.

[@goelnikhils]

Amazing Content

[@adityashrivastava5018]

hey patrick , great stuff , can you upload the ppt for this part to on your github repository , i checked it's not there , it'll be extremly helpful

[@ahmedchaoukichami9345]

thank u so much it is very good explanation thnks a lot

[@tz8904]

Great Tutorial, but why does 3 sample look like [2, 0, 1] but not [[2], [0], [1]], thanks!!

[@lamnguyentrong275]

a little bit confuse, i thought that the Y true value must be the same size as Y_predicted value. As far as i know cross entropy will be a list of probability of true value and predicted value multiply each other then sum up. But in your case the y_predict value consists of three value, does it mean the y_true have to be the same as well ? Hope you can clear that for me

[@keroldjoumessi]

The CrossEntopyLoss function provided by PyTorch implicitly does the one-hot encoded with Softmax for us...

[@piotr780]

does it work fot multi-label classification problem ?

[@n45a_]

well, my first error i figured out before watching this video was using softmax at the end of my model, and the second one was using logits as Y target, and i learn it just now after playing with my modle for weeks... Thanks for poiting that out for others

[@LearningWorldChatGPT]

Hi Patrick! One question please! I trained a model and I'm making some predictions. On the other hand I'm trying to use the Pytorch library: "from torchmetrics.functional import auroc" To assim be able to know how my model is performing. But when applying that Pytorch function... I receive a tensor with a value of zero --> tensor(0.), I expected a higher value. My research is a Classification task with 48 speakers (Multiclass) (Speaker Identification). If you could give me a suggestion, I thank you in advance! # We perform a Classification: audio_file = '../CC_Sara.WAV' signal, fs = torchaudio.load(audio_file) # test_speaker: 1 output_probs, score, index, text_lab = classifier.classify_batch(signal) #print('Target: , Predito: ' + text_lab[0]) preds = output_probs.softmax(dim=-1) print(preds.sum()) # --> tensor(1.0000) print(preds) # --> tensor([[9.9889e-01, 1.2311e-05, 6.8829e-07, 9.4551e-06, 1.1589e-05, 8.6421e-07, 1.0865e-05, 3.1053e-06, 3.6530e-06, 1.7232e-06, 1.4130e-05, 4.6458e-06, 3.5194e-07, 2.0069e-05, 4.8732e-06, 1.1485e-06, 6.5748e-04, 3.0096e-05, 2.5761e-06, 1.6100e-06, 6.5871e-07, 2.6747e-06, 8.3195e-07, 3.0935e-06, 1.1512e-06, 2.0719e-05, 1.1157e-05, 3.2084e-07, 3.7289e-05, 3.2228e-07, 2.2807e-06, 1.1052e-05, 9.6865e-05, 1.0193e-05, 2.5367e-06, 6.8669e-06, 6.2059e-07, 7.3362e-05, 1.1659e-05, 1.7374e-06, 1.2059e-06, 4.7137e-06, 1.2199e-07, 9.8617e-07, 3.7428e-06, 2.1497e-06, 7.7288e-06, 4.6681e-07]]) target = index # --> tensor([0]) print(target) import torch from torchmetrics.functional import auroc auroc(preds, target, num_classes=48).item() # --> 0.0

[@penugondasaichand692]

super(car1,self).__init__() can you explain this with example what is difference between class and super class

[@DanielWeikert]

if Y is [0] and we have a prediction of 1,3 then Y is automatically one hot encoded?

[@patloeber]

Not sure what you mean. Do you mean you have two predictions 1 and 3, and 1 is the correct class? Then [0] is NOT one hot encoded, however it is the right shape for the CrossEntropyLoss. [1, 0] would be one hot encoded...

[@shadowzabyss]

@@patloeber So, I do not understand what the prediction values represent if the correct class is not one hot encoded..

[@sunnybwoy4547]

would multiple samples mean batch size greater then 1? That one confused me a bit

[@_jiwi2674]

Dont you have divide by N (number of classes)? It seems that you just added the true Y and log of pred Ys

[@keroldjoumessi]

I think N is the number of sample. Thus, he didn't take the mean because it was just one sample.

[@turboytytyt]

thanks!

[@patloeber]

thanks for watching!

[@TheOraware]

i got it now after viewing different videos for cross-entropy, one confusion is in case of numpy implementation at 5:38 the predictions Y_pred_good or Y_pred_bad are an output of softmax/sigmoid right?

[@patloeber]

yep!

[@manalihiremath2805]

i am gretting this error while using softmax function: Dimension out of range (expected to be in range of [-2, 1], but got 3)

[@patloeber]

Your tensors do not have the correct shape

[@manalihiremath2805]

Just a general question even after adding np.random.seed(1) my accuracy value is changing after evry run whyy?

[@joaobarreira5221]

Hi, Could y_pred_good have negative values?

[@islambennani6782]

Y_pred_good = torch.tensor([2.0, 1.0, 0.1]).unsqueeze(0) in case it's not working

[@roshankumargupta3711]

torch.max returning index right?

[@patloeber]

It returns a tuple with (value ,index)

[@elsaorange9341]

Bi ok ni ok ni