Introduction to Deep Learning with Python

[@slatervictoroff3268]

Feel a little silly for not having commented on this before, but despite the fact that deep learning has become such a buzzword, I think this tutorial provides and excellent example of how deep learning can be practically applied to real world problems. In my mind this is the go-to resource for beginners (especially python beginners) trying to get started with deep learning, or machine learning in general. That said, I'm biased.

[@alexandermcfarlane2109]

Slater Victoroff thanks for that download shell script. You earned me a coffee break whilst it did all the search and find for me :)

[@gilbertobatres-estrada5119]

This is a very good tutorial and introduction to Theano and neural networks. Deep learning is about greedy layer wise pre-training of for example Restricted Boltzmann Machines (RBM). Deep Learning is about constructing Deep Belief Networks or Auto-encoders with RBMs as building blocks. If you want to know about deep learning you have to read the papers by Hinton, Bengio and company to understand the subject of Deep Learning.

[@TodorTodorovv]

Fantastic video! It parallels my current course but is of a much higher quality.

[@roberttrueblood6168]

Awesome guide! Is there part 2?

[@dicksonneoh5563]

These codes are a lot simpler than the one on the tutorial! Awesome guide. Is there any snippet of code to print the hidden activation?

[@jundong6789]

The first t-SNE plot is based on the Raw Input, it shows the error is 8.28%. How was this raw input error calculated?

[@alexmcdonald7892]

As prgrammer I really enjoyed that video

[@edwardelric9107]

I ran into a problem with the modern neural net usimg the rectifier function, when I only take the gradients of the first two weight matrices it works perfectly. if I include updating the last one then the neural net gets stuck around 10% accuracy

[@hudaalamri266]

Great lecture, thank you for sharing, I have question about the first code you ran in the lecture, what did you use to get this simulation ? (I'm very new to python and Theano)

[@robertschrock450]

Love this kind of tutorials ! Thank you

[@teweishieh]

Thanks for this video a lot! I have a little question about the softmax function because I can't understand why we should use this numerically stable softmax instead and why it is stable?

[@Chr0nalis]

This implementation of the conv net gives me a 'ValueError: dimension mismatch' for images higher than 34x34 pixels. Can anyone give a hint why this happens?

[@tianqi]

Hi Alec, Thanks for the very nice video, you explained the Python/Theano code line by line! It was a surprise the whole convolutional network code is only 92 lines, I wonder how long it would be if coded with C++? I would guess at least 10 times longer. Any thought? Also I wonder what was the recognition rate for the convolutional net? I could not find the performance figure from the video. I assume it should be higher than the 99%.

[@Ed-bi4oo]

Hi Alec, Really enjoyed this video, I have a question about learning rates. at 29:53 do all these methods have the same (initial) learning rate? I'm wondering because I'm trying to implement rmsprop in another net (also using theano) and I'm finding I get instability unless I pick a really small learning rate. I also notice you use a (relatively) small learning rate for rmsprop here. Is there some kind of general rule for what learning rate to use for each method? And, if I am right that rmsprop requires a small learning rate, can you shed some light on why this might be the case? Thanks!

[@orrymr]

Hi, thanks for the great tut. At around 47:10, you reshape the data into a 1*28*28 matrix, since it's just grayscale, by using trX.reshape(-1, 1, 28, 28). What's the first parameter for? I'm confused as to how this works, since the numpy.reshape() method only takes in 3 parameters?

[@CloseCourse]

The -1 is a placeholder to say "whatever shape you need" given the other shape constraints. Since it's being called on an array, the interface is different from numpy.reshape which expects a tuple for shape I think - sorry never use it?

[@cherryentertainment9549]

please help to me . i want deeplearning tools (deeplearning 4j). how can i download it.

[@a_sobah]

how you have done the graphs and animation of the learning can can you show that code please

[@a_sobah]

Thanks

[@antonmbk]

For the "modern" net at 46:30, I think I am missing something: w_o is not passed in as an argument to the model function although it is used. Why is this?

[@CloseCourse]

I passed all the parameters in explicitly to make it clear what's going on for the function, but there's actually no need to since they were declared as globals, that's just sloppy coding on my part for forgetting to pass one in.

[@antonmbk]

Alec Radford Ah I see, thank you for explaining

[@nadaghanem4627]

Thanks, that helpful!. Imagine that we have only images of digits , and we want to build a model to figure out if any coming image is either a " digit" or "not digit". a problem when you only have one class label , how can we deal with it?

[@AayushThokchom]

+Nada Courses It wont be a one class problem. It will be a binary classification problem. You can use the images in column one and column 2 as label ( 1 for a digit, 0 for not a digit). You can use naive bayes, logistic regression, SVM for binary classification.

[@nadaghanem4627]

Thanks Aayush . Actually I'm working on few images , it's kind of comparing pairs of images from different sources - I want to show how likely the images are closed to each other. I still don't know how to apply machine learning !. Regards

[@David_Lloyd-Jones]

When recording a speech, please check your echo. This is really painful to listen to.

[@ahmedmazari9279]

Hello; Is there any code for the graph and visualization of 3_net.py, 4_modern_net.py and 5_convolutional_net.py ? Thank you

[@indicoioData]

+Ahmed Mazari (ⴰⵀⵎⴻⴷ ⵎⴰⵣⴰⵔⵉ) Sure, code is available from Alec's Github page: github.com/Newmu/Theano-Tutorials . It's MIT licensed, so hack away!

[@liqunyu8260]

Hey, I don't understand the dropout example at 31:15. At line 26-32, it seems to me that acc resets to 0 at beginning of every iteration and hence acc_new is always 0.9*g. Could anyone clarify? Thanks!

[@alapanbag5200]

at 7:00, what is line number 5 and 6 doing?

[@sergulaydore114]

I don't understand why you multiply p by 0 in RMSprop function around 31.14? acc = theano.shared(p.get_value() * 0.) Can you please explain? Thanks

[@drmgordon]

Sergül Aydöre The line is for initializing an empty variable that then is updated with the RMS moving average. The reason that he uses p.get_value() is simply to get the matrix of the apropriate size and since this is only done once the cost of *0 is negligable. The confusion that happened in my mind was to deduce the symbolic from regular. Note that as the T.grad is symbolic the only line actually evaluated will be the theano.shared line, since all other contain the grads or g variables. There is a nice description of the RMS method here: climin.readthedocs.org/en/latest/rmsprop.html#tieleman2012rmsprop

[@liqunyu8260]

Hi, I don't understand the dropout example at 31:15. At line 26-32, it seems to me that acc resets to 0 at beginning of every iteration and hence acc_new is always 0.9*g. Could you clarify on this? Thanks!

[@mattgenkin7929]

Hi, big fan of the video and sample code, which I'm using as my main guide around the theano library. Could you please post the training times for the models in the video so that I could compare mine?

[@mattgenkin7929]

Not necessarily precise times, just let me know if it should take minutes, hours, or days

[@AIgineer]

I am also interested in it for comparison

[@matejkopar5017]

Does anybody know where he get weight 625 in 22:28? I watched a few times but couldn't get where he get this and how it is connected with algorithm/data.

[@todanilt5518]

+Matej Kopar 625 is the number of hidden units.

[@indicoioData]

+Matej Kopar Neural networks have input units, hidden units, and output units. Each unit in each layer has a "weight"; weights are the parameters that the network updates in order to "learn" to minimize loss. Models have one input layer, and the shape of your input layer is determined by the dimensions of your input data. So in this case, MNIST images are 28x28 pixels = 784 total pixels. The shape of the hidden layer is the choice of the author, with smaller values representing a more compact "hidden" or latent representation of the data. Models might have many hidden layers, but here there is only one. Alec chose to use 625 hidden units here. Finally, each neural network needs an output layer to translate the model's "hidden" or latent representation into predictions for each class. For MNIST, there are 10 possible digits to predict, so you need 10 output classes.

[@kevin_machine_learning]

+indico perfect explanation !

[@LucGendrot]

I'm having trouble understanding the dropout function defined around 32:00, what exactly does MRG_RandomStreams.binomial() return? I can't seem to find any sort of explanation in the documentation and looking at the source code is beyond me at this point (I tried). The fact that we're multiplying the output by X tells me that it probably returns a matrix of 1s and 0s where the probability of getting a 0 (dropping out) is p. How wrong am I?

[@mattgenkin7929]

***** Your question is answered in this blog post: blog.wtf.sg/2014/07/23/dropout-using-theano/ Specifically the quote is: "The approach I use is to sample a vector of the same size as the hidden layer from a binomial distribution with a probability of 0.5. What happens when you then multiply this with the reLUs is that it randomly sets half of the outputs to 0, sort of like a random mask, which is what we need for dropout." In this context, reLU means "rectified linear unit", which is also what the guy is using in the video.

[@Voidroamer]

hory shiet. y only 5k views? the internet needs to know about this

[@Eltimjohn]

Is there a problem with the RMSprop algorithm? The variable "acc" is always keep reseting to 0 at line 26. (28:34)

[@indicoioData]

+Eltimjohn acc is a theano shared variable. Theano will create one copy of this variable and initialize it to 0. (The code there is a trick to initialize it to the same shape as the parameters except all zeros.) Each step it will be updated to acc_new by passing it into updates in the theano.function. Hope this helps!

[@Thomas-hv4bi]

Interactive plotting, 1_linear_regression.py example plt.ion() .... .... for i in range(100): plt.clf() plt.plot(trX,trY, 'o') plt.plot(trX, trX * w.get_value(),"--", linewidth=2) plt.draw() print("i:", i) for x,y in zip(trX, trY): train(x,y)

[@tatianaefremova5154]

I am waiting for the next video !!!!!!!1

[@hatrixyesa]

Anyone know how the animated plots were generated at 20:10 ?

[@Eltimjohn]

+Bhaarat Sharma Save image files of the weights at the end of each epoch, then combine images as animated gif. ( You can created animated gifs at gifmaker.me/ )

[@dratgulit2261]

You are a master

[@ivandeyanov2204]

Thank you! That's exactly what I needed.

[@ProkopHapala]

is it possible to download the python source codes for this tutorial somewhere?

[@slatervictoroff3268]

Absolutely! You can find all of the source code here: github.com/Newmu/Theano-Tutorials

[@philtoa334]

Thanks.

[@Daejii96]

I am so lost lol. I really want to understand this though.

[@alexandermcfarlane2109]

See below code to create an output in pyplot for each iteration for the first graph I found Alec Radford 's github useful too: Newmu/Theano-Tutorials import matplotlib.pyplot as plt # create pyplot graphfig = plt.figure() ax = fig.add_subplot(111) ax.axis([-1.5,1.5,-3,3]) plt.ion() plt.scatter(trX, trY) plt.show() # iterate through the data 100 times and train the model on # each example of input and output pairs for i in range(100): for x, y in zip(trX, trY): c = train(x, y) # print to plot if ax.lines: del ax.lines[0] ax.plot(trX, trX * w.get_value(), "-", linewidth=2, color='red') plt.title('slope is currently {0:4.2f}, cost is currently {1:6.4f}'.format(float(w.get_value()),float(c))) plt.draw() time.sleep(0.5/(float(i)+1)) print 'plotting for {0:0d}/100'.format(i) raw_input()

[@alexandermcfarlane2109]

Logistic Regression Plot: Adjust the loop to define this class and create a plot instance just before looping. The use plot.update(w.get_value()) to update for each iteration. Lets you see how the Logistic Regression learns the data. class plot_output(object): def __init__(self, X): self.X = X self.n = X.shape[1] # number of digits self.pixels = (28,28) # pixel shape self.fig, self.ax = plt.subplots(1,self.n) plt.ion() # cycle through digits from 0-9 # X input vector is reshaped for each 10 digits # to a (28,28) vector to plot self.img_obj_ar = [] for i in range(self.n): wi=X[:,i] # w = weights for digit wi=wi.reshape(*self.pixels) self.ax[i].imshow(wi,cmap=plt.cm.gist_gray, interpolation='gaussian', aspect='equal') self.ax[i].axis('off') self.ax[i].set_title('{0:0d}'.format(i)) plt.tick_params(\ axis='x', # changes apply to the x-axis which='both', # both major and minor ticks are affected bottom='off', # ticks along the bottom edge are off top='off', # ticks along the top edge are off labelbottom='off') plt.tick_params(\ axis='y', # changes apply to the y-axis which='both', # both major and minor ticks are affected left='off', right='off', # ticks along the top edge are off labelleft='off') plt.show() def update(self, X): # cycle through digits from 0-9 # X input vector is reshaped for each 10 digits # to a (28,28) vector to plot for i in range(self.n): self.ax[i].cla() wi=X[:,i] # w = weights for digit wi=wi.reshape(*self.pixels) self.ax[i].imshow(wi,cmap=plt.cm.gist_gray, interpolation='gaussian', aspect='equal') self.ax[i].axis('off') self.ax[i].set_title('{0:0d}'.format(i)) plt.draw() plot = plot_output(w.get_value()) # train on mini-batches of 128 examples for i in range(100): for start, end in zip(range(0, len(trX), 128), range(128, len(trX), 128)): cost = train(trX[start:end], trY[start:end]) print 'Training Run: {0:0d}, Accuracy: {1:6.3f}%'.format( i, np.mean(np.argmax(teY, axis=1) == predict(teX))*100.) plot.update(w.get_value())

[@RoxanneDSwaim]

This video is good. simplifies it quite well. But even deep learning is a cell net, so its still a neural network, in my principles. just like drum and bass, its a kind of electronic music, and I call it techno, a toy word of the general group.

[@ГеоргиКандев]

Python is my favourite developing I can help you if have questions

[@johnallard2429]

where is a link to the code?

[@Eltimjohn]

+John Allard github.com/Newmu/Theano-Tutorials

[@minorinxx]

thank you! thank you! thank you!

[@brhgvishal]

The load package is not working...can't get the data.

[@slatervictoroff3268]

The load package doesn't actually download the data for you, it only refers to an existing data file. If you're looking to download the data you can find it here: yann.lecun.com/exdb/mnist/. You should place these files in the /media/datasets/mnist directory. In the future if you have any issues, you should probably raise them directly on the Github issues page as you'll get a better response time. You can find that page here: github.com/Newmu/Theano-Tutorials/issues If you wanted to open a pull request that would automatically download mnist for you that would certainly be appreciated.

[@slatervictoroff3268]

I actually went ahead and opened a pull request that can help get the data. You can look at it here: github.com/Newmu/Theano-Tutorials/pull/2

[@micahturner1441]

Great work!

[@davidschrock5762]

Really useful!

[@GiohLSilva]

someone run code of convolutional network ? i got some errors, like theses: Traceback (most recent call last): File "/home/giovanni/PycharmProjects/Theano-Tutorial/5_convolutional_net.py", line 78, in l1, l2, l3, l4, py_x = model(X, w, w2, w3, w4, 0., 0.) File "/home/giovanni/PycharmProjects/Theano-Tutorial/5_convolutional_net.py", line 44, in model l1a = rectify(conv2d(X, w, border_mode='full')) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/nnet/conv.py", line 151, in conv2d return op(input, filters) File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 507, in __call__ node = self.make_node(*inputs, **kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/nnet/conv.py", line 628, in make_node "inputs(%s), kerns(%s)" % (_inputs.dtype, _kerns.dtype)) NotImplementedError: The image and the kernel must have the same type.inputs(float32), kerns(float64) can anybody help me?

[@mattgenkin7929]

Giovanni Silva I haven't run the code myself, but it looks like some of your variables are being stored as 32-bit floats and some as 64-bit floats. In particular, it looks like the image is in float32 and the kernel is in float64. The kernel is initialized using the init_weights function, which uses the floatX function defined above it, which uses the value stored in theano.config.floatX. Try adding the line " theano.config.floatX = 'float32' " after your imports so that the kernel is stored as float32.

[@GiohLSilva]

I will see, thank you all

[@internationalscholarhw]

Good !!!

[@harshitsharma6781]

Can we use theano without having a GPU... ?

[@Eltimjohn]

+Harshit Sharma Yes you can

[@valken666]

Subtitles? please

[@valken666]

Thank you, it's appreciated.

[@mccloud35]

Just auto-generate them with a NN

[@slatervictoroff3268]

4 people missed the upvote button. Or maybe they just think Elon Musk is right.

[@michaelcastillo6828]

⌨️

[@isospin8u]

Not so good...hard to hear what the presenter is saying much of the time and he makes a lot of assumptions in his explanations.

[@billkillernic]

So this kiddo is the head of research at indico data solutions? lol

[@slatervictoroff3268]

Astonishing that someone so young has progressed so far, no?

[@billkillernic]

yup

[@billkillernic]

***** well thats not a big deal especially if you are much older now (so you hacked lets say the 80's norad "firewall" :P )