How to Learn from Little Data - Intro to Deep Learning #17

One-shot learning! In this last weekly video of the course, i'll explain how memory augmented neural networks can help achieve one-shot classification for a small labeled image dataset. We'll also go over the architecture of it's inspiration (the neural turing machine).
Code for this video (with challenge):
github.com/llSourcell/How-to-Learn-from-Little-Data
Please subscribe! And like. And comment. That's what keeps me going.
More learning resources:
kzbin.info/www/bejne/eau0hISVZb9gh7c
arxiv.org/abs/1605.06065
futuristech.info/posts/differential-neural-computer-from-deepmind-and-more-advances-in-backward-propagation
thenewstack.io/googles-deepmind-ai-now-capable-deep-neural-reasoning/
Join us in the Wizards Slack Channel:
wizards.herokuapp.com/
And please support me on Patreon:
www.patreon.com/user?u=3191693
Follow me:
Twitter: sirajraval
Facebook: sirajology Instagram: sirajraval
Signup for my newsletter for exciting updates in the field of AI:
goo.gl/FZzJ5w
Hit the Join button above to sign up to become a member of my channel for access to exclusive content! Join my AI community: chatgptschool.io/ Sign up for my AI Sports betting Bot, WagerGPT! (500 spots available):
www.wagergpt.co

[@xPROxSNIPExMW2xPOWER]

I learn from little data all the time, it's called college education

[@xPROxSNIPExMW2xPOWER]

Lemmas are the bane of my everything lol kmn

[@swapanjain892]

That's not learning,thats called "How do you waste your time"?

[@userou-ig1ze]

ProSurviver you're too optimistic

[@marvlousdasta2566]

and most of this little data are noise

[@Aer0xander]

Get little data for going in debt

[@khanfor]

You make ML so accessible and convenient. Keep it up Siraj! You're an awesome educator!

[@SirajRaval]

thanks Abdullah!

[@PsRafael84]

I have to say: im not a programmer, im a doctor with programming as some short of a hobby. But your videos actually make me rethink about my profession! The way you teach, the quality of your videos! Man! Thank you for sharing your knowledge. Hope for more videos. Success!!

[@SirajRaval]

thats aweosme thanks!

[@pdsnk1]

Thanks a lot for your videos! They are so many! I even don't have enough time to watch them all!

[@SirajRaval]

thanks Albert

[@SchultzC]

Someone give this man a medal. These videos have been amazing.

[@markph0204]

Mix of learning and humor nicely done.

[@secondsandthings]

Woo! Was waiting for this for so long. Great video Siraj

[@SirajRaval]

thanks!

[@avatar098]

Dude, Siraj, you helped me with one of my kaggle projects that contains little data (like 8-10 examples/label). This is my first time being exposed to concepts like neural turing machines :) You have earned a subscriber

[@RoxanaNoe]

Great great video. You totally rock Siraj!!!

[@bernardogalvao4448]

I missed watching your videos :') Recently I decided to finally read the paper (which I learned about because of your video back then) after a year long of putting it off... thankfully Siraj comes to the rescue breaking it down in more digestible pieces :) keep up the good work

[@chroniclesofave2137]

Great video Siraj... You're like my role model in my Deep Learning journey and ML in general. I've learnt so much within such a short time. Keep up the great work and hope you reach far. Watched all your videos by the way.

[@TheOjayyy]

loved the course, thanks, keep up the good work man!

[@SirajRaval]

thx ojay will do

[@michaelyadchuk4989]

Congrats on the completion of the course. I'm looking forward to taking your Udacity Nanodegree!

[@SirajRaval]

awesome thx

[@I77AGIC]

such a unique channel. and always great info. thanks!

[@SirajRaval]

thanks!

[@bobcrunch]

And the crowd goes wild! Siraj spikes the ball and the crowd pours out on the field!

[@SirajRaval]

thx Bob!

[@florentinalexandruiftimie8214]

Siraj, you are freaking awesome!!!

[@SirajRaval]

thx florentin!

[@misterj9817]

man, i love this guy ! I think i am NOT the only one who likes the video first then watches it.

[@SirajRaval]

love u

[@sryan2640]

Great channel dude, gonna fucking tear through all your ML videos when my exams are over

[@waxwingvain]

Finally! One shot!

[@SirajRaval]

finally

[@DojoGiantShanNY]

Thanks Siraj, really helpful vid. you sings awesome, by the way:)

[@barbolo]

Great video, Siraj. I'd like to suggest you to explain Generative ladder networks (REGEL) for problems with small labeled datasets available, but also with big unlabeled datasets.

[@larryteslaspacexboringlawr739]

thank you for deep learning video

[@datasciencewithr1039]

Thanks for the video! Get well soon Siraj!

[@SirajRaval]

thx. get well? im not sick tho lol

[@andreamarkos]

brilliant!

[@chrisoffersen]

Can't wait until I understand this!

[@chrisoffersen]

For context: I'm only on Submission 2 in Udacity's Deep Learning ND

[@SirajRaval]

so close

[@olfmombach260]

I have three questions: 1. So there is really nothing more than the RNN (controller) that uses its memory database to achieve the best results? 2. Is this implementable in keras? 3. Should I be more focused on keras (for efficiency) or tensorflow (for flexibility or accessibility)? Are there things that can be done only in tensorflow?

[@HmmContemplates]

I love it, never change Siraj. Keep on learning though

[@SirajRaval]

thx i wont and will

[@franklinbradfield9131]

Very powerful concepts here. It's a lot to package into a 9 minute video, so I've been going over the arXiv paper to get a better grasp of the fundamentals. If you have a moment, can you comment on the difference(s) between one-shot learning and transfer learning?

[@brofessorsbooks3352]

you are so good! god bless u!

[@SirajRaval]

thx u 2

[@AnanthNatarajanS]

YOU ROCK.

[@vaer-k]

Can you please share links in the description to the journal articles you highlight in your videos?

[@bliss766]

Please do a video on zero-shot-learning.

[@elisayao668]

Hi Siraj, this tis a great one! Could you do one in information leakage? e.g. processing testing dataset using training set statistics.

[@troy3393]

SIRAJ! Not sure if you remember me commenting a few months back about starting my Data Science degree and a bunch of Math being involved but I need some more guidance. We have a project where we're exploring simple regression and multivariable regression models on a dataset of our choosing. I want to try take it a step further. I have a bunch of individual player data for a sports league (seasons 2012 - 2016), and I have a source of the data so I could probably grab the 2017 results with a little wget magic. ANYWAY, my goal is to predict a future win or loss when one team is playing another (sounds like a 'simple' classification problem - win or loss) taking as many individual player statistics into account as possible. Is there any easy ways I can just throw the data into a pile, tell the computer a WIN is good and a LOSS is bad (a DRAW is okay); and then get it to spit back out some correlations that it found with a win or a loss when specific teams are up against one another? I need a little bit of a starting point - what should I be looking at? Would love to shock my class and lecturer with a semi-intelligent model!

[@BrunoJuncklaus]

Note that you also need pytest, matplotlib, Pillow and scipy to run the code. And also make sure to add a __init__.py in both MANN/ and MANN/Utils/ folder. And also make sure you create a file ~/.matplotlib/matplotlibrc there and add the following code (if you're using virtualenv): backend: TkAgg And even though when you're trying to sample the self.character_folders it raises an error, because the variable is empty.

[@ionmosnoi]

in order to make it work for tensorflow v1, is it enough to change batch_matmul to matmul, mul to multiply, concat_v2 to concat and add labels, logits keywords for softmax_cross_entropy_with_logits?

[@SirajRaval]

thx bruno yes this code was tough AF

[@LaSalaProductions]

if I want to create a Q&A system that answers from a book chapter, should a use seq2seq or DMN?

[@logicboard7746]

Listening to Siraj = time well spent

[@SirajRaval]

thx logic

[@danieldeleon5804]

Siraj, I would like to hear a bit more about the life cycle of a deep learning paper. Can you make a video or point me in the direction where I can learn more?

[@SirajRaval]

ill make a vid on this

[@UsmanAhmed-sq9bl]

Siraj: great tutorial as always. After this course, what is inline ? Please give detail of your upcoming videos.

[@SirajRaval]

thx figuring it out

[@shivambhardwaj5599]

Hey Siraj, I just want to know how I can learn Machine learning for autonomous cars from scratch and what language should I choose. Thanks Your Videos are really intersting.

[@calebkirschbaum8158]

Could you use this to train a network to make a network that is supposed to make networks using the original network as the data? Basically, Make 'A' be able to make 'B' that can make 'C', by using 'A' as the data for 'A'. Then using all subsequent programs use that for more data to become better? Or is that way too complicated for one shot learning?

[@artr0x93]

Great video as always! Is there a discord channel for Siraj-related things? If not I would really like to see one created, so that ppl can go there and discuss the challenges. Discord is really good at bringing communities together in my experience.

[@SirajRaval]

we use slack see vid desc thx

[@nitink4245]

siraj please let us know if we can use , one shot learning or mann for fraud or loan defaulter prediction ??? . if so please make a comprehensive video ?

[@vineetkothari2839]

As you show it is used for image classification can it be used with CNN and GAN if yes how ?

[@SirajRaval]

hasn't been tried, good experiment!

[@ngrislain]

Yes, the gaussian is mathematically convenient, but it is also special in the sense it maximises Entropy for a given (mean, std-err) pair, among so many special properties...

[@SirajRaval]

good points

[@MaduSudananB_N]

So who taught you machine learning in the first place? Are you self learnt ? P.S : Thanks for all these videos, they are amazing.

[@cidroldan6732]

Hello, great vid! also where can i see the code for this lesson's challenge winner?

[@SirajRaval]

Thanks! none this video

[@siddharthyadav5682]

Most of the times, I watch your videos just for recreational purposes.

[@manalmohammed27]

Siraj, is it feasible to implement it with text classification ?

[@vigneshyaadav6322]

@Siraj: please do make vedios about robotics u may have a grate. knowledge as u had been in a start up

[@JakeDownsWuzHere]

[@SirajRaval]

thanks Jake!

[@macot79]

Doesn't the central limit theorem kinda indicate that using a Gaussian distribution can often times be a good choice of distribution when trying to model a lot of phenomenon?

[@SirajRaval]

indeed just saying we r still figuring out things

[@ebtesamh9624]

Can we use this for translation

[@Skythedragon]

I sadly doesn't take one shot for me to understand this, But great video, I've been wanting to train models on tiny datasets!

[@SirajRaval]

thanks!

[@Skythedragon]

+Siraj Raval also, what's the difference between a MANN and DNC?

[@kingpopaul]

Has anyone tried the Hierarchical Temporal Memory (HTM) algorithms from Numenta, I'm curious about their tech but I don't know if they work better than the more common neural networks solutions.

[@SirajRaval]

numenta is kind of a joke since they hardly publish but make big claims

[@kingpopaul]

Yeah they publish a little but I'm keeping an eye on them, their github is active and they are gradually adding stuff to their nupic libraries. Maybe once they have implemented enough algorythms it'll be interesting to compare and see if their claim are true. Though for now they mostly seem to focus on anomaly detection for their products, which is pretty lame in my opinion. But if their 'novel' approach is any good, it could turn out to be interesting. Thanks for the reply kind sir!

[@theosalmon]

The claim is that they're reverse engineering the neo-cortex. If that becomes successful, it will be an invaluable contribution. Practical applications are minimal so far, but the mechanisms they propose are very different and interesting.

[@Renardbardhi1]

Siraj nice work. I would like to ask for learning from a small data set. I am using Corell5k data set that contains 50 classes each class contains 100 images and its small data for CNN to achieve a high score of accuracy. How can I increase accuracy? Thank you very much

[@charansai9421]

nice video siraj and?? can u suggest a good book for learning algorithms??

[@allanng78]

Hi Sir Siraj, I have downloaded and run the code in github. It seems to take forever. Is it suppose to be like that? Please help..... I am running this in a vn in an i7 laptop.

[@yeahokaysureyoubet]

Oh MANN!

[@SirajRaval]

vietMANN

[@ebtesamh9624]

Hi, can i apply this code to text data

[@aryopradiptagema6677]

This is what i was waiting for!!! BTW, mathematically speaking, any kind of memory augmented neural networks actually can be used even to solve a problem with a moderate amount of data (not too big, not too small), right? Any thought about the performance comparison between other deep learning algorithms and MANN in task with a moderate amount of data?

[@SirajRaval]

awesom thanks and yes. sounds like a good experiment

[@calebkirschbaum8158]

Can you not combine the 2 so as to make deep learning really effective at lower levels, and it just keeps getting better with more data you throw at it?

[@elfamosodpereira]

Would this work for a chatbot project?

[@chinmayeejoshi2119]

Was he trying to start a the equivalent of getting rick rolled with the Barbra Streisand thing?

[@xzhao5824]

Hi,Siraj.where can I find the winner code for this week's competition?

[@Lindomcgud]

How big is you dataset?

[@machapuis]

yaayyy

[@SirajRaval]

yay

[@gabrielsb15]

it doesnt work for python 3.5 right?

[@bonesfelipe93]

Just like Oracle Hahahaha :')

[@SirajRaval]

lool

[@vaer-k]

Here is the link to the Neural Turing Machine deepmind article arxiv.org/abs/1410.5401

[@AkashMishra23]

You should link the arXiv/ Research Papers in the Video in the Description, It'll be a lot easier to get to them...

[@whatthefunction9140]

Behave like an excited neuron.

[@TidalMaker]

I died at the Pork Pig homage.

[@SirajRaval]

haha

[@bf2825]

Hi, Siraj. I want to ask about why people in ML always discourage people from learning Deep Learning?

[@Pietrabentivi]

HI LEVEL CHANNEL !!!!!

[@AhmedShariffGplus]

"just like oracle" lol dude :D

[@alisyariatimohdsalleh1848]

The video and github code not the same. Simplified version i think.

[@Arty66Click]

>That Chappelle meme tho) lmao)

[@SirajRaval]

lolz

[@alonkellner5375]

Could you make a video about ZERO DATA LEARNING too? A very interesting topic in the realm of meta-learning.

[@terryr9052]

Can you explain that more?

[@alonkellner5375]

Terry R Zero data learning is the concept of learning to do new given tasks without ever training on it (therfore the name zero data learning) using the knowledge learned from previous tasks, for example learning to classify images of digits to 0-9, and then being asked to classify to new classes like letters A-Z, and with the knowledge learned from the digits classify the new letters images without ever being trained on letters at all. The paper about zero data learning is in this link: www.cs.toronto.edu/~larocheh/publications/aaai2008_zero-data.pdf

[@terryr9052]

Ah! So a specific type of transfer learning... That is interesting - thanks for the link.

[@SirajRaval]

nice i should

[@walterhugolopezpinaya5641]

Who won this Coding Challenge?

[@diegoantoniorosariopalomin4977]

hey Siraj , i was asigned to be the representative of programing 101 in my college , any tips

[@diegoantoniorosariopalomin4977]

what to post on the google + group i created and such

[@diegoantoniorosariopalomin4977]

there is an exam on monday btw ( any resources you know ?)

[@SirajRaval]

thx diego, make a git commit everyday

[@Eddie-rf4tp]

First half was great. I don't know why, the singing kills it for me. Suggestion, sing at the end.

[@SirajRaval]

thx eddie. i gotta do me. love u

[@Eddie-rf4tp]

Haha, fair enough. Love you the same 🤓

[@shutinghuang1398]

Love your video! I have no background in computer science or coding, but managed to train my own image classification through your tutorial! Can you do a tutorial on semantic segmentation? something like this is really fun: github.com/warmspringwinds/tensorflow_notes/blob/master/fully_convolutional_networks.ipynb

[@synetic707x]

Programs that learn how to learn... it's getting out of hand

[@htconezm4501]

just like Oracle lol

[@SirajRaval]

;)

[@God-po4rt]

I... I don't understand.

[@SirajRaval]

lmao

[@terryhintz4521]

memes overflow

[@SirajRaval]

so many

[@roshanrane3112]

Misleading title alert! This is about meta learning and one shot learning and it's definitely not an "introduction to deep learning" topic

[@moazim1993]

Eww the code is in Python 2

[@oneroadgoodwalk]

nice video but the memes get very distracting

[@SirajRaval]

thx for feedback

[@andrewkoh6124]

I think the memes are a fresh breath of air! otherwise it would be kinda dry. thanks siraj!

[@user-zi3nb1bv3e]

Who won this Coding Challenge?