Deep Learning for Natural Language Processing (Richard Socher, Salesforce)

The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (www.bayareadlschool.org) and full live streams below.
Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:
1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - • Foundations of Deep Le...
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - • Deep Learning for Comp...
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - • Deep Learning for Natu...
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - • TensorFlow Tutorial (S...
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - • Foundations of Unsuper...
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - • Nuts and Bolts of Appl...
7. Deep Reinforcement Learning (John Schulman, OpenAI) - • Deep Reinforcement Lea...
8. Theano Tutorial (Pascal Lamblin, MILA) - • Theano Tutorial (Pasca...
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - • Deep Learning for Spee...
10. Torch Tutorial (Alex Wiltschko, Twitter) - • Torch Tutorial (Alex W...
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - • Sequence to Sequence D...
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - • Foundations and Challe...
Full Day Live Streams:
Day 1: • Video
Day 2: • Video
Go to www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

[@angelachikaebirim8894]

How lucky are we to be present when all of these advances are being made ! Very exciting!

[@TheJuliaLaRocheShow]

@LexFridman, would love to see an updated conversation between you and Richard Socher!

[@kevinurban1016]

Some links for quick reference: * 1997: Hochreiter & Schmidhuber: Long Short-Term Memory: citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.676.4320&rep=rep1&type=pdf * 2010: Mikolov et al: Recurrent Neural Network Based Language Model: www.isca-speech.org/archive/archive_papers/interspeech_2010/i10_1045.pdf * 2013: Mikolov et al (word2vec ref): Efficient Estimation of Word Representations in Vector Space: arxiv.org/pdf/1301.3781.pdf * 2013: Mikolov et al (word2vec ref): Distributed Representations of Words and Phrases and their Compositionality: papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf * 2013: Mikolov et al (word2vec ref): Linguistic Regularities in Continuous Space Word Representations: www.microsoft.com/en-us/research/publication/linguistic-regularities-in-continuous-space-word-representations/?from=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F189726%2Frvecs.pdf * 2013: Mikolov et al: Exploiting Similarities among Languages for Machine Translation: arxiv.org/pdf/1309.4168.pdf * 2013: Socher et al: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank: www.aclweb.org/anthology/D13-1170/ * 2014: Cho et al: On the Properties of Neural Machine Translation: Encoder-Decoder Approaches: arxiv.org/abs/1409.1259 * 2014: Chung et al: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling: arxiv.org/abs/1412.3555 * 2014: Graves et al: Neural Turing Machines: arxiv.org/abs/1410.5401 * 2014: Irsoy & Cardie: Opinion Mining with Deep Recurrent Neural Networks: www.aclweb.org/anthology/D14-1080/ * 2014: Irsoy & Cardie: Deep Recursive Neural Networks for Compositionality in Language: papers.nips.cc/paper/5551-deep-recursive-neural-networks-for-compositionality-in-language * 2014: Kalchbrenner et al: A Convolutional Neural Network for Modelling Sentences: arxiv.org/abs/1404.2188 * 2014: Kim: Convolutional Neural Networks for Sentence Classification: arxiv.org/abs/1408.5882 * 2014: Le & Mikolov: Distributed Representations of Sentences and Documents: proceedings.mlr.press/v32/le14.pdf * 2014: Pennington et al (GloVe ref): Glove: Global Vectors for Word Representation: www.aclweb.org/anthology/D14-1162/ * 2014: Sutskever et al: Sequence to Sequence Learning with Neural Networks: papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks * 2014: Weston et al: Memory Networks: arxiv.org/abs/1410.3916 * 2014: Zaremba et al: Recurrent Neural Network Regularization: arxiv.org/abs/1409.2329 * 2014: Zaremba & Sutskever: Learning to Execute: arxiv.org/abs/1410.4615 * 2015: Antol et al: VQA: Visual Question Answering: openaccess.thecvf.com/content_iccv_2015/html/Antol_VQA_Visual_Question_ICCV_2015_paper.html * 2015: Gal & Ghahramani: A Theoretically Grounded Application of Dropout in Recurrent Neural Networks: papers.nips.cc/paper/6241-a-theoretically-grounded-application-of-dropout-in-recurren * 2015: Grefenstette et al: Learning to Transduce with Unbounded Memory: papers.nips.cc/paper/5648-learning-to-transduce-with-unbounded-memory * 2015: Hermann et al: Teaching Machines to Read and Comprehend: papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend * 2015: Huang et al: Bidirectional LSTM-CRF Models for Sequence Tagging: arxiv.org/abs/1508.01991 * 2015: Sukhbaatar et al: End-To-End Memory Networks: papers.nips.cc/paper/5846-end-to-end-memorynetworks * 2015: Tai et al: Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks: arxiv.org/abs/1503.00075 * 2015: Weston et al: Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks: arxiv.org/abs/1502.05698 * 2015: Zhang et al: Structured Memory for Neural Turing Machines: arxiv.org/abs/1510.03931 * 2015: Zhou et al: Simple Baseline for Visual Question Answering: arxiv.org/abs/1512.02167 * 2016: Andreas et al: Neural Module Networks: openaccess.thecvf.com/content_cvpr_2016/html/Andreas_Neural_Module_Networks_CVPR_2016_paper.html * 2016: Andreas et al: Learning to Compose Neural Networks for Question Answering: arxiv.org/abs/1601.01705 * 2016: Kumar et al: Ask Me Anything: Dynamic Memory Networks for Natural Language Processing: proceedings.mlr.press/v48/kumar16.pdf * 2016: Merity et al: Pointer Sentinel Mixture Models: arxiv.org/abs/1609.07843 * 2016: Noh et al: Image Question Answering Using Convolutional Neural Network With Dynamic Parameter Prediction: openaccess.thecvf.com/content_cvpr_2016/html/Noh_Image_Question_Answering_CVPR_2016_paper.html * 2016: Yang et al: Stacked Attention Networks for Image Question Answering: openaccess.thecvf.com/content_cvpr_2016/html/Yang_Stacked_Attention_Networks_CVPR_2016_paper.html * 2017: Zilly et al: Recurrent Highway Networks: arxiv.org/pdf/1607.03474.pdf

[@kozzuli]

Thank you for sharing!

[@richarddownes9037]

Thanks for this!

[@MelvinKoopmans]

Fascinating work! Makes me wonder how you can best integrate this with speech as an input. You could use a speech-to-text model and just use the text as input, but maybe it would work better if you take the spectrogram directly (spectrogram -> CNN -> RNN -> output to episodic memory).

[@kenichimori8533]

Thanks Point

[@maosun7474]

In the GRU section, the update gate and reset gate nearly do the same thing, i.e. choose what to keep and what to forget. So why do we need both of them? If rt=0 and zt=1, the information can still be kept according to the equation, right?

[@HsenagNarawseramap]

Correct, update gate has precedence over reset (a reset is also an update). So if zt is 1, that means you are not going to update the state. If zt is less than 1, that means you are going to update, and then the value of of reset will determine to a certain extent what part of the previous state is going to be remembered. It is not very intuitive. Reset should be called "reset a little".

[@Smoshfaaaaaaaaaaan]

Lex - invite him! Greetings, Linus

[@danlan4132]

in 17:34 what is u_w means ?

[@riccardoandreetta9520]

nice but not for dummies ... and couldn't see the neural networks here, seems to be more related to baysean networks

[@riccardoandreetta9520]

Anh Nguyen OK...started 1.5 months ago to study some AI, lot of work still to do

[@lizeyu4444]

最后一个哥们的英语让人好笑，不过anyway，作为一个从视觉转到nlp的人，当初提了一个相同的问题，算是帮我解答了疑问。

[@shubhamchandel1742]

Used a language model to translate: The last guy's English is funny, but anyway, as a vision to nlp, had raised the same question, be considered to help me answer questions.

[@lukelee4510]

Guess NLP is the future. Maybe a milestone for AI complete that interacts with human

[@robosergTV]

yep, vision is basically solved