Efficient Computing for Deep Learning, Robotics, and AI (Vivienne Sze) | MIT Deep Learning Series

Lecture by Vivienne Sze in January 2020, part of the MIT Deep Learning Lecture Series.
Website: deeplearning.mit.edu
Slides: bit.ly/2Rm7Gi1
Playlist: bit.ly/deep-learning-playlist
LECTURE LINKS:
Twitter: / eems_mit
KZbin: / @miteemsviviennesze
MIT professional course: bit.ly/36ncGam
NeurIPS 2019 tutorial: bit.ly/2RhVleO
Tutorial and survey paper: arxiv.org/abs/1703.09039
Book coming out in Spring 2020!
OUTLINE:
0:00 - Introduction
0:43 - Talk overview
1:18 - Compute for deep learning
5:48 - Power consumption for deep learning, robotics, and AI
9:23 - Deep learning in the context of resource use
12:29 - Deep learning basics
20:28 - Hardware acceleration for deep learning
57:54 - Looking beyond the DNN accelerator for acceleration
1:03:45 - Beyond deep neural networks
CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- Twitter: / lexfridman
- LinkedIn: / lexfridman
- Facebook: / lexfridman
- Instagram: / lexfridman

[@lexfridman]

I really enjoyed this talk by Vivienne. Here's the outline: 0:00 - Introduction 0:43 - Talk overview 1:18 - Compute for deep learning 5:48 - Power consumption for deep learning, robotics, and AI 9:23 - Deep learning in the context of resource use 12:29 - Deep learning basics 20:28 - Hardware acceleration for deep learning 57:54 - Looking beyond the DNN accelerator for acceleration 1:03:45 - Beyond deep neural networks

[@gggrow]

Looking forward to watching this, but shouldn't the Vladimir Vapnik lecture be coming first?

[@createchannel8815]

Me too. Invite her again.

[@createchannel8815]

Great talk. The Speaker Vivienne was clear and concise. Very informative.

[@NomenNescio99]

Thank you for sharing the lecture, this is the type of content I really enjoy.

[@gonzalochristobal]

thank you lex, the amount of information you already shared is invaluable, eternally grateful

[@JonMcGill]

I used to be a Field Apps engineer for telecom, and she's certainly correct about the power problem with respect to chip technology. Very likeable lecturer!!

[@burkebaby]

Impressive amounts of information delivered by this lady!. To watch a such high densely packed informative video I had to take more than few breaks.

[@colouredlaundry1165]

I am not an expert in the field of Vivienne Sze, however, she was an extremely good lecturer. Every concept was extremely clear.

[@samuelec]

Impressive amounts of information delivered by this lady!. To watch a such high densely packed informative video I had to take more than few breaks. I wonder how she managed to go through those 80 slides so fast and if there is someone that watched it all in one go without lose the focus !

[@jayhu6075]

Many thanks for sharing to the people that not study can afford at the MIT. Respect.

[@colouredlaundry1165]

Agree with you. Respect.

[@pierreerbacher4864]

The density of neurons in this channel is incredibly high.

[@colouredlaundry1165]

Vivienne is incredibly smart, it is a pleasure to listen to her.

[@davidvijayramchurn1860]

Ironically, if you call someone 'dense' in English slang, it would imply the opposite.

[@summersnow7296]

Excellent lecture 👏👏👏. Things that we don’t usually think about as a ML practitioner but highly important. Great insights.

[@nikhilpandey2364]

I was researching about this on my own. I have been doing the network pruning wrong. I wouldn’t mind a hit in accuracy if my latency budget were met but now I think I can be far more frugal with the decrease in accuracy. Thanks a lot.

[@JousefM]

Thanks for a rather "exotic" topic I need to learn about as an AI newbie, much appreciated Lex!

[@warsin8641]

I love this I will rewatch everything when I'm older and hopefully understand better and deeper I'm only a junior in high school 😖

[@tomfillot5453]

Maybe start by looking at Crash Course computer science. They give a good overview of how a computer actually works, and should give you more context for what are the different types of memory, operation and stuff like that. Then 3Blue1Brown has an excellent video series on neural networks. A lot of understanding comes from calculus, but fortunately he also has an excellent video series on that !

[@alterna19]

Warsin I like your avatar

[@thusspokeshabistari]

Try to watch the video slowly in segmented chunks, and then write down what you understand and don't understand about the particular segment of the video(s), and then you can Google what you don't understand and then get back to viewing the video again later.

[@ayushdutta8050]

Haha .. senior year here 😅 . . AI has no age cutoff thank god haha

[@UglyG82]

Great stuff Lex. Thank you !

[@UglyG82]

And Thank you Vivienne for the fantastic insight

[@XCSme]

Great video and an interesting problem. Why stop at architecture? What about using different materials for specialized DNN hardware? Maybe using some lower power transistors that are less accurate but good enough for inference. I don't think the brain neurons are always 100% accurate and consistent, but the brain seems to be somewhat fault tolerant.

[@merlinmystique]

Thank you, every video you post is incredibly useful. Though, it is really hard to enter this field from scratch: everything you learn forces you to go learn thousands other things, it gets really frustrating sometimes. I hope in time this will go better

[@BlackHermit]

FastDepth is really interesting. Could be useful for many people.

[@Happy-wi7ml]

Brilliant thank you

[@ganeshdongari7098]

Excellent

[@machinimaaquinix3178]

This was a great talk, thank goodnees KZbin has a .75 speed mode. She talks fast!

[@colouredlaundry1165]

She has very high knowledge throughput: 10Gb information per second xD

[@Soulixs]

thanks lex

[@punyaslokdutta4362]

Trade off between number of filters on the 3D Convolution and a 4D Convolution ? . Convolution is a matrix operation (w*Imap+ Bias). RELU Activation is mostly used to provide non-linearity . I feel the number of Filters is needed to see higher abstracted stuff . For instance, The initial layer of a CNN understands pixels based information primarily for edges , cuts, depths. The layer following it understands shape , structures. The further layers help us understand semantic meaning of eyes, skin, ears, nose, face. But, How will the model perform when instead of multiple filters , we having more layers . That is, the information in filters in stuffed inside the CNN layers. Or is it done for easing computation while training ?

[@kitgary]

Genius!!!!!

[@adamsimms8528]

I'm trying to imagine how this structure will become half relevant as we move into UltraRAM which is as close to as fast as DRAM but NOT volatile like memory stick type RAM. What are the implications if the data can be laid out and accessed in place where it is saved. Suddenly the whole structure is no longer useful.

[@masbro1901]

1:10:37 its 100x faster than FPGA ?? wth, wow, thats blowing my mind, i thought designing custom hardware for specialized algorithm on FPGA its the fastest way on the planet, is it really??

[@minhongz]

So essentially power consumption and speed are almost equivalent for AI chips. Does anyone know what architecture Tesla chips employ?

[@paulrautenbach]

While watching this I was seeing parallels with what I know about the Tesla chips from their Autonomy Investor Day presentation. The Tesla chips were designed with an energy budget in mind and so address many of the same things. One advantage the Tesla chips have is they do not need to be general purpose - so, to a large extent, only need to support a single architecture or configuration. In some cases the Tesla chips avoid storage and retrieval of intermediate data by passing outputs directly to inputs via hardware channels between successive computational stages implemented as separate hardware. A large proportion of the Tesla chips are used for static memory to implement what she called global memory. This avoids going off-chip for most values.

[@samoha0812]

This is what I exactly wanted to hear. Thank you. I expected to hear about how AI chip is designed to minimize energy consumption attracted by her presentation title but lot of content is focused on computing algorithm rather than hardware design but great presentation providing comprehensive understanding about computing and energy consumption. Thank you

[@jessenochella4309]

REEVERSIBLE computing uses less power! But you need new chip architecture and algorithms.

[@jkobject]

what about neuromorphic computing?

[@alexanderpadalka5708]

🗽

[@fayssalelansari8584]

not bad

[@vuththiwattanathornkosithg5625]

Tesla hardware 3.0???

[@dapdizzy]

I don’t think the content presented by those youngsters is on par with the talks to the legends. You know what I mean.