#86 - Prof. YANN LECUN and Dr. RANDALL BALESTRIERO - SSL, Data Augmentation [NEURIPS2022]

Support us! / mlst
Yann LeCun is a French computer scientist known for his pioneering work on convolutional neural networks, optical character recognition and computer vision. He is a Silver Professor at New York University and Vice President, Chief AI Scientist at Meta. Along with Yoshua Bengio and Geoffrey Hinton, he was awarded the 2018 Turing Award for their work on deep learning, earning them the nickname of the "Godfathers of Deep Learning".
Dr. Randall Balestriero has been researching learnable signal processing since 2013, with a focus on learnable parametrized wavelets and deep wavelet transforms. His research has been used by NASA, leading to applications such as Marsquake detection. During his PhD at Rice University, Randall explored deep networks from a theoretical perspective and improved state-of-the-art methods such as batch-normalization and generative networks. Later, when joining Meta AI Research (FAIR) as a postdoc with Prof. Yann LeCun, Randall further broadened his research interests to include self-supervised learning and the biases emerging from data-augmentation and regularization, resulting in numerous publications.
Pod version: anchor.fm/machinelearningstre...
Note: We have another full interview with Randall, which we will release soon as part of a show focussed on Spline Theory of NNs.
TOC:
[00:00:00] LeCun interview
[00:18:25] Randall Balestriero interview (mostly on spectral SSL paper, first ref)
References:
[Randall Balestriero, Yann LeCun] Contrastive and Non-Contravention Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
arxiv.org/abs/2205.11508
[Randall Balestriero, Ishan Misra, Yann LeCun] A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments
arxiv.org/abs/2202.08325
[Bobak Kiani, Randall Balestriero, Yann LeCun, Seth Lloyd] projUNN: efficient method for training deep networks with unitary matrices
arxiv.org/abs/2203.05483
[Randall Balestriero, Richard G. Baraniuk]A Spline Theory of Deep Networks
proceedings.mlr.press/v80/bal...
Learning in High Dimension Always Amounts to Extrapolation [Randall Balestriero, Jerome Pesenti, Yann LeCun]
arxiv.org/abs/2110.09485
• #61: Prof. YANN LECUN:... [MLST special edition show on extrapolation and this/spline paper]
[Mathilde Caron et al] DINO - Emerging Properties in Self-Supervised Vision Transformers
arxiv.org/abs/2104.14294
[Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton] A Simple Framework for Contrastive Learning of Visual Representations (SIMCLR)
arxiv.org/abs/2002.05709
MLST show with Simon Kornblith: • #032- Simon Kornblith ...
[Yann LeCun] A Path Towards Autonomous Machine Intelligence Version
openreview.net/pdf?id=BZ5a1r-...
[Patrice Y. Simard, Yann A. LeCun et al]
Transformation Invariance in Pattern Recognition - Tangent Distance and Tangent Propagation
link.springer.com/chapter/10....
[Kaiming He et al] Masked Autoencoders Are Scalable Vision Learners
arxiv.org/abs/2111.06377
[Radford et al] Whisper - Robust Speech Recognition via Large-Scale Weak Supervision
cdn.openai.com/papers/whisper...
RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank [Quentin Garrido, Randall Balestriero, Laurent Najman, Yann Lecun]
arxiv.org/abs/2210.02885
[David Silver, Satinder Baveja, Doina Precup, Richard Sutton] Reward is Enough
www.deepmind.com/publications...

[@AICoffeeBreak]

Just wow. I love it how you let everybody get a taste of the best bits at NeurIPS. ❤

[@arnokhachatourian8928]

I'm glad we're starting to see so many previous interviewees again, especially when they are such interesting thinkers. Good to hear iteration on previous conversation and their new ideas.

[@paxdriver]

I can't believe how many specifics Tim remembers from all these papers and conversations. He's so meticulous in his prep even for seemingly ad hoc interviews

[@MachineLearningStreetTalk]

@@paxdriver Cheers Kristopher! These meetings were all ad hoc, but luckily in many cases the MLST shows we did were still fresh in my memory. We covered LeCun's paper on our Chomsky show.

[@MachineLearningStreetTalk]

@@paxdriver By the way - I a kicking myself that I forgot to ask him about his thoughts on the limits of language i.e. see www.noemamag.com/ai-and-the-limits-of-language/

[@user-ru5dj7pl8j]

Great interviews!

[@paxdriver]

Yan is so so funny sometimes lol "the purpose of reinforcement learning should be to minimize the use of reinforcement learning, because reinforcement learning is so damn inefficient; *pardon my French*" 😂

[@maloxi1472]

It's no accident that a life well-lived and good sleeping habits tend to positively feedback into each other

[@simonstrandgaard5503]

Thank you for covering this conference.

[@bissbort]

Great content and production quality once again! Keep up the great work.

[@gr8ape111]

Very lucky to interview someone who is never, ever wrong!

[@RoyceFarrell]

Thanks Tim legendary 🙏

[@johntanchongmin]

Really inspired by Yann Lecun! Hope we can find a better way to do RL in the future!

[@xXKM4UXx]

This interview is the definition of 'ML street talk' aha. With engineering, biology etc its right or wrong. When it comes to A.I it is a debate.

[@FergalByrne]

The duality is due to the duality between normals to the manifold and vectors in the tangent space. The latter is more efficient as any other data point (or an augmented data point) will approximate geodesics on the manifold. Most efficient (as Yann mentions) is use of previous or next frames which actually are on geodesics. Both the neocortex and the Feynman Machine use this succession and are thus orders of magnitude more efficient than these methods.

[@opusdei1151]

Wow mind blowing

[@bytesizedbraincog]

Is reinforcement learning really inefficient? I really liked the architecture where reinforcement learning is brought in the scope of text generation, especially chatGPT model, it looked like it had some plausible reasoning because of reinforcement learning.

[@fyodorminakov6092]

Nearest neighbor does extrapolate... that's what the k value is for. The problem with KNN is the k is just not all that predictive, because it's fixed for all, and even worse, each N is unweighted in a given context, so there is massive information loss. Converting KNN to a neural net fixes the problem.

[@nafizabdoulcarime5082]

Does anyone know from where the clip at 17:17 is taken ?

[@MachineLearningStreetTalk]

See twitter.com/imtiazprio/status/1635350552967254016 - we have an unreleased interview with Imtiaz

[@nafizabdoulcarime5082]

@@MachineLearningStreetTalk Thank you for your kind answer. I look forward to this upcoming interview with Imtiaz

[@TheReferrer72]

When are you guys going to have a play with ChatGPT? Would love if you include Dr. Walid Saba if you do.

[@MachineLearningStreetTalk]

We are publishing a show with Walid later today, we didn't discuss ChatGPT in particular but let's say, he has shifted his position a little bit in response to testing these recent LLMs :)

[@andybaldman]

People making the machines smarter without realizing what the repercussions will be.

[@andybaldman]

@@AB-wf8ek That's flawed logic. Electricity doesn't have the power to be conscious, more intelligent, and more powerful than humans. We should have stopped viral research before creating covid and accidentally letting that escape into the wild. This will be much worse, and kill far more people.

[@charlesb.1969]

Come on Yann ! Even the French accent is distortioned . 😂 Tell us instead what is the real purpose of all this IA predictions.I n real life IA will be sadly used in order to control private life of people around the world against their will or real consentement .

[@michaelhartjen3214]

He is copying from jeff hawkins with his HTM's. Where the key part is the Temporal Adjancency Matrix.

[@Extys]

It's much older than that and he himself says "these ideas have been floating around for a long time".

[@MachineLearningStreetTalk]

Are you Schmidhubering Yann on Jeff's behalf?! Clearly similarities I will grant - but very different in many ways i.e. Jeff's system predicts complete objects at each level rather than progressive levels of representation (see our video on Jeff)

[@johntanchongmin]

@@MachineLearningStreetTalk Is Schmidhubering now a verb haha?

[@johnnypeck]

Great interviews!