#046

Academics think of themselves as trailblazers, explorers - seekers of the truth.
Any fundamental discovery involves a significant degree of risk. If an idea is guaranteed to work then it moves from the realm of research to engineering. Unfortunately, this also means that most research careers will invariably be failures at least if failures are measured via “objective” metrics like citations. Today we discuss the recent article from Mark Saroufim called Machine Learning: the great stagnation. We discuss the rise of gentleman scientists, fake rigor, incentives in ML, SOTA-chasing, "graduate student descent", distribution of talent in ML and how to learn effectively.
With special guest interviewer Mat Salvaris.
Machine learning: the great stagnation [00:00:00]
Main show kick off [00:16:30]
Great stagnation article / Bad incentive systems in academia [00:18:24]
OpenAI is a media business [00:19:48]
Incentive structures in academia [00:22:13]
SOTA chasing [00:24:47]
F You Money [00:28:53]
Research grants and gentlemen scientists [00:29:13]
Following your own gradient of interest and making a contribution [00:33:27]
Marketing yourself to be successful [00:37:07]
Tech companies create the bad incentives [00:42:20]
GPT3 was sota chasing but it seemed really... "good"? Scaling laws? [00:51:09]
Dota / game AI [00:58:39]
Hard to go it alone? [01:02:08]
Reaching out to people [01:09:21]
Willingness to be wrong [01:13:14]
Distribution of talent / tech interviews [01:18:30]
What should you read online and how to learn? Sharing your stuff online and finding your niche [01:25:52]
Mark Saroufim:
marksaroufim.substack.com/
robotoverlordmanual.com/
/ marksaroufim
/ marksaroufim
Dr. Mathew Salvaris:
/ drmathewsalvaris
/ msalvaris
Pod version: anchor.fm/machinelearningstre...

[@stalinsampras]

I'm loving the editing in the intro section, Good job guys

[@AICoffeeBreak]

I'm wondering how long did it take @Machine Learning Street Talk to make it? Do you have any insider stats for us?

[@AICoffeeBreak]

One thing I also wonder: How to tag channels/people from mobile?

[@shivapundir7105]

I loved the way Mark smiled and nodded his head in agreement/delight. I kept smiling looking at his face.

[@DerekWoolverton]

Malcolm Gladwell has this excellent podcast episode called "The Big Man Cant Shoot", and it goes into how we can know that doing something will be better for us and sometimes we will still not do it. It came down to the fact that society requires a certain amount of conforming for it to survive, and as social creatures we are affected by this to varying degrees. A very few could care less what others think, and those people often run against the grain (or are the crackpots wandering around the edges). A great many people got into the software field because it seemed like a chance to work with inherently logical constructs; but that was a lie, and to our horror we discover that our successes or failures are going to depend on how well we work with and understand people (including ourselves).

[@popupexistence9253]

God bless YT for recommending this

[@AICoffeeBreak]

May the Algorithm be with you.

[@Shikhar_]

Dang! Loved Mark's perspective on risks, incentives, and going independent. Also, Dr. Mathew's presence was so delightful. (40 mins in) Thank you for the amazing guests!

[@akshayshrivastava97]

Great article, thanks Mark Saroufim. I think the community needs to hear more opinions of the sort expressed in your article.

[@henrikvendelbo1117]

This is one of the best discussions in ages

[@henrikvendelbo1117]

This is relevant for so much more than ML. It’s excellent advice on how to approach a technology “career”. Details matter on how to exactly express yourself online, how much open source to publish and the value of original work

[@markryan2475]

Awesome discussion. Mark Saroufim's ML Stagnation article was like a great sermon - parts of it had me nodding in agreement (praise for Keras and fastai), and other parts had me squirming with discomfort (DL is an empirical field). The video really examined the article in an interesting way, with interesting, accessible ideas, great pacing, and noteworthy quotes ("Share the wrong thing and people will correct you" and "not everyone is interesting"). I could have easily listened to another couple of hours of this discussion.

[@drhilm]

I loved this episode, thanks for introducing Mark Saroufim. haven't read his excellent content before.

[@LiaAnggraini1]

this kind of talk that we never discuss with other peers in ML or lecturers/ professors. looking forward to more intriguing topics :)

[@joshuapatterson5095]

The most inspirational video I've watched in a while!!! I am audibly saying 'yes - Yes - YAAASSSS'

[@Baron-digit]

Very interesting episode and conversation!

[@eismscience]

Very interesting discussion and very valuable service you're putting out. Thank you.

[@bethcarey8530]

I share your thoughts Mark. The incremental improvement that is necessitated by todays system of incentives and leader boards, has lead us to hype and sameness of academic papers. Unfortunately, there has been no 'move the needle' improvement in products for enterprise & customers in NLU. Stuck with todays AI of chatbots & intent based NLU, we can't have dependable goal oriented conversations. Instead, the system's incentives leads to more 'chat' without substance.

[@Arthaxninja]

I really enjoyed the whole episode, and I think I just saw this in a critical moment of my career when I am trying to balance salary / interest... Thank you. You made my day

[@brofessorsbooks3352]

love the intro! how has this page not gotten more viewership!

[@TaimurAhmadRana]

I wish I had a mentor like that telling me to stop worrying about being right, doing something is better than thinking for a long time to get to that elusive perfection

[@AdrianCosmaAI]

Just found out about your channel. Amazing content, a breath of fresh air from all the bullcrap in the ML community.

[@abby5493]

Very interesting! Great video 😍

[@GagandeepSingh-rz7ue]

13:33 - 14:00 is the "region" where I laughed the hardest. Your delivery style is awesome. Subscribed the channel.

[@CristianGarcia]

Hey Tim, I'd love to hear more about the engineering aspects of ML at some point. I feel the opinions here reflect mostly the worries of top researchers. That said, I enjoyed the discussion. Cheers.

[@MachineLearningStreetTalk]

Watch this space, I've been sitting on an ML DevOps episode with AWS for quite a while

[@CristianGarcia]

@@MachineLearningStreetTalk awesome! MLOps is not a sexy topic but its a bit more real as of what you ACTUALLY do in the industry 😅.

[@MachineLearningStreetTalk]

@@CristianGarcia Exactly! I also think it's the single most misunderstood thing in our space

[@kevalan1042]

great conversation! ++

[@minma02262]

Very good street talk. Refreshing to hear more pragmatics in this field.

[@deepblender]

This is easily my favorite episode so far! Thanks a lot!

[@AICoffeeBreak]

As Mark Saroufim said in the podcast: It is easily relatable in one way or another to anyone in the field. I think this is also what makes this episode so enjoyable.

[@zZthefishZz]

Mark's stagnation blog post really resonated with me but I hadn't come across any of his work before so was still a bit unsure about his credibility. Having seen him in this show I have to say I was very impressed by his insights and perspective and definitely had no reason to doubt him! Found this very motivating as well! Thanks for the great guests keep up the great work!

[@demohub]

Really enjoyed watching this. Mark is super brilliant 👏👌

[@AlbertHulk]

You guys should take this content all the way up to some sort of "academic ESPN"

[@nikoshazaridis2766]

This is academic ESPN what do you mean? :P

[@stalinsampras]

"Academic ESPN" lol😂😂, We need to have someone like Stephen A Smith in that to spew off something nonsensical and we need someone like Max Kellerman to combat that and give well thought out analysis on the subjects discussed Now I would pay to watch that thing!! Put it on a Academic Pay per view lol

[@machinelearningdojowithtim2898]

Brilliant 😃

[@AlbertHulk]

And the debates would be on whether Schmidhuber is greater than Bengio, just as Jordan vs Lebron

[@someonespotatohmm9513]

About the willingness to be wrong: I think context matters. If you have a project you work on with other people, offering ideas/ suggestions that might be wrong is not a bad thing. If you didn't think about any ideas that are wrong you probably didn't explore enough ideas. But when saying things in general, like on a blog or news articles, saying something wrong means that it could get picked up by ppl and taken as true by them. It is still on them to make sure their information is correct. But I always feel that when talking about something I should, or am expected to know what I am talking about, it is better to stay on the side of caution and clearly communicate when I am not sure about stuff.

[@trinhvg6099]

Great talks guys

[@dfountain2000]

Targetting incremental improvements is probably a side-effect of marketing limitations. You can't really market something unless the use case is able to be communicated effectively but generally profitable use cases for new techniques are an emergent phenomenon or occur by chance exploration. So obviously the vast majority of improvements that get developed are those where the researcher/engineer/etc. guesses a potential improvement that can lead to a better result in a well-defined setting. Unfortunately the only improvements that researchers can see will lead to a better result are almost always marginal, unless you're Terence Tao or someone similar who can see many steps ahead reliably. Whether this is dynamically efficient is a whole other argument.

[@ratsukutsi]

It seems that this kind of debate like what Mark proposes (and also Sara or Kenneth) have a much deeper root on the economics of society. ML is quite of a privileged field in terms of financial rewards, and as it saturates with people coming from every other niches because of the very disparity presented everywhere else, the reward channels tend to have their bottlenecks narrowed. There is also something related to the way education is being presented, which is changing the value universities have as a prime factory of refined workers. The experience of learning something technical is far different than it was 10 years ago, when the only way to get the information was to be in the classroom of a certain post-graduate course at a certain institution. Much of that information is not so exclusive anymore. I don't know, I've never seen so much technological innovation and a culture with so many agents and platforms available for the average citizen, it seems to me that society have never been such an interesting place. The problem is that there is no money for everyone, or maybe it is better to phrase that by saying that the problem is that money is way to exclusive. That's the real deal.

[@sandraviknander7898]

You talk a little bit about interviews and I just got to say that I was on an interview at Ericsson in Sweden which had just that format you describe where I was asked to prepare a presentation on an interesting project I had done. And basically it was an hour of me nerding out on all the aspects of the project I liked and we talked about the process and what could be improved or made differently, it was defiantly one of the best interviews I had. As far as I understand it that’s the direction interviews are heading here at least.

[@willd1mindmind639]

The industry for ML is primarily based around the big silicon valley companies who are able to sell such algorithms as part of a growing ML ecosystem of data, models, APIs, chips and frameworks that produce that salary for those interested in learning it. It is almost like such models that require large amounts of compute and data are inherently beneficial to this era of big data and big data companies because that is the problem space they are working in and the solutions make sense in that context. And we all know that once something becomes adopted as a norm in industry it is hard to change because of all the investment. So there is a natural synergy there. And the academics that proposed these models are happy because their work is getting the attention and traction to allow them to continue their research. But in terms of whether or not a particular algorithm is the most efficient, smartest or really "intelligent" that is not the really the priority vs producing profits in providing services related to managing and making sense of large data sets.

[@aibonsai7045]

WHAT? Making videos this good is illegal

[@flaskapp9885]

amazing liked todays episode

[@ahmedmagdy2932]

this is a great talk, touches some serious points, it makes me worry honestly, I mean this "media person" idea will require you simply to work every day more than what is required to stand out. i feel ML/AI is very competitive in that you spend a lot of time working, blogging, summarizing,... I don't think it should be like that in other fields, why I can't just work normally 8~10 hours and then go back home enjoying other stuff without needing to worry about my standing out level, of course, self-development is necessary but not like that.

[@AICoffeeBreak]

ML is the field where you cannot go home and just relax because this is when GPT-3, GLOM, DALL-E, CLIP, etc are coming out. Partner: Come, dinner is getting cold! Me: I can't, ML papers cool faster.

[@AHermas]

agreeeeeed

[@SisypheanRoller]

In future videos I think it would be helpful to consider adding links to the articles being referenced in the video (e.g., the reddit post)

[@soccerplayer922]

I'm so happy to hear these ideas get more attention. The success of of AI will be the cause of its own hibernation. In some sense the more money being poured into these disciplines the less they resemble the discipline.

[@j.erickson8571]

When Mark talks I need to press the pause button to make notes. His brain operates at least three times faster than a normal human.

[@billgalen9014]

Starting an internal blog today summarizing street talk. Using hand drawn mind map to begin. My safety net - I can retire anytime I want. And I have a rich data source at my day job. Thinking about Medium as a blog host for future public facing content (not street talk). Is Mark happy with the service?

[@marksaroufim]

I like substack better because you get people's email, so people will stick with you even if substack no longer ends up being popular. Also gives you a good onramp to paid subscribers

[@ChrisOffner]

Does anyone know what Yannic uses for his background replacement? Is he chroma-keying with a proper greenscreen behind him or does he have some image segmentation software that's vastly superior to that of the other three guys?

[@MachineLearningStreetTalk]

Yannic actually has a greenscreen, we wrote some software to segment the background -- github.com/ecsplendid/rembg-greenscreen

[@dinoscheidt]

Why in hell did I see this channel just now?!? Subscribed

[@LucasDimoveo]

Are there any analogs for this in the physical sciences and engineering? It seems like the main way to a senior research scientist position in Materials Science, for example, is through a PhD

[@mattizzle81]

I can definitely tell from interacting with GPT-3 that to say it is just verbatim copying its dataset was just hubris. I'm not an expert but it wasn't hard for me to find examples from it that definitely were not just "copied" from the data. Some word puzzles that some very qualified researchers dismiss GPT-3 would be unable to answer correctly, it does answer when the prompt, temperature, etc is set up correctly

[@CrazyAssDrumma]

I wanna break into ML so bad :( Working towards experience on RL with my game also, but its so difficult

[@maxpayne8708]

@25:10 what name does he say regarding forks of bert?

[@MachineLearningStreetTalk]

amitness.com/2020/05/git-log-of-bert/ (Amit Chaudhary). Yannic Kilcher already translates papers into this format for us, we just need him to make us an API 😃

[@DasGrosseFressen]

I love how you guys talk of science but mean ML research... What if I told you, there is a vast universe existing outside ML 😎 BTW: I agree that many of the issues are found in sience in genera.

[@eelcohoogendoorn8044]

Im never quite sure what to think about ML researchers complaining about lack of reproducibility. I suppose its good to be self critical of ones own field. But having dabbled in a few fields, let me give some perspective. You know that publishing working code, happens in essentially no other field, right? Like nontrivial code you can clone and expect to run without putting months of work into compiling / deciphering it? Where did you see that before in science? I never did. You know that using standardised published and public datasets is something the ML community pretty much invented, right? It does not exist anywhere else, to decent approximation. You know spending 1k on cloud TPU's to train a published model isnt actually a lot of money right? Ok perhaps spending 100k on cloud TPU's to independently reproduce is a lot in some sense. But its still a bargain compared to redoing 4 years of phd biology or physics experiments, the raw data, materials or equipment you dont have access to. (aka stuff that never happens; reproducibility has always been more of an ideal than something people actually do in practice). Not that the reproducibility of biology papers is something to strive for; its good that people are setting a higher bar. And the bar for effective reproducibility is never high enough. Its just that the comparison to other implied to be superior fields of science is kinda ridiculous, since ML is at least 1 or 2 order of magnitude ahead of them, in terms of any measure of effective reproducibility.

[@TheReferrer72]

What fields are you talking about? Maths?, Physics? Biology? At the moment machine learning seems like a branch of Computer science, If you can't show working code/containerised/micro-service version of your research how did you do the research in the first place?

[@eelcohoogendoorn8044]

​@@TheReferrer72 Im talking about all of these fields. Even within branches of computer science, inability to show working code is depressingly common. Try finding a reference implementation of recent papers in fluid dynamics or other simulation/computational fields, from researchers that consider themselves computer scientists (this field tends to be a mix of lots of backgrounds). 9/10 you will be left completely empty handed; and the other 1/10 it will be some c++ without docstrings or makefiles, with a massive 'it works on my machine' factor, which is months of work removed from reproducing any figures in the paper. ML is an absolute exception, given that 9/10 publications come with working code. Thats 1/10 too few; but still light years ahead of any other field.

[@TheReferrer72]

@@eelcohoogendoorn8044 That's because ML are toy examples, where as the papers I have read in computer science have direct commercial applications.

[@eelcohoogendoorn8044]

​@@TheReferrer72 commercial applications probably apply a role in peoples mind, but its honestly does not make a lot of sense. I cant think of a single CS paper that actually had commercial potential in isolation; best case scenario your method ends up in free software like blender, as far as impact of simulation papers go. I can think of countless ML papers that would have had a comparatively strong case for commercial exploitation; yet most ML researchers just actually publish without excuses about commercial interests. Its a culture thing mostly. Do you care about making your work available in a reproducible manner? Does your field force you to, in order to be taken seriously? If the answer is no, its easy to find excuses.

[@TheReferrer72]

@@eelcohoogendoorn8044 An example I have read over 100 Bin Packing research papers because I write software that uses heuristics to solve this NP=Hard/ NP Complete problem ( one of the many reasons i'm interested in AI). The authors would be ignorant to post working code.

[@1wisestein]

This “weak papers” conclusion surprises me 0% and is the attitude that had me backing away 2 years ago. Also, the professors expecting you to “volunteer your time” to the lab was a little messed up.

[@AICoffeeBreak]

Hey, this is interesting. How did this attitude discourage you exactly, could you please elaborate on that?

[@user-or7ji5hv8y]

Does it make sense to also have shorter clips like Lex from MIT?

[@MachineLearningStreetTalk]

Thanks for the suggestion. I have never been a huge fan of that. We have chapters too. Yannic has made a clips channel now for his stuff. We probably will do this when the channel grows bigger.

[@citiblocsMaster]

38:00 I think the real issue is that Mark created a product no-one wanted (granted, no way to promote it doesn't help) but I don't think communication was the root issue in this case

[@mikeCavalle]

5 years of retirement from 50 years of programming - randomly youtubing MI AI Math languages API's -- yep you got a point here.

[@karimtit4897]

Thanks for this great content! "What should you read online and how to learn? Sharing your stuff online and finding your niece ?" I guess you mean "niche"!

[@TimScarfe]

Whoops! Yes! 😀

[@HoriaCristescu]

1:08:12 I used to say that you could fit all the ML people in a single bus in my country.

[@machinelearningdojowithtim2898]

First! 😎💥✌ What do you think folks, is Mark on the money with this one?

[@stalinsampras]

uhh, Second😅... This time i have pressed the bell icon, I hope it gives me an edge

[@afafssaf925]

He's exaggerating a bit in general. It's a "great stagnation" that has been going on for... 2-5 years? That's nothing. By that criterion, most fields of science are stagnating.

[@AbdennacerAyeb]

AGI will never be solved without giving a definition about what is intelligence.

[@doppelrutsch9540]

How much time have you spent researching different definitions of intelligence before giving this opinion? Because there are.

[@AbdennacerAyeb]

@@doppelrutsch9540 There is a lot, but what I meant is a mathematical definition. Like when we define gravity or electromagnetic fields... We should have a mathematical definition because math is the language of nature.

[@doppelrutsch9540]

@@AbdennacerAyeb Sure and there are people doing exactly that.

[@scottmiller2591]

I've learned over the years to repress most of the opinions that Mark Sarofim expressed in "Stagnation," simply because otherwise I sound like "Old Man Yells At Cloud."

[@IngolfDahl]

I see some similarities between the opinions of Mark Saroufim and of Stephen Wolfram.

[@shivasrightfoot2374]

I am curious about who these young AI hotshots are that are vaguely mentioned several times in the video. NAME NAMES!!! I am sure they would love the visibility.

[@DasGrosseFressen]

Wait. Why shoukd ML move towards software engineering? It is ceartaily good that tools are developed for ML applications, but you need more than just good software engineering skills for ML...

[@tacopacopotato6619]

Stagnation redefined to mean "the plebs are at it".

[@DasGrosseFressen]

Be yourself, but only if you are likable and interesting 😂

[@ratsukutsi]

Mark looks like Yuval Harari's little brother

[@AICoffeeBreak]

🤣🤣🤣

[@666andthensome]

GPT-3 and DALL-E are definitely cool, but I'm surprised you guys couldn't find "avocado chairs" -- and how it made those isn't surprising anyhow, since certain shapes work nicely (if you cut an avocado in half, and add four legs, it's automatically chair-like). But in cases where the elements don't fit together so nicely, the systems seem to fail or produce accidentally awesome surrealist art. But "understanding" is always absent. I asked GPT-3 if suicide was a good option, and it agreed that it was. Boy, I hope it didn't have "understanding" otherwise, it's a real jerk! I suspect what we'll find is that for any topic where there are "computable" solutions that are known, we can automate the hell out of that, and optimize. But whenever we run into issues that are murky, such as certain ethical situations, or emotional states (which can be a black box since even humans don't always understand their subjective emotional experiences), systems like this will simply fail. And maybe that's fine. Maybe we should think of these systems as tools, as powerful extensions of our cognition, but in a different class from us mere humans, which is fine and good.

[@satychary]

True. DIRECT (via experience, via 3D body in our 3D world) understanding (not just objective things or computable things) is what's been missing for 60 years (since the dawn of AI).

[@NextFuckingLevel]

Fight me as you want, but i tell ya scaling laws work, and openAI SHOULD scale up CLIP DALL-E and GPT3 until it reach the heaven

[@adavirro]

There's no M in FAANG dude.

[@MachineLearningStreetTalk]

😃 Tim used to work at Microsoft, so we count the M

[@telesniper2]

didn't age well