The moment we stopped understanding AI [AlexNet]

Thanks to KiwiCo for sponsoring today's video! Go to www.kiwico.com... and use code WELCHLABS for 50% off your first month of monthly lines and/or for 20% off your first Panda Crate.
Activation Atlas Posters!
www.welchlabs....
www.welchlabs....
www.welchlabs....
www.welchlabs....
Special thanks to the Patrons:
Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti
Welch Labs
Ad free videos and exclusive perks: / welchlabs
Watch on TikTok: / welchlabs
Learn More or Contact: www.welchlabs....
Instagram: / welchlabs
X: / welchlabs
References
AlexNet Paper
proceedings.ne...
Original Activation Atlas Article- explore here - Great interactive Atlas! distill.pub/20...
Carter, et al., "Activation Atlas", Distill, 2019.
Feature Visualization Article: distill.pub/20...
`Olah, et al., "Feature Visualization", Distill, 2017.`
Great LLM Explainability work: transformer-ci...
Templeton, et al., "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", Transformer Circuits Thread, 2024.
“Deep Visualization Toolbox" by Jason Yosinski video inspired many visuals:
• Deep Visualization Too...
Great LLM/GPT Intro paper
arxiv.org/pdf/...
3B1Bs GPT Videos are excellent, as always:
• Attention in transform...
• Transformers (how LLMs...
Andrej Kerpathy's walkthrough is amazing:
• Let's build GPT: from ...
Goodfellow’s Deep Learning Book
www.deeplearni...
OpenAI’s 10,000 V100 GPU cluster (1+ exaflop) news.microsoft...
GPT-3 size, etc: Language Models are Few-Shot Learners, Brown et al, 2020.
Unique token count for ChatGPT: cookbook.opena...
GPT-4 training size etc, speculative:
patmcguinness....
www.semianalys...
Historical Neural Network Videos
• Convolutional Network ...
• Perceptron Research fr...
Errata
1:40 should be: "word fragment is appended to the end of the original input". Thanks for Chris A for finding this one.

[@WelchLabsVideo]

Thanks to KiwiCo for sponsoring today's video! Go to www.kiwico.com/welchlabs and use code WELCHLABS for 50% off your first month of monthly lines and/or for 20% off your first Panda Crate.

[@samuelspace101]

AI today is growing exponentially, just curios, do you think we will ever hit a peak where innovation on AI will start to flat out, or hit a wall, and if so where and when do you think AI will hit its peak. You kind of skimmed over this in the end, i just wanted a bit of a longer explanation.

[@KWifler]

Oops, I farted 4o

[@michaelmangion6187]

Was really keen to sign up for a crate for my daughter, but after 10 minutes of glitches on their system I just gave up. Not your fault of course, but you might want to let them know that their system is pants.

[@jeffg4686]

could have done it with ASIC a long time ago. Just living out THEIR best life possible first...

[@ryvyr]

These days non-adsense being placed some ways into video, rather than with mutual consideration at very front/back/both where plenty people and myself would still watch, is instant skip/cliff off - though I wish success

[@EdgarVerona]

30 years ago, I used to work with an older guy who retired from IBM. I was barely out of high school, and he used to tell me that neural networks were going to change the world once people figured out how to train them properly. He didn't live to see his dream become reality unfortunately, but he was totally right.

[@spartaleonidas540]

Same except the guy was at Toronto and his name was Hinton

[@EdgarVerona]

@@spartaleonidas540 guy I knew was named David Modlin. I wonder how many folks who had their prime years in the 60's and 70's saw this coming? I wish they had been able to see it. I suppose some of them might have lived to see it. Crazy to think about.

[@squamish4244]

@@EdgarVerona Hinton's most important contributions came from the 80s onwards, but he has lived to see it, for one. He was working on neural nets in the 70s as a postdoc. It's all happened well within a human lifetime, is the crazy part.

[@EdgarVerona]

@@squamish4244 Nice, that is very cool. Glad to hear he's still kicking! The guy I knew helped create handwriting recognition software in the 70's. It is crazy to think that someone could see basically the dawn of modern computing and also its progression to this crazy time we're in.

[@squamish4244]

@@EdgarVerona Ray Kurzweil did too, but he's obsessed with mind-uploading, cryonics and resurrecting digital copies of his father etc. which is distracting, and he has trouble with being challenged on the practical implications of his predictions. He was right about the computing revolution but he's also a very strange dude. Hinton was running circles around him recently in a debate when both of them were onstage.

[@JustSayin24]

That real-time kernel activation map was life-changing. If, whilst editing these videos, you've ever questioned whether the vast amounts of effort are worth what amounts to a brief, 10s clip, just know that it's these moments which have truly stuck with me.

[@prabhatp654]

Ikr, shows the hard work of this guy and that is something I respect.

[@JoseJimeniz]

I wanted to say this too. You actually did it, you make that animation. That is an amazing thing you've done, you've really added to the sum of human knowledge. The amount of effort must have been phenomenal. Really: thank you. Nobody else has done this. I know the effort of huge, but I'd love more even on just Alexnet. Animations on creating the node activation image generation. I'd love one of Resnet

[@amarissimus29]

The irony being, of course, that the script popped out of ChatGPT in about three seconds, editing by submagic slightly more, and images by stable diffusion in much less. But I agree, those few moments are worth it.

[@kapsi]

@@JoseJimeniz While I'm sure it took a lot of work, someone else already did most of the work for the Activation Atlas.

[@marklorinczy4412]

Same, this was truly eye opening

[@drhxa]

I've been in the field for 10 years and never had anyone describe this so clearly and visually. Brilliant, thank you!

[@TheStickofWar]

same here (9 years)

[@TheRealMcNuggs]

Would you say it is still worth it going into the field (studying AI) even after progress is made so incredibly fast nowadays that after the maybe 3-4 years of studying everything could have already changed again?

[@drhxa]

@@TheRealMcNuggs I say, if you love it (or have a strong interest) then absolutely! It's been changing quickly since I started, but the underlying fundamentals stay the same 👍

[@GRIM787]

3blue1brown made a whole gen ai series which goes much deeper and visualises things better, I do recommend to have a look, really interesting stuff

[@ineedpills]

Im still confused 😭

[@somnvm37]

"one way to think about this vector, is as a point in 4096 dimentional space" give me a minute, I now gotta visualise a 4096 dimentional space in my head.

[@autohmae]

Are you done yet ? 🙂

[@Muhammed_English314]

"One way to think about a point in 4096 dimensional space, is as a vector"

[@adamrak7560]

high dimensional spaces are crazy. A hypercube with the sides size=2, would have absolutely enormous surface and volume in 4096 dimension. size = 1, volume: 1 size = 1.01, volume, approx 501587856585103488.

[@RandomGeometryDashStuff]

@@adamrak7560what does volume mean for non-3d thing?

[@thenonsequitur]

Easy, image a 3-dimensional space and pretend it's 4096 dimensions. I mean, that's basically what the visualizations in the video are doing.

[@samuelspace101]

Most people think AI is a brand new technology, while in reality there have been studies on Computer Neural Networks all the way back in the 1940s, that's insane.

[@louis-dieudonne5941]

But the real issue is that only now has computing power become strong enough to support everything, allowing research ideas to be realized into reality, and truly transforming these ideas into technologies with such astonishing effects.

[@samuelspace101]

@@louis-dieudonne5941 makes you think, what are we studying now that will only be possible years in the future because of the lack of resources.

[@empathogen75]

It’s new in the sense that neural networks are relatively inexpensive and for the first time broadly applicable to a wide range of tasks.

[@davidaugustofc2574]

@@empathogen75 Its just a popularity phase, KZbin paid for itself when it was rapidly gaining users, we'll have Adobe level subscriptions in the future.

[@gljames24]

​@@louis-dieudonne5941Not just hardware, but data as well.

[@khanghoutan4706]

Fun fact, the kernels used in vision models work pretty much the same way as how our retinas perceive objects. In a similar structure, our eyes have cells that perceive edges at certain angles, then as shapes, then as objects in increasing abstraction.

[@PallasTurrets]

only edge detection occurs in the retina, anything more complex than that happens higher up in the various visual areas of the brain

[@pyropulseIXXI]

They don’t at all; you are confusing a low level explanation for how our eyes really work Humans don’t work like the kernel at all; biology is far more efficiency and works in ways we don’t even understand yet

[@khanghoutan4706]

@@PallasTurrets Whoops I forgot to mention but yeah, more complex stuff still occurs in the brain. Thanks for correcting me

[@ВалентинТ-х6ц]

Their similarity is less than between an airplane and a bird.

[@schok51]

​@@ВалентинТ-х6цmeaning? Do you have a more detailed understanding of human vision to share to compare and contrast ?

[@kellymoses8566]

Computers not being fast enough to make a correct algorithm practically usable reminds me of Reed-Solomon error correcting codes. They were developed in 1960 but computers were too slow for them to be practical. They went unused until 1982 when they were used in Compact Discs after computers had become fast enough.

[@jimktrains0]

RS codes were used on the Voyager probes in 1977. CDs were the first large scale usage. Your basic point is still true: it took a while for computers to be complex enough to use them.

[@T3sl4]

Bayesian models have followed a similar path; the basic idea is so fundamental as to be trivial, but actually using it in practice requires a high level (uh, I don't know what the big-O complexity is -- quadratic? worse?) of detail and thus computation to truly harness. The parameters might be trivial (individually, or conceptually), but there are so many of them for a problem of modest scale that it's only recently we've made much use of it.

[@kellymoses8566]

@@jimktrains0 I should have specified first wide-spread use.

[@ron5948]

Logix programming same prediction, eill be viable in a yeR year and I will do it ???😮❤😂🎉🇨🇭😘💶💶💶🍆🍑🍆🥑⛔⛔⛔🪬🤣😅🏳️‍🌈✡️💪🏾👯♂️♂️🔯✡️🔯👬🕎♀️⛔

[@afterthesmash]

It has always been an easy decision tree. Will the interesting case fit in system memory at all? It not, wait for the next system refresh. Can I tolerate the latency? Predicting tomorrow's weather a week from now is a good example of not being able to tolerate the latency. If it fits in memory and I can tolerate the latency, am I willing to pay for the computer time? I recall hearing stories in the 1980s about a power station with an entire Vax 11/780 devoted to running an FFT kernel on generator shaft vibration. There was no legal way to ship a replacement shaft. They had barely been allowed to truck in the first one over existing roads. Hence they spent the moon looking after the one they had.

[@ernestuz]

I was working with deep neural networks at the university during the late 90s, the main issue that stopped all progress was the use of a kind of functions between layers (the sigmoid as activation function), this effectively stopped the learning backpropagating from the output layers and limiting how many layers you can use (the problem is called the vanishing gradient). Once people rediscovered ReLU (it was invented in the early 70s, I believe, but I think the inventor published it in Japanese, so it went unnoticed) deep neural networks became possible. High computation needs were only a problem if you wanted real time or low latency, those days we used to leaving the computer calculating during nighttime to get something next day.

[@chiyembekezomaunjiri3278]

Thank you for all the work you did.

[@dest5218]

Thank you for all your work, cant imagine doing all this back then

[@yannickhein9788]

While this video perfectly explained how the networks work during recognition, I don't understand how they are actually training all the layers. Does anyone have a similar good source about teaching neural networks / backpropagation?

[@The_Quaalude]

Bro was working on a toaster 😭

[@ernestuz]

@@yannickhein9788 Hi, the most common algorithm used today, backpropagation, is based on propagate the "error" (the difference between the neural network, now on nn, prediction and real value) backwards, from the output to the input. One way of seeing it is for every layer in the nn (though not all nn can be divided in layers, but lets simplify) the error at its output is transformed to an error at its input, having into account the contribution of each neuron to the result. Performing a search in YT, there are two videos on top: kzbin.info/www/bejne/f53KZJp9mtyEa7c&pp=ygUZYmFja3Byb3BhZ2F0aW9uIGV4cGxhaW5lZA%3D%3D kzbin.info/www/bejne/f3-ViaB4na5_qpY&pp=ygUZYmFja3Byb3BhZ2F0aW9uIGV4cGxhaW5lZA%3D%3D

[@optiphonic_]

Your visualisations helped a few concepts click for me around the layers and activations Ive struggled to understand for years. Thanks!

[@4.0.4]

The visualization is what takes this video from good to fantastic. It's very evident you put a lot of effort into making this visually engaging, which is very didactic!

[@grapehool]

i had to search what didactic means

[@frostebyte]

I really appreciate how well you communicate non-verbally despite using very little A-roll. You're expressions are clear yet natural even while reading, enunciating and employing tone, and there's no fluff; you have a neutral point for your hands to signal that there's no gesture to pay attention to. I couldn't find anything to critique in your vids if I tried and this seems particularly easy to overlook. Thanks for every absolute banger!

[@MathGPT]

@@frostebyte he is truly a master teacher we can all learn from

[@sntslilhlpr6601]

The vocal fry is annoying. A shame, because his vids are such fantastic quality otherwise. But I've literally just noped out of his vids before because it grates me so heavily. Use your lungs, my good man!

[@StormTheSquid]

Half of these comments read like they were written by chatgpt lmao

[@codycast]

@@sntslilhlpr6601I don’t know what “vocal fry” is but his voice doesn’t sound annoying to me.

[@JorgetePanete]

Your*

[@talkysassis]

My dad graduated around 2009. One of his teachers (that was my teacher at Computer Science too) said at the time neural networks would thrive if they find their place in practical applications, but at the time most computer work as analogue to human work, but we've been learning how to abstract everything and work from there. That was when programs designed for computers instead of digital versions of physical media got very very popular. As he said: the tools were already there, we just needed to know how to use them, and we would shift from designing computer programs from real world to design the real world around computers. This teacher is still one of the smartest people I've ever seen so far. To put that into perspective, digital document control until a few years ago was very tied to how we handled them with paper and programs for dedicated digital control were a massive change of paradigm when they got adopted. We now tie small databases with documents, link them to project files, communicate projects with attributes... What a good place for something like NNs.

[@michaelala4924]

Awesome video! Funny how the moment we stopped understanding AI also appears to be the moment it started working lol

[@andybaldman]

An astute observation.

[@MrAvaricia]

It works like the brain. And like the brain, the moment the results are interesting is when they have enough oomph

[@ObjectsInMotion]

"If the brain were so simple we could understand it, we would be so simple that we couldn't" The same is true for AI.

[@Sqlldude]

AI cant verify the truth of the answers it gives. It often gives shit answers.. So saying it works is a bit of a reach

[@ObjectsInMotion]

@@Sqlldude Humans can't verify the truth of the answers they give either. Both need an external source.

[@crownoffyre894]

<a href="#" class="seekto" data-time="160">2:40</a> dude this single picture right here the way you described it this way literally was like the the thing that truly helps me understand how this all worked thank you

[@siddharth-gandhi]

Stellar video, you’re gifted at communication. Keep at it!

[@WelchLabsVideo]

Thank you! Will do!

[@jackwhitestripe7342]

sir, thank you

[@azurius_]

wonderful explanation and visualization! Thank you! A very interesting parallel i found is that these activation atlas images look very similar to what psychedelic drugs make you experience visually, especially on higher doses where everything kinda "blends into each other" and you dont necessarily are able to discern between different objects.

[@michaelm358]

Very clever and nice vizualisations! Excellent as usual. Thank you!

[@TheEVEInspiration]

It is pretty amazing that these systems consume their own output to set the next step in their "reasoning". This implies that much of the true decision of the final output is actually already made in the first pass-through. And that the extra passes are just needed for us to extract the output in a way we can process. It also implies there is a "hidden" boundary of how far the first pass through can "reason", any cycles beyond that are "improvisations" of the path the output was already set on.

[@clinttube]

Very astute observation. And it gets to my biggest concern with any kind of recent AI model I encounter whether at work or in the wild: "what was this trained on"? Much like if you get a group of highly trained but inexperienced students together, the range/frequency of potential answers to a question near their field of expertise is likely to be a lot lower/tightly clustered than if you do the same with a less trained group. That initial lens through which the question gets passed (the training) can severely limit novel outputs. There are deeper connections between concepts it theoretically can make that humans may not, which is super cool, but fundamentally it's synthesizing. Various permutations and combinations of + - , * / , powers/square root, derivatives, and vectors. Another question I always have that is much harder to answer simply (if at all): "out of the various potential modes/models tested, what was it about this one being used that made it get selected for production". Haven't gotten a good answer yet; I'm sure if I dove deep I'd find some, but at least at work our AI folks aren't capable of explaining it.

[@StefanQuandt]

What I also assume, is that this also implies, that the first words of a response to a prompt will immensely constraint the possible variation of the whole response (or let's call it the result space).

[@Aofekiko]

The visual aid in this video is unlike I've seen anywhere else, it really helps grasp the ideas presented easily, wonderful video!

[@SeanStClair-cr9jl]

It's rare to find an AI video this informative and interesting. Great pacing great focus, this is wonderful. I'm a particular fan of the sort of stop-motion / sped-up physical manipulation of papers on your desk with that overhead lighting. Very clean and engaging effect. Seeing the face-detecting kernel emerge after so few blocks was also mind-blowing!

[@WelchLabsVideo]

Amazing - thank you!

[@CollinReinking]

The amount of work you must put into videos is mind boggling. Thank you for making them.

[@TheSoylentGreen]

GREAT video. Your crystal clear script and visuals make a very complex topic approachable and your production values are top notch. Kudos!

[@emrahe468]

Amazing intro with scissor and carboards 👏

[@WelchLabsVideo]

Thank you 😁

[@opusdei1151]

I was also amazed by this

[@beautifulsmall]

A great learning experience i had was to deep dive into bitmap format and multiply greyscale images with 3x3, 5x5 arrays with simple patterns, ie all zero with a -1 in the middle. Different array patterns highlight edges or remove edges. it was a really eyeopening experience any software person should try that shows these fundamental operations. Great video.

[@ben9089]

This was an incredible introduction in just 18 minutes. I continue to be blown away by this channel.

[@WelchLabsVideo]

Woohoo!

[@thevulture5750]

​@@WelchLabsVideothe KJV Bible is mathematically encoded by God

[@AlvingGarcia]

I've been studying AI for the past year and the first 2 minutes was the best explanation I have see of how Transformers and ChatGPT works so far. Ive studied everything from Andrew Ngs Coursera courses, to Andrej Karpathy and more. Thank you for this great video!

[@Sam_Saraguy]

I stopped understanding AI around the six minute mark.

[@valentineezegwu9668]

Lol it’s wild technical stuff

[@projunder]

98% do

[@P4INKiller]

What stopped you?

[@thymii]

@@P4INKiller that's a legitimately good question

[@wearefromserbia9714]

same XD

[@elektrolysid]

What a remarkably well thought out, well organised, well created video! Just stumbled upon this channel and glad i did !

[@coscostan3334]

I've never seen AlexNet this way with a live preview of what happens inside. I'm sure it required a lot of time and programming: great job!

[@martinchakarov]

The quality of this content is remarkable - great job! Looking forward to watching more awesome videos like this one.

[@iccuwarn1781]

Fantastic presentation on the inner workings of machine learning!

[@WelchLabsVideo]

Glad you enjoyed it!

[@HenriFaust]

After watching a couple of your videos, you have some of the best explanations of AI concepts on KZbin. I hope you'll do more on the subject in the future.

[@thicksteve1761]

<a href="#" class="seekto" data-time="64">1:04</a> the moment I stopped understanding this video

[@sachinpandeyatd]

It made me chuckle, lol. Thank you.

[@DavidAspden]

You nailed it with this one. I'd love to see how much of your video was 'effect' and how much was real computation and composition. Seeing the third layer change with the video on that angle was very impressive.

[@WelchLabsVideo]

I try to "show the real thing" whenever I can, only thing that was really "effect" i think was showing the 96 kernels learning. I thought about actually doing a training run, but didn't have the time!

[@DavidAspden]

@@WelchLabsVideo thanks for that honest insight. I'd put this above Grant's effort, which is a rock solid series. I really enjoyed the whole pace and b roll inserts of historical research.

[@millanferende6723]

<a href="#" class="seekto" data-time="137">2:17</a> - THIS symbolizes the difference between "AI generated content KZbin channels" and real human-beings KZbin channels, like yours.

[@johnconner4695]

Dude your videos are amazing and that office space looks amazing.

[@manic-pixie]

I was literally talking to my roommate about this last night!! Thanks for the excellent video

[@bjordan429]

As a visual learner, I deeply appreciate the effort put forth here.

[@stoggi27]

Yeah! And they ways he uses the AI to do funny things like when it responds to completeing his sentence...

[@robert75019]

You were there 7 years ago when i started my AI journey in images….back at it when i start in language. Truly great work….love your content 👏👏👏

[@TheBooker66]

Great video! I've been subbed ever since I've watched your amazing series on imaginary numbers, and the quality hasn't dropped and even improved. Looking forward to your next videos.

[@PunmasterSTP]

The stop-motion and animation, including visualizing AlexNet's activation, were incredible!

[@bradleyfallon6847]

Very informative video. Thanks!! It's great to see content that actually includes in depth research and knowledge and not just enthusiastic speculation.

[@zhoudan4387]

It is not that the neural networks magically “understands” what is important. It is that the information is not random, so can be synthesized into smaller chunks. The synthesis process is what creates patterns, thus understanding.

[@rhackathorn]

Way to go... This is most informative and easy to comprehend tutorial of both the evolution and nature of our current AI innovations. I love the pivot using Latent Space as "the moment we stopped understanding AI". So, how should we wander in this new Latent Space to discover an intuition into its AI? It is as if we create a machine that does wonderful things. We can construct and disassemble the machine, but we are clueless of how the pieces work together. Kinda scary...

[@Riksia]

There will be some point in time, when people stop call statistic models an AI, but it will not be today for sure.

[@tim40gabby25]

I bet on 6 months after fusion solved...

[@waylandsmith]

I finally gave up about a year ago in trying to hold the line on the definitional shift of "AI".

[@givingin2G4Courtz]

@@waylandsmith yep, it's seriously tiring. And frustrating beyond belief. And depressing.

[@VampireFlutist]

I believe that might happen when AGI becomes possible and more widespread, and the distinction between today’s AI and the next evolution into AGI becomes necessary.

[@gregoryk_lite]

+++

[@90whiterabbit]

Phenomenal video!! Love the cut out approach to showing how the different elements of the algorithms come together!

[@alessi4249]

The amount of work that went into that visualisation i would love a behind the scenes video!

[@WelchLabsVideo]

I have some on TikTok!

[@TinkerRaw]

Been trying to learn and implement CNNs in my PhD research and work, this has been the best video for helping me visualize each step of the process in my head :) Going to be trying to replicate these visualizations for my presentations as I think they're great ways to show how these 'black box' models function. As mentioned, the real time kernel mapping is super helpful. Wonderful work.

[@denrober]

Love this video. First one where I finally understand how gpt-4 works. Thank you.

[@alexvisan7622]

Wow, so much effort has been put into the animations. Subscribed.

[@jamesknapp64]

understatement.

[@Gabriel_CeS]

Hey, thanks for this presentation. This video is a great example of how to teach about neural networks and their development. From the audio to the level of detail, everything was top notch. I hope you make more videos. I wish you much success, health and knowledge.

[@JulianKingman]

This is such a good explanation of AI, and the production value is so high. I'm bookmarking this so I can show it to my friends who ask me if I think AI is developing sentience.

[@NIHAR743]

Amazing explanation and history and best of all the visual way of explaining the things. Loved it😌😌 Keep up the good work👍

[@ar4hm4n]

Visualization was just wonderful, but what attracted me more is the way of delivering the information. Excellent work! Keep it up!

[@1ceBreaker99]

Truly amazing video, really great explanations and way of telling these hard to understand concepts. This got me more exited to learn more about this than an entire year of ai at university

[@youssefladeb]

this is youtube premium

[@the1JM]

Yes indeed

[@ubiquitous9105]

This is a fantastic video. Thanks for visualising the kernels so well. I enjoyed every minute of it. I've re-watched a couple of times now :)

[@amarokorama]

Great video, insightful visualizations! Regarding your footnote at <a href="#" class="seekto" data-time="375">6:15</a>, though: the myth that mantis shrimp have great color vision has been debunked long ago. They're worse at it than we are. Just because they have many photoreceptor types doesn't mean they combine them in a way similar to humans or other animals. Shouldn't have been too surprising, given their lack of brainpower.

[@kellymoses8566]

Yes. Human vision takes up a huge part of our brain.

[@bubbleman2002]

Say that to a mantis shrimp's face, I dare you.

[@GalaxyTracker]

Earned a sub for sure. The visualizations, and especially those of the real time activation maps are just incredible tools for a better understanding. Got into DL out of a hobby and now I am using it for my research in my scientific field, especially image processing. Visualizing exactly how they map data to vectors in each layer was eye opening.

[@Ben-nx7yy]

I stopped understanding this video at about 2 minutes

[@johndaviddeatherage2232]

thank you for taking the time to make this video. What a great explanation!

[@Paul0937]

Best dynamic illustrations yet. Using highlights on physically printed research papers is a wonderful story telling technique.

[@RussellDeming]

I absolutely appreciate the way in which you present this information in an easy to consume and understand format. Brilliant my friend.

[@raxirex6443]

A math professor of mine actually worked on many of the papers coming out of AI lab at MIT and he also worked on AI to play Minecraft. At the time it was really interesting to me as a sophomore, many years after I can write my own GPT, how the times haves changed!

[@youerny]

<a href="#" class="seekto" data-time="355">5:55</a> your 3d animations (thanks manim!) are truly awesome for the educational value embedded in visualization of math pattern and flow

[@machinate]

hehe, "hotdog / not hotdog".

[@premkumarsanamala]

😂😂😂 zin yaang

[@ciao934]

SEEfood

[@ArtyMars]

Hotdog / NotDog

[@TheDreadLab]

When you said “the image and kernel are no longer aligned” my mouth fell open and I got it for the first time.

[@stratfanstl]

Fantastic visualizations. It is very appropriate to try to think through this transformation process as you illustrate to first see how the algorithm first reorganizes info as we perceive it into info optimized for the algorithm to recursively refine. Once you see this first iteration, then "lose sight" of the next abstraction, it becomes apparent how impossible it will be for any human to identify and correct a "flaw" in an AI model. The only approach for "correcting" a flaw in "learned data" is to somehow feed the AI more data. That assumes an imperfect system WON'T become MORE imperfect by consuming more input. This defies logic.

[@backwashjoe7864]

How does that defy logic?

[@stratfanstl]

@@backwashjoe7864 Round #1 of the example showed that the algorithm is capble of creating flawed "links" or probabilities that lead to "incorrect" information being spit out for a given set of inputs. All of the inputs processsed in round #1 aren't "right" or "wrong," they just ARE. If the solution to (data)===> (partially incorrect output) is to feed more data in, there's no reason to expect round #2 to ELIMINATE the type of probabistic mistake encontered in round #1. It might REDUCE it but NEW errors can creep in, creating new errors in output, either for the original topic or some other prompt given the system.

[@kurtu5]

I have been having this conversation for the last 2 years. Thanks for putting it in video form and expanding the conversation past layer 1, as most people are totally lost on layer 1.

[@JamesTiberius-pw1du]

As a very young engineer I got involved in NN with the publication in the Signal Processing IEEE journal an article on the MLP by Lippman. I also worked at a small company for the president who was at Cornell name Frank Rosenblatt. It became my job to integrate NN into our product. I developed a cool way to deal with regularization and realized how critical regularization was as we had very little data. Did not have a billion images of cats. I wrote early FORTRAN code for back propagation that ran in a Sky Warrior array processor. No one knew what would become of the field and the history of Rosenblatt v Minsky. I am sad that Frank never lived long enough to see the word 'Perceptron' on t-shirts. He won. Minsky is a foot note.

[@gabemulero3962]

Great video thanks for taking the time to make it

[@eiwaransichsein]

I agree, I understood nothing.

[@diegosilveirarosa1213]

Thats a really in depth and interesting content, hands down!! Great work bud

[@svendtveskg5719]

The moment I stopped understanding a single word: <a href="#" class="seekto" data-time="1">0:01</a>.

[@nicholascasey6473]

"This is an activation atlas" Understandable, have a good day.

[@nicholasn.2883]

it’s not that hard you’re not doing any math just concepts

[@logandarnell8946]

@@nicholasn.2883which means without prior knowledge you will not understand much of it. at least with math, it can be applied pretty universally except at extremely high levels.

[@alejandrocelery3091]

Superbly marvelous! Fantastic explanation, thanks a lot!

[@huub_al]

Fun fact: Neural Networks are based on Neurons in the brain (hence the name), which we also do not know a lot about. Theory suggests that the neurons in our brain work very similar compared to a neural network in combining millions upon millions of simple transformations into something meaningful. This is also why research in fields like Cognitive Psychology go hand in hand with AI research. Very interesting to see where both fields are headed, because the key to understanding human intelligence is in understanding the unthinkable depths of neurons.

[@karen-7057]

you are talking out of your arse. there are NO connections between NN and biological NN.

[@woolfel]

feature activation visualization aren't interpretable and there's papers that have addressed this issue. Even before Alexnet, researchers couldn't interpret the weights of a deep neural network. There wasn't really a moment when we stopped understanding neural networks, we never really understood them. We understand back propagation and the frameworks (tensorflow, pytorch, tinygrad), but we don't understand the weights.

[@logandarnell8946]

thats why neural nets are a thing to begin with, manually programming things that specific and abstract is not a job for a human brain, way too complex, too many moving parts, too much trial and error. its likely impossible for a single human to ever actually understand the step by step process of a neural net after training data.

[@pcp351]

The video -> layers -> activation map animation is one of the best clarifying animations I have seen describing this process.

[@kellymoses8566]

The 3D visualizations of the neural network activation is incredible. What did you use to do it?

[@WelchLabsVideo]

Really crappy VPython code I wrote.

[@jvo1464]

​@@WelchLabsVideo it's not crappy if it works!

[@BRUXXUS]

The explanations, visualizations, and animations in this are incredible! You really got me to feel a much more intuitive understanding of a lot of the concepts I've been reading about for years.

[@demolle87]

so basically AI sees in LSD

[@moggadah]

Yes, I've never taken LSD, but I understood that it is similar when I saw so called deep dream images ca 2015. Holy s*** I thought, if AI can hallucinate like that it must be working like the brain.

[@invven2750]

It's possible that LSD screws some higher order of operations indeed

[@ruul_in_bahrain]

Thank you for all the work and effort youve put into this. Amazing teacher.

[@rotors_taker_0h]

Great video. The only nitpick is with title: we haven't stopped understanding AI at AlexNet (and video clearly shows that we only getting better at understanding since that moment), we finally had working "AI" starting from AlexNet. All those "expert handcrafted" AIs before were no simpler to understand (if not harder) despite being handcrafted. And they largely didn't work and it was very hard to understand why.

[@Anonymous-df8it]

Why didn't they work?

[@rotors_taker_0h]

@@Anonymous-df8it too simple and brittle to capture the real world, I think. I started working on computer vision right after deep learning started to solve problems one by one but was not yet commonly accepted. So for some time people tried to use old and new methods and every single time classic methods only worked with toy versions of the problem and broke apart in real world when anything changed that you as human don't even notice, like different lamp temperature or some reflection.

[@Anonymous-df8it]

@@rotors_taker_0h Why would it be difficult to understand how they "work" or why they didn't? Also, what were the 'classic methods' and could people in the soft sciences who know programming create an image identifier or chatbot that actually thinks like us (which should work since people can do those things, and the code should be intuitive since it's our own thought processes)? I don't know about you, but I don't remember multiplying giant matrices together a bunch of times when thinking about how to respond to you (you could argue that I did but aliens wiped my memory or something, but that would be unfalsifiable), and whilst glare and monochromatic light sources would probably make it hard to see things, those are extreme cases, and I can certainly handle sunlight vs indoor lighting

[@gilbert2720]

@@Anonymous-df8ityou didn’t do any math in your head, but your brain did all sorts of calculations behind the scenes.

[@Anonymous-df8it]

@@gilbert2720 Like what?

[@tobycortes]

bro how is this not a long form video, GOLDEN INFO !!!!!!

[@stoggi27]

It is, did it just feel like it went by fast? It's almost 20 minutes long and most are like 5 minites on the platform that are considered "long-form"

[@ligz.3437]

<a href="#" class="seekto" data-time="144">2:24</a> no way you actually asked if it was mad 💀

[@alegian7934]

i really like the direction this channel has taken the last few months! keep it up :)

[@WelchLabsVideo]

Thanks, will do!

[@jonanderirureta8331]

After seeing how much computational power is needed for a language model to correctly identify an elephant, for example, one has to wonder about the computational power of the human brain, since a toddler can do the same.

[@behnamjabbarizadeh4536]

I presented Alex Net in my ML class and I am surprised at how good and intuitive you explained it.

[@tommyshobalongdong]

“No one told Alex what a face was, we just forced it to see millions of them over and over again, crazzzyyyyy!”

[@tommyshobalongdong]

“What’s even crazier is that the math from a photo is similar to other similar photos, crazyyyyyyyy”

[@zelda_smile]

How can we make Alex unlock 100% of its intelligence?

[@radupopovici118]

absolutely amazing video, with great explenation and visuals. Keep up the good work!!

[@Qstate]

What an insanly high production value.

[@AdvantestInc]

This video is a fantastic resource for anyone interested in AI. Your ability to explain the intricate workings of AlexNet and GPT is commendable. Keep up the great work!

[@theskinegg9168]

alr why does the right poster <a href="#" class="seekto" data-time="1058">17:38</a> look like Africa

[@HomelessPank]

Pre continental drift heatlands

[@marcel-dennisboerzel7902]

brilliant didactic visualizations. I directly subscribed

[@iceboorg9737]

Man im too dumb for these videos so sad

[@chloefourte3413]

really beautiful visual aids! both the digital motion graphics and the physical cards! great video!

[@djayjp]

<a href="#" class="seekto" data-time="833">13:53</a> Curious how logic operations look so much like the brain's own neural network....

[@Hiyori___]

incredible video. I'm working with a ResNet on a project and sometimes, focusing on minor bugs and programming headeaches, I tend to lose the perspective on the amazing tool I'm dealing with. This video was a pleasure to watch

[@ViralKiller]

What people fail to explain is, the training has 2 core chunks. The first stage is 'pre-training' when it is fed millions of words, to understand general relationships between them. No strcuture just words and letters. The second stage is secret but we can speculate this is the 'fine-tuning' stage where data is provided as a JSON file containing a question and answer parts. I mean this is how they would do it if smart. There are also other 'experts' like code maths etc....

[@rahul_siloniya]

Why is it secret now? Can't we look at Llama and check what that "secret" step is?

[@jcm2606]

@@rahul_siloniya Because OpenAI and Meta keep their training datasets and procedures secret. We can't learn anything meaningful about how LLaMA or GPT was trained by looking at the model, as the model is just a set of seemingly random weights with no indication of how the weights were arrived at or what the weights actually mean. Anthropic are trying to reverse engineer the weights to figure out what they mean, but that still leaves us in the dark regarding how these models were trained.

[@tyruslawhorn]

The second step probably involves virgin sacrifices, demon summonings and an assortment of scented candles from Bath & Body Works