Hopfield network: How are memories stored in neural networks? [Nobel Prize in Physics 2024]

[@ginqus]

bro made a channel dedicated to neural networks, posted one video, gained 24k subs, decided this was enough and left

[@bernhardstelzhammer8636]

Hahahaha 😂

[@avikchattopadhyay1483]

HahAAA

[@maxave7448]

To be fair tho, this video is really good and detailed while remaining relatively not-too-complicated for people who have a basic understanding of neural nets

[@masterofnone597]

obviously way more was planned however was cancelled

[@PreyumKumar]

He mentioned the following video.😢

[@aegontargaryen597]

As a neural network, I confirm that this is how we store memories

early

[@ugestacoolie5998]

Lmao

[@claudiamanta1943]

All the screws of those who are OBSESSED to think that people are nothing but cybernetic machines are rather loose…

[@aegontargaryen597]

@@claudiamanta1943 I haven't seen anyone thinking that people are cybernetic machines lol

[@merozemory]

LMAO

[@gianlucazinarof8617]

You cannot fathom my disappointment when I looked at your channel and discovered that this is your only video. What do you mean there aren't any others? I need more! Seriously, interesting topic, crystal clear presentation, keep up the great work!

[@Mangputt]

Do you know, what software able to produce and animate these animation? adobe after effects or any software else?

[@layerwiselectures]

It is a custom toolbox written in Python, called Manim. Here it is the community edition, but Manim was originally devised and open-sourced by 3blue1brown. There is a link in the description if you want to learn more.

[@Mangputt]

@@layerwiselectures thankyou... I still waiting for your next videos. Keep it going Bro 🔥

[@sebastianmuller4649]

"Contrast this with what we believe about the brain..." I like how careful you are with how you make your statements.

[@murilospineli2241]

is this a first video? it's surprisingly good

[@jasoneel76]

My god I need to subscribe immediately

[@jaumeivars]

Oh my god i didn't figures it out until you commented!

[@ryanpmcguire]

When something is made by someone who cares, you get a good result. Caring is the first ingredient in the recipe for success.

[@fadonoso]

I thought the same!!! This guy is really good. Thanks!!

[@NickSklias]

The first video uploaded to a channel does not mean the person/people editing/producing the video had no prior experience. That being said, it's a fantastic first upload and this channel is off to a tremendous start!

[@mmarrotte101]

The amount of effort required to not only curate such complex information in an intuitive way, but also to animate the concepts in the way you've done, is clearly tremendous - thank you for taking the time to produce this! I'm inspired by your work, really looking forward to more!

[@piggy9539]

This guy explained such a complex topic so well, great job.

[@alexanderhandjiev2845]

It's going viral soooo ye

[@mihailmilev9909]

@Yatish38 yeah and the subs for one video is crazy but deserved too

[@RoseOnFire]

@Yatish38 Really good, yes, deserves more, also yes

[@TheRainHarvester]

Yeah i had trouble with the super/subscripts in all the math. So i just made a video easily explaining multilayer back prop with only 1 subscript. It's so easy now!

[@pizzainc.1465]

If humans know how it works, it’s a simple subject that is explained badly. If they don’t, it’s too complicated anyway. For example, we know nothing about how the brain works because it is simple too complicated. However, we know plenty about gravity, other areas of anatomy, computers, and more.

[@minh9625]

Guy, it's 1 year already. Cmon comeback bro

[@sciencelab4225]

This is one of the most important pieces of media I have yet consumed to better understand this world. It is incredible to realize how one's brain works in principle. Thank you so much!

[@keithmiller3770]

This is remarkable. How the heck does this channel only have 1 video?!

[@eda1198-w6m]

Still waiting :( ...

[@alexb3617]

yeah i knew this video would be good. i noticed it few months back when i was beginning to study neural networks, just stored it, knew wasn't ready yet. now finely watched, as expected, it turned out good. nothing like clickbat stuff that usually out there on youtube

[@Mikee512]

Everyone must start somewhere :)

[@dunar1005]

Because it was a submission for a convention, and not his goal to continue this

[@luke.perkin.online]

You nailed the 3blue1brown feeling whilst being ligther on the maths. I wish you the best of luck with future videos! 🙂

[@ianthehunter3532]

He likely uses Manim, the software made and used by 3b1b.

[@poultrypants]

Are you guys also watching the 3B1B videos about GPTs?

[@luke.perkin.online]

Aww, you never did a follow up!

[@Sazazezer]

I've been idly musing about the brain's memory in terms of binary and bytes for years and always thought it was such a limited way of looking at the subject but didn't know any better. I asked around in the past about alternative ways of looking at the subject but never came away with a decent answer. This video has been such an eye opener and i think i'm going to have to take a deeper dive into the subject. Thanks!

[@Daniel-ih4zh]

You should haken's book on synergetics

[@andrewhancock2451]

You mean, like, deep into neural nets? ;P

[@SpringySpring04]

I heard in an scifi anime (which is surprisingly more accurately scientific than it may seem despite being scifi) called Steins;Gate that said the brain can hold a little over 4 terabytes of memories, but that was just a claim, I'm not sure if that's true.

[@pyropulseIXXI]

Memory doesn't just exist in the brain; there is evidence it is dispersed throughout the entire body. People have had large chunks of their brain removed and their memory was unaffected. People born with basically no brain, just a small layer around the skull and the 'cavity' where the brain should be was filled with cerebral-spinal fluid. And yet their IQ and memory was well above average. I know it is hard for purely materialistic to understand, but consciousness is more fundamental than physical matter, with the brain merely being an interface for the physical world A true ghost in the machine; consciousness does not arise from the machine being complex; it is controlling the biological machine

[@goldnutter412]

@@pyropulseIXXI Exactly, I posted a heap of times about this already, this video really inspired.. brain caching, permanent information, rinse and repeat with new context. Optimization is often overlooked. It will be obvious sooner or later, there is no other way to efficiently battle entropy. We get sense data and need a nervous system and brain in general, but uncertainty is the ultimate optimization. Probabilistic by design.. part of why the speed of light is so high. We see the very very distant past, and are disconnected from it.. we get data and interpret it and wonder how the heck physical matter organized itself a certain way and why the math doesn't work 0 to 1 is a logical impossibility.. the Big Bang is a misinterpretation, but a plausible back story for the universe and adds to the immersion. We process things bound by our brain capacity (see Einstein's freak brain!) and easily fall into materialist traps of our own design. Logically the universe and biology is simple, but obfuscated by so many LAYERS of background processes ! we call it subconscious.. yeah it's busy !

[@frenk3928]

I studied Hopfield networks last year in the framework of spin glasses and all assumptions were dropped seemingly out of nowhere in a big confusing mess (as usually happens in lectures). This video really tied up all together and now makes so much sense beyond memorising proofs. Excellent video and brilliant demonstration.

[@NajibElMokhtari]

This is amazing! I finished the video and immediately went looking for the next one where you'd explain the more accurate models. Then realized this is the first and only video on this channel. Can't wait for the upcoming stuff. Congrats for the launch!

[@martingisser273]

Brilliant exposition for both mathematicians and nonmathematicians. It took a few decades to get me (mathematician) interested in neural networks. Now I'm hooked. Musing about connections to Markov processes and PDE...

[@macchiato_1881]

It's really amazing how something as simple as a neuron has highly emergent properties that can do very complex tasks.

[@Wonders_of_Reality]

Looking forward to watching the second part! And many thanks for choosing dark text on a light background. It looks like a calm creamy ocean of fascination!

[@jboj8430]

Kind sir. Thank you for this video. If you have the time you should totally make that follow up video. The announcement of the Nobel prize is going to rake in views!

[@TR-kn3sn]

This is absolutely incredible. Just immense, wow. I was initially skeptical of neural networks because they lacked the "logic" side of the problem (as I think a lot of people were at first) but I knew they were unreasonably effective, so I assumed they had their place. But now, I think they have more than a place, they are officially critical to our understanding of the world. In short, they were always cool, but now we've proven it. Thank you for making this video! It is so well made by the way!

[@Tigrou7777]

By looking at this video, you just updated the neural network in your brain to be able to remember stuff about neural networks. Brilliant.

[@tf3confirmedbuthv54]

Love it! Hopefully this video will get big, this question is wonderful and really makes you think outside the box, and I think you explained it perfeclty.

[@RafaelBeraldoPU2URT]

I'm a PhD student in language acquisition and neural networks are in our theoretical toolbox. However, and bear in mind this is an oversimplification, there's a dispute between what we could call “cognitivists”, whose models are generally at a higher description level (whatever that means), and “connectionists”, whose models seek to show any complex behavior/knowledge of language is an epiphenomenon of the functioning of neural networks. I've read a number of connectionist papers and never have I seen such a clear and honest presentation of these topics. Thanks for making me smarter. Keep 'em coming, I'll be excited to watch anything else you make.

[@BrandonWong55]

Wow, this video is really eye-opening. As someone who has been interested in both memory sports and math, it really intrigues me how these false memories arise. Whenever I make a mistake memorizing a deck of cards, I'm always trying to figure out what when wrong, and it's cool to see a mathematical model also make similar mistakes in your video.

[@question_mark]

guys, just wanted to remind you that this video is easily in the top 1% videos ever produced by a human

[@anon_y_mousse]

I don't know why you haven't made another video in 9 months, but I'm hoping you continue this series. Maybe go into the math a little bit more, but keep it simple so even a simpleton like me can understand it.

[@FelipeCampelo0]

When a video catches your curiosity in the first two minutes that’s because it’s going to be a piece of art !

[@konradd8545]

Please continue making videos! This was not only super interesting but also very well explained and animated. I hope to see more videos on this channel soon!

[@hvdveer]

Amazing, how the brain stores memories is one of the biggest enigmas in neuroscience and you've just provided an answer using it's digital equivalent. This is worthy of a scientific publication. Brilliant stuff.

[@layerwiselectures]

I should clarify: the video is based on an existing scientific publication (Hopfield, 1982) as mentioned in the video. See the link in the description. It's brilliant, but not *my* brilliance, is what I should make clear.

[@HansLemurson]

Different "seed" states will call up different memories. I wonder if this could be analogous to how certain ideas link together, and maybe some of the underlying information of related ideas is actually a correlated section of their memory patterns.

[@MrNatsuDragneel]

This is wrong, our brain can excite and depress certain neurons with neurotransmitters, hormones, data from sensors of the environment and our own body etc...

[@hufficag]

This explains why I have an idea at my desk, get up to walk across the room and forget what I was thinking about. Get back to sitting at my desk seed state, and remember it, get up and forget it again.

[@mihailmilev9909]

@@hufficag omg we figured it our also lmfao

[@floriantesto4925]

@@hufficag just take a look on en.wikipedia.org/wiki/Encoding_specificity_principle

[@woolfel]

the video is meant to explain how modern artificial neural networks work. biological brains don't work that way. Take for example a deer. Within an hour of being born, a deer can walk and within a few hours it can run. Clearly the baby deer didn't learn how to run in the womb. The evidence suggests some kinds of memory are genetic, while other skills are learned. Even if there was seed states, keep in mind memories are stored redundantly in biological brains. the obvious example is stroke patients that temporarily loose skills, but then recover them. Assuming there something like a seed, it would be seeds or parts of a seed stored in multiple places. then there's the fact that brains are analog, not digital. In classic computer science, associative memory is two elements linked with an address that points to both elements. In neural networks like GPT3, the association is captured by many weights. One could consider those weights as a seed, but the analogy breaks down. A common problem with neurons in a neural network is that a single neuron may participate in multiple patterns. If you change 1 weight, it can cause the network to perform better on one type of task and worse on a different task. The problem is called polysemantic neurons. There are people working on fixing polysemantic neurons, but there's no general purpose fix today.

[@hengchunsong301]

Very rarely KZbin has video with this much deep insight. Treasure.

[@LearnwithAnnaN]

Dude made one video and then stops uploading. You have a talent.

[@YZhou-mq1bw]

You definitely should keep making the vids if you are still interested, this is insanely amazing! You must be a big name there!

[@redaabakhti768]

Perfect video. I didn't know you could look at neural networks from this neat mathematical vantage point. I thought about eigenvectors and how the eigenvector with the largest eigenvalue would influence the network sort of as a dominant weight that increase exponentially , this would need a slightly different f the function you operate on the resulting weighted sum this would work for storing one memory maybe with increase possible states of single neuron and the right activtion function you will be ablse to store many patterns. Amazing video once again.

[@Prince-xp9vk]

had an idea to write about this, "is the brain really a RAM or ROM?", but seeing you do justice to it, I can safely procrastinate for another day. it's kinda sad too that you only have one video. please do more.

[@dererzherzog]

Even a reductionist model of neural network (disregarding the dynamic spatio-temporal interplay of a plethora of neurotransmitters, neuromodulators, receptors, secondary messengers and downward pathways) leaves one awe-struck by hinting toward the tremendous scale of capabilities in a biological neural system. Thank you for this wonderful presentation and thanks to the YT algorithm for suggesting it! May your channel and its viewership grow exponentially!

[@LuaanTi]

And don't forget, the _in_ capabilities. Even from this extremely simplified model, you can see why it's hard to write memories, and keep them stable against change (which of course happens even when you're just _reading_ the memory). The computer on your desk can easily store terabytes of information without a glitch just by "remembering" the information one by one, while the computer in your skull needs to keep reinforcing what you want to remember, and keeps overwriting related memories as others are read or changed. Mind, this is most likely a critical component of learning and intelligence in the first place - your memories aren't just static data stored elsewhere, they're ultimately a part of the computational network itself. Every experience has _some_ effect on your decision-making, and memories interact with each other (and with other inputs from all over the brain and the outside world).

[@goldnutter412]

@@LuaanTi apparently the brain throws out TONS of data all the time.. anyway won't repeat the other posts.. maybe read those first above.. but it's a good filter/cache for information processing.. short, long and permanent information. Only what is really significant to us, lasts.. and we come back to evolve past our problem with entropy. The unknown is enticing.. fear of the unknown is like getting stuck in loops on tangents.. or outright stuck and unable to deal with whatever patterns we're experiencing. The optimal configuration for experience is of course permanent state, temporary brain and universe. Next time, the patterns will have a completely different basis, while still understandable ! almost every random set of DNA and environment will be the same pattern. Choose, which gives result (quantum mechanics delivering data on demand), JIT type of interpretation with the brain cache, more choices, more life experience, on and on we go. Because we are self aware, we are aware of entropy.. time advances, we collect more data. More entropy unless we evolve away from "fear" or.. I guess you could call it over management of the data structure. That's the nature of information ? "Let it be" playing in the background..

[@artysanmobile]

The thumbnail image makes a very important point on the subject of human thought. We don’t look for the room, shelf, book, page. We build a complete path to every experience we have ever had since birth. If we contextualize it by re-use, or by interconnecting it with another path, we improve its clarity, its importance, and remember that address. The classic explanation for having no idea what you ate this morning. Breakfast has not yet been made important.

[@d0tz_]

This is the best SoME video I’ve seen so far. Such an intuitive explanation for a niche yet mindblowing idea. Maybe it’s my lack of intuition for high level math or my affinity to computer science, but most of the other videos had me closing out of it mid way, while this one was interesting in its entirety.

[@AlphabetAYT]

Speed running getting an headache. Awesome video!

[@YasheshBharti]

Such a unique topic, so beautifully explained! Thank you so much for this piece of art!

[@cmilkau]

I guess you have to make a distinction between short term memory (stored on activity patterns) and long term memory (this video? stored in the network weights). The latter is much simpler and in theory there could be a single neuron for a single memory, although in practice there are more efficient encodings.

[@chanyan2024]

Congratulations on winning the honorable mention prize for the Some2 competition. Your entry is my favorite

[@brandonzhang5808]

What I'm getting out of this is that there is still a scalar memory parameter in the number of neurons or size of the weight matrix, and storing patterns in it is similar to file compression, allowing the same network to represent multiple patterns, but with a compression ratio that is actually disadvantageous, requiring O(n^2) weights to store O(n) patterns. Something of a Rube-Goldberg for memory systems.

[@dominikwitczak1655]

Eactly, this does not prove that our memory can't be measured in Gb of information.

[@mo_i_nas]

in a neural network of 3 words... THIS WAS AMAZING

[@schnitzelsemmel]

Yes yes yes this is exactly the video i didnt even know i needed. incredibly good explanation and animation

[@Red_Fox_Miro]

same

[@asanzramaxi]

didnt realize the views and subscribers until casually saw comments. keep it up, such a good video!

[@br.3250]

This is just an outstanding explanation and it was a very interesting subject. And it's even more amazing that even though I know very little about this subject I could understand almost everything Keep up with the great work :)

[@MyHi1997]

Dont mind me, just leaving a comment so that the algo will pick up this amazing first video from the channel

[@Ambixious]

Beautiful explanation and visualizations. I don't normally comment, but it happens that this is what my PhD thesis is on, and I hope to be able to emulate the clarity with which you have presented this during my defence. I see lots of people in the comments saying they learned a lot from this. Really gives me hope that my work wasn't for naught. The Hopfield network truly does draw a beautiful relationship between physics, mathematics and neuroscience. Just want to add that with the right learning function, and the right structure in the encoding, the number of memory states can grow faster than linear with respect to the size of the network.

[@Personal-M.I.S.]

Well done! I'm sure I'm not the only one who would appreciate it if you did more =)

[@somedudeok1451]

Love the presentation. I think that the mistakes you were frustrated and fascinated with are maybe a chance to generate something new from something stored. It reminds me of our ability to be creative by morphing our memories over time. It's like when you have memory A and memory be and over time they fade and become memory AB which then turn into memory C. And sometimes memory C might actually be something useful. It also reminds me of generative AI. Of course, this is all just layman talk and I have no idea if this has anything to do with how human memory actually works.

[@khasmosis]

I don't usually comment on youtube but.. I don't know what to say. This is by far one of the most enlightening videos I've watched ever. I just subscribed and I'm looking for your next video

[@venamaybe]

This was super interesting and awesome!! I'm excited for the next video :)

[@LoongBerries]

looking forward to the next video too

[@Ryush806]

I just learned your video was for 3Blue1Brown's contest by watching his video about the contest. Makes sense why you only have 1 video. I originally watched it weeks ago and was dumbfounded by it being your debut. PPLLLEEEAASSSEE make more! You have a knack for explaining a very complex topic in terms us commoners can understand!

[@nulled7888]

Super interesting video, got me really thinking about the correlations to real neurons.. very well made as well :)

[@fabioteixeira868]

Wow! Awesome first video! Subbed and eager for more!

[@eaudesolero5631]

would you continue ?

[@rafaeladivapinho2399]

You are by far the best teacher for soft soft . It's very complicated at first - overwhelming, actually - but, you make it doable for

[@theClaytron]

If you change the procedure to use a synchronous update, and make the activation function the identity, then you can analyze the dynamics of the weights matrix W. That is, v_{n+1} = W*v_{n}. In this case, a memory which you converge to is simply a stable eigenvector of the matrix W. When you "train" the network with multiple memories, you are performing multiple rank-one updates to the matrix. This also explains why the maximum number of memories you can store in W varies linearly with the size of W, since a matrix of size NxN cannot have a rank greater than N. In other words, if you tried to train W on N+1 memories, you are guaranteed that at least one of those memories is a linear combination of the other memories. Not only that, if you add a memory with a rank-one update to W, that does not necessarily mean that memory will be an eigenvector of W. The memories can combine such that an eigenvector emerges which is not equal to any of the memories. This approach also gives another perspective on why you can converge to the "flipped" version of the memory. An eigenvector defines a 1-dimensional space which is simply a scalar times a vector, where the scalar can be positive or negative. Also, the properties of the W that emerges from Hebbian Learning are very interesting. That is, W is symmetric and has a zero-diagonal. It is a property of symmetric matrices that their eigenvalues are all real. So we would not expect the network to oscillate around some solution. Also, the trace of a symmetric matrix is equal to the sum of the eigenvalues, so that means if we have negative eigenvalues, we must also have positive eigenvalues. Again, this is just some conclusions for the case of synchronous update and identity activation function. It may not be so relevant to the case you discuss.

[@layerwiselectures]

That's perfect, thank you. The original paper does not use this reasoning to arrive at the linear memory capacity, but the similarity of the conclusions is striking. Thank you for this addition!

[@theClaytron]

@@layerwiselectures Thank you for your excellent video! I am looking forward to the sequel.

[@isaacaraya3848]

@@layerwiselectures Have you heard of adaptive resonance theory? Could the stable states in the network be harmonics? Like if each pattern is stored not just spatially but at a specific frequency of oscillation then you could maximize the number of memories a single neuron could contribute to. Kind of like the tv's that play multiple images on the same screen at once

[@isaacaraya3848]

Great insight to complement video. Could you expand on why symmetric matrices can't oscillate around a solution? Might save me some googling

[@guineapig1016]

I had a similar idea, over 10 years ago... i was extremely young and no way to learn how to create such a system. Life moved on for me. Glad to see people are creating this stuff.

[@HubertRozmarynowski]

That's very interesting. Maybe you should learn some basic Python so it'd be possible for you to bring your future ideas to life? You decide if it's worth it. Just saying the barrier to entry might be lower than you think. I've implemented the Hopfield Network as described in the video in Python without external libraries in under 100 lines using basic things like: state = [ [1, -1], [-1, 1] ] weights = [ [0, 0.1, 0.1, 0.1], [0, 0, 0.1, 0.1], [0, 0.1, 0, 0.1], [0.3, 0.1, 0.1, 0] ]

[@arkadius228]

​@@HubertRozmarynowskiNah, just use numpy and make it 50 lines instead of 100

[@aankitdas6566]

bro just dropped one video and vanished. Bro is the Gotye of youtube tutorials

[@awritrojitbanerjee1055]

you knocked it out of the park dude! need more of the good stuff!

[@ofrigal9557]

Amazing video! Great job, i would love you to explain autoencoders

[@wakandangunner465]

you`ve gotta make another video, i felt sad seeing this was your only video yet, the way it was goood

[@alzario]

Awesome video, comparatively digestible explanations and really good visualisation of what you're describing.

[@pafloxyq]

Hopfield nets were the first thing that got me interested into studying neural networks some years back, but I hadn't seen such an eloquent presentation on it since now. Will wait for the subsequent videos of this series.

[@dominicbaumer6038]

Thanks for this great lecture! My idea for a next video topic is that you can talk about differences between artificial perceptrons and biological neurons

[@layerwiselectures]

Yes, thank you! Okay, goes on the list.

[@brandontylerburt]

An excellent explanation. I'm starting to realize what a profound influence this type of neural network model has had in recent decades on all kinds of disciplines ranging from machine learning to the development on new theories on PTSD and how traumatic experiences alter brain function. Thank you for providing such a succinct and illuminating video on a complex and vitally important subject.

[@Kaepsele337]

That's really neat. I wonder what happens if you add "eyes" to the network, i.e. some subset of neurons whose activation is driven by an external signal instead of the update function.

[@layerwiselectures]

Good point. The original paper actually mentions an input current to the neurons. This would then be combined with the summed input of all other neurons, so it would modify the update function rather than replacing it. I left this out for simplicity. However, it's actually an important detail once you start thinking about it. It sort of biases the network to a certain state and would explain where the starting state of the network comes from, if it's not random. The story would then go like this: the "eyes" of the network (love the metaphor btw) would cue the network with a partial state and the network would then autocomplete to its memory state. Kinda-sort-of-almost-like seeing a visual cue of some sort and then being reminded of a scene from your past - if we're being a little bit fanciful. Thanks for the comment.

[@samcoding]

We need this guy to make more videos.

[@nikhil_kr97]

Who all are here after the Nobel Prize in Physics 2024 announcement?

[@acyned8079]

The conclusion really was what I was here for. I didn't get most of it but I got the point of it!

[@drdca8263]

Hm, I don’t know that this really shows that the memory capacity can’t be appropriately described in terms of “how many bits”. That there aren’t addresses for physical locations for different chunks of memory, sure. But, information theory isn’t fundamentally inapplicable to the brain’s ability to remember things? Like, if there is some random distribution of possible observations (e.g. a sequences of symbols, or other combinations of e.g. color, shape, texture, whatever) at one time, and then at a later time the person says their guess about all the information needed to describe which of the observations were randomly selected, then, ((the entropy of the distribution of observations) - (the conditional entropy of the distribution of observations given the person’s recollection of the observation)) should give a lower bound of how much information about the observation the person remembers on average. This can be expressed in bits. Of course, this isn’t describing the person’s entire memory capacity! Just how much they remember from these observations in such tests. This could, I think, be used to measure an estimate of a lower bound on how much information (measured in bits) a person can rapidly memorize for a short amount of time. Presumably how much info one could memorize would depend on the way the information is expressed. For example, I would expect a random sequence of words would be easier to memorize than a random sequence of literal bits with approximately the same entropy. And that’s part of why I say it would only be an (approximation of) a lower bound. (Approximation because there’d be measurement error due to the random variation.) We can also probably put an extremely weak upper bound on total capacity based on the amount of information theoretically needed to describe the physical state of the brain. If nothing else, there is at least the Bekenstein bound based on just the size of the brain along with the fact that it isn’t part of a black hole. This is so weak of an upper bound that it is barely worth mentioning. I don’t know how to get a better estimate of “the entire memory capacity of the brain” than these though. Well, of course, you can look at how much data people have memorized over longer periods of time. Oh, maybe some estimates could be made based on people with eidetic memory. Like, if we query facts about randomly selected dates and check how accurate what is recalled is, then, we could extrapolate that and assume it would be equally accurate for all the many dates in the same range which we didn’t ask about, then we could estimate from that how much they remember overall from that range of time. This could probably give an estimate which reveals a rather large amount of information stored. And, presumably, people with an eidetic memory don’t have just, an inherently larger memory capacity, so much as, just more stuff gets committed to memory? (I could be wrong about that.) So I think this would still give a good estimated bound on how much memory capacity a brain in general has.

[@layerwiselectures]

Wow, thanks for the comment. There's a lot to think about. My favorite part is the black hole information bound. No you're right, information theory isn't fundamentally inapplicable here and I didn't mean to imply it was. Personally I'm not clear on whether the "bits" in a hard disk are information theoretic bits. Are they? Maybe in the sense that, if I didn't know the state of the hard disk, and then I looked to find a certain pattern there, I would have gained that many bits of information? But then on the other hand I wouldn't expect every pattern to be equally likely to appear on a hard disk or in RAM. What I'm getting at is that, yes, there must be a way to compare the information in both memory systems (computer and neural) in bits, but no, it isn't usually being done when we talk about computer memory. My goal was to show that casually speaking of bits of memory has a tendency to obscure real differences between the two memory systems. If in the process I appeared to discount information theory ... oh my, not my intention. Thanks for the comment again!

[@drdca8263]

@@layerwiselectures I think I phrased my comment a bit too critically given how much I liked the video. I quite liked the video, and subscribed (before your reply). I think computer bits can be related to bits in information theory by, if you sample from a distribution many times, the optimal way to store such a sequence on disk, would have its length be on average, the number of samples times the entropy of the distribution. And, I think if the number of samples is large, then the variance in how much storage is needed overall should go down, eventually to the point where you can regard it as nearly always the same. ...uh, assuming the variance of the number of bits each sample contributes, is finite (which it will be if the number of possible outcomes is finite). So that’s at least *a* connection. It perhaps isn’t quite as close a correspondence as one might hope? Well, the amount of information in an information theory sense that can be stored by a hard drive, is bounded by the storage capacity, and at least for some distributions of a random variable, this bound is exact. But for other random variables where the outcomes aren’t just 2^n equally likely outcomes, then I guess there can be some overhead compared to the ideal, in storing a finite number of samples. But in the limit of many samples, I think the two notions of “amount of information” become the essentially the same? In some sense anyway.

[@sullivan3503]

@@layerwiselectures You're correct in saying that information theoretic bits don't always correspond exactly to hard drive bits. In order to calculate the amount of information stored, you need to also understand the encoding. For instance, ASCII text uses 7-bits of information per character to store text, but these 7 bits of information are often stored at byte boundaries, resulting in each character taking up 8 bits of space on the disk. So the leading bit is wasted. Compression can also reduce the amount of bits needed to store a certain piece of text drastically. So, as long as your data is not maximally compressed (which is hard to tell), then your hard drive bits is not the best measure of the amount of information stored. However, in practice, we typically use file size as a good proxy for information.

[@LamNguyen-zc8xs]

This is brilliant. I've never seen any video about Hopfield network as interesting as this video. All they show are just math. Man you should do more video on brain and neuron networks like this :D

[@anjaney_007]

Anyone here after the 2024 Physics Nobel Prize 🥇 👍🏻

[@loganpowell]

Dude, this is the best video on the topic I've found (and I've looked). When can we expect a follow-up? Would love to watch it.

[@anarakberov7173]

Who came from the Nobel Prize announcement?

[@Frost_Byte_Tech]

Please upload more videos like this. You have yourself a new subscriber sir 🙏

[@霍金本人]

Watched

[@simbaji-hj9zv]

Seems there is only one video, hoping the next will come soon, great!

[@okopyl]

Too complicated for somebody with 0 knowledge in math

[@BrandonMcCurry999]

Do you have the ability to learn? I do, and I'm slow, especially on things with many steps that must be done in an exact way or the whole "thing" will be incorrect

[@BrandonMcCurry999]

Or were you not talking about you

[@BrandonMcCurry999]

Difficult doesn't mean it can't be done, many things have taken alot of work for me specifically Some things come naturally I tend to go with what comes naturally, Plato my strengths

[@dorionporath4363]

Elementary school

[@Dorstamo]

For YT recommendation: good job, great recommendation. For author: Thanks for the stimulating video!

[@25_26]

This is just perfect. I studied maths and quit later and even with that knowledge i understood the concept easily thanks to you.

[@zoraamethyst2147]

the level of this video is so very close to 3 blue 1 brown videos. similar topics, similar brain, and voice. it should be taken as a compliment since 3 b1b is one of best intelligence channels on yt

[@sinaomidvar2807]

The content you provide is greatly appreciated. We are looking forward to your new contents. ❤

[@ariondorin4480]

This is awsome ! Man, you really understand things deep and what is more, you really explain the essential!

[@fabiospinelli4179]

Wow this video is Gold, I'm impressed by the fact that this is your first! I will look at your career with great interest

[@TheOnlyHatchet101]

1s and 0s... computers are as simple as KZbin posters. The mind is near 26 million bits, come back in 5,000 years to explain what you still can't understand. Great video.

[@exinangai2216]

The best illustrated explanation on Memory modelling of inter-connected neuron (switch), bit and quirks of Neural Networks , formation and significance of multiple "stable states", and how they are presented as row & col vectors, update function as Matrices. Your video is both clear and excellent visual intro to the topics. I am really looking forward to your next video. Helping more ppl to visualize how transformers, recurrent net, back propagation, main idea of ML papers, etc are being modeled.

[@waitingforgodot355]

That was phenomenal. Fingers crossed for the "follow-up video"!

[@noahcordier7002]

This video was clearly made by a perfectionist, and it's very enjoyable. I can't imagine the time taken to produce it

[@HubertRozmarynowski]

You're absolutely right about him being a perfectionist.. This video is literally among the best I've watched on youtube and he's still not happy with it (judging by his replies to criticism)

[@Rowan69]

I love the animation and presentation style. Keep it up!

[@spudsT]

Dude. The aesthetic of this video is sooooo slick

[@jslevenson101]

Pieces of images and thoughts put back together in our brains. Makes a clearer picture.

[@circuitgamer7759]

Took me a bit to locate the subscribe button due to KZbin's layout changes, but I'm subscribed now. You're definitely worth it :)

[@symbolspangaea]

One of the best explanations I've seen

[@parmesanzero7678]

Thank goodness I found this. I’ve been talking about the parallels between LLMs and natural minds and either the people with the neuroscience backgrounds don’t understand the tech side or the tech people don’t understand the neuroscience side. We will never achieve AGI without people spanning those disciplines.

[@LiquidMasti]

can't believe it's your first video. it's very well explained and the background sound makes it even more compiling.

[@MaciejAleksandrowicz-s5g]

What a fantastic explanation. I'm looking towards more content from you!

[@pumkin7222]

great first video! keep it going I'll be here to support you!

[@bakdiabderrahmane8009]

the IDEA explained in this video is something a was looking for for a looong time, thank you so much you have no idea how helpful you have been.