#54 Prof. GARY MARCUS + Prof. LUIS LAMB - Neurosymbolic models

Professor Gary Marcus is a scientist, best-selling author, and entrepreneur. He is Founder and CEO of Robust.AI, and was Founder and CEO of Geometric Intelligence, a machine learning company acquired by Uber in 2016. Gary said in his recent next decade paper that - without us, or other creatures like us, the world would continue to exist, but it would not be described, distilled, or understood. Human lives are filled with abstraction and causal description. This is so powerful. Francois Chollet the other week said that intelligence is literally sensitivity to abstract analogies, and that is all there is to it. It's almost as if one of the most important features of intelligence is to be able to abstract knowledge, this drives the generalisation which will allow you to mine previous experience to make sense of many future novel situations.
Also joining us today is Professor Luis Lamb - Secretary of Innovation for Science and Technology of the State of Rio Grande do Sul, Brazil. His Research Interests are Machine Learning and Reasoning, Neuro-Symbolic Computing, Logic in Computation and Artificial Intelligence, Cognitive and Neural Computation and also AI Ethics and Social Computing. Luis released his new paper Neurosymbolic AI: the third wave at the end of last year. It beautifully articulated the key ingredients needed in the next generation of AI systems, integrating type 1 and type 2 approaches to AI and it summarises all the of the achievements of the last 20 years of research.
We cover a lot of ground in today's show. Explaining the limitations of deep learning, Rich Sutton's the bitter lesson and "reward is enough", and the semantic foundation which is required for us to build robust AI.
Pod: anchor.fm/machinelearningstre...
Tim Epic Intro [00:00:00]
Main Intro [00:38:05]
Gary introduces the field [00:42:12]
Luis introduces his thoughts on Neurosymbolic methods [00:47:56]
On the history of achieving a logical foundation and mathematical foundation for semantics [00:54:12]
Will emulating discrete reasoning break optimizability?
Buzzwords without basis [01:04:34]
We have known for decades about the statistical regularities in language [01:07:02]
Intension vs extension [01:09:14]
Easy to demand abstraction, but what is a workable definition? [01:13:33]
Abstraction is a "terrorist attack on neural networks" [01:20:38]
To succeed we need both, we are the moderates [01:30:14]
What would the future world look like with better semantics? [01:31:32]
Promising current approaches to discrete reasoning systems [01:39:58]
The challenge of machine knowledge acquisition [01:47:32]
Prof. Lamb's more on relational learning [01:53:06]
The role of vector embeddings and neural symbolics [02:02:30]
Humans seem both good and bad at reasoning, what's going on? [02:09:06]
Is reasoning a first-class citizen in the human brain? [02:15:06]
Does reasoning happen on the same substrate as system 1? [02:17:08]
GM papers:
The Next Decade in AI
arxiv.org/abs/2002.06177
Innateness, AlphaZero, and Artificial Intelligence
arxiv.org/abs/1801.05667
Deep Learning: A Critical Appraisal
arxiv.org/abs/1801.00631
Rule learning by seven-month-old infants
www.researchgate.net/publicat...
Rethinking Eliminative Connectionism
nyuscholars.nyu.edu/en/public...
GM YB Debate
The Best Way Forward For AI
montrealartificialintelligenc...
GM:
Rebooting AI
www.amazon.com/Rebooting-AI-B...
Kluge: The Haphazard Evolution of the Human Mind
www.amazon.com/Kluge-Haphazar...
The Birth of The Mind
www.amazon.com/Birth-Mind-Cre...
The Algebraic Mind
www.amazon.com/Algebraic-Mind...
LL:
Neurosymbolic AI: The 3rd Wave
arxiv.org/pdf/2012.05876.pdf
Understanding Boolean Function Learnability on Deep Neural Networks
arxiv.org/pdf/2009.05908.pdf
Graph Neural Networks Meet Neural-Symbolic Computing
arxiv.org/abs/2003.00330
Discrete and Continuous Deep Residual Learning Over Graphs
arxiv.org/pdf/1911.09554.pdf
Learning to Solve NP-Complete Problems
arxiv.org/abs/1809.02721
Neural-symbolic Computing
arxiv.org/pdf/1905.06088.pdf
Neural-symbolic learning and reasoning
arxiv.org/abs/1711.03902
Neural-symbolic learning and reasoning
openaccess.city.ac.uk/id/epri...
LL books:
Neural-Symbolic Cognitive Reasoning
www.amazon.com/Neural-Symboli...
A Uniform Presentation of Non-Classical Logics
www.amazon.com/Compiled-Label...

[@AICoffeeBreak]

First reaction: Yeas, finally a new episode! 🤩 Second reaction: Oh, now I have to find 2 and a half hours to watch this. 😳

[@G12GilbertProduction]

I'm ablazed from this second reason, too. :)

[@Shikhar_]

Dang! Just 2 mins into the video, and Yannic mentions Abstraction, which I have been thinking for the past couple of days (`what is it & how to measure it`). Also, loving the intro!

[@markryan2475]

Thanks very much for another great video. The production values go from strength to strength, and the topic deserves more attention than it gets.

[@dr.mikeybee]

Is it provable that our current deep learning architectures lack internal symbolic structure? I don't think so. We know the perceptron can be used to construct a NOR gate. That means the primitive is functionally complete. Perhaps we have only built inference only DNNs with architectures like transformers, but I have serious doubts about this. Fast slow network architectures do seem to have "hidden" internal loops and variables. We have no direct access to them, but they exist within the architecture. My intuition for this is basically that at some point, just as has been the case with machine translation and speech recognition, end-to-end (e2e) systems will outperform heterogeneous ones, and at some point, extrapolation will fall out of a new e2e architecture.

[@katerynahorytsvit1535]

This KZbin channel is gold, thank you, guys, for all your efforts!

[@shanicecodner7138]

6

[@DavenH]

I simply cannot wait for the Hutter / Legg podcast. Still working through episode in pieces, thank you so much for the effort Tim et al!

[@MachineLearningStreetTalk]

That would be amazing! We will try and get in contact with them.

[@sabawalid]

Another great episode. It goes without saying that I am a huge fan of Gary Marcus. I also loved what Luis Lamb had to say (it was certainly refreshing to hear of people like Richard Montague, Dana Scott and Christopher Strachey ... and the contributions such people made to formal semantics, programming, etc.) Also, an awesome introduction to the video, Tim Scarfe.

[@dan7582]

What a awesome video. Congrats on this great work guys!

[@dr.mikeybee]

Thank you for producing these fascinating episodes. I get a bit excited watching them, and my idiot savant emerges. That means, I don't always think about the niceties or social graces. I'm very sorry if I ever offend anyone here. I really rarely know about that kind of thing. So let me take the time to simply thank you. That being said, I believe the greatest hope for knowledge graphs is that we will be able to get first class training sets out of them. I believe that, ultimately, intelligence will reside in more advanced recursive transformer like neural nets. And that the agents that use these ANNs will run on policy networks created from reinforcement learning systems. Where we do use symbolic programming will be in brute-forced and evaluation-net-led synthetically generated code. Eventually, however, I believe that we will discover, and when I say we, I mean our machines, an end-to-end architecture that will render symbolic programming obsolete. I believe these things because I don't see humans as capable of designing a symbolic system directly. We don't have the storage capacity or the bandwidth; so if we ever do discover real AGI, it will be because our machines find it for us. Basically, I'm a pessimist when it comes to believing anything about humans being particularly special. We may be the top of the local food chain, but consider what's below us.

[@mattizzle81]

I agree. I think it sounds great in principle, but too idealistic. Realistically to achieve AI you would want to emulate what the human brain does. Anything else is just handwaving and speculation, we know the human brain works. So far the human brain seems to be a neural network.

[@andres_pq]

You should host a second Bengio vs Marcus debate

[@ozgurtanriover600]

Pkk

[@PrpTube]

One every 6 months!

[@earleyelisha]

To Dr. Lamb's and Dr. Marcus' points about contextual information: how does any system develop context without an embodied experience to create a grounding of the natural world. All the language that we use is founded on experience much before we arrive at more abstract ideas like mathematical principals. This, to me, seems to be at least one element lacking with systems like GPT-3 - they have no experiential grounding of the world and thus the words they they are manipulating have no subjective(or analogous) meaning.

[@judgeomega]

i imagine a human being would have a VERY hard time making sense of a world when only given a sequence of nonsequential snapshots from random positions and vantage points.

[@earleyelisha]

@@judgeomega Definitely agree.

[@baskaisimkalmamisti]

It was a great talk. thanks for all your efforts.

[@srh80]

I think I can understand why the community does not seem to respect some people. Its easy to say, guys we need to do this, this thing will not work etc. People will respect the viewpoint if one provides a well thought approach which would work. Granted deep learning is most probably not the answer. But, opinions dont carry even close to same weight as tangible ideas and implementations. Consider Chollet, he provided a great starting benchmark for analogies and a very good arguments in his paper. Its a pleasure to listen to him talk about limitations of deep learning. This is just a lot of, i have been wronged and people better respect me yapping.

[@breaktherules6035]

EXCELLENT ideas!!! THANK YOU so much!!!

[@abby5493]

Wow love your shirt in the intro 🦁 Another amazing video 😍

[@MachineLearningStreetTalk]

🦁 🦁 🦁

[@pennyjohnston8526]

As always an awesome introduction to the video and thank you for the fantastic list of go to resources !

[@reinerwilhelms-tricarico344]

When you talked about the "discovery" of the FFT by a neural net, I had to think about an apparently much simpler problem: How could an AI figure out how to compute the square root of a complex number? Here is one way to compute the square root of a complex number. Say z = a + i b, and v = x +i y is the square root, that is, you have to get v^2 = z. Thus z = a + i b = v^2 = x^2 - y^2 + 2i x y , So you have to solve two equations: x^2-y^2 = a, and 2 x y = b for (x,y). And then you need to know how to solve a quadratic equations. Even to turn this into one quadratic equation isn't entirely trivial: You find out that you actually have to replace y = b/(2x) in the first equation, which results then quickly in an equation x^2 - b^2/(4x^2) = a, and thus you have a 4th order equation: x^4 - a x^2 - b^2/4 = 0. So you need to first use u=x^2, then solve u^2 - a u - b^2/4=0. And once you get that, you find that the original problem has actually 4 possible solutions, which all look pretty horrendous. In the end the result is this: v = (z + |z|)/ sqrt(2 (a + |z|)), where |z| = sqrt(a^2+b^2). Which looks a bit like something that fell from the sky when you see it the first time. (there are other methods to get there, using the polar representation of complex numbers, but you need plenty of trig to reproduce the above formula). On the other hand, once you have this formula for v, the equality of v^2=z can be confirmed easily by carrying out that formal computation. But as I know from trying 🙂 it's a bit of a mess to get to the formula for v, while proving it to be a correct solution is no big deal. Now. If I wanted an AI to find this formula, it would have to at least have all the common rules of complex algebra built in as possible operations, as well as methods for solving quadratic equations built in. I have my doubts that some of the existing AI models would find the proper path through the jungle, and I bet they'll just look it up, because you can find the above calculation probably in many books and online tutorials and what not, so that all the heavy duty AI's can cheat. Of course, you might be lucky if you build an AI that turns such queries like "What is the square root of a complex number z = a + i b" into a query to Mathematica. But that might be seen as cheating. BTW, there's a very similar formula to compute the square root of a quaternion. I tried ChatGPT a while ago and it didn't know this formula, even though it could deliver a C++ class for Quaternion algebra. Q = a + b i + c j + d k. And |Q| = sqrt(a^2+b^2+c^2+d^2) . a is the scalar part of Q or a = Re(Q). Find V, with V^2 = Q, has a solution: V = (|Q| + Q )/sqrt(2 (a + |Q|)).

[@jondor654]

The intentionality of the underlying programmatic infrastructure is a prior to the associated probabilistic layers

[@ikiphoenix9505]

Thanks for these. What Ohad Asor (now joined by Pr. Franconi and Pr. Benzmüller) is doing is very promising as a backbone.

[@muhammadaliyu3076]

Advice to Machine Learning Street Talk: please stop having more than 4 people in a single podcast. It's distracting, and not everybody is giving the opportunity to talk indepth. If you have 2 guests, then have 2 people asking questions. If you have 1 guest, then you can have 3 people asking questions. And also I believe I get alot more knowledge from you guys when you have a single guest in which that guest will have enough time to share his or her views and knowledge.

[@TimScarfe]

Thanks for the feedback, we agree

[@firstnamesurname6550]

Ops, These dudes are "ML street dudes", they can - simultaneously - process Non linear information in 10 ^ 500 parallel layers, in random discrete fragmented chunks, probabilistic, fractal, monster group vertex algebra, non commutative algebraic geometry, topological in n-dimensions with surreal numbers with boolean q-bit algebras, interuniversal tech-muller theory exponential attractors.... etc, etc, etc Why they have to constrain themselves to satisfy your cognitive bottlenecks??? 🤖

[@victorsmirnov876]

My 2c. There is no problem with "rewards for everything". There is a problem with rewards at the level of symbols, because subjectively, rewards are feelings. Formalization of feelings (reducing feelings to symbols) _is_ the Hard problem of consciousness. One of the reasons why connectionists try to avoid symbols, is they are trying to escape from the Hard Problem. In ontologies, there is the problem of Observer, that is _apparenty_ is hardly reducible to anything we can name symbolic. Neural codes are _apparently_ more expressively powerful than plain symbols, but if they are implemented in a digital computers, they are the same symbols, just different interpretation of. More specifically, phenomenal part of the human knowledge (represented ans feelings and intuition) is very rich and (hence) functionally important, but hardly reducible to _both_ classical symbols AND neural codes :) Today, the problem is not _that_ hard as it used to be. Just need some more advanced psychology and math.

[@danielbigham]

Ever since the rise of deep learning in the earl 2010s, my intuition has been that we need to learn how to integrate and synergize symbolic approaches with continuous/connectionist approaches. I think many people have that intuition, including Gary Marcus. So let's do it! Sometimes I feel like Gary is trying to energize a tribalistic feud, rather than bring integration among researchers. That's good for getting attention, and maybe some attention is due, but it comes at a cost.

[@nmc8187]

The "what" (integration of DL & symbolic approaches) makes perfect sense, the "how" is what's lacking, and for that Gary Marcus is not the one to blame :)

[@michaelwangCH]

Tim, how can we visualize the parameter space under the neural network, could you make some recommandations? Thanks.

[@fraserg]

These guys yap on and on while saying very little. I didn’t here any new or even & actionable ideas. Maybe we could have a language model write a graph neural net tree? At least give some testable ideas plz.

[@ai._m]

In hindsight, was platforming GM a good idea? It's been a net negative for the world. (Lex blocked him as penance for platforming him years ago)

[@setlonnert]

Yet another interesting talk. Good work! You are always very informative and present the current concepts and topics (and the historical) with great detail, in/from machine learning. I have learned a lot each time. Thank you! However as I have had a previous experience from analytical philosophy, and some experience from the previous AI "revolution" in the 80'ties, there seems to be a slightly rudimentary approach to philosophy in many of the discussions. Which is a pity. For instance, as an example, when you often discuss language it seems to stay, or stays, within the boundaries of the brain of individuals. Or in the brains of many individuals. But if you would pick a philosopher such as Willard van Orman Quine, he states (stated) that "philosophy is a social phenomenon". If you, for the sake of argument, assume such an approach the usual neural network implementation of pouring data over a single but maybe complex network wouldn't get you anywhere as it lacks any "social interaction". It might be that we could think of language in machine learning or maybe some any other system, animal or whatever in other ways than natural language in men. Many such small but limiting assumptions of what a language might be, might impede restrictions on your discussions. But as always: they are very refreshing to me, at least!

[@diegofcm6201]

Although I do agree with the knowledge base approach being necessary, we would first need GARANTEES that it’s both in the right format (encoded correctly) and in the right time to make sure that the system WILL get it once it’s exposed to it. Building lots of priors without making sure they will be useful could just be a waste of time: imagine it needs, for some reason, to be in a certain code and we build a knowledge base that’s in another one that’s not directly translatable to the needed one.

[@willd1mindmind639]

Part of the confusion owes to the fact that the idea of "intelligence" and "the mind" or "consciousness" have always been considered within the world of philosophy not the discrete sciences. So there has to be a clear distinction between trying to arrive at a common definition of "intelligence" from a philosophical perspective, versus a discussion of intelligence from a biological perspective (ie. are cells intelligent?) and separate from "intelligence" in a computer science perspective. And even within computer science there are additional sub areas related to this discussion, whether it be algorithms and mathematical models or physical hardware and systems design. And in that vein, from a computer science perspective, what is the most common paradigm of AI today in computing is basically something based on convenience, not something intended to be the end all and be all of intelligence in computers. That is because in concrete terms there is no direct correlation between cells in the brain and how they function and "computer neural networks" either in physical terms or logical terms. And to get to that will take many, many years and at the end of that computers with such "intelligence" will still be a physically separate physical taxonomic entity from biological organisms. But for the time being we don't need to wait for that evolution in order to make use of some practical models of "intelligence" to build solutions that are practical and portable for modern use within the industry. And because "big data" is growing exponentially, it only makes sense that these models are the most dominant because the data is going to be there regardless. It makes practical sense to use that data as part of the process of evolving towards a future development of more human like "intelligence" in computing. The problem is that because of this only certain models and algorithms dominate the R&D and scholarly space which actually encompasses many different domains within computing from hardware to software. And the problem there is even with promising research, what is the viability of using it from a business perspective? How many companies can actually train and build their own GPTx model? It is bad enough just getting businesses to adopt the current models of "AI" let alone getting them to build more advanced "systems" of computing with some sort of "intelligence". Which is why it is the big data companies that are the ones currently most poised to "bridge the gap" in the between all of these various areas and providing some kind of useful "products" for businesses to use in the AI space. I think the "debate" more or less boils down to what paths should be taken in the R&D and scholarly community across the various computing domains that will ultimately lead to the result of truly intelligent forms of computing. Because the biggest problem here is the idea of a 'one size fits all solution" for anything and the idea that the current models are the best solution for everything in AI which I don't think anyone is really saying. As opposed to those models being a good solution for certain types of problems and solutions which have the most "bang for the buck" in the current computing ecosystem.

[@willd1mindmind639]

That said another way of describing the problem in modern usage of neural networks is that they encompass domains that are too wide. So within GPT 3 it has hundreds of classes of data that it contains and can use. However the problem is it doesn't have DEEP knowledge about any specific class of data. Trying to do an identity function correlating to those classes learned without labels is difficult because the feature space is too wide and includes too many mappings and meanings. So, for example, if you had just a network that segmented images into individual objects and then sent that data to a neural network representing a "class" of objects, then you would be getting closer to what human brains do. There is no one neural network (or set of neural signals) for all things in discrete space. There are separate sets of signals or networks for each thing in the brain which is the basis of logic and reasoning. Having a segmentation network be trained to distinguish objects (labels are irrelevant) as inputs to other networks for specific things would provide a much better approximation solving for better reasoning and understanding. That is not a single monolithic model just mapping data and patterns to labels. One of the benefits of this is it removes the need for mechanical turks and only handles visual knowledge. But this idea of collections of networks having deep knowledge about specific things is a better approach that ties into this idea of a knowledge database. Because each network has "features" specific to a class of objects that are more relevant in reasoning and understanding those objects than a generalized model that only understands high level label mappings of things to patterns across a large data domain. Of course it can even go deeper where there are additional networks within those top level class networks but that is one way of combining both approaches. However the problem with that is most companies aren't going to invest in developing such complex systems and there probably isn't going to be any sort of "AI" that glues all of these together at least in the short term vs traditional programming frameworks.

[@lenyabloko]

@@willd1mindmind639 you can consider GPTx a commercial attempt to build an underlying layer of intelligence. No one expected it to "understand" full stack of human intelligence. And I agree that no one wants to invest in human-like AI for a sake of being exactly like humans. The question is: what should the architecture be?

[@willd1mindmind639]

@@lenyabloko You started off with the million dollar question. First there has to be a consistent definition of intelligence in computing as a standard or set of standards. And I don't believe that exists right now. As time goes on I guess that coming up with that set of standard definitions will help define new architectures. But right now the word "intelligence" is more of a marketing term mostly referring to classes of algorithms and frameworks that can do certain things well, such as GPTx. And when I talk of standards for measuring computer "intelligence", I mean something like having an AI system with no coding being able to read an entire book and then generate a narrative summary of the book on its own. Or being able to answer questions about what it "read" in the book. GPTx by itself cannot do this.

[@sgramstrup]

I don't know enough about this, but Semantics/abstractions are caught by our language, and can be extracted from the neural network. So why aren't we talking about an 'addon' to what we have ? We are not going to get much better datasets soon, so we need an addon that have focus on meaning. So why all the debate, if we can 'just' produce an addon ?

[@ThiagoSTeixeir4]

Best podcast about ml.

[@LiaAnggraini1]

reading the title be like '.. it's gonna be a tough topic'

[@NextFuckingLevel]

Maybe we should ask GPT-3 to summarize this topic 😂😂

[@NextFuckingLevel]

Also i hope in the next few years we could see Xtra large language models akin to GPT-3, but trained solely on ""bahasa indonesia"

[@LiaAnggraini1]

@@NextFuckingLevel yes that's right. We can build it as long as there's enough resource 😅

[@thantin4348]

A THEORY OF MIND IN A NUTSHELL Mind-Body and Wave-Particle are terms that are integral to the discussions of philosophy and physics, and we have acknowledged the obvious difference in our usage. However, what is not obvious is the similarities (the analogies) that exist between the two terms, like for example: What Mind is to Body is similar to what Wave is to Particle. Translated into the format of ordinary language, the analogy can be read as: Mind is wave-like, and Body is particle-like. With a larger list of duals/terms, we have the potential to generate many interesting propositions, and in fact infinitely many: “RC- FC” (Roughly Correct - Finely Correct) ~ “Mind-Body” ~ “Wave-Particle” ~ “Analogy-Reason” ~ “Subjective-Objective” ~ “Freedom-Determinism” ~ “Variation-Selection” ~ “Software-Hardware” ~ “Same-Difference” ~ “Chance-Necessity” ~ “Action-Reaction” ~ “Quantum-Classical” ~ “Boson-Fermion” ~ “Random-NotRandom” ~ “Empiricism-Rationalism” ~ “Induction-Deduction” ~ “Imaginary-Real” ~ “Vector-Scalar” ~ “Order-Disorder” ~ “Space-Time” ~ Discrete-Continuous” ~ Unconscious-Conscious” ~ “Automatic-NotAutomatic” ~ “Linear-NotLinear” …and so on ad infinitum. See for yourself and it’s fun!

[@dr.mikeybee]

What we've learned from CNNs, for example, is that as they get deeper, higher and higher abstractions are learned. Why should we assume that this will not be the case with transformers? Is extrapolation just a higher abstraction? Is metaphor? I believe that they are. As I've said, with GPT-3, we are at least two orders of magnitude away from our own biological neural nets. What higher level abstractions will be learned as we go deeper and wider? I'm currently using a distilbert model for question answering, and I'm symbolically curating context, storing, and retrieving, but it's a hack, IMO. Eventually, I feel fairly certain these will be handled e2e by an extremely deep wide fast slow ANN architecture.

[@SimonJackson13]

Analogy as a transfer of verb action to an alternate set of meta-motor neurons perhaps including symbolic transformative actions of "makable" futures to evaluate?

[@SimonJackson13]

What question does an AGI solve first? The extraction focus of existential risk within the model?

[@mobiusinversion]

Imho the problem is related to the task. Next token generation is simply unrelatable and uncontrollable. There are algorithms that convey common sense reasoning in language understanding tasks, but they are not open ended token streaming. I’ll demo one of those algorithms for my interview on MLST one day ;)

[@SLAM2977]

Epic Intro !

[@RavenAmetr]

Watching this made me hopeful about the future of AI. I can see real critical thinkers over there. Really sick of stupid hypesters already.

[@dr.mikeybee]

If I enter "The identity function is f(a) = a ∀" Will GPT-3 produce "f(a) = a ∀ a ∈ R?" If so, it does knows the identity function. Can it be trained to produce python code for the identity function? Yes it can. GOFAI advocates will always say ANNs don't understand, but what is meant by understanding? Moreover, why should we ever expect that a computer will do anything the way a human does? It's not human. Judea Pearl said, "It takes real intelligence to fake intelligence."

[@lenyabloko]

I think the problem of identity is to find all features of a given extension. That means to learn a latent representation of a training set. And that requires an objective function. In addition, ANN can only learn a differentiable objective function. There is a large body of work on Inductive Logic Programming, including neural architectures. But is remain a fundamental chellange.

[@robbiero368]

Wouldn't you get all the same issues GPT3 has if you just took a baby and gave it all the data from the Internet and no schooling. Which is to say isn't the order and manner in which we build up the data and structure stored in these gigantic networks not given nearly enough attention.

[@satychary]

Si, 100%. But there is something much bigger that is missing - the baby has a body, GPT-* does not :) BTW GPT-3 has been surpassed by Google's Switch Transformer, which in turn has been leapfrogged by China's Wudao [悟道] that has 1.75 trillion parameters - but still, no dice. Bigger is not better, there is no switch that will flip when one of these reaches 100 trillion params, for ex.

[@robbiero368]

@@satychary I wonder if the embodiment needs to be physical though, would a virtual one be enough as long as the physics of the world are consistent. At least to create a simple general system

[@satychary]

@@robbiero368 - it does, because the virtual version is computation based, whereas the real world is directly experienced by real bodies, so the two aren't equivalent. Also, FYI :) www.researchgate.net/publication/334521819_A_VR-Based_System_and_Architecture_for_Computational_Modeling_of_Minds

[@robbiero368]

@@satychary I'll read the paper but my immediate question would be that all VR systems don't yet create the richness of inputs that the real world does but that doesn't mean they cannot especially if it's decided that's what is required

[@lenyabloko]

@@robbiero368 I think it requires a combination of VR/Oracle and selection pressure. The main chellange is to find a basic architecture that includes in a simple an practical way.

[@eazioeasille9598]

That's the guy from Rambo

[@keithkam6749]

2:17:00 @Yannic RE: idea of reasoning as pattern matching - I agree, but I think you're missing a jump that the interactionist theory of reasoning from Sperber and Mercier's 'The Enigma of Reason' bridges- It's a psychology book so maybe not the ML/AI crowd's cup of tea but many of the ideas are very applicable: The core idea proposed is that we humans do not have what Kahneman describes as 'system 2', logical thinking. (in 'Thinking fast and slow' he proposes that we have two types of cognition, system 1 = fast cheap intuitions, system 2 = slow expensive logical deduction). Instead, Sperber and Mercier suggest that all we have are intuitions - the ability to pattern match. ** Specifically, our ability to reason is actually intuitions about reasons, combined with intuitions for evaluating the validity of reasons. ** They argue that the primary purpose of reasoning from an evolutionary perspective is not to generate new knowledge from existing knowledge, but instead to generate emergent consensus and allow for cooperation between non-related individuals. 1. Alice wants to convince of Bob something - e.g. a new idea, a proposal to do something together, a justification for an action. 2. Of course, Bob would not accept everything Alice proposes. If that was the case they would be gullible and easily taken advantage of. 3. However, it is not beneficial to reject everything Alice proposes, since the knowledge could be useful (for Bob or for both of them). 4. To get around this, Alice proposes clear to follow, logical reasons that Bob then has to evaluate. Perhaps the key to reasoning in an ML context would be this generative, adversarial process, combined with an ability to direct attention to existing knowledge bases or new experiments. (paraphrasing of a comment I made a few episodes ago)

[@TimScarfe]

Thanks a lot for the comment!

[@Dr.acai.jr.]

"Knowledge is a justified true belief' I do not understand this. What is a true belief? If you know something is true it ceases to be believed. It is understood. It is understood through computation.

[@dr.mikeybee]

I recently watched a bunch of old MLST episodes. This one stood out. I found myself getting really frustrated. Prof. Marcus keeps saying that machines don't abstract well. What does that mean? He doesn't describe what it means to create an abstraction or an analogy. We need that. It's silly to say, my eight-year-old does better. Eight-year-olds have evolutionary priors; so of course they do better. So here's what I mean by abstraction. You chop up a bunch of data by a feature, and then you name the characteristic of the feature. You put all the odd numbers here, and the even numbers there, and you have two abstractions and their instances. For analogies, you do semantic encoding. You get all the tokens into a vector space and arrange them by predicting dropped tokens, and you end up with a semantic vector space where king is to queen as man is to woman. That's an analogy. We can do that. The combination of the two methods produces probabilistic reasoning that traverses as far as we've gone in creating layers of abstraction and as wide as we've gone by increasing the number of dimensions in our semantic encoder. Am I missing something? I don't think so.

[@MachineLearningStreetTalk]

"you end up with a semantic vector space where king is to queen as man is to woman. That's an analogy. " that's one analogy which is learned from data, the key is *forming* new analogies, and there is an infinite space of possible analogies

[@dr.mikeybee]

@@MachineLearningStreetTalk The learning happens in the training. That's where new analogies are discovered.

[@kristinabliss]

It humors me that so many people are agreeing to the absurd premise that humans can confine & control another intelligence.

[@antonschwarz6685]

Some really interesting stuff, but also alot of partisan, ad hominem attacks on straw men. I agree with Gary's views 100% but this barely gets off the ground, addressing none of the interesting and reasonable replies deep learning advocates might give, and proposing few concrete solutions. The episode with Bob Coecke was totally fascinating- outstandingly original thinker, Francois has the beginnings of what could be a powerful formal framework for understanding what DL will/wont be capable of. This episode mostly just criticised people and recited tired and vague introductory points, most of which DL people would strongly agree with, not at all indentifying the sources of divergence. Sorry for the negative comment, I love the show but the epistemic health of the field is crucial and as mammals, we need strong norms that prevent the politicisation of conversation.

[@MachineLearningStreetTalk]

Thanks for the comment, but I don't think there were any ad hominem attacks? What are you referring to?

[@antonschwarz6685]

@@MachineLearningStreetTalk First, I should have mentioned that I loved Luis' contribution and would like to listen to him a lot more, that large part towards the end was interesting and I greatly admire his predisposition to give his interlocutors the benefit of the doubt. Gary's comments about Connors views, those of Hinton/Bengio/DL folk generally were all made in an unhelpfully scathing tone and attacked weak versions of their beliefs which they certainly do not hold. The suicide example or invoking the 'terrorist attack' reviewer comment portray those he disagrees with extremely negatively, and greatly amplify the exact incivility of these conversations which he himself dislikes. Almost all reasonable DL fans agree basically that generalisation is abstraction, and that weak models latch onto imperfectly predictive, causally downstream epiphenomena. The question is how much of the data's underlying generative functions can be expressed/learned by these distributed models, an extremely difficult question, especially when we understand little about either. As with Gary, my guess is nowhere near enough, but he basically spent an hour slam-dunking over views which no-one interesting would defend. Perhaps I'm being too harsh, but conversational norms are critical. for reference see fox news/cnn etc......

[@MachineLearningStreetTalk]

​@@antonschwarz6685 Connor is a very close friend of ours and I am pretty sure he wouldn't be offended by Gary's comments here. We did discuss this with Connor just before the show. We are sorry if we didn't get the tone right. It's our policy to edit out any ad hominem stuff. Perhaps there are sensitivities and history on this one. I (Tim) think Gary is super respectful of the DL folks, he does feel somewhat vindicated that the DL narrative has moved to support his position more in recent years but that's only because he respects the leaders in the DL space so much. And in my opinion Gary is (if anything) more of a connectionist than a symbolist! Gary is way closer to the deep learning folks than, say, Walid Saba is. Embedding symbols in connectionist models is still connectionism IMO 😀

[@antonschwarz6685]

@@MachineLearningStreetTalk Ok, good to hear. I too apologise if my tone here has been less than perfect. Sublime researchers are reliably also exceptionally gracious (Max Welling for instance!) so its worth looking out for this stuff. Anyways, very kind of you to respond. I've listened to many episodes several times as getting away from the laptop, getting outside and switching to audio is extremely refreshing. Literally an invaluable resource. Big thanks.

[@ayushgarg70]

19:19 tf is that gloomy music

[@oualadinle]

show me one piece of literature that lS justified in arrangement of characters

[@fritz3388]

"Knowledge is a justified true belief", or self-delusion like so much "knowledge" in medicine and science.

[@alainportant6412]

Nerd