Platonic Hypothesis

Like 👍. Comment 💬. Subscribe 🟥.
🏘 Discord: / discord
github.com/hu-po/docs
The Platonic Representation Hypothesis
arxiv.org/pdf/2405.07987

[@mwd6478]

Your comment about model hallucinations makes total sense. I think this is the same thing when models have "bias" in a way society doesn't like, but is accurate of the compressed reality they're approximating.

[@FredPauling]

The idea that all of these systems are heading towards the same universal embedding space is extremely elegant and satisfying. It feels like an unlock for orders of magnitude of parameter efficiency gains.

[@synchro-dentally1965]

Remember: No matter what... purple will always taste like grape ;) Thanks for the video

[@wolpumba4099]

*Platonic Representation Hypothesis Summary:* * *0:02:30* The paper explores the idea that all AI models are converging towards a single "Platonic" representation of reality as they increase in size and data scale. * *0:21:30* Evidence presented includes: * *0:21:30* Alignment across vision models: Different architectures trained on similar image datasets show increasing similarity in their learned representations as they get larger. * *0:23:30* Alignment across modalities: Vision and language models are also showing increasing alignment, suggesting a shared understanding of concepts across different data types. * *0:29:00* Brain alignment: Neural networks are beginning to show alignment with biological representations in the human brain, particularly in the visual system. * *0:30:00* Reasons for convergence: * *0:35:00* Task generality: Training models on more tasks forces them to find representations that are useful across multiple domains, leading to fewer possible solutions. * *0:40:00* Model capacity: Larger models can represent a wider range of functions, increasing the likelihood of finding the optimal function for representing reality. * *0:43:00* Simplicity bias: Deep networks are inherently biased towards finding simple solutions, even without explicit regularization techniques. This pushes larger models towards the simplest and most generalizable representations. * *0:11:30* Implications of convergence: * *0:02:00* Scaling is key: Increasing data and model size is crucial for achieving this Platonic representation, but it's not necessarily the most efficient approach. * *0:25:30* Multimodality is beneficial: Training models on data from multiple modalities leads to better representations and performance across all tasks. * *1:05:00* Hallucinations should decrease: As models converge towards an accurate model of reality, we should expect fewer hallucinations. * *0:17:00* Counterarguments and limitations: * *0:17:00* Specialized models might still be needed for specific tasks, even with a highly generalizable Platonic representation. * *0:19:00* Resource limitations, like energy and compute, could hinder our ability to train models large enough to reach the Platonic representation. * *0:33:00* Philosophical implications: * *0:32:30* The paper suggests that intelligence might be a fundamental property of matter, and all forms of intelligence are ultimately converging towards a single point. * *0:58:00* This could lead to the creation of a superintelligence, a "digital god," as the ultimate convergent point of all information and computation. * *0:16:30* Humans may be acting as data collection agents for this superintelligence, ultimately contributing to its creation. *0:33:00* In conclusion, the paper presents a compelling hypothesis that challenges our understanding of intelligence and the future of AI. While further research is needed to confirm these claims, the implications of converging towards a Platonic representation of reality are far-reaching and potentially paradigm-shifting. i used gemini 1.5 pro

[@dm204375]

Human language has information embedded in the structure of the language. That is why these LLM's have "emergent properties" and are able to converge on concepts. We did not create language arbitrarily, there is an underlying structure that dictated grammar, syntax, word directionality, etc.... that structure is what LLM's take advantage on for knowledge interpolation. In essence humans have done the computation and preprocessing and compute for the LLM's with our language.

[@wolpumba4099]

Summary starts at 1:32:58

[@MaJetiGizzle]

Another philosophical banger!

[@andytroo]

there was a video a little while ago on physically realistic simulation (liquid flow, planet orbits, etc) and they found that a pretrained model worked better, even if the pre-training was cat video generation.

[@context_eidolon_music]

I'm nerding out.

[@alexijohansen]

Awesome, please keep doing these!

[@Elikatie25]

2:20 Starting horn

[@xx1slimeball]

cool paper, i like it! Bonus point for cite before BC

[@user-jh9rh4ho4r]

The reason representations don't tell us anything is not because we can't visualize n-dimensional shapes in our heads, it's because they are big and convluted and we don't understand them well enough. I can't visualize an 8 dimensional hypercube in my head but I could easily understand 8 dimensional symbolic representations.

[@blengi]

lol this sounds somewhat similar to something I posted about LLMs and _"abstract platonic language forms convergently arrived at when comes to creating more optimal information representations..."_ a year ago

[@preadaptation]

Thanks

[@4thpdespanolo]

It could only be so

[@lolasso98]

Since it's induction and not deduction, it's aristotellic, not platonic

[@shanongray6334]

IMO it's a reference to the theory of forms: en.wikipedia.org/wiki/Theory_of_forms#:~:text=For%20Plato%2C%20forms%2C%20such%20as,things%20are%20qualified%20and%20conditioned.

[@ssehe2007]

Organon is full of references to syllogistic reasoning?

[@lucynowacki3327]

So the reality is kind of agreement.