29 - Science of Deep Learning with Vikrant Varma

In 2022, it was announced that a fairly simple method can be used to extract the true beliefs of a language model on any given topic, without having to actually understand the topic at hand. Earlier, in 2021, it was announced that neural networks sometimes 'grok': that is, when training them on certain tasks, they initially memorize their training data (achieving their training goal in a way that doesn't generalize), but then suddenly switch to understanding the 'real' solution in a way that generalizes. What's going on with these discoveries? Are they all they're cracked up to be, and if so, how are they working? In this episode, I talk to Vikrant Varma about his research getting to the bottom of these questions.
Patreon: patreon.com/axrpodcast
Ko-fi: ko-fi.com/axrpodcast
Topics we discuss, and timestamps:
0:00:36 - Challenges with unsupervised LLM knowledge discovery, aka contra CCS
0:00:36 - What is CCS?
0:09:54 - Consistent and contrastive features other than model beliefs
0:20:34 - Understanding the banana/shed mystery
0:41:59 - Future CCS-like approaches
0:53:29 - CCS as principal component analysis
0:56:21 - Explaining grokking through circuit efficiency
0:57:44 - Why research science of deep learning?
1:12:07 - Summary of the paper's hypothesis
1:14:05 - What are 'circuits'?
1:20:48 - The role of complexity
1:24:07 - Many kinds of circuits
1:28:10 - How circuits are learned
1:38:24 - Semi-grokking and ungrokking
1:50:53 - Generalizing the results
1:58:51 - Vikrant's research approach
2:06:36 - The DeepMind alignment team
2:09:06 - Follow-up work
The transcript: axrp.net/episo...
Vikrant's Twitter/X account: / vikrantvarma_
Main papers:
- Challenges with unsupervised LLM knowledge discovery: arxiv.org/abs/...
- Explaining grokking through circuit efficiency: arxiv.org/abs/...
Other research discussed:
- Discovering latent knowledge in language models without supervision (CCS): arxiv.org/abs/...
- Eliciting Latent Knowledge: How to Tell if your Eyes Deceive You: docs.google.co...
- Discussion: Challenges with unsupervised LLM knowledge discovery: www.lesswrong....
- Comment thread on the banana/shed results: www.lesswrong....
- Fabien Roger, What discovering latent knowledge did and did not find: www.lesswrong....
- Scott Emmons, Contrast Pairs Drive the Performance of Contrast Consistent Search (CCS): www.lesswrong....
- Grokking: Generalizing Beyond Overfitting on Small Algorithmic Datasets: arxiv.org/abs/...
- Keeping Neural Networks Simple by Minimizing the Minimum Description Length of the Weights (Hinton 1993 L2): dl.acm.org/doi...
- Progress measures for grokking via mechanistic interpretability: arxiv.org/abs/...

[@nowithinkyouknowyourewrong8675]

Love it, I've run CCS and I learned a lot. You guys had a tranquil vibe too