Thank you so much. Your channel is a gem.

Interesting, but It's not entirely clear how R works?.... </eos>

Awesome paper and video! I didn't realize what a cool innovation the NTK approach was and the fact that it first originated on reddit r/LocalLlama! Crazy (awesome) times.

great one agn :)

basically openai used deepseek research to develop o1 and o3

It's Chinese engineers against Chinese engineers....😂

You were doing so well until you "inadvertently" used dismissive language. Hindsight is always 20/20. Why didn't I think of this?!

Liked; for the details in this review.

Excellent!

What is most important (as some people claim, BIG IF: IF this is true), the RL part was for a largely trained on synthetic data. Because if that is true, the more time and compute you throw at it, the more performance you can probably get. Which means the sky is the limit. Which is maybe what o3 is...?

Skip the prelude and go straight to 3:42 Lot of preamble, skip to 14:50 Base Model V3 Innovations: MoE Top-k vs Top-1 Routing: 18:50 Base Model V3 Innovations: Parallelization: 23:23 Base Model V3 Innovations: Use of FP8 training: 28:02 DeepSeek R1 35:39 Quotable 40:39: "infuriatingly simple...write down a checker (verifiability)...run a model against that checker...it'll learn to get the right answer and do it in a way using it CoT to think about the answer"

Fantastic review!!

Western led technology communities??? For who?? Deepseek sounds like a game for the LLM story. This is a very expensive story.

Tanks Sasha 🎉😊

Thank you so much Sasha!!!

amazing review

I am not able to understand the compress function. Can someone please explain?

followup?

13 (pad_to) could also be - where(arange(j) < i, a[arange(j)%i],0)

```python def bucketize(v: TT["i"], boundaries: TT["j"]) -> TT["i"]: i = v.shape[0] j = boundaries.shape[0] return (1 * (v[:, None] >= boundaries)) @ ones(j) ``` Is this a matmul of shapes (i, j) and (j,) resulting in (i,)? I'm not used to this because when it comes to matmuls I think of row vectors having shape of (1, j) Yet when I try `ones(i) @ (1 * (v[:, None] >= boundaries))` I get a (j,) tensor back So in this case it is interpreted as a matmul of shapes (1, i) and (i,j)? What is going on here?

Great video. One question tho? isn't computation of attention 2TD? instead of TD? I say that because when you do a QK_transpose. You compute in TD and then when you multiply with V values you again multiply with TD? making it 2 TD for each time step?

'The innovations driving rapid AI research' [9:41-12:34]

Shouldn't equation in 18:07 be E_(y~p(·|,z_(1:t),x))[Ver(y)]? Adding z_(1:t) into the expectation value equation's subscript.

Nice

🔥

Why do you think that spamming subscribers' "Subscription" lists with a dozen videos a day is something they want?

Hype playlist😤 How're u finding shorts as opposed to horizontal video? Tried it for awhile and didn't seem worth the extra effort but I'm considering revisiting the idea

Thats what i am looking for. Thanks

"When paralyzing a neural network...." 🤨 -- it's a common thing to hear in ML videos and podcasts due to relaxed speech patterns (though I believe some people think it's genuinely the correct way to say 'parallelizing"), but it always catches me off guard and sounds so incorrect. Random thought of the day. Looking forward to the rest of the videos on this series. Thanks for uploading!

🎉

Thanks for this video! It's really useful. Something I don't fully understand is how for Fine-tuning (bottom) at 10:07 you get 2x5BxD for the activations. Could you please help me to understand it?

Thank you, this is super useful and very insightful! As a new industrial phd student with limited support from my professor and not a lot of expertise in my specific field (3D generative models) in the team, I really appreciate this video.

Welcome to the beautiful world of shader coding... has your brain started to see the world around you in little code snippets of procedural functions? I wrote my bachelor thesis on how awful language models are for shader code completion, which is also accepted at a conference workshop now and will get published eventually. I didn't look at instruction models or closed models, so it's promising to see Claude being somewhat useful with WGSL even. There is a leaderboard for all the open models I run on HF, look for a space called Shadermatch. While I didn't start the python implementation, I worked on it for the past year and half and still do. The repo might look a bit abandoned but there is a large PR in the works that adds more functionality that is present on the website: multipass shaders (called Buffers on the Shadertoy website). Feedback greatly welcome

This is probably the 3rd time I've watched this since you posted it. I still can't shake it, you capture the 5Ws+H perfectly imo. It seems to have been overlooked in the general as I keep seeing people conflating things across models and architectures - which I find annoying. It could be a byproduct of the development cadence or the many hype trains littering the internet, but I always feel this is so significant that it's worth trying to portray it in such discussions. I have this itching question that I'm curious what your thoughts are on it. Thinking beyond what you've covered: in the same manner that LLMs derive these language patterns and then are leveraged, do you think that o1, in it's TTC graphs and results, along with supplying the output is there a more general heuristic pattern being derived from the process? If so, wouldn't these general heuristics effectively apply to different scopes and across domains, at least inevitably?

Coming back a month later, this is still the goat (We've gotten more of a consensus since then about what's happening, but who's to say that some of these strategies aren't (or couldn't) be using at train-time [though, in usual OAI fashion, it's probably just the simple thing scaled up -- RL with ORMs]).

Hey, great video! I've been trying to wrap my head around O1 for a while, and this really helped me put things into perspective. I'm surprised I haven't seen more discussion about using special tokens for reasoning. It seems like trying to generate these really long, abstract sequences for reasoning can be difficult and hard to evaluate. I have a strong feeling that we could make LLMs more stable by using special tokens as anchors to keep them from going down the wrong path during reasoning.

Since you strike me as the type of person who responds well to thoughtful feedback: It would've been nice to have a 90-second overview of the paper and its core ideas before starting. I know this isn't a video about GPT-Q, but I'd have found it useful context when going through the sections and studying the authors' approach to writing. Regardless, another amazing video!

Hi Sasha, Thank you for your Module 1 & 2. That's really interesting! And can't wait to enjoy Module 3 & 4!

Thank you so much for your detailed information. 🙏

i love the fact that not only does this research exist, but someone went through the effort to distil it in such an intelligible way. Thank you!

Did you guys just recently read the original BERT paper, added a random masking, a few repeats and done?

Sasha is the tops.

🎉❤terrific video, thank you

Amazing talk professor , recently attended your talk at Penn state .

I’d really like to see this combined with nGPT as that model seemed to innately have an ability to generalize to out of distribution sequence lengths

@srush_nlp Great explanation! How do you think discrete diffusion models should be modified to enable long context sequence generation comparable to LLMs?

Cool lecture, thanks!

The o1 test time compute plot x-axis is on the log scale, that means that you will need exponential compute to make a linear improvement, so it will be grinding to a halt

Great video! Looking forward to more academia advice content.

My way of doing compress! def compress(g: TT["i", bool], v: TT["i"], i:int) -> TT["i"]: small = v[g] i = g.shape[0] j = small.shape[0] return small @ eye(i)[:j,:]