Speculative Decoding: When Two LLMs are Faster than One

[@dev.22.-1]

Super clear explanation of speculative decoding. I have been working with this for a while but this clarified some of my questions

[@kaenovama]

Really well done, my brain bulb went light up when you show the table! Thank you, keep it up!

[@oc1655]

i recently wanted to brush up on this because it's been a while i read this paper. browsed a few tutorials/blog posts. it's funny how many people wrote about this without understanding it. you certainly did understand, and do a great job at breaking it down. thank you very much

[@decycle2912]

really informative! One thing that I don't understand is how does the LLM knows the previous probability distributions in a single pass? I thought decoder llm's only outputs the new token's probability distribution

[@EfficientNLP]

This is due to the way the transformer architecture is set up: during decoding it takes as input all previous tokens and computes the hidden states for all previous tokens at each layer. Since we have the final layer hidden states, it is possible to obtain the probability distributions for all previous tokens.

[@henkjekel4081]

@@EfficientNLP Now I get it. Did you also explain this somewhere in the video? Maybe link to it in the describtion because this is also where I didnt understand it.

[@shairuno]

We can compute the probability for each token in a batch fashion. The trick is to use mask attention.

[@igorfilippov1221]

Very clear explanation, thank you!

[@einsteinsapples2909]

Thank you for the video, when i first heard this idea in February i was wondering how it made sense because i was picturing a large K, now seeing that the recommended K is about 3 I understand how most of the output will be the same.

[@meisherenow]

Clear and succinct. Well done.

[@kevintai8656]

Great work! Thank you

[@rajmankad2949]

What an amazing explanation! Thank you so much

[@vukrosic]

Thank you for explaining it!

[@shairuno]

I haven’t read the paper yet but my understanding is that we sample from q(x) - p(x) because we want the most surprising token that the draft model does not anticipate. It should maximize the entropy but then it should have log on the equation, anyway, I gotta read the paper to understand the math.

[@甘楽-u7v]

Very easy to understand. Thanx so much.

[@420_gunna]

Love your video, thanks! If I had to give one request/critique, it'd be that I wish there were some slides in here similar to Samuel Albanie's videos that are quite information-dense recaps that could be lifted out of the presentations and put into our notes (or into a powerpoint for a paper club, or something).

[@EfficientNLP]

Interesting idea, though my videos often contain animations, drawings, screencasts, etc., and are not directly a recording of PowerPoint slides. Feel free to take screenshots of my videos for any educational purposes though!

[@magalodontestnoneworld9043]

Very good video!

[@paull923]

great explanation thank you

[@kylewilliams9214]

Google and DeepMind doing the Spiderman meme 😅

[@jackzhang9053]

Thanks for creating this amazing video! I’m wondering if you could open source the slides as well?

[@EfficientNLP]

I'm glad you enjoyed it! I'm not planning to release the slides, but I'm happy to answer any questions.

[@rexyl547]

Great video! Keep it up!

[@anshumansinha5874]

Hi, thanks for the great content. I have a question, Let's say during speculative decoding (vocab size = 5 token only) we got to a stage where draft model has the next token distribution as = [0.35,0.3,0.15,0.2] and target_model = [0.4,0.5,0.05,0.05] so now the prob of token 1 in draft = 0.35 and prob of toke 1 in target = 0.4. what will the speculative algo do? Now if the speculative algo picks token 1 from the vocab, can we still say that we decode the exact same tokens what the larger model would decode? Thanks

[@EfficientNLP]

Yes, the algorithm will always decode the same tokens (more precisely, generate tokens in the same distribution) as the larger model, no matter what the draft model does. However if the draft model picks token 1, it does not always mean token 1 ends up in the final output, whether it gets accepted or not depends on the ratio of probabilities; that's the rejection sampling procedure described in the video.

[@waynelau3256]

Thanks for this! I've been enjoying your videos! Do you think you do a review / explanation on flash-decoding by tri dao? I have been reading the pytorch blog but I don't really understand it

[@EfficientNLP]

Thanks for the suggestion, I will add it to my list of future topics!

[@dorianlin491]

Really helpful video!

[@Basant5911]

made very simple, but one more variable is choosing right draft model. Suppose if one chooses that is too too away from larger one's distribution then its also a problem.

[@EfficientNLP]

If the draft model is far from the target model's distribution, then speculative decoding will be less effective because it will have a higher rejection rate, thus reducing the speedup. However, the algorithm guarantees that the output sequence will be identical; therefore, even if the draft model is of poor quality, the text generation quality will not be affected.

[@mingzhou2213]

thank you for the explanations and the visuals. Does speculative decoding work with beam search? I understand that for LLM we generally just do greedy decoding in one pass, but for translation models like whisper, the performance increase significantly if we use beam search. I see even from hugging face official post discussing how speculative decoding improve whisper large inference speed by 2x, but to be honest, for non english audio data, with greedy decoding whisper is barely usable...

[@EfficientNLP]

Interesting idea, but I don't know of any attempts to combine them. Speculative decoding relies heavily on random sampling, whereas beam search is deterministic, so probably they are incompatible. For speeding up whisper inference, you might try using smaller or quantized models on faster engines like CTranslate2 or faster-whisper.

[@laulinky334]

Thanks for sharing, I am wondering how target model check the generated tokens of draft model and produce probability distribution q of x for each token?

[@EfficientNLP]

This is due to the parallel nature of transformers - when given a sequence of tokens, it can generate the logits for all of them in parallel, unlike generation which must be done autoregressively.

[@saiashwalkaligotla6639]

Interesting. Curious if we can use mutiple different fine-tuned small models to do the same task along with a bigger model.

[@EfficientNLP]

That is an interesting idea-like using multiple smaller models to generate several completions and having the large model choose the best one. I'm not sure if anyone has tried this.

[@saiashwalkaligotla6639]

@@EfficientNLP Yeah, hopefully might increase (or match) accuracy than original model.

[@ArtemAlekhin]

Great video 👍

[@feixyzliu5432]

why does target model running with K new tokens spend almost the same computation than with just 1 new token? I know K new tokens can be computed in parallel at one single forward pass, but self-attension with K new tokens indeed need more works than 1 token (suppose KV-cache is used), isn't it?

[@EfficientNLP]

It's true that the computation time in the two scenarios might not be exactly the same due to KV cache and other implementation details; however, for simplicity, we can assume one forward pass through a model as taking one unit of time. Both decoding one token and checking the probabilities of multiple tokens require one forward pass.

[@ariellubonja7856]

Thanks! But doesn't the google paper define Mq as the draft model i.e. flips the definitions?

[@EfficientNLP]

You are right; the Google paper uses a different notation from the DeepMind paper, in this video I'm using the DeepMind notation.

[@domenvake3077]

this is great! is there any chance you could demonstrate something like this in code?

[@EfficientNLP]

Good question - it appears that neither of the deepmind or google papers have an official source code implementation, but there are several implementations of this idea on GitHub, but I have not looked at them.

[@gnorts_mr_alien]

For this to work, the two models need to have identical tokenizations right? Is there any way around it?

[@EfficientNLP]

That's right - the two models need to use the same vocabulary so that we can compare their logits meaningfully.

[@gnorts_mr_alien]

@@EfficientNLP thank you for the quick response. that makes sense!

[@murtazanazir9997]

Not necessarily. We can retokenize the predicted text by draft model. That can be slow though.

[@christospapadopoulos7894]

Nothing really new about this, it seems that big tech companies really do have it easier when publishing research

[@EfficientNLP]

That’s the way it tends to go! One small step at a time