Fast LLM Serving with vLLM and PagedAttention

[@hemanthsethuram6740]

Beautiful adaptation of a fundamental idea of paging, reference counting and copy -on-write.👌

[@dinoscheidt]

Full circle dynamic memory management and garbage collection. Great talk!

[@simonguo1048]

Such an elegant idea and amazingly clear explanation!

[@sherlockho4613]

very helpful and distinguish presentation!

[@TheAIEpiphany]

Great talk and amazing work guys!

[@keshmesh123]

It was great. Thank you!

[@RahulJain-wr6kx]

Awesome 👍

[@harshadkunjir5800]

This is so great!

[@vaporeon2822]

Interesting sharings. Curious about the underlying implementation for KV blocks sharing part you have a copy-on-write mechanism, but how does it avoid dirty-read condition, where both request reads that ref count is 2 and both request copies the block simultaneously.

[@alankhor2000]

I think the last question was asked on impact on latency

[@erkinsagroglu8519]

7:25 How is it possible to compute attentions separately block by block? Softmax (attention weight) is calculated based on all of the previous tokens and then those softmax scores are multiplied with all of the previous tokens' value vectors to calculate the attention score for the new token. So it should use all of the previous tokens on other blocks twice. What am I missing here?

[@erkinsagroglu8519]

I read the paper. Turns out the illustration is not 100% accurate (probably for the sake of making it intuitive). It indeed uses every previous block (in case sliding windows is not used) while computing the attention for the next layer.

[@LiangyueLi]

great work

[@mshonle]

It seems like there would be a performance increase for beam search as well? (That is, in addition to the memory savings it gets.) Would be great to see some benchmarks for that!

[@erkinsagroglu8519]

If sequences of different sizes can be processed in parallel (say request 1 is generating 11th token and request 2 is generating 3rd token), how come those two operations (Query vector of request 1 - say dimension 1x50 - dot product with previous tokens' key vectors matrix 11x50) and (1x50 dot product 3x50) can be batched together?

[@Karthikprath]

How do we calculate memory used by kv cache in paged attention.Example for input 500 and output 1000

[@billykotsos4642]

sick

[@julien3578]

brilliant guys

[@ameynaik2743]

Is vLLM engine running on the host?

[@fxhp1]

you run the server on the host that has the GPU installed, the server can be accessible over an API remotely using openai's client. follow me for more AI vids

[@FoxTheodore-b4x]

Rosamond Plain

[@LizzieSimpson-g5p]

Kiehn Crescent

[@RollandWensman-s3y]

Stroman Walks

[@SydneyThomson-p3y]

Tyrell Mountain

[@VickySpears-g4u]

Farrell Stravenue

[@HelenJackson-r6n]

Lebsack Light

[@CarllyleLynn-b4y]

Thurman Terrace

[@RafaelaKrahulec]

470 White Branch

[@CookSuzanne-t3w]

Breitenberg Pines

[@CurmeHayden-p8o]

Cloyd View

[@MadgePapiernik-c6d]

Fae Harbors

[@ConnorAlma-s5f]

Pink View

[@KennethSmith-o7j]

Terrance Villages

[@KimberlyAllen-d5u]

Dejah Corners

[@LarryYoung-b9c]

Steuber Lakes

[@BeckyMarvin-l7t]

Francis Track

[@FitzGeraldMamie-d6f]

Lenora Isle

[@HarrodAmos-j6n]

Durgan Mews

[@MaryTaylor-d8r]

Ettie Road

[@LeonardBuck-s3l]

Declan Mews

[@WyattWayne-g8w]

Magnus Ridges

[@SmollettTaylor-c6s]

Wunsch Vista

[@LucasNoah-r7y]

Jennyfer Cliff

[@KittyDelia-g7m]

Quitzon Walk

[@ChaplinBobby-g7n]

Kunze Junctions