Fast LLM Serving with vLLM and PagedAttention

[@hemanthsethuram6740]

Beautiful adaptation of a fundamental idea of paging, reference counting and copy -on-write.👌

[@dinoscheidt]

Full circle dynamic memory management and garbage collection. Great talk!

[@simonguo1048]

Such an elegant idea and amazingly clear explanation!

[@sherlockho4613]

very helpful and distinguish presentation!

[@RahulJain-wr6kx]

Awesome 👍

[@TheAIEpiphany]

Great talk and amazing work guys!

[@keshmesh123]

It was great. Thank you!

[@harshadkunjir5800]

This is so great!

[@LiangyueLi]

great work

[@vaporeon2822]

Interesting sharings. Curious about the underlying implementation for KV blocks sharing part you have a copy-on-write mechanism, but how does it avoid dirty-read condition, where both request reads that ref count is 2 and both request copies the block simultaneously.

[@alankhor2000]

I think the last question was asked on impact on latency

[@erkinsagroglu8519]

7:25 How is it possible to compute attentions separately block by block? Softmax (attention weight) is calculated based on all of the previous tokens and then those softmax scores are multiplied with all of the previous tokens' value vectors to calculate the attention score for the new token. So it should use all of the previous tokens on other blocks twice. What am I missing here?

[@erkinsagroglu8519]

I read the paper. Turns out the illustration is not 100% accurate (probably for the sake of making it intuitive). It indeed uses every previous block (in case sliding windows is not used) while computing the attention for the next layer.

[@mshonle]

It seems like there would be a performance increase for beam search as well? (That is, in addition to the memory savings it gets.) Would be great to see some benchmarks for that!

[@Karthikprath]

How do we calculate memory used by kv cache in paged attention.Example for input 500 and output 1000

[@erkinsagroglu8519]

If sequences of different sizes can be processed in parallel (say request 1 is generating 11th token and request 2 is generating 3rd token), how come those two operations (Query vector of request 1 - say dimension 1x50 - dot product with previous tokens' key vectors matrix 11x50) and (1x50 dot product 3x50) can be batched together?

[@julien3578]

brilliant guys

[@billykotsos4642]

sick

[@FoxTheodore-b4x]

Rosamond Plain

[@VickySpears-g4u]

Farrell Stravenue

[@LizzieSimpson-g5p]

Kiehn Crescent

[@RollandWensman-s3y]

Stroman Walks

[@CarllyleLynn-b4y]

Thurman Terrace

[@HelenJackson-r6n]

Lebsack Light

[@SydneyThomson-p3y]

Tyrell Mountain

[@MadgePapiernik-c6d]

Fae Harbors

[@CurmeHayden-p8o]

Cloyd View

[@RafaelaKrahulec]

470 White Branch

[@ConnorAlma-s5f]

Pink View

[@CookSuzanne-t3w]

Breitenberg Pines

[@HarrodAmos-j6n]

Durgan Mews

[@MaryTaylor-d8r]

Ettie Road

[@KimberlyAllen-d5u]

Dejah Corners

[@KennethSmith-o7j]

Terrance Villages

[@BeckyMarvin-l7t]

Francis Track

[@LarryYoung-b9c]

Steuber Lakes

[@FitzGeraldMamie-d6f]

Lenora Isle

[@ameynaik2743]

Is vLLM engine running on the host?

[@fxhp1]

you run the server on the host that has the GPU installed, the server can be accessible over an API remotely using openai's client. follow me for more AI vids

[@SmollettTaylor-c6s]

Wunsch Vista

[@LeonardBuck-s3l]

Declan Mews

[@WyattWayne-g8w]

Magnus Ridges

[@LucasNoah-r7y]

Jennyfer Cliff

[@KittyDelia-g7m]

Quitzon Walk

[@ChaplinBobby-g7n]

Kunze Junctions