How to code long-context LLM: LongLoRA explained on LLama 2 100K

[@sambhavkhur]

Amazing video! I had 1 doubt regarding the Shift Short Attention though and was hoping that you could clarify. Consider that we have 8000 sequence length and we divide it into 4 groups of 2000 length each. Group 1 contains tokens from 1 to 2000, group 2 contains the next 2000 tokens and so on. The way I understand S2-Attention is that if we have 4 heads for each group, we keep 2 heads as it is and for the other 2 heads we shift them by half the group size. So for these 2 shifted heads, 1st row (for group 1) will not contain information about the 1st token but that information will be present in the row number 1001. The 1st row will instead contain information about the token number 7001. Similarly 1st row in these shifted heads (for group 2) will contain information about token number 1001. Once we have shifted these 2 heads, we can continue with the original self attention mechanism for each of these groups independently where we concatenate these heads and produce input for the next layer. Am I correct or is there an error in my understanding?

[@sandhyas2033]

Thanks a lot!!! Great videos with indepth explanation.Please share the Presentations as well for reference.

[@gunwooo]

Tnks for your slow speaking eng. im korean, im not well eng. but i do little understanding 😊😊

[@jdoejdoe6161]

Lots of errors in trying out the fine-tuning code. Please try it out and see

[@generativeresearch]

Thanks for the valuable information.

[@li-pingho1441]

thank you so much!!

[@htr-g8r]

Paper links please?

[@sergialbert97]

Do you see feasible a simple implementation for a "small"model within a notebook?

[@comediansguidetotruecrime3836]

are there any results on the long context learning, like test results?

[@AzureUnagi]

Have any one tried to conduct an experiment to inspect the lost in the middle effect? I wonder if the S2 attention mechanism affects how the model utilizes the results of attention.

[@icestika]

how doese it effect intefernce time

[@echofloripa]

I'd be wrong to imply that faster training would mean faster inference? 36:01

[@jimhrelb2135]

Inference time should be similar to running the foundation model at the same context length. LoRA-based modes only changes couple weights from the foundational model in-place, so (production) inference time and memory remains unchanged

[@fabianaltendorfer11]

Hey, do you have an idea how I could use Screenshots in the documents I want to use in a RAG-Usecase? Thanks

[@jmirodg7094]

Extremely interesting Thanks, The context length is a real limitation for now

[@li-pingho1441]

nice work!