LLM Foundations (LLM Bootcamp)

New course announcement ✨
We're teaching an in-person LLM bootcamp in the SF Bay Area on November 14, 2023. Come join us if you want to see the most up-to-date materials building LLM-powered products and learn in a hands-on environment.
www.scale.bythebay.io/llm-wor...
Hope to see some of you there!
--------------------------------------------------------------------------------------------- In this video, Sergey covers the foundational ideas for large language models: core ML, the Transformer architecture, notable LLMs, and pretraining dataset composition.
Download slides from the bootcamp website here: fullstackdeeplearning.com/llm...
Intro and outro music made with Riffusion: github.com/riffusion/riffusion
Watch the rest of the LLM Bootcamp videos here: • LLM Bootcamp - Spring ...
00:00 Intro
00:47 Foundations of Machine Learning
12:11 The Transformer Architecture
12:57 Transformer Decoder Overview
14:27 Inputs
15:29 Input Embedding
16:51 Masked Multi-Head Attention
24:26 Positional Encoding
25:32 Skip Connections and Layer Norm
27:05 Feed-forward Layer
27:43 Transformer hyperparameters and Why they work so well
31:06 Notable LLM: BERT
32:28 Notable LLM: T5
34:29 Notable LLM: GPT
38:18 Notable LLM: Chinchilla and Scaling Laws
40:23 Notable LLM: LLaMA
41:18 Why include code in LLM training data?
42:07 Instruction Tuning
46:34 Notable LLM: RETRO

[@amirmohammadi572]

So far this has been the simplest brief explanation of transformers I have seen. Not complete though but goos starting point.

[@USONOFAV]

This is the best explanation so far for Language Model and its parameters. The speaker though didn't attributed Google for their Attention is all you need white paper.

[@MsWorldMine]

People from Google not Google

[@zrebbesh]

He attributed the researchers. That's all that matters.

[@andersw69]

As a data scientist prepping to retrain a transformer for a classification task at work, this is so helpful in refreshing what I learned in school!

[@oleksandrasaskia]

Amazing explanation of Transformers! And everything else... LOVED the talk! Thank you so much, Sergey!

Great presentation! Thanks!

[@robertcormia7970]

This was amazing! As a "neophyte" to the field of AI and LLMs, highly recommend this for foundational knowledge and perspective.

[@allanrp100]

really cool content. the part about training the models on code enhancing non-code capabilities is mind blowing. congrats!

[@shihabullah2475]

Nice attempt! 😊

[@aldomatic]

Good stuff, thank you!

[@enesbol6569]

amazing content!

[@dsagman]

excellent

[@kocokan]

Sound. Very small Subtitle. Will help

[@m65917]

Whats the presentation date please ?

[@The_Full_Stack]

April 21, 2023

[@ScientiaFilms]

Pls use AI to enhance this audio tnx. Either way, thanks for the vid!