The 10,000x Yolo Researcher Metagame - with Yi Tay of Reka

Pod: www.latent.spa...
It’s easy to get de-sensitized to new models topping leaderboards every other week - however, the top of the LMsys leaderboard has typically been the exclusive domain of very large, very very well funded model labs like OpenAI, Anthropic, Google, and Meta. OpenAI had about 600 people at the time of GPT-4, and Google Gemini had 950 co-authors. This is why Reka Core made waves in May - not only debuting at #7 on the leaderboard, but doing so with all-new GPU infrastructure and 20 employees with no more than 5 people on pre-training and a relatively puny $60m in funding.

[@KevinKreger]

This channel is incredible❤

[@prasannad5719]

keep em coming 🔥

[@soupbob5813]

[00:00:00] Intro [00:01:57] Yi Tay Intro [00:03:02] Path into LLMs [00:09:41] Google Brain: PaLM, UL2, DSI, Emergent Abilities [00:11:54] PaLM 2 [00:15:27] Emergent Abilities [00:18:26] Quoc Le [00:24:16] Marketing Research: How to Start from Zero with No Reach [00:27:34] What's needed to be a successful AI Researcher? [00:30:31] Reka Origin [00:33:24] Starting Reka Infra [00:35:04] Why not to use TPUs outside Google [00:36:29] Chaotic vs Stable Infra [00:38:04] Risk Sharing of Bad Nodes [00:41:05] Checkpointing and Orchestration [00:43:39] Reka Flash/Core/Edge [00:46:59] Recruiting the team [00:47:22] Noam Architecture - Swiglu, GQA, RMSnorm, ROPE [00:52:26] Encoder-decoder vs Decoder-only [00:55:52] LLM Trends - Llama 3 and Phi 3 Glowup [00:57:46] LLM Trends - Benchmarks and Evals [01:03:25] LLM Trends - Early vs Late Fusion Multimodality [01:07:22] LLM Trends - Scaling Laws [01:09:41] LLM Trends - Long Context vs RAG [01:12:31] Long Context vs Finetuning [01:14:14] If emergence is real, when does Efficiency work? [01:17:41] MoEs and Upcycling [01:20:47] The Efficiency Misnomer - Efficiency != Speed [01:25:05] Open Source vs Closed Models [01:28:08] Personal Productivity [01:33:19] Singapore vs US Academic Scene [01:37:42] Building Silicon Valley outside Silicon Valley [01:40:29] TechInAsia Meetup

[@swyxTV]

that is the audio timestamps, but the video has different editing therefore i didnt put it there

[@NA-sd8bw]

I am a beginner in the LLM world , but I have 5 years experience in ml and data science. Can you a get a and learn from this amazing guy in his start up ? I can work for free for him.

[@LatentSpaceTV]

Audio pod and show notes: www.latent.space/p/yitay

[@paulmcgee4176]

Don't forget to "like" it.