Stanford CS224N | 2023 | Lecture 10 - Prompting, Reinforcement Learning from Human Feedback

For more information about Stanford's Artificial Intelligence professional and graduate programs visit: stanford.io/ai
To learn more about this course visit: online.stanford.edu/courses/c...
To follow along with the course schedule and syllabus visit: web.stanford.edu/class/cs224n/
Jesse Mu
cs.stanford.edu/~muj/
Professor Christopher Manning
Thomas M. Siebel Professor in Machine Learning, Professor of Linguistics and of Computer Science
Director, Stanford Artificial Intelligence Laboratory (SAIL)
#naturallanguageprocessing #deeplearning

[@user-qk7wr2ny1f]

It's one of the most awesome lecture that I have ever watched! The lecturer is wonderful!

[@uraskarg710]

Great Lecture! Thanks!

[@ReflectionOcean]

- Utilize prompting and instruction fine-tuning to align language models with user intent (start: 25:29). - Implement penalty terms in RLHF to prevent models from deviating too far from pre-trained baselines (start: 52:14). - Train reward models on human comparisons instead of direct human responses for more reliable reinforcement learning (start: 47:09). - Normalize reward model scores post-training for better reinforcement learning outcomes (start: 49:27). - Explore reinforcement learning from AI feedback to reduce human data requirements (start: 1:11:12).

[@ningzeng2239]

great lecture, tks!

[@user-ui4ye1qn8x]

Great lecture! very helpful!

[@philippvetter2856]

Amazing lecture, really well presented.

[@mavichovizana5460]

great lecture! very helpful!

[@willlannin2381]

Fantastic lecture, thank you

[@gemini_537]

Gemini: This lecture is about prompting instruction fine-tuning and RLHF, which are all techniques used to train large language models (LLMs). LLMs are trained on a massive amount of text data and are able to communicate and generate human-like text in response to a wide range of prompts and questions. The lecture starts with going over zero-shot and few-shot learning, which are techniques for getting LLMs to perform tasks they weren't explicitly trained for. In zero-shot learning, the LLM is given a natural language description of the task and asked to complete it. In few-shot learning, the LLM is given a few examples of the task before being asked to complete a new one. Then the lecture dives into instruction fine-tuning, which is a technique for improving the performance of LLMs on a specific task by fine-tuning them on a dataset of human-written instructions and corresponding outputs. For example, you could fine-tune an LLM on a dataset of movie summaries and their corresponding reviews to improve its ability to summarize movies. Finally, the lecture discusses reinforcement learning from human feedback (RLHF), which is a technique for training LLMs using human feedback. In RLHF, the LLM is given a task and then asked to complete it. A human expert then evaluates the LLM's output and provides feedback. This feedback is then used to improve the LLM's performance on the task. The lecture concludes by discussing some of the challenges and limitations of RLHF, as well as the potential future directions for this field. One challenge is that it can be difficult to get humans to provide high-quality feedback, especially for complex tasks. Another challenge is that RLHF can be computationally expensive. However, RLHF is a promising technique for training LLMs to perform a wide range of tasks, and it is an area of active research.

[@Pingu_astrocat21]

Thank you for uploading this lecture :)

[@akhileshgotmare9812]

The question at 50:10 is interesting! To combat this to a certain extent, what Llama2 authors did was to collect annotator preference responses on a scale of 4 points, and use that to include a margin component in the RM training loss. See Section 3.2.1 and 3.2.2 in the Llama2 paper. They report that the margin component can improve reward model accuracy.

[@marshallmcluhan33]

Cool 😎

[@isalutfi]

💙💙💙

[@munzutai]

Are there any promising strategies to reduce the amount of data that's necessary to do RLHF?

[@theneumann7]

👌

[@HoriaCristescu]

1. RLHF is updating the model for whole sentences. Does that carry special meaning? Because next token prediction is focusing too much on short term, this changes the focus to whole answer. 2. RLHF uses model generated outputs for training, so it is on-policy data. Does that make it more effective than training on random internet texts?

[@ThamBui-ll7qc]

If a model is just instruction-finetuned without any RLHF, does hallucination occur?

[@buoyrina9669]

At 44:35, does theta_t refer to the LM's entire parameters ?

[@DrumsBah]

In the case of Instruct GPT, theta was the full parameter set of the foundation model. However, there's no reason RLHF couldn't be performed on the head or adapter parameters (e.g. lora) instead.

[@Andrewlim90]

I really like the question at 00:34:20! Anyone know if this is being explored? People who can produce questions like this seem like they'd make excellent researchers. I'm jealous.

[@DrumsBah]

The embedding distance is commonly utilised in contrastive loss based optimisation. It's actuslly been shown to be useful in training sentence embeddings, see: SimCSE. Of course in this setting, the embeddings are directly of interest. I could imagine a possibility of it being used for alignment. However, it has significant disadvantages compared to reward model approaches in terms of the salience of embedding distance actually relating to human preference.