Can LLMs Learn by Teaching Other LLMs?

Can LLMs Learn by Teaching? A Preliminary Study
arxiv.org/abs/...
Support my learning journey either by clicking the Join button above, becoming a Patreon member, or a one-time Venmo!
/ tunadorable
account.venmo....
Discuss this stuff with other Tunadorks on Discord
/ discord
All my other links
linktr.ee/tuna...

[@lexer_]

This one sounds big if true. But there are some gaps here. First, with any rl-esque training like this, how do you prevent the teacher from reward hacking by just learning how to get the student to output the optimal answer? Lots of students could maybe somewhat limit this but I would still be very worried the teacher just reward hacks itself into a dead end if you run this for long enough. The second question would be if the teacher actually generalizes to the solution or if this is just a more effective way to memorize. There are some hints here that some sort of generalization might be happening but none of it seems conclusive as the only thing that is really being evaluated is the performance on the benchmark which it could still overfit to. Aren't the graded student answers just indirectly feeding information about the benchmark answers into the teacher model? There are some levels of indirection here but it might very well be that the teacher is essentially just training on the benchmark here. I am very skeptical for now but this for sure deserves deeper exploration because the gains might be huge if this really works.

[@rasen84]

The issue always seems to be that you need a true held out set that you only use at the end to know if overfitting has occurred.

[@foxfining4210]

Hey, thanks for your interest and questions from the authors! Your question is very worth discussing. As I myself usually rely on LbT to learn, (no matter it's explaining to oneself or others, just trying to output something can help learning), and I benefit a lot from teaching newbies, so I'm very interested in exploring whether LLMs can do it. If so, this is exciting opportunity for self-evolving, right? So we begin this exploration excitedly. I first reduce this problem to whether or not "the students' performance" produce signal for measuring the quality of a teaching material in a better and fine-grained way. That's why we design M1. But actually, we do encounter some issues on this direction. For example, for both math and code task, it's very hard to tell whether the student is learning something from the teacher or he just can solve one problem by itself (as the student is trained on similar data). So we have tried to construct some synthetic task that we're sure LLMs haven't been explicitly trained on. But the one we construct is too easy for current LLMs for us to get interesting results . So we do not include the discussion. In M3, our workflow is very resemble to some sort of prompt tuning. So maybe the teacher isn't find high-quality teaching material, but just overfit a prompt for a student. In this case, the discovered prompt might not help for the teacher himself. But in our exp, we find it indeed has some improvements on the teacher. That is to say, the teaching material (or ICL examples) that are good for students are actually better ICL examples for the teacher. This somehow gives evidences on "a teaching material that can make weaker teachers learn better is better". But indeed, maybe it's just because of some similarity between the teacher and the students. This is something we haven't explored. There might be more efficient way of reaching the results. Actually, we think M1 is not practical now (its inference cost is high, as we discussed in the limitation). So we don't promote our current method to be used directly, but introduce them as the initial execution of this idea. Indeed, there are many things to carefully explore in the future to see where do the benefit actually come from, and whether other LbT implementation can help LLM, and whether the helping is resemble to why LbT helps human. We regard this direction to be very interesting, not only it might be a way to continuously improve AI. Even if it cannot, explorations towards this direction raises a very interesting gap between the learning of human and learning of AI, which might shed some lights on, maybe, how to make LLM learn in a more data-efficient way, like human's learning. That's why we spend a lot of texts in discussing the exploration roadmap in Section 6. We hope to introduces this interesting idea, give the roadmap of future exploration, and introduce some of our initial execution of this idea.

[@leptir1]

Haven't commented yet but I really appreciate what you're doing. Super helpful and efficient to boot. Stay golden. What are you doing in the space more broadly? For career, what stage are you at in it, etc?

[@Tunadorable]

basically unemployed rn but this just started making money recently & im hoping it’ll be enough to live off of by the end of the year. if not i’ll be applying to jobs. either way i hope to start publishing papers soon, goal is this fall for first one but we’ll see. glad you’re enjoying the videos!

[@cammccauley]

@@Tunadorablehave you started building any models or expirements yourself yet? If not I’ve got plenty of ideas and not a lot of time!

[@Tunadorable]

hahaha yeah i’ve also got more ideas than time. i’ve posted videos on some failed ones before and i’m hoping to have time to really iterate quickly come september october time

[@leptir1]

@@Tunadorablei recently graduated (neuroscience postgrad) and am in a similar boat. Looking to pivot to leveraging/contributing to the AI space. Are you publishing solo or within an institution? I can also ask by email or another method if you'd prefer that. Anyways, wishing you the best of luck, financial, academic, and otherwise!

[@Tunadorable]

i’d be posting to arxiv independently and not thinking about an actual journal until after that’s up. yeah my email and linkedin are both in my linktree which is in the video description. i’ll be testing minimal versions on my own before inviting any subscribers who are interested to contribute code, writing, and/or funding for larger experiments. however those in-progress minimal versions will always be live on my github while i’m building them so feel free to look around there if you’re curious

[@zinanlin46]

Thank you so much for sharing our paper! The summary and discussions are awesome!!

[@Tunadorable]

love when authors find a video, you’re welcome!

[@zinanlin46]

@@Tunadorable ❤

[@GNARGNARHEAD]

cool, cool cool cool

[@Ivan.Wright]

"Best way to learn is to teach", I think the idea is true in that it requires the teacher to develop an abstract stack of understanding where the core idea can be translated such that an individual of any knowledge background can understand to the degree that they can with their knowledge background. As a teacher you have to have an idea of what the learner knows and use the core principles from that set of ideas to describe the new idea. Analogy seems to me to be the best tool we have for this besides direct observation. Just my initial thoughts anyway.

[@tornyu]

This feels like it could be used to iteratively improve alignment instead of capability. If that's the case can we please do that first

[@Tunadorable]

do you feel like alignment is currently falling behind capability? I'd disagree; it seems to me like we're still very much in the pushing capabilities without caution time period

[@tornyu]

@@Tunadorable oh interesting, yes that's the assumption I'm working from. I'm not sure how to quantify the current state of alignment. But it seems plausible that capabilities could improve fast just with scale* - whereas there's no indication of anything similar for alignment. In any case capabilities seem to be on a very steep trajectory, and I'm not seeing that for alignment. And we're far from having aligned systems: when an LLM refuses a reasonable request, or when an image generator produces too-diverse historical images, it's unaligned. * E.g. maybe an order of magnitude increase in scale unlocks planning and suddenly we have powerful agents. I'd want to have much better alignment before that happens.

[@GodbornNoven]

Ive got a question. What if we just straight up skipped the training data for LLMs and jumped to tests directly? Never letting an iteration of the problem in the training data be repeated twice. Or maybe a bit more but not too many times. I reckon this would have the effect of skipping the memorization process and moving on to the learning the concept itself. I've never seen it done. Any idea what it might do?

[@Tunadorable]

problem is how do propose to look through the entire internet and cross-reference every single piece of text with every other single piece of text to make sure there are no duplicates (I'm talking about conceptual duplicates rather than plagiarism duplicates)? And then if a model has only seen it once how can it be expected to learn it? I certainly could not learn a mathematical concept after only seeing one example, I'm going to need at least a couple

[@envynoir]

Very sexy indeed, nice job broski

[@foxfining4210]

Many thanks for sharing our paper! We're very happy~ We need to use KZbin more haha.

[@Tunadorable]

love when an author finds the video!