Fine-Tuning Text Embeddings For Domain-specific Search (w/ Python)

[@ShawhinTalebi]

Excited to share another fine-tuning video! Check out links to the code, dataset, and model in the description :)

[@TonyCerone]

Thank you @Shaw ! Very good material. Your pedagogy is powerfull 🙂

[@ShawhinTalebi]

Thanks Tony! Glad it was clear :)

[@ifycadeau]

Love this video Shaw!

[@pauliusztin]

Amazing video, Shaw 🤟

[@ShawhinTalebi]

Thanks Paul 😁

[@sndrstpnv8419]

very good material thanks for sharing

[@ShawhinTalebi]

Thanks! Glad it was helpful :)

[@gustavojuantorena]

Great!

[@pasan-i5e]

Can you please explain me what actually do you suggest here? Use fine tuned llm instead of just using a llm for output generation ?

[@ShawhinTalebi]

It's always worth exploring non-finetuning-based improvements since these will be relatively quicker to iterate on e.g. improving prompts, chunking strategy, retrieval strategy. However, if further optimizations are needed then fine-tuning can make sense.

[@sndrstpnv8419]

can you share code and video for Fine-Tuning Text Embeddings For Domain-specific on openai? or other good quality cloud based LLM (google , aws )? or recent bert model modernbert ?

[@ShawhinTalebi]

Unfortunately, OpenAI doesn’t have their embeddings models available for finetuning. But I’ll take a look at those other options you mentioned 😁

[@sndrstpnv8419]

@@ShawhinTalebi you are the best. but you do video for recent bert model modernbert ?

[@cynorsense]

Why only BERT?

[@ShawhinTalebi]

There is a rich ecosystem of BERT-based embedding models and tools to develop them (e.g. the Sentence Transformers lib). One benefit of BERT is that is relatively small so it's easy to experiment with. In principle, however, you can do exactly the same thing with latent representations from more modern LLMs. Happy to do a video on that if there is interest :)

[@sndrstpnv8419]

your write w/ Python , but where is link to python code ?

[@ShawhinTalebi]

GitHub repo link (and other resources) in the description! Repo link: github.com/ShawhinT/KZbin-Blog/tree/main/LLMs/fine-tuning-embeddings

[@believer8754]

Do u put the same content in a blog ?

[@ShawhinTalebi]

Yes! Coming soon :)