Why vector search is not enough and we need BM25

[@endre777]

Thanks for the explanation, was super clear. We just planning to move from vector search to hybrid, and your explanation on BM25 helps a lot to understand what edge cases it can solve. Appreciate a lot! Guess we will see a surge on BM25 due to Anthropic Contextual retrieval paper .

[@ChocolateMilkCultLeader]

Great video. This is why when building a search engine- I like to use BM25 for sparse search, and use Vector based search later, once most of the corpus has been filtered out. This allows me to stay precise and efficient. One additional thing- people often assume that you need a Vector Db for vector search, but you can do completely without. Just store the vectors in a normal DB.

[@notsojharedtroll23]

I mean, at the end of the day, the embedsings are data period

[@stxnw]

It should be the other way around. Most prompts may not have exact matches. Use vector search first, then BM25 and rerank the results.

[@microburn]

Nice video. I’ve been on the opposite side of the coin, but I like hearing the balanced argument to keep me educated

[@shiholololo1053]

Waiting for the next videp. I enjoyed the format.

[@oncedidactic]

Nice discussion, thanks! I wish there was more structure to the video so the “why” of the title I served as a main dish, ie let’s define the terms up front, explain how each works, then do why discussion and give a teaser for hybrid approach discussion. Instead there are some gaps and jumps around, which leaves it feeling incomplete or maybe not quite capturing the essence? I have a feeling this is partly a result of editing many clips, so don’t take this feedback too seriously. Cheers

[@jameswigglesworth8132]

Thank you for delving into this important topic!

[@matveyshishov]

Thanks, guys, YT recommended me this video, a very pleasant snippet of explanation. Trying to work through your website to understand what the service is.

[@aproperhooligan5950]

Excellent presentation/explanation. Very useful. Thank you!

[@andydataguy]

Great video! This is one of the most misunderstood concepts. Will def share this next time it comes up!

[@roopad8742]

This is so easy to understand, thank you!

[@dougunderwood569]

Great overview, thank you!

[@NicolasEmbleton]

Wonderful explanation. Thank you.

[@badashphilosophy9533]

this is an amazing explanation. im an instant follower

[@marka5215]

Great explanation. Thank you so much!

[@weirdsciencetv4999]

Oh man you are amazing!! Love channel I subscribed. Please do a video on working with such graphs using a vector database

[@ashraf_isb]

thats insightful, thank you so much boss

[@ShadowD2C]

Hi, I liked the video and the explanations, I wouldve liked it to show more visuals about the topic instead of the presenters face tho

[@theepicosityofpizza]

BM25 doesn't do anything to address any of the issues you bring up at the beginning of the video. TF IDF is dumber than vector search in every aspect. It's just much cheaper to run. Not saying it doesn't have value as part of the toolkit but not sure why you spend the first half setting all thes problems with vector search up as if BM25 addresses any of them.

[@stxnw]

is English not your first language?

[@PongsiriHuang]

how does bm25 help with ranking the words excellent, good, decent, and numer like 50, 100, 150 or 150-100=50? I thought the video would discuss that

[@amortalbeing]

thanks this was great!

[@broccoli322]

Thanks for the video.

[@MathsSciencePhilosophy]

The mathematics behind chatGPT is amazing

[@andrewwalker8985]

Why don’t we include semantic dimensions in vectors

[@BleachWizz]

Oh no, this is going to make texts like I do!!! ok, drama aside, I do believe this will improve things a lot. I still see some caveats that would be left for luck, but huges amount of data might overcome that. I do believe we already have enough with GPT and a few previous ideas, still improving the language model itself is always a plus.

[@АндрейАндреевич-з7т]

BM25. Frequency-weighted by sponsored-definition-tag vector search. Yeah google search do that too, you know. If you ever did seo optimization for your website or some kind of smm you know that it works

[@MLGJuggernautgaming]

I believe a vector search is still better for rag applications. Bm25 is better for more literal matches. Also what does this have to do with LLMs doing math?

[@Howoulduknow841]

This is something Anthropic has shared with their contextual retrieval.

[@pratikerande4808]

super

[@shizheliang2679]

wait...I think I am in love...

[@bmm8213]

Golden nugget

[@Isaacmellojr]

Otima exemplificacao de como word2vec não é a solucao definitiva.

[@tempname-dr2bm]

Poland mentioned

[@themax2go]

ty for the insight to "pair" numerical rep (vector) w/ MB25... can the same be achieved w/ just using a knowledge graph? i'm experimenting w/ sci/phi triplex... what do you think, do you have any preliminary ideas, or have you already tested it and found using "entities_and_triples" not as effective / not effective at all? 6 mo ago you did a vid on knowledge graphs, i haven't watched it yet, i'll check it out...

[@knucker3]

TURN YOUR VOLUME UP

[@NLPprompter]

i love this bot...

[@815TypeSirius]

But vs is enough to scam dummies and create a market bubble.

[@ValidatingUsername]

Try tokenizing engendered languages 😂

[@rontheoracle]

Excuse me, but your volume is just too low. Just saying.

[@martin777xyz]

Seems fine to me

[@sladeTek]

No it’s not, your device is the issue

[@rontheoracle]

@@sladeTek It's just this video and a few others that play with very low volume. I try other videos in youtube, in general, they sound acceptably loud. Dunno why.

[@rontheoracle]

@@sladeTek Try watching the video in youtube with this title: "The Best RAG Technique Yet? Anthropic’s Contextual Retrieval Explained!" It is significantly much louder. Just my 2 cents.

[@csmac3144a]

Her audio is fine. Turn up your volume.

[@Ruhgtfo]

Contributed 3blue1brown