SPLADE: the first search model to beat BM25

[@jamesbriggs]

To install the naver labs splade library you need `pip install git+github.com/naver/splade.git`

[@JulianHarris]

Came here curious about SPLADE, discovered a super understandable introduction to transformers and attention networks. Thank you!

[@jamesbriggs]

I really wanted to get the point across about SPLADE but there was a lot of foundational stuff to cover from sparse vs. dense, transformers, etc - so I'm glad the extra info helped :)

[@zazouille2264]

Agreed. Great video. Nicely layered. Thank you OP

[@magicofjafo]

I agree!

[@shamaldesilva9533]

dude you are a gold mine when it comes to these topics 😍😍 .

[@jamesbriggs]

thanks man it's appreciated!

[@alivecoding4995]

Which graphics library do you use for these Transformer illustrations? Are these pre-built assests?

[@ברקמנחם-ק3ק]

Thank you! when using embeddings and asking the model gpt-3.5 some question like "write me some code that use this and that" does the model automaticlly search in the embedding too in order to give the answer?

[@jamesbriggs]

gpt 3 doesn't, you need to add a knowledge base to do this, like I do here kzbin.info/www/bejne/qKOkdJuml9ysnac

[@ArnavJaitly]

James, this is awesome and very relevant to my current project! Thank you for your efforts in putting this together and sharing it, much appreciated!

[@jamesbriggs]

awesome, good timing!

[@danrosher6658]

Great talk, thanks James ... Would an alternative to the cosine sim to compare query/doc is to index the tokens and weights for docs (from SPLADE model outputs) , also convert a query to tokens(and weights) , then return docs having the query tokens where the doc weight > query token weight for each token? .. would this work ?

[@MaheshJha-y3j]

Hello James, the above pinned method for pip install splade is not working and giving error like "error: subprocess-exited-with-error" so, Can you please let what is the issue or what alternate we can use if not this.

[@williamxion2806]

Hi james. I know this video is already a year old and there has been a lot of new development, but didn't contriever already outperform BM25 at the time on most benchmarks? I believe Contriever fine tuned on MS MARCO basically outperformed BM25 on everything.

[@IamalwaysOK]

Hey James, as usual, thanks a ton for your awesome videos! I've got a quick question for you. Have you ever thought about using a knowledge graph alongside SPLADE to expand terms? And is there any way we can embed that knowledge into sparse vectors using transformers? Curious to hear your thoughts on this!

[@ttharita]

Super informative. Thank you so much!!!

[@kevon217]

Great tutorial as always. Your slide animations are next level!

[@avidlearner8117]

Fantastic content! Especially since I'm building an app and need to find a proper solution for data retrieval....

[@gorgolyt]

Great video. But you should link to the SPLADE paper(s). Are you just talking about the original paper here?

[@lutune]

Have you built any of these apps? Your content is so great, as you get into more media, some development of those apps could really help with putting this into a visual space

[@jamesbriggs]

started building some demos and testing splade a couple months ago, will be sharing more soon - it's really cool though and I intend on making it a big part of my "go to toolkit" in the future

[@lutune]

@@jamesbriggs Your DC seems to be getting a lot of new people! ill get some things updated on there today for ya

[@SinanAkkoyun]

How does this compare to the new OpenAI embeddings?

[@aurkom]

Really enjoyed this one.

[@abhayr]

Amazing explanation. Thx for sharing

[@thedude9270]

Thanks for the tutorial! Is it possible that you could also share a colab or video explaining what would then be upserted as a Pinecone vector?

[@Sky-ec9eu]

This is incredible. Thanks James!

[@jamesbriggs]

you're welcome!

[@salesgurupro]

Amazing. Thanks for such a great explanation 😊

[@jamesbriggs]

you're welcome!

[@johannamenges3095]

But is Faiss still a solid solution for a semantic search engine? Cause I am at the moment working on a search engine with Faiss algorithm

[@biaoliu9297]

Is there a multi-language version model?

[@snack711]

i am surprised how "orangutans" got split into tokens. i thought "orangutan" surely had to be a token itself.

[@AnonymousIguana]

So, SPLADE vector generation is just as computationally intensive as dense vector generation? My understanding is that SPLADE requires real-time inference from a sophisticated model like BERT at query time. Isn't that very problematic?

[@RatafakRatafak]

Looks like so. Sentence-BERT is equally computationally intensive thant this SPLADE.

[@ylazerson]

very fascinating - thanks!

[@jamesbriggs]

glad you enjoyed it!

[@EkShunya]

what tool do you use to make the diagrams ?

[@jamesbriggs]

excalidraw!

[@nhatpham4053]

awesome works

[@leventk.1611]

13:02: low proximity = high semantic similarity. Not high proximity. :D

[@kayalvizhi8174]

How has the results of SPLADE been. Has it been proven to be effective?

[@abhinavkulkarni6390]

Hey James, Can you please compare SPLADE with ColBERTv2 - both of which are designed to alleviate the problems of desnse passage retrievers?

[@jamesbriggs]

I haven't read into the colbert models, I understood them to not be hugely scalable? I can look into it if they're of interest

[@jeffsteyn7174]

That's interesting. What does pinecone use, sparse or dense?

[@jamesbriggs]

now it can use both, I'll talk about it in the coming days or you can refer to here github.com/pinecone-io/examples/blob/master/search/hybrid-search/medical-qa/pubmed-splade.ipynb - for an example

[@sndrstpnv8419]

code deleted pubmed-splade.ipynb @@jamesbriggs

[@RatafakRatafak]

Is it important? If you use cosine similarity for both dense and sparse embeddings, it should work in any case.

[@BuFu1O1]

vocabulary mismatch can be fixed with sub-embeddings

[@ramsescoraspe]

Multilingual??

[@jamesbriggs]

I don't think there's a multilingual splade *yet*

[@RubenAlvarezMtz]

My thoughts exactly

[@klammer75]

Keywords and page rank are dead! The information landscape is undergoing a seismic shift and everyone better put a helmet on!!!🤔🤪😉🤖

[@jamesbriggs]

things are moving so fast rn

[@klammer75]

@@jamesbriggs seems we’re getting closer and closer to the inflection point of the exponential….next stop, ludicrous speed!🤯🚀

[@hoangphanhuy1992]

I thought CLIP no need to finetune so why cons of dense is need to finetune sir? @jamesbriggs