Choosing Indexes for Similarity Search (Faiss in Python)

[@Nick-vs1zp]

Great explanations, especially for IVF - it's probably the best explanation for how it works that I've seen.

[@jamesbriggs]

thanks Nick!

[@narayansharma8797]

Thanks a bunch for this, James! Would be really great to see a couple of them get explored in depth. Also, if you could benchmark FAISS against ScaNN, it will help a few of us noobs a hell lot. Great content! Lovely command over your content. Really need more of this.

[@jamesbriggs]

Hey Narayan, there is a video released already covering the 'traditional' version of LSH, and two more videos that will be released at 1200 ET today on the random projection version of LSH (used in Faiss) - and there are plenty more of these on the way ;) I love the FAISS vs ScaNN idea too, will be working on it soon!

[@narayansharma8797]

@@jamesbriggs Sold!

[@harshitjaitly6850]

Super Informative Content! Thank you so much for this.

[@katehan9623]

Thank you for your video. Most Valuable Channel. Do you use GPU for indexing in this projects?

[@ChrisZuo]

Thank you! The drawings are cute!

[@grayrigel7091]

Hi James. Thanks for such a wonderful tutorial. Really useful. A quick question, For a new query vector, is it possible to return the IVF cell/partition that it belongs to, instead of returning the neighbors? I think I can measure the distances with centroids and return the closest centroid. However, I was thinking if there is built-in way.

[@mohammadyahya78]

Does the IVF algorithm works with high dimensional data please like 100?

[@haneulkim4902]

Thanks for amazing video! Do you know why simple K-means are not used for these MIPS problems?

[@itheenigma]

Super useful! Thanks for this video James. For IVF, can we retrieve the clusters that each datapoint belongs to after training (also cluster centroids)?

[@jamesbriggs]

Yes you can, there is info on it here gist.github.com/mdouze/904e0b538ef7767c9e83a45ac1b57d1b The code you need to write (after training and adding your data to 'index') is: invlists = index.invlists all_ids = [] for l in range(ind.nlist): ls = invlists.list_size(l) if ls == 0: continue all_ids.append( faiss.rev_swig_ptr(invlists.get_ids(l), ls).copy() )

[@itheenigma]

@@jamesbriggs legend. Will give it go. Ta!

[@viorelteodorescu]

What does IP stand for?

[@nareshsandrugu6057]

Can share the video assume I have binary data of train and test, so need to calculate the haming distance, I didn't found any videos using faiss ,if share the video that may more helpful

[@mohammadyahya78]

what is nbits please at 10:21?

[@basedscienxe6632]

This is the amount of bits for the precision of each component in the vector I believe