Metadata Filtering for Vector Search + Latest Filter Tech

[@temiwale88]

Your videos and teaching style are top quality. Thanks James!

[@jamesbriggs]

Appreciate the comment, thanks!

[@number1ice]

Hey James. Nice video. Is it possible to filter just using the metadata? Without providing the vectors? In my use case I want to check if the doc is already in pinecone and not add it if it already exists

[@johnharlow5090]

This is great, it would be helpful to have the notebook file you used to follow along :)

[@opalkabert]

Awesome explanation 👍

[@alexbriggs1366]

Well done on the vid

[@bastabey2652]

can you turn off ANN and force an exhaustive vector search in Pinecone?

[@jamesbriggs]

no just ANN in Pinecone, best to use something like Faiss for exhaustive

[@MrTulufan]

can the filters be built dynamically by prompts?

[@carlosalberto-mo1wj]

not sure if this will help, but in my case i had an agent responsibile for looking up in the user query, then on the llm prompt asked for the llm to build a response of a metadata filter format, then aplying a post-filtering to the search, based on key points inside the user prompt, not sure if this is the ideal way to work things out, but i was prety hapy with the results it is generating

[@mohitaggarwal3625]

Apart from Pinecone, is there any Open Source Vector DB which allows to filter the documents using Metadata and then run vector search on filtered documents ?

[@fabianaltendorfer11]

He works for Pinecone so probably not ;D

[@stefanmisic7405]

Pg vector i think

[@nickthequick13]

Where can I read about the actual implementation of single-stage filtering ?

[@jamesbriggs]

The details haven't been released - eventually they will be - but there's no plans as to when afaik

[@alexbriggs1366]

Very good content

[@AndJusTIceForRob]

Hi, James. I've arrived here from your Udemy course. I'm having difficulty locating the English/Italian multilingual Squad dataset (I'm assuming it's from Squad v1.1). Can you assist?

[@jamesbriggs]

hi Robert! You can find an Italian SQuAD here huggingface.co/datasets/squad_it and to use it (as I know this is from a different source than that used in the course) you can follow the same steps I do in this video at ~13:14: kzbin.info/www/bejne/rWLHfniNopydeMU

[@AndJusTIceForRob]

@@jamesbriggs thanks for the assist. I have a couple follow-up questions. I can find the English SQuaD dataset and the Italian one you referenced. In this video, it appears that you are using one dataset that has both English and Italian, which I cannot seem to find. When I load either of the single-language datasets and print samples to the screen or use the keys() function, I do not see 'lang', 'vector', or 'metadata'. It also looks like I am missing a step where I need to create an index in pinecone (I set index = pinecone.Index('squad-test'), and when I run index.query(xq, top_k=3) I get PineconeProtocolError: Failed to connect; did you specify the correct index name?). Is there a prior video that covers the preparatory work to run the code in this video?

[@jamesbriggs]

@@AndJusTIceForRob ah I see what you mean now, I believe I pulled in the two different datasets separately - this video is not a good demo for how to do all of this as it was intended to just demo the metadata filtering, I did not include the indexing step, so for indexing you should follow this video: kzbin.info/www/bejne/Y6XHZmqPq6-XfsU from around 10:45 I hope that helps, let me know if there's anything else that isn't clear

[@lennonfoster9588]

Nice vid