Langchain TextLoader, TextSplitter, Embeddings, and PGVector

[@RealEstate3D]

I appreciate your work. Surprised to have found your channel so “late”. From my POV your channel is underrated.

[@DerekArends]

I appreciate that!

[@patruff]

OMG I was looking for this exact thing and you did it, thanks friend

[@DerekArends]

No problem 😊

[@emailthistorob]

Thanks for a great video. I'm struglling to get the metadata filters (e.g. $eq, $in) to work. Could you do a video on that?

[@DerekArends]

Hey hey! Let me take a look at that and see if that is something I can create a quick demo on.

[@CristianMolina]

Nice video. I liked that you just limited the do one thing and not a complete RAG chat. I'm a bit overwhelmed with so many parameters and diff. classes to do something, for instance, there is RecursiveCharacterTextSplitter. How it differs?

[@LearningWorldChatGPT]

Thank you very much, Derek! Excellent tutorial ! Using PGEmbedding would be very similar to PGVector, right ? ... and from what I've read, PGEmbedding has a better performance, right?

[@DerekArends]

Yeah, I believe it would be very similar. From a code perspective Langchain would need to have a tool for it but otherwise the concept is the same. I haven't tried it out but it does say it is more performant than PGVector. I may have to try it out and create a video on it. Thank you for pointing this project out!

[@shah_mori]

This was very easy to understand, I love it! I have a question tho, is it possible to update the postgresql vector database based on a collection name?

[@DerekArends]

If you want to update the database collection when loading the documents you can set the parameter "pre_delete_collection" to true and it will delete that collection and insert a new one db = PGVector.from_documents( documents=data, embedding=embeddings, collection_name=collection_name, connection_string=connection_string, distance_strategy=DistanceStrategy.COSINE, openai_api_key=api_key, pre_delete_collection=True )

[@domyu6418]

Hi Derek, quick question. When I try to install the "vector" extension in VS Code, it tells me that "extension 'vector' is not available. Even when I manually search for it, I cannot seem to find it. Do you have any insight here? Thank you and great video!

[@DerekArends]

Hey! Are you running "ankane/pgvector" as the docker image? If not, try that or github.com/pgvector/pgvector as steps for how to install it in other situations.

[@arpitapramanik4679]

How to access a user defined table where i have stored vector of text.with vector type as a column.

[@DerekArends]

So sorry for delay in reply. The easiest way I have found to do it is use the VS Code extension called SQL Tools, install the Postgres Connector, and connect to the database that way. This will give you the ability to see database schema.

[@ernie_hudds]

So just to be clear - its not really using AI fanciness its just finding a close match from documents based on a query, right?

[@DerekArends]

Correct, nothing too fancy here. It is just using vector embeddings and doing a similarity match. Next steps would be to take those matches and send it to AI to do summary/answer questions. I will get a video out about why you may want to do this and how to do it.

[@ernie_hudds]

@@DerekArends I think that would be a superb and helpful video.

[@SMCGPRA]

fed up of seeing openai why can't use open-source LLMs

[@DerekArends]

I will try to make a few with other LLM and see what magic happens!

[@CristianMolina]

at least he used an OSS vector DB ;)