Python RAG Tutorial (with Local LLMs): AI For Your PDFs

Learn how to build a RAG (Retrieval Augmented Generation) app in Python that can let you query/chat with your PDFs using generative AI.
This project contains some more advanced topics, like how to run RAG apps locally (with Ollama), how to update a vector DB with new items, how to use RAG with PDFs (or any other files), and how to test the quality of AI generated responses.
👉 Links
🔗 GitHub: github.com/pixegami/rag-tutor...
🔗 Basic RAG Tutorial: • RAG + Langchain Python...
🔗 PyTest Video: • How To Write Unit Test...
👉 Resources
🔗 Document loaders: python.langchain.com/docs/mod...
🔗 PDF Loader: python.langchain.com/docs/mod...
🔗 Ollama: ollama.com
📚 Chapters
00:00 Introduction
01:06 RAG Recap
03:22 Loading PDF Data
05:08 Generate Embeddings
07:16 How To Store and Update Data
10:46 Updating Database
11:45 Running RAG Locally
15:12 Unit Testing AI Output
20:29 Wrapping Up

[@musiitwaedmond1426]

this is the best RAG tutorial I have come across on youtube, thank you so much man💪

[@pixegami]

Thank you! I appreciate it!

[@tinghaowang-ei7kv]

It's hard to find such high quality videos on China's Beep, but you've done it, thank you so much for your selflessness. Great talk, looking forward to the next video. Thanks again, you did a great job!

[@pixegami]

Thank you! Glad you enjoyed it!

[@denijane89]

That was the most useful video I've seen on the topic (and I watched quite a lot). I didn't realise that the quality of the embedding is so important. I have one working code for local pdf ai, but I wasn't very impressed by the results. That explains why. Thank you for the great content. I'd love to see other uses of local LLMs.

[@pixegami]

Glad you liked it! Thanks for commenting and for sharing your experience. And absolutely - when building apps with LLM (or any kind of ML/AI technology), the quality of the data and the index is really non-negotiable if you want to have high-quality results.

[@frederichominh3152]

Best tutorial I've ever seen in a long time, maybe ever. Timing, sequence, content, logic, context... everything is right in your video. Thank YOU and congrats, you are smart as hell.

[@heesongkoh]

agreed.

[@pixegami]

Wow, thanks for your comment. I really appreciate it, and I'm glad you liked the video.

[@JaqUkto]

Thank you very much! I've started my RAG using your vids. Of course, much of your code needed to be updated, but it was simple even given my zero knowledge of Python.

[@sergiovasquez7686]

You may could share the updates to us 😅

[@pixegami]

Nice work, glad you got it working!

[@nachoeigu]

Your content is amazing! Keep it going. I would like to see the continuation of this video in terms of how to upload and automate the workflow in the cloud AWS and how to integrate the chat interface with telegram bot

[@pixegami]

Glad you liked it, and thanks for the suggestions. My next video will be focused on how to deploy this to the cloud - but I hadn't thought about the Telegram bot idea before, I will look up how to do that.

[@nascentnaga]

Suuuuuper helpful. I need to test this for a work idea. thank you!

[@pixegami]

You're welcome!

[@paulham.2447]

Very very useful and so much well explained ! Thanks.

[@pixegami]

Thank you!

[@careyatou]

I got this to work with my own data. This was so cool. Thanks!

[@pixegami]

Awesome! Glad to hear it worked for you :)

[@fabsync]

Oh man.. by far the best tutorial on the subject.. finally someone using pdf and explaining the entire process! You should do a more in-depth series on this...

[@pixegami]

Thank you for the feedback :) Looks like with the interest this topic has received, I'm definitely keen to dive into it a bit deeper.

[@fabsync]

One of the questions that I was asking myself with pdf.. do you clean the pdf before doing the embeddings .. or this is something that you can resolve by customizing the prompt? What would be a good way to do semantic search after using pgvector..? I am still struggling with those answers

[@pixegami]

@@fabsync Yeah I've had a lot of people ask about cleaning the PDFs too. I think if you have PDFs that have certain structural challenges, I'd probably recommend to find a way to clean/augment it for your workflow. And LLM prompt can only go so far, and cleaning noise from the data will always help.

[@joxxen]

Very nice, I wish I had this guide few weeks ago, had to learn it the hard way xD

[@pixegami]

You got there in the end :)

[@user-xk3tj5cj8p]

Recently discovered your channel 🎉 , subscribed 😊 keep up the awesome content

[@pixegami]

Thank you! Welcome to the channel!

[@Mykyta-Korniienko-CS]

Deploying the model on the cloud would definitely be interesting! thank you for the video :D

[@NW8187]

Simplifying a complex topic for a diverse set of users requires an amazing level of clarity of thought, knowledge and communication skills, which you have demonstrated in this video. Congratulations! Here are some items on my wish list for you when you can get to it. 1. Ability for users to pick among a selected list of open-source LLMs. A list that users can keep it updated. 2. build a local RAG application for getting insights from personal tabular data, which stored in multiple formats e.g. excel/google sheets, PDF tables

[@pixegami]

Thanks for your comment, I'm really glad to hear it was helpful. I appreciate you sharing the feedback and suggestions as well, I've added these items to my list of ideas for future videos :)

[@zhubarb]

Crystal clear. Great video.

[@pixegami]

Thank you! Glad to hear that :)

[@muhannadobeidat]

Great video and nicely scripted. Thanks for the excellent effort. I find that nomic 1.5 is pretty good for embedding and lightweight as well. I did not do actual performance metric based analysis of that but actual recall and precision testing is pretty impressive with 768 dimensions only.

[@pixegami]

Thank you! Glad nomic text worked well for your use case :)

[@gustavojuantorena]

Great content as always!

[@pixegami]

Thanks for watching!

[@basselkordy8223]

High quality stuff. Thanks

[@pixegami]

Glad you liked it!

[@jial.5245]

Thank you so much for the content👍🏼 very well explained! Would be great to see a use case of using autogen multi-agent approach to enhance RAG response.

[@pixegami]

Glad you liked it, thank you! And thanks for the suggestion and project idea :)

[@AlexandreBarbosaIT]

Smashed the Subscribe button! Awesome content! Looking forward for the next ones.

[@pixegami]

Thank you! Glad you enjoyed it, and welcome!

[@durand101]

Such a helpful tutorial, thank you!

[@pixegami]

Glad you enjoyed it!

[@RasNot]

Great content, thanks for making it!

[@pixegami]

Glad you enjoyed it!

[@kam3580]

Thanks, great video! To avoid duplicate documents and to look for updated documents I use a document hash to check.

[@pixegami]

Yup! Great idea :)

[@60pluscrazy]

Excellent 🎉🎉🎉

[@pixegami]

Thank you! Cheers!

[@carlosalberto-mo1wj]

I simply love the hole video! for the next Rag tutorial can you make a deploy on a azure cloud or any other cloud, just to see in depth how this works! thanks so mutch for the content man!

[@pixegami]

My upcoming video is actually about how to deploy a RAG app like this to the AWS cloud :) Stay tuned!

[@namaefumei]

This is great!

[@elvistolotti45]

great tutorial

[@pampaniyavijay007]

Superb bro 🤩

[@pixegami]

Thank you!

[@sergiovasquez7686]

I just subscribed to your channel… very high vids on KZbin

[@pixegami]

Thank you! Welcome.

[@KrishnaKotabhattara]

For evaluation, use RAGAs and Langsmith. There is also an SDK for azure which does same things as RAGAs and Langsmith.

[@pixegami]

Oh, thanks for the recommendation. I'll have to take a look into that.

[@ManuelJimenez1]

Thanks for the whole tutorial, I would suggest adding speed at the queries of the vector database with PostgreSQL with pgvector, pg_embeddings plugin.

[@JohnBoen]

He he he... tests are easy. I was wondering how to do those. Prompt: State several facts about the data and construct a question that asks for each fact. Create tests that look for the wrong answer... Give me 50 of each... Give me some examples of boundary conditions... Formatting... In an hour I will have fat stack of tests that would normally take a day a day to create. This is awesome :)

[@mehmetkaya4330]

Great tutorial! And if you could please do a tutorial on when/how to know data within the documents (pdf or csv etc) has changed?

[@pixegami]

Thanks for the suggestion :) That's a good idea, I think I'll have to plan it...

[@ishadhiwar7636]

Thank you for the fantastic tutorial! It was incredibly helpful and well-explained. I was wondering if you have any plans to release a video on fine-tuning this project using techniques like RLHF? It would be great to see your insights on that aspect as well.

[@pixegami]

Thank you! Glad you enjoyed the video. I've noted the suggestion about fine-tuning-I hadn't considered it yet, but thanks for sharing that idea with me.

[@samho9831]

Thanks for sharing and I can have a grasp of the concept. If possible, it will be deeply appreciated if you can show how to do it in the cloud. Thanks.

[@pixegami]

Stay tuned for my next video! kzbin.info/www/bejne/opWpgIF4pJd2iJI

[@mrrohitjadhav470]

After searching 100s of videos journey ends here. 😍Please would you make a tutorial making a knowledge graph using Ollama?

[@pixegami]

Thanks, glad your journey came to an end :) Thanks for the suggestion - I've added the idea to my list :)

[@mrrohitjadhav470]

@@pixegami Aweeeeeeeesome, Just want to slightly change the knowledge graph based on pdf,txt (own data). Sorry for not elaborating, but too much own data makes it difficult to find connections between many sources.

[@mrrohitjadhav470]

??/

[@kozark875491]

Very high quality video! Thank you!!! What are the min requrements to download and run locally llama3?

[@iainhmunro]

This is pretty good. I was wondering how I could integrate this with my current python scripts for my AI Calling Agent, so if someone wanted to call the number, they could chat with the PDF.

[@pixegami]

I think that certainly should be possible, but it's quite complicated (I haven't done anything like that before myself). You'd probably need something to hook up a phone number/service to an app that can transcribe the text in real like (like what Alexa or Siri does), then have an agent to figure out what to do with that interaction. And eventually hook it up to the RAG app. After that, you'll need to seriously think about guard-rails for the agent, otherwise you could end up with it getting your business into trouble. An example of this is when Air Canada's chatbot promised a customer a discount that wasn't available: www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know

[@ayoubfr8660]

Great stuff as usual! Could we have a video about how to turn this RAG app into a nice and proper desktop app with a graphic interface? Cheers mate.

[@pixegami]

Good idea, thanks! I'll note it down as a video idea :)

[@ayoubfr8660]

@@pixegami Thank you for the reply and reactivity! Have a nice day!

[@J3R3MI6]

@@pixegamiI subbed for the advanced RAG content

[@mingilin1317]

Great video! Successfully implemented RAG for the first time, so touching. Subscribed to the channel already! In the video, you mentioned handling document updates. Do you have plans to cover this topic in the future? I'm really interested about it! Also, is "ticket_to_ride" and "monopoly" sharing the same database in example code? What if I don't want them to share? Is there a way to handle that?

[@pixegami]

Awesome! Glad to hear about your successful RAG project, well done! I've had a lot of folks ask about vector database updates, so it's something I definitely want to cover. If you want to store different pieces of data in different databases, then I recommend put another layer of logic on top of the document loading (and querying). Have each folder use a different database (named after each folder), then add another LLM layer to interpret the question, and map it to which database it should query.

[@MrRoflHamster]

Thank you so much for the tutorial! How would you go about creating a webinterface for the application to serve it locally for example?

[@felixkindawoke]

Thank you! Could you do a tutorial on how to talk to the data? So based on this create a voice chat with it.

[@maikoke6768]

correct and improve: The issue I have with the Rag is that when I ask about something in a document that I know doesn't exist, the AI still provides a response, even though I would prefer it not to.

[@danielcomeon]

Thanks a lot. Great video!!! I want to know how to add new data to the existing database with new unique IDs.

[@pixegami]

Thanks! Glad you liked it. If you just want to **add** new data, the chapter on updating the database should already cover this. You just need to add new files into the folder, and run the `populate_database` command again. Any pages/docs on in the database will be added. But if you meant updating existing pages/segments in the existing data, then yes I'll have to make a video/tutorial about that :)

[@SmooveFinance]

Hi, loved the video. I was wondering if it would be possible to populate the data with markdown files as well as pdf. this would greatly improve my training data :) thanks for your time

[@pixegami]

Yes, my first RAG tutorial actually uses Markdown :) kzbin.info/www/bejne/qpTUdoiJg6ihbqs You can pretty much load any type of data, just need to search Langchain for the appropriate dataloader: python.langchain.com/v0.1/docs/modules/data_connection/document_loaders/markdown/

[@davidgortega3734]

For the unit tests you can use tools or grammars to limit the output and that way you can fix some issues that you are showing

[@pixegami]

Good idea. I haven't actually explored testing LLM output in detail yet, and I think it will be a fascinating topic.

[@NgocDung211]

That's amazing video, thank you so much for your effort making those free. I want to know that if i have my own finetuned LLMs and want to use RAG. Is it something else diffrent.

[@pixegami]

Glad to hear it was helpful! I don't know how fine-tuned LLMs would interact with RAG. They are very different techniques - RAG is just a way to engineer better prompts. Both RAG and LLM fine-tuning can work together. But for general purpose use-cases, I'd probably avoid fine-tuning because it's harder and more expensive to operate/maintain.

[@AiWithAnshul]

This is an impressive setup! I'm currently using Weaviate as my Vector DB along with Open AI Models, and it's working really well for handling PDFs, Docs, PPTs, and even Outlook email files. However, I've been struggling to integrate Excel and CSV files into my Knowledge Base. For small Excel files, the vector approach seems fine, but it's challenging for larger ones. I'd love to get your input on how to build a system that incorporates Excel files along with the other formats. I've considered using something like PandasGPT for handling the Excel and CSV files and the traditional RAG approach for the remaining file types (PDFs, Docs, etc.). Perhaps adding an agent as the first layer to determine where to direct the query (to the RAG model or PandasGPT) would be a good idea? What are your thoughts on this?

[@pixegami]

Thanks for your comment and for sharing your challenges and ideas. I think if you are mixing free-form text (like documents) and something more traditionally queryable (like a DB), it does make sense to engineer some more modality into your app (like what you suggested). I haven't explored that far myself so I can't share anything useful yet. But I'll be sure to keep it in mind for future videos. Good luck with your project!

[@maxflokinho]

I would like it not only to be able to read PDFs but also if the final information was 'weak' or missing information, it would do an internet search on the topic provided in the 'query' and complete the final answer with the collected data. Do you think this is feasible to do? I thought about using agents for this, like crewai. I looked but couldn't find any tutorial that used both methods.

[@nickmills8476]

To update the chromadb data for PDF chunks whose data has changed, store the PDF document contents hash in the metadata field. In addition to adding IDs that don't already exist, select records whose metadata.hash has changed and update these records, using collection.update()

[@user-xz1jh9qv1k]

Very good content. Thanks for making it. Actually I liked your validation idea but How about for the descriptive answers to evaluate? Does pytest and prompt together works? Also did you make tutorial on how to update the vector database when file content changes.

[@pixegami]

Thanks, I appreciate it! I think you can also try to use the same evaluation strategy for descriptive answers. I've also seen other commenters mention frameworks for evaluating LLM responses so that might be worth looking into as well.

[@JorgeGil-qf6zy]

thank you for the video pixegami, question, how will you do to implement follow up questions based on the last answer?

[@pixegami]

You'd probably need a way to store/manage memory and use it as part of the next prompt. I haven't explored this much myself, but it's a topic I'm interested to look into as well, so thanks for the comment :)

[@HimSecOps]

Amaizing bro! Thank you I request you to tell how to connect the prompt part to ui

[@pixegami]

Thanks for the suggestion :) This is on my list to work on as well, stay tuned!

[@MatsAlexanderNissen-Lie]

Hey, what is the best free model for embedding out there? Cool video, great content!

[@LaptopiaLTD]

Thank you for making this concise, clear, and helpful video. Is there a limitation on the quantity and size of PDF files? I'm currently using ChatRTX; however, it has limitations due to being beta. Perhaps a video exploring this question and ChatRTX limitations would be useful?

[@pixegami]

Glad you enjoyed it! I haven't had a look at ChatRTX yet, but thanks for the suggestion.

[@edgarallik9995]

Thank you for the video! Would there be an advantage in an LLM pre-processing the user query before it is embedded and matched against the database of documents? I'm thinking perhaps embedding the more useful parts of the user question to help in the matching process.

[@pixegami]

Yes, there's actually quite a bit of research on "query transformation" to improve RAG results (and it's effective too). E.g. there's a technique that uses an LLM to create a "fake answer" to the question, then use that fake answer to query (instead of the question). Here's a paper on that: boston.lti.cs.cmu.edu/luyug/HyDE/HyDE.pdf

[@chhil]

Thank you for your content and access to your github repo. I tried modifying the code to read java file from the folder and keep getting an error Unsupported mime type: text/x-java-source. I use the GenericLoader to load the java file. Any pointers on where to look for a solution?

[@nickmills8476]

Using a local embedding model: mxbai-embed-large, got me similar results to your monopoly answer.

[@pixegami]

Thanks for sharing! I hadn't tried that one yet.

[@shaigrustamov5115]

Great videos 👍 Do you think RAG is better than fine tuning for invoice data extraction? If I have 5000 invoices and want to train a model for data extraction, do you know if I need to prepare both OCR and labels? Do the labels also need to contain bounding boxes? Is not this very time consuming? Are there models that can be trained without bounding boxes?

[@pixegami]

I don't have a ton of experience with fine-tuning LLM models so I can't compare them or advise them for your use-case. But in general, I prefer not to fine-tune, since that is quite a slow/expensive process and it'll bind your use-case tightly to your model (a RAG approach lets you switch LLMs more easily). For the OCR issue, I think you'll probably want to separate that process out as a different problem. You might want a computer-vision process to first extract the data for your document and standardize them somehow for your RAG workflow later.

[@rob679]

On model param size, 7B models are enough. Not related to this video, but I'm using Llama3 8B with OpenWebUI's RAG and it works but it sometimes have problems to refer to correct document while giving correct answer (it will hallucinate document name), but its how its RAG implementation are.

[@pixegami]

Interesting, I haven't tried this with the 7GB models yet. Thanks for sharing!

[@mo3x]

So it is just an advanced ctrl+f ?

[@pixegami]

Yes, that's one way to think about it. Still, incredibly powerful.

[@patoury4392]

Very interesting, thanks for this video. I have a question : Once you realize that your model has many failures. How can you adjust back the model ? Should there be document replacement ? like starting over ?

[@pixegami]

Thank you for watching! That's an excellent question. I haven't explored all the options to rollback and limit blast radius in detail. Top of my head, I'd probably: - Separate my data-sources and DBs by topic/subject to limit the blast-radius (if something goes wrong). - Store every single version of the embedding DB and source data as a back-up. Maybe make them last 1-2 years before they get deleted (if storage cost is a problem). Then if I have an issue, just rollback to the previous version and try again.

[@gianmarco-lr7wc]

hank you! It would be great to see how to deploy this to the cloud, for examplw aws!

[@pixegami]

That's a great idea!

[@vibesforyou8230]

Hey , can yolu tell me which python version do you use for such project ? I am curious about working with LLMs , so this best video which gave me a decent understanding of communicating with LLMs but i never used pyhton so need to know what version is better for such project . BTW it great have your in feed and go through it 🙂

[@pixegami]

That's awesome. I use Python 3.10 right now, but anything higher than 3.8 should be fine. When you are learning, the version doesn't matter as much (as long as it's > 3). I also have an entire beginner tutorial series for Python as well if you're looking to learn (all 100% free!): kzbin.info/aero/PLZJBfja3V3Rsbiz84Z63IXnTQZH_Rnfuo

[@srikanthdongala6922]

I am trying to implement source CRUD in my application, but I am looking for an approach to embed docs where there is an update in the document, you he raised the question in the video butI would really like to listen to an approach to take here, creation and deletion is fine but what about the update? I don't want to embed the entire document Any responses will be very helpful, Thanks:)

[@E.X.P.JP_roblox]

If my word documents contain mostly tables instead of text, and the format of the tables are all over the place and may be difficult to work with (with merged cells and sub tables), would the chunking and embedding steps stay the same? Or would you recommend cleaning up the word documents to kind of “flatten” the contents and perhaps make it easier for AI model to understand? However, manually cleaning up the files doesn’t sound like a scalable solution.

[@pixegami]

Very good question, that's a real challenge I've run into as well in other projects. It's hard to answer-the chunking/prompting strategy really depends on the data, and the best way to know is to just test different strategies. Of course, you probably don't want a "manual" solution, but it might make sense to do things manually for a small part of the data just to see what works. With tables, I'd probably attempt to pre-process it somehow so it's in a format that could be embedded more easily. For example, I might normalize the headers/columns of the table into every row. E.g. instead of "30", I might change it to "price: 30", because I notice most table formats have the headers at the top, but that is cut-off if the table is split into two or more chunks. Complex problem! Let me know if you find something that works.

[@ziadbensaada]

hi, it give me this problem when I run python populate_database.py: Could not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid. Bedrock error: The config profile (default) could not be found (type=value_error)

[@MaliciousCode-gw5tq]

Question will RAG method works even if the data are too big? Example 1k pages each file?

[@beatsofbinary]

I love your video. For the problem of updating a chunk, would a timestamp make sense? For example save the last modified date of a PDF file to each chunk Then after reading the documents again, compare if it's newer than that one in the database -> overwrite the chunks?

[@pixegami]

I'm not sure how that would work - if you had a PDF that splits into 100 chunks, and you updated chunk 57, then how does your system know that the chunk was updated? If you *knew* it was updated you can store timestamp, but how do you know when the chunk is updated in the first place? I think you're quite close - rather than a timestamp, you can consider using a hash of the chunk (e.g. MD5), then use that to figure out if the chunk has change or not.

[@beatsofbinary]

@@pixegami I thought I could save metadata such as Last Modified directly when importing data. If the Vector database already contains a PDF document that has the same name but is older, it will be overwritten. If it is the same age, the PDF is skipped. I would therefore query the database to see which documents are already in the database. But then all chunks of the older PDF would have to be deleted, you're right. The idea with the hash is good, read in the PDF, split it, create a hash, save it as metadata and then compare the hashes of the chunks. But that leads to further problems, doesn't it? If, for example, only a few words change within a chunk, but the overlap changes as a result, will several or all chunks change at the same time?

[@NicolaRomano]

Thanks for the great video! This might be a silly question, but say I want to extract some information from a bunch of PDF files. So, for example in your example of manuals I have manuals for 100 different games and want to know for each of them how many players is the game for. Is there a difference in processing all of the documents at once, putting them in a vector store and then query this versus processing one document at a time? I can see the advantage of having a vector store in case you want to ask another question, so you don't have to reprocess all of the documents, but aside from that? Also, can you somehow limit what context the LLM uses? Say I want to ensure the LLM is using file1.pdf but not file2.pdf how would I go about that?

[@REINOSO195]

Amigo eso es tema de tesis, lul

[@pixegami]

Thanks for watching and for the great questions. You are asking about 3 different use-cases I think, so it's probably best to tackle them one at a time as separate problems. 1) If I wanted a very specific piece of information from all my documents, and I need it be accurate, then I'd probably write purpose-specific logic to extract it rather than make it a general-purpose thing. 2) If you still wanted to run general queries across all the knowledge, then that could be a separate project (maybe using a knowledge graph instead of just a vector DB). 3) You could store separate DBs, or use the tags/meta-data to filter out page that aren't from the one you want. These are all very basic solutions to you problems, but you get the idea - a good solution will depend on your use-case and how strong you need the solution to be.

[@NicolaRomano]

​@@pixegamithanks I've been looking into metadata filtering and it seems to be a good direction indeed! I'm currently testing it and seems to do a good job!

[@uwegenosdude]

Thank you very much for the really helpful video. I also made bad experience when not using OpenAI to create embeddings. Can you recommend a free embedding model that produces good results and that I could run locally? I tried to use anythingLLM but RAG results were really bad. But I‘m not sure if the reason might be the language of my documents, they are all German.

[@pixegami]

Thanks, glad you found it helpful! For local Embedding models, I haven't tried anything other than nomic-text so far. Have you tried all the other embedding models on ollama.com/library? There is a couple there. For non-English text, I heard Mistral (based in France) is quite good. I don't know if it's available on Ollama (officially), but it's open source so you should be able to get a copy: docs.mistral.ai/capabilities/embeddings/

[@uwegenosdude]

@@pixegami thanks for your help. I will try this out.

[@user-zl5dl7bu8j]

We can duplicate edited files upon an edit trigger , delete original file and have the duplicate being added to our existing database.

[@edwardtse8631]

This is a very good tutorial, how do you solve the problem of edit a data file? to store sha-1 of the file?

[@pixegami]

Exactly right - you can use an even simpler hash function like MD5 just to check that the content of each chunk hasn't changed. You'll still have to loop through all the chunks though to calculate the hash and to compare them. That should be fine for 100s or 1000s of chunks, but it might not scale too well beyond that.

[@Queenmaya99]

hello! do you know if there is a way to filter the db.similarity_search_with_score, based on metadata? for example, if you had a query and you wanted it to only reference the monopoly pdf to answer it.

[@pixegami]

I think it might be possible. I haven't tried it, but looks like Chroma has an underlying API for filtering (and Langchain should surface it too): docs.trychroma.com/usage-guide#querying-a-collection collection.query( query_embeddings=[[11.1, 12.1, 13.1],[1.1, 2.3, 3.2], ...], n_results=10, where={"metadata_field": "is_equal_to_this"}, where_document={"$contains":"search_string"} )

[@siswanto4045]

Wow, this is what I wanted. But can it be running on open webUI? Like running ollama locally with webUI? Thank you

[@pixegami]

Glad to hear that! I do plan to do a tutorial later on how to build a a web UI for your app, so stay tuned.

[@siswanto4045]

@@pixegami can't wait to watch the video

[@EduardoJGaido]

Hello! Thank you for a great video. I ask you or the community, I have a hard problem to solve: i want to make a chatbot using a local LLM with RAG (keep reading please!) BUT i want to use it for my business so the clients of my physiotherapy clinic can ask it and it responds JUST WITH the information that is fed. Otherwise, it says "Oh, i don't know, wait please" so the secretary can answer instead. Just with that, I would be happy. I have a lot of FAQ listed with the answers (in a json friendly format). I can't find this answer anywhere. If you have any information, would be apreciated. Cheers from Argentina.

[@pixegami]

I see, if it's an FAQ I'd structure each question or piece of information as a separate file, and just use a directory loader to turn it into Langchain documents. Then you can try using the technique in this video to see if works. To get it to respect the answer boundary, you can probably write an explicit prompt or an evaluation step to say "I don't know" if the answer isn't clear from the queries.

[@rude_people_die_young]

Great work 🎉❤

[@pixegami]

Thank you!

[@TrevorDBEYDAG]

Thank you for the tutorial, I guess it should be a better way to create chunks, not only by character count because it cuts the paragraph in disruptive way. May be another library to split in paragraphs or at least end of the sentence?

[@pixegami]

The Recursive Text Splitter actually attempts to do what you suggest: python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/ "It tries to split on them in order until the chunks are small enough. The default list is [" ", " ", " ", ""]."

[@TrevorDBEYDAG]

@@pixegami That's cool.

[@AstigsiPhilip]

Hi, this python rag can handle 70,000 pdf index files? thank you for your response.

[@Farzaneh-fq6fn]

Thank you so much for the useful content. I have a question: I want to query on a pdf file which is in German language, but it answers in English and translates all the sentences. Do you know how can I fix this to answer in the original language of pdf file?

[@pixegami]

Hmm, did you also update all the prompts to be in German as well? Or explicitly add instruction to the prompt to use German? If it still answers in English, I recommend just adding a second request to the AI to translate the entire response to German (assuming the answer is correct). Sometimes it's easier to get the AI to do two simple tasks and to do one complex one.

[@JoseLuisCornejoRivas]

Now how to choose the correct document and not mix contexts from different documents? Thinking about a large documentary database

[@pixegami]

Good question. I haven't looked into this myself, but I suspect you'd need to upgrade how you store/query data, maybe using something like a knowledge graph instead (e.g. neo4j): python.langchain.com/v0.1/docs/use_cases/graph/

[@natemiles4951]

maybe a dumb question but im curious why the embeddings would be any different if using the same model to generate locally vs the cloud (IE bedrock)?

[@pixegami]

The same model should generate the same embeddings, so if your local model is EXACTLY the same as using a cloud model it should work. In this video I'm using different models: 1) Titan (via AWS Bedrock) for the embeddings and 2) Mistral for the LLM agent.

[@albertozacchini3388]

Hi! Thanks so much for the video, it was super useful. However, i face that is a big problem for my laptop give all the context to the LLM, it will blow my CPU. Do you know any API of other free LLM instead of a local one?

[@pixegami]

I haven't used any free LLM APIs myself, but the OpenAI and the AWS Bedrock ones are pretty cheap if you use the turbo models. Otherwise, I found this Reddit thread discussing free LLM APIs: www.reddit.com/r/deeplearning/comments/1350qtu/what_are_some_small_llm_models_or_free_llm_apis/

[@albertozacchini3388]

@@pixegami yeah I give up on searching but I found the Mistral API really nice for embeddigs, still cheap and super fast to use. By the way thank you so much!

[@maxi-g]

Hey, I have a question. I tried to load a fairly large PDF (100 pages) into the database (approx. 400 documents). However the add_to_chroma function seems to be excruciatingly slow. The output from ollama shows that the embeddings only get requested once every two seconds or so. There is also no CPU or GPU load on my system when this process is running. Is there any way to improve this? Thank's already

[@pixegami]

This is most definitely because of the time it takes to embed each page (since you mentioned embeddings get requested once every two seconds). Your Ollama model might not be able to fully leverage your hardware, which is potentially why your don't see your CPU/GPU load rise up. You could experiment by switching this to use an online embedding API (like OpenAI or AWS Bedrock) and see if it's faster. Or you could double check to see if Ollama is using your GPU correctly (github.com/ollama/ollama/blob/main/docs/gpu.md)

[@Ammarsays]

I am just a layman and I want to know if text splitters count the characters, words or sentences for a given chunk size? And if the text splitters can identify sentences or paragraphs in text?

[@pixegami]

Yup, the splitter should attempt to do that. Here's the documentation: python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/ "It is parameterized by a list of characters. It tries to split on them in order until the chunks are small enough. The default list is [" ", " ", " ", ""]."

[@coreybarron907]

Did anyone find out how to get the langchain packages to work with pydantic?? I can't seem to do that

[@xspydazx]

Question : once loading a vector store , how can we output a dataset from the store to be used as a fine tuning object ? as this is the most important part : producing the data from documents that will be used to fine tune the model : as the rag will have the domain documents etc embedded into the vector store: As you know retrieval time will be reduced after fine tuning the data into the model : but after deciding if the data is good to update of course , hence all historical chat should also be uploaded into the rag at the end of the chat?

[@pixegami]

It really depends on what you want your training data to look like. But honestly I haven't done a lot of work in LLM fine-tuning myself so I can't really give advice on this question yet.

[@xspydazx]

@@pixegami in truth the beauty of langchain is it's document loaders , hence it enables for you to create datasets from your documents ... Or sites , so it just needs to be saved to JSON . Or your rag will continue to grow . As we know the rag is to fill the training gap ! But if we have converted our docs we can run a fine tuning session and basically update the model .. so only for large docs or new docs should be in the rag .... The rag ...IE augmented query's ... This.can be done with your chains !! By your chain of questions to your chain , a still augmented response !

[@jerkmeo]

awesome video...thanks mate.

[@pixegami]

You're welcome!

[@Nord-su]

Thanks a lot. How can I make RAG interact with Open WebUI as an interface?

[@pixegami]

Very interesting. I haven't looked into docs.openwebui.com/ yet, but it looks like a great use-case for integrating with RAG. Thanks for the idea!

[@derekpunaro2422]

Hi Pixe! I was wondering how would you write the get_embedding_function for Chat GPT OPEN AI?

[@pixegami]

My first RAG project actually uses OpenAI embeddings: kzbin.info/www/bejne/qpTUdoiJg6ihbqs Here is the documentation and code examples from Langchain: python.langchain.com/docs/integrations/text_embedding/openai/

[@DCW09]

Cursory glance says - add a hashing function to the chunk metadata, this way the chunk should have a unique identifier (MD5, SHA, Etc) if anything changes the hash will also change. Then its just simple logic to validate current chunk page.index against an existing one's hash. If its different, overwrite. If its not, dont waste the cycles. In practice, I am not 100% sure that this would be the approach but at least the theory here should be pretty on point for identifying changes with few compute cycles.

[@pixegami]

Yup! I think that's probably the way I'd do it too. If there's too documents and you need to scale it, then I guess you can hash the entire document as well first to narrow the search space each time.

[@MichaelTanOfficialChannel]

@@pixegami I just want to add that I would apply the hash to the entire page and not the chunk. A page can be edited in a way where the content is shorter than previous version, thereby causing the number of chunks to be less that what it previously was. And I would also remove all chunks belonging to the said page before adding new chunks, so as not to have an orphaned chunk from the previous lengthier page.

[@phizicks]

md5 of the data to the index, if it matches, no update needed. unless I didn't understand the requirement

[@pixegami]

Yup, I think that's probably how you'd go about indexing/updating specific chunks. You might also need a tree structure (e.g. hash the entire document or categorize them first) if you want to work in the scale of 10k+ documents.

[@vidfan1967]

I want to verify an existing PowerPoint document against a PDF, e.g. with an updated law text. I want to find out, whether my statements on each PPT slide are still true. Challenge: each slide contains one or more statements, which should be verified against the PDF, for example you have 6 or 10 bullet points on one slide. To use RAG I cannot use them all for the query as they might be quite diverse and would knowledge from the PDF, that is not specifically matching any one point but all of them together. Also: the context on the slide should be considered together with each statement, e.g. the title of the slide, an intro text above the bullet point list, or the upper level information for a statement that sits in a sub-structure of bullet points. I guess I would somehow need to split the statements in the PPT in logical chunks, preserving the context. Is there a python function I could use? Or can this be done with AI (e.g. few shot) after the slide text has been extracted? If this is of wider interest, I would appreciate a video on this 🙂

[@pixegami]

Thanks for sharing your use case. Yup, this is definitely something you need to solve during the "chunking" phase of the process. For example, there are some experimental "chunking" functions from Langchain you could try: python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker/ Also, you could "bake" in custom information to each chunked document itself. E.g. something like (each variable name is made up): chunk.metadata["page_content"] = title_of_doc_str + context_str + actual_page_text_str

[@roxtonjhon]

How to retain context information in multi-turn conversations

[@liqwis9598]

any idea how this can be implemented in Java / Spring boot

[@GregBreak]

How can I add multi language support? So I can type in spanish and get response in the same language

[@nvajay3829]

Brother, this is working well for PDFs, but PDFs with tables and json files it's not able to do even if i use json loader and modify the code. Can you make a video for it ? Local RAG for large json files

[@pixegami]

Ahh yes, any weird formatting inside a PDF will be challenging. I'd probably try to approach it by seeing if I can first parse it into Markdown or HTML, because I think data in those formats are a little easier to work with. If you have any examples of PDFs you'd like to parse, please feel free to share them here and I'll see if it gets interest for me to cover in a future video. Thank you!

[@KaiJaz-lm8oj]

The vscode theme is looking cool! Can I know the theme name?

[@pixegami]

I use Monokai Pro :)

[@nhtdmr]

Could you make video same project with LLAMA 3?

[@pixegami]

You can do it here! Just change the Ollama model to Llama 3: ollama.com/blog/llama3

[@nirmalkumar007]

which part of the code makes the API call to the OLAMA server ? Kindly help

[@pixegami]

The Langchain Ollama wrapper class (e.g. python.langchain.com/docs/integrations/text_embedding/ollama/ for the embedding) wraps all the code to call Ollama for you.