Python RAG Tutorial (with Local LLMs): AI For Your PDFs

Learn how to build a RAG (Retrieval Augmented Generation) app in Python that can let you query/chat with your PDFs using generative AI.
This project contains some more advanced topics, like how to run RAG apps locally (with Ollama), how to update a vector DB with new items, how to use RAG with PDFs (or any other files), and how to test the quality of AI generated responses.
👉 Links
🔗 GitHub: github.com/pixegami/rag-tutor...
🔗 Basic RAG Tutorial: • RAG + Langchain Python...
🔗 PyTest Video: • How To Write Unit Test...
👉 Resources
🔗 Document loaders: python.langchain.com/docs/mod...
🔗 PDF Loader: python.langchain.com/docs/mod...
🔗 Ollama: ollama.com
📚 Chapters
00:00 Introduction
01:06 RAG Recap
03:22 Loading PDF Data
05:08 Generate Embeddings
07:16 How To Store and Update Data
10:46 Updating Database
11:45 Running RAG Locally
15:12 Unit Testing AI Output
20:29 Wrapping Up

[@tinghaowang-ei7kv]

It's hard to find such high quality videos on China's Beep, but you've done it, thank you so much for your selflessness. Great talk, looking forward to the next video. Thanks again, you did a great job!

[@pixegami]

Thank you! Glad you enjoyed it!

[@frederichominh3152]

Best tutorial I've ever seen in a long time, maybe ever. Timing, sequence, content, logic, context... everything is right in your video. Thank YOU and congrats, you are smart as hell.

[@heesongkoh]

agreed.

[@pixegami]

Wow, thanks for your comment. I really appreciate it, and I'm glad you liked the video.

[@musiitwaedmond1426]

this is the best RAG tutorial I have come across on youtube, thank you so much man💪

[@pixegami]

Thank you! I appreciate it!

[@denijane89]

That was the most useful video I've seen on the topic (and I watched quite a lot). I didn't realise that the quality of the embedding is so important. I have one working code for local pdf ai, but I wasn't very impressed by the results. That explains why. Thank you for the great content. I'd love to see other uses of local LLMs.

[@pixegami]

Glad you liked it! Thanks for commenting and for sharing your experience. And absolutely - when building apps with LLM (or any kind of ML/AI technology), the quality of the data and the index is really non-negotiable if you want to have high-quality results.

[@JaqUkto]

Thank you very much! I've started my RAG using your vids. Of course, much of your code needed to be updated, but it was simple even given my zero knowledge of Python.

[@sergiovasquez7686]

You may could share the updates to us 😅

[@pixegami]

Nice work, glad you got it working!

[@careyatou]

I got this to work with my own data. This was so cool. Thanks!

[@pixegami]

Awesome! Glad to hear it worked for you :)

[@nachoeigu]

Your content is amazing! Keep it going. I would like to see the continuation of this video in terms of how to upload and automate the workflow in the cloud AWS and how to integrate the chat interface with telegram bot

[@pixegami]

Glad you liked it, and thanks for the suggestions. My next video will be focused on how to deploy this to the cloud - but I hadn't thought about the Telegram bot idea before, I will look up how to do that.

[@nascentnaga]

Suuuuuper helpful. I need to test this for a work idea. thank you!

[@pixegami]

You're welcome!

[@paulham.2447]

Very very useful and so much well explained ! Thanks.

[@pixegami]

Thank you!

[@user-xk3tj5cj8p]

Recently discovered your channel 🎉 , subscribed 😊 keep up the awesome content

[@pixegami]

Thank you! Welcome to the channel!

[@NW8187]

Simplifying a complex topic for a diverse set of users requires an amazing level of clarity of thought, knowledge and communication skills, which you have demonstrated in this video. Congratulations! Here are some items on my wish list for you when you can get to it. 1. Ability for users to pick among a selected list of open-source LLMs. A list that users can keep it updated. 2. build a local RAG application for getting insights from personal tabular data, which stored in multiple formats e.g. excel/google sheets, PDF tables

[@pixegami]

Thanks for your comment, I'm really glad to hear it was helpful. I appreciate you sharing the feedback and suggestions as well, I've added these items to my list of ideas for future videos :)

[@joxxen]

Very nice, I wish I had this guide few weeks ago, had to learn it the hard way xD

[@pixegami]

You got there in the end :)

[@fabsync]

Oh man.. by far the best tutorial on the subject.. finally someone using pdf and explaining the entire process! You should do a more in-depth series on this...

[@pixegami]

Thank you for the feedback :) Looks like with the interest this topic has received, I'm definitely keen to dive into it a bit deeper.

[@fabsync]

One of the questions that I was asking myself with pdf.. do you clean the pdf before doing the embeddings .. or this is something that you can resolve by customizing the prompt? What would be a good way to do semantic search after using pgvector..? I am still struggling with those answers

[@pixegami]

@@fabsync Yeah I've had a lot of people ask about cleaning the PDFs too. I think if you have PDFs that have certain structural challenges, I'd probably recommend to find a way to clean/augment it for your workflow. And LLM prompt can only go so far, and cleaning noise from the data will always help.

[@jial.5245]

Thank you so much for the content👍🏼 very well explained! Would be great to see a use case of using autogen multi-agent approach to enhance RAG response.

[@pixegami]

Glad you liked it, thank you! And thanks for the suggestion and project idea :)

[@muhannadobeidat]

Great video and nicely scripted. Thanks for the excellent effort. I find that nomic 1.5 is pretty good for embedding and lightweight as well. I did not do actual performance metric based analysis of that but actual recall and precision testing is pretty impressive with 768 dimensions only.

[@pixegami]

Thank you! Glad nomic text worked well for your use case :)

[@zhubarb]

Crystal clear. Great video.

[@pixegami]

Thank you! Glad to hear that :)

[@basselkordy8223]

High quality stuff. Thanks

[@pixegami]

Glad you liked it!

[@gustavojuantorena]

Great content as always!

[@pixegami]

Thanks for watching!

[@AlexandreBarbosaIT]

Smashed the Subscribe button! Awesome content! Looking forward for the next ones.

[@pixegami]

Thank you! Glad you enjoyed it, and welcome!

[@nickmills8476]

To update the chromadb data for PDF chunks whose data has changed, store the PDF document contents hash in the metadata field. In addition to adding IDs that don't already exist, select records whose metadata.hash has changed and update these records, using collection.update()

[@RasNot]

Great content, thanks for making it!

[@pixegami]

Glad you enjoyed it!

[@KrishnaKotabhattara]

For evaluation, use RAGAs and Langsmith. There is also an SDK for azure which does same things as RAGAs and Langsmith.

[@pixegami]

Oh, thanks for the recommendation. I'll have to take a look into that.

[@mrrohitjadhav470]

After searching 100s of videos journey ends here. 😍Please would you make a tutorial making a knowledge graph using Ollama?

[@pixegami]

Thanks, glad your journey came to an end :) Thanks for the suggestion - I've added the idea to my list :)

[@mrrohitjadhav470]

@@pixegami Aweeeeeeeesome, Just want to slightly change the knowledge graph based on pdf,txt (own data). Sorry for not elaborating, but too much own data makes it difficult to find connections between many sources.

[@namaefumei]

This is great!

[@ishadhiwar7636]

Thank you for the fantastic tutorial! It was incredibly helpful and well-explained. I was wondering if you have any plans to release a video on fine-tuning this project using techniques like RLHF? It would be great to see your insights on that aspect as well.

[@pixegami]

Thank you! Glad you enjoyed the video. I've noted the suggestion about fine-tuning-I hadn't considered it yet, but thanks for sharing that idea with me.

[@elvistolotti45]

great tutorial

[@iainhmunro]

This is pretty good. I was wondering how I could integrate this with my current python scripts for my AI Calling Agent, so if someone wanted to call the number, they could chat with the PDF.

[@pixegami]

I think that certainly should be possible, but it's quite complicated (I haven't done anything like that before myself). You'd probably need something to hook up a phone number/service to an app that can transcribe the text in real like (like what Alexa or Siri does), then have an agent to figure out what to do with that interaction. And eventually hook it up to the RAG app. After that, you'll need to seriously think about guard-rails for the agent, otherwise you could end up with it getting your business into trouble. An example of this is when Air Canada's chatbot promised a customer a discount that wasn't available: www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know

[@ayoubfr8660]

Great stuff as usual! Could we have a video about how to turn this RAG app into a nice and proper desktop app with a graphic interface? Cheers mate.

[@pixegami]

Good idea, thanks! I'll note it down as a video idea :)

[@ayoubfr8660]

@@pixegami Thank you for the reply and reactivity! Have a nice day!

[@J3R3MI6]

@@pixegamiI subbed for the advanced RAG content

[@mehmetkaya4330]

Great tutorial! And if you could please do a tutorial on when/how to know data within the documents (pdf or csv etc) has changed?

[@pixegami]

Thanks for the suggestion :) That's a good idea, I think I'll have to plan it...

[@JohnBoen]

He he he... tests are easy. I was wondering how to do those. Prompt: State several facts about the data and construct a question that asks for each fact. Create tests that look for the wrong answer... Give me 50 of each... Give me some examples of boundary conditions... Formatting... In an hour I will have fat stack of tests that would normally take a day a day to create. This is awesome :)

[@sergiovasquez7686]

I just subscribed to your channel… very high vids on KZbin

[@pixegami]

Thank you! Welcome.

[@kozark875491]

Very high quality video! Thank you!!! What are the min requrements to download and run locally llama3?

[@mingilin1317]

Great video! Successfully implemented RAG for the first time, so touching. Subscribed to the channel already! In the video, you mentioned handling document updates. Do you have plans to cover this topic in the future? I'm really interested about it! Also, is "ticket_to_ride" and "monopoly" sharing the same database in example code? What if I don't want them to share? Is there a way to handle that?

[@pixegami]

Awesome! Glad to hear about your successful RAG project, well done! I've had a lot of folks ask about vector database updates, so it's something I definitely want to cover. If you want to store different pieces of data in different databases, then I recommend put another layer of logic on top of the document loading (and querying). Have each folder use a different database (named after each folder), then add another LLM layer to interpret the question, and map it to which database it should query.

[@MrRoflHamster]

Thank you so much for the tutorial! How would you go about creating a webinterface for the application to serve it locally for example?

[@JorgeGil-qf6zy]

thank you for the video pixegami, question, how will you do to implement follow up questions based on the last answer?

[@pixegami]

You'd probably need a way to store/manage memory and use it as part of the next prompt. I haven't explored this much myself, but it's a topic I'm interested to look into as well, so thanks for the comment :)

[@maikoke6768]

correct and improve: The issue I have with the Rag is that when I ask about something in a document that I know doesn't exist, the AI still provides a response, even though I would prefer it not to.

[@60pluscrazy]

Excellent 🎉🎉🎉

[@pixegami]

Thank you! Cheers!

[@pampaniyavijay007]

Superb bro 🤩

[@pixegami]

Thank you!

[@NicolaRomano]

Thanks for the great video! This might be a silly question, but say I want to extract some information from a bunch of PDF files. So, for example in your example of manuals I have manuals for 100 different games and want to know for each of them how many players is the game for. Is there a difference in processing all of the documents at once, putting them in a vector store and then query this versus processing one document at a time? I can see the advantage of having a vector store in case you want to ask another question, so you don't have to reprocess all of the documents, but aside from that? Also, can you somehow limit what context the LLM uses? Say I want to ensure the LLM is using file1.pdf but not file2.pdf how would I go about that?

[@REINOSO195]

Amigo eso es tema de tesis, lul

[@pixegami]

Thanks for watching and for the great questions. You are asking about 3 different use-cases I think, so it's probably best to tackle them one at a time as separate problems. 1) If I wanted a very specific piece of information from all my documents, and I need it be accurate, then I'd probably write purpose-specific logic to extract it rather than make it a general-purpose thing. 2) If you still wanted to run general queries across all the knowledge, then that could be a separate project (maybe using a knowledge graph instead of just a vector DB). 3) You could store separate DBs, or use the tags/meta-data to filter out page that aren't from the one you want. These are all very basic solutions to you problems, but you get the idea - a good solution will depend on your use-case and how strong you need the solution to be.

[@NicolaRomano]

​@@pixegamithanks I've been looking into metadata filtering and it seems to be a good direction indeed! I'm currently testing it and seems to do a good job!

[@felixkindawoke]

Thank you! Could you do a tutorial on how to talk to the data? So based on this create a voice chat with it.

[@edgarallik9995]

Thank you for the video! Would there be an advantage in an LLM pre-processing the user query before it is embedded and matched against the database of documents? I'm thinking perhaps embedding the more useful parts of the user question to help in the matching process.

[@pixegami]

Yes, there's actually quite a bit of research on "query transformation" to improve RAG results (and it's effective too). E.g. there's a technique that uses an LLM to create a "fake answer" to the question, then use that fake answer to query (instead of the question). Here's a paper on that: boston.lti.cs.cmu.edu/luyug/HyDE/HyDE.pdf

[@AiWithAnshul]

This is an impressive setup! I'm currently using Weaviate as my Vector DB along with Open AI Models, and it's working really well for handling PDFs, Docs, PPTs, and even Outlook email files. However, I've been struggling to integrate Excel and CSV files into my Knowledge Base. For small Excel files, the vector approach seems fine, but it's challenging for larger ones. I'd love to get your input on how to build a system that incorporates Excel files along with the other formats. I've considered using something like PandasGPT for handling the Excel and CSV files and the traditional RAG approach for the remaining file types (PDFs, Docs, etc.). Perhaps adding an agent as the first layer to determine where to direct the query (to the RAG model or PandasGPT) would be a good idea? What are your thoughts on this?

[@pixegami]

Thanks for your comment and for sharing your challenges and ideas. I think if you are mixing free-form text (like documents) and something more traditionally queryable (like a DB), it does make sense to engineer some more modality into your app (like what you suggested). I haven't explored that far myself so I can't share anything useful yet. But I'll be sure to keep it in mind for future videos. Good luck with your project!

[@danielcomeon]

Thanks a lot. Great video!!! I want to know how to add new data to the existing database with new unique IDs.

[@pixegami]

Thanks! Glad you liked it. If you just want to **add** new data, the chapter on updating the database should already cover this. You just need to add new files into the folder, and run the `populate_database` command again. Any pages/docs on in the database will be added. But if you meant updating existing pages/segments in the existing data, then yes I'll have to make a video/tutorial about that :)

[@ziadbensaada]

hi, it give me this problem when I run python populate_database.py: Could not load credentials to authenticate with AWS client. Please check that credentials in the specified profile name are valid. Bedrock error: The config profile (default) could not be found (type=value_error)

[@nickmills8476]

Using a local embedding model: mxbai-embed-large, got me similar results to your monopoly answer.

[@pixegami]

Thanks for sharing! I hadn't tried that one yet.

[@LaptopiaLTD]

Thank you for making this concise, clear, and helpful video. Is there a limitation on the quantity and size of PDF files? I'm currently using ChatRTX; however, it has limitations due to being beta. Perhaps a video exploring this question and ChatRTX limitations would be useful?

[@pixegami]

Glad you enjoyed it! I haven't had a look at ChatRTX yet, but thanks for the suggestion.

[@user-xz1jh9qv1k]

Very good content. Thanks for making it. Actually I liked your validation idea but How about for the descriptive answers to evaluate? Does pytest and prompt together works? Also did you make tutorial on how to update the vector database when file content changes.

[@pixegami]

Thanks, I appreciate it! I think you can also try to use the same evaluation strategy for descriptive answers. I've also seen other commenters mention frameworks for evaluating LLM responses so that might be worth looking into as well.

[@shaigrustamov5115]

Great videos 👍 Do you think RAG is better than fine tuning for invoice data extraction? If I have 5000 invoices and want to train a model for data extraction, do you know if I need to prepare both OCR and labels? Do the labels also need to contain bounding boxes? Is not this very time consuming? Are there models that can be trained without bounding boxes?

[@pixegami]

I don't have a ton of experience with fine-tuning LLM models so I can't compare them or advise them for your use-case. But in general, I prefer not to fine-tune, since that is quite a slow/expensive process and it'll bind your use-case tightly to your model (a RAG approach lets you switch LLMs more easily). For the OCR issue, I think you'll probably want to separate that process out as a different problem. You might want a computer-vision process to first extract the data for your document and standardize them somehow for your RAG workflow later.

[@E.X.P.JP_roblox]

If my word documents contain mostly tables instead of text, and the format of the tables are all over the place and may be difficult to work with (with merged cells and sub tables), would the chunking and embedding steps stay the same? Or would you recommend cleaning up the word documents to kind of “flatten” the contents and perhaps make it easier for AI model to understand? However, manually cleaning up the files doesn’t sound like a scalable solution.

[@pixegami]

Very good question, that's a real challenge I've run into as well in other projects. It's hard to answer-the chunking/prompting strategy really depends on the data, and the best way to know is to just test different strategies. Of course, you probably don't want a "manual" solution, but it might make sense to do things manually for a small part of the data just to see what works. With tables, I'd probably attempt to pre-process it somehow so it's in a format that could be embedded more easily. For example, I might normalize the headers/columns of the table into every row. E.g. instead of "30", I might change it to "price: 30", because I notice most table formats have the headers at the top, but that is cut-off if the table is split into two or more chunks. Complex problem! Let me know if you find something that works.

[@TimC00k]

Thank you for making great video. but I can't even start first step because the terminal displays error.. please help me

[@derekpunaro2422]

Hi Pixe! I was wondering how would you write the get_embedding_function for Chat GPT OPEN AI?

[@pixegami]

My first RAG project actually uses OpenAI embeddings: kzbin.info/www/bejne/qpTUdoiJg6ihbqs Here is the documentation and code examples from Langchain: python.langchain.com/docs/integrations/text_embedding/openai/

[@Queenmaya99]

hello! do you know if there is a way to filter the db.similarity_search_with_score, based on metadata? for example, if you had a query and you wanted it to only reference the monopoly pdf to answer it.

[@pixegami]

I think it might be possible. I haven't tried it, but looks like Chroma has an underlying API for filtering (and Langchain should surface it too): docs.trychroma.com/usage-guide#querying-a-collection collection.query( query_embeddings=[[11.1, 12.1, 13.1],[1.1, 2.3, 3.2], ...], n_results=10, where={"metadata_field": "is_equal_to_this"}, where_document={"$contains":"search_string"} )

[@rob679]

On model param size, 7B models are enough. Not related to this video, but I'm using Llama3 8B with OpenWebUI's RAG and it works but it sometimes have problems to refer to correct document while giving correct answer (it will hallucinate document name), but its how its RAG implementation are.

[@pixegami]

Interesting, I haven't tried this with the 7GB models yet. Thanks for sharing!

[@chhil]

Thank you for your content and access to your github repo. I tried modifying the code to read java file from the folder and keep getting an error Unsupported mime type: text/x-java-source. I use the GenericLoader to load the java file. Any pointers on where to look for a solution?

[@HimSecOps]

Amaizing bro! Thank you I request you to tell how to connect the prompt part to ui

[@pixegami]

Thanks for the suggestion :) This is on my list to work on as well, stay tuned!

[@natemiles4951]

maybe a dumb question but im curious why the embeddings would be any different if using the same model to generate locally vs the cloud (IE bedrock)?

[@pixegami]

The same model should generate the same embeddings, so if your local model is EXACTLY the same as using a cloud model it should work. In this video I'm using different models: 1) Titan (via AWS Bedrock) for the embeddings and 2) Mistral for the LLM agent.

[@xspydazx]

Question : once loading a vector store , how can we output a dataset from the store to be used as a fine tuning object ? as this is the most important part : producing the data from documents that will be used to fine tune the model : as the rag will have the domain documents etc embedded into the vector store: As you know retrieval time will be reduced after fine tuning the data into the model : but after deciding if the data is good to update of course , hence all historical chat should also be uploaded into the rag at the end of the chat?

[@pixegami]

It really depends on what you want your training data to look like. But honestly I haven't done a lot of work in LLM fine-tuning myself so I can't really give advice on this question yet.

[@xspydazx]

@@pixegami in truth the beauty of langchain is it's document loaders , hence it enables for you to create datasets from your documents ... Or sites , so it just needs to be saved to JSON . Or your rag will continue to grow . As we know the rag is to fill the training gap ! But if we have converted our docs we can run a fine tuning session and basically update the model .. so only for large docs or new docs should be in the rag .... The rag ...IE augmented query's ... This.can be done with your chains !! By your chain of questions to your chain , a still augmented response !

[@uwegenosdude]

Thank you very much for the really helpful video. I also made bad experience when not using OpenAI to create embeddings. Can you recommend a free embedding model that produces good results and that I could run locally? I tried to use anythingLLM but RAG results were really bad. But I‘m not sure if the reason might be the language of my documents, they are all German.

[@pixegami]

Thanks, glad you found it helpful! For local Embedding models, I haven't tried anything other than nomic-text so far. Have you tried all the other embedding models on ollama.com/library? There is a couple there. For non-English text, I heard Mistral (based in France) is quite good. I don't know if it's available on Ollama (officially), but it's open source so you should be able to get a copy: docs.mistral.ai/capabilities/embeddings/

[@uwegenosdude]

@@pixegami thanks for your help. I will try this out.

[@albertozacchini3388]

Hi! Thanks so much for the video, it was super useful. However, i face that is a big problem for my laptop give all the context to the LLM, it will blow my CPU. Do you know any API of other free LLM instead of a local one?

[@pixegami]

I haven't used any free LLM APIs myself, but the OpenAI and the AWS Bedrock ones are pretty cheap if you use the turbo models. Otherwise, I found this Reddit thread discussing free LLM APIs: www.reddit.com/r/deeplearning/comments/1350qtu/what_are_some_small_llm_models_or_free_llm_apis/

[@albertozacchini3388]

@@pixegami yeah I give up on searching but I found the Mistral API really nice for embeddigs, still cheap and super fast to use. By the way thank you so much!

[@beatsofbinary]

I love your video. For the problem of updating a chunk, would a timestamp make sense? For example save the last modified date of a PDF file to each chunk Then after reading the documents again, compare if it's newer than that one in the database -> overwrite the chunks?

[@pixegami]

I'm not sure how that would work - if you had a PDF that splits into 100 chunks, and you updated chunk 57, then how does your system know that the chunk was updated? If you *knew* it was updated you can store timestamp, but how do you know when the chunk is updated in the first place? I think you're quite close - rather than a timestamp, you can consider using a hash of the chunk (e.g. MD5), then use that to figure out if the chunk has change or not.

[@beatsofbinary]

@@pixegami I thought I could save metadata such as Last Modified directly when importing data. If the Vector database already contains a PDF document that has the same name but is older, it will be overwritten. If it is the same age, the PDF is skipped. I would therefore query the database to see which documents are already in the database. But then all chunks of the older PDF would have to be deleted, you're right. The idea with the hash is good, read in the PDF, split it, create a hash, save it as metadata and then compare the hashes of the chunks. But that leads to further problems, doesn't it? If, for example, only a few words change within a chunk, but the overlap changes as a result, will several or all chunks change at the same time?

[@siswanto4045]

Wow, this is what I wanted. But can it be running on open webUI? Like running ollama locally with webUI? Thank you

[@pixegami]

Glad to hear that! I do plan to do a tutorial later on how to build a a web UI for your app, so stay tuned.

[@siswanto4045]

@@pixegami can't wait to watch the video

[@JoseLuisCornejoRivas]

Now how to choose the correct document and not mix contexts from different documents? Thinking about a large documentary database

[@pixegami]

Good question. I haven't looked into this myself, but I suspect you'd need to upgrade how you store/query data, maybe using something like a knowledge graph instead (e.g. neo4j): python.langchain.com/v0.1/docs/use_cases/graph/

[@mo3x]

So it is just an advanced ctrl+f ?

[@pixegami]

Yes, that's one way to think about it. Still, incredibly powerful.

[@maxi-g]

Hey, I have a question. I tried to load a fairly large PDF (100 pages) into the database (approx. 400 documents). However the add_to_chroma function seems to be excruciatingly slow. The output from ollama shows that the embeddings only get requested once every two seconds or so. There is also no CPU or GPU load on my system when this process is running. Is there any way to improve this? Thank's already

[@pixegami]

This is most definitely because of the time it takes to embed each page (since you mentioned embeddings get requested once every two seconds). Your Ollama model might not be able to fully leverage your hardware, which is potentially why your don't see your CPU/GPU load rise up. You could experiment by switching this to use an online embedding API (like OpenAI or AWS Bedrock) and see if it's faster. Or you could double check to see if Ollama is using your GPU correctly (github.com/ollama/ollama/blob/main/docs/gpu.md)

[@nirmalkumar007]

which part of the code makes the API call to the OLAMA server ? Kindly help

[@pixegami]

The Langchain Ollama wrapper class (e.g. python.langchain.com/docs/integrations/text_embedding/ollama/ for the embedding) wraps all the code to call Ollama for you.

[@Ammarsays]

I am just a layman and I want to know if text splitters count the characters, words or sentences for a given chunk size? And if the text splitters can identify sentences or paragraphs in text?

[@pixegami]

Yup, the splitter should attempt to do that. Here's the documentation: python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/ "It is parameterized by a list of characters. It tries to split on them in order until the chunks are small enough. The default list is [" ", " ", " ", ""]."

[@DCW09]

Cursory glance says - add a hashing function to the chunk metadata, this way the chunk should have a unique identifier (MD5, SHA, Etc) if anything changes the hash will also change. Then its just simple logic to validate current chunk page.index against an existing one's hash. If its different, overwrite. If its not, dont waste the cycles. In practice, I am not 100% sure that this would be the approach but at least the theory here should be pretty on point for identifying changes with few compute cycles.

[@pixegami]

Yup! I think that's probably the way I'd do it too. If there's too documents and you need to scale it, then I guess you can hash the entire document as well first to narrow the search space each time.

[@MichaelTanOfficialChannel]

@@pixegami I just want to add that I would apply the hash to the entire page and not the chunk. A page can be edited in a way where the content is shorter than previous version, thereby causing the number of chunks to be less that what it previously was. And I would also remove all chunks belonging to the said page before adding new chunks, so as not to have an orphaned chunk from the previous lengthier page.

[@edwardtse8631]

This is a very good tutorial, how do you solve the problem of edit a data file? to store sha-1 of the file?

[@pixegami]

Exactly right - you can use an even simpler hash function like MD5 just to check that the content of each chunk hasn't changed. You'll still have to loop through all the chunks though to calculate the hash and to compare them. That should be fine for 100s or 1000s of chunks, but it might not scale too well beyond that.

[@EduardoJGaido]

Hello! Thank you for a great video. I ask you or the community, I have a hard problem to solve: i want to make a chatbot using a local LLM with RAG (keep reading please!) BUT i want to use it for my business so the clients of my physiotherapy clinic can ask it and it responds JUST WITH the information that is fed. Otherwise, it says "Oh, i don't know, wait please" so the secretary can answer instead. Just with that, I would be happy. I have a lot of FAQ listed with the answers (in a json friendly format). I can't find this answer anywhere. If you have any information, would be apreciated. Cheers from Argentina.

[@pixegami]

I see, if it's an FAQ I'd structure each question or piece of information as a separate file, and just use a directory loader to turn it into Langchain documents. Then you can try using the technique in this video to see if works. To get it to respect the answer boundary, you can probably write an explicit prompt or an evaluation step to say "I don't know" if the answer isn't clear from the queries.

[@MaliciousCode-gw5tq]

Question will RAG method works even if the data are too big? Example 1k pages each file?

[@TrevorDBEYDAG]

Thank you for the tutorial, I guess it should be a better way to create chunks, not only by character count because it cuts the paragraph in disruptive way. May be another library to split in paragraphs or at least end of the sentence?

[@pixegami]

The Recursive Text Splitter actually attempts to do what you suggest: python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/recursive_text_splitter/ "It tries to split on them in order until the chunks are small enough. The default list is [" ", " ", " ", ""]."

[@TrevorDBEYDAG]

@@pixegami That's cool.

[@gianmarco-lr7wc]

hank you! It would be great to see how to deploy this to the cloud, for examplw aws!

[@pixegami]

That's a great idea!

[@phizicks]

md5 of the data to the index, if it matches, no update needed. unless I didn't understand the requirement

[@pixegami]

Yup, I think that's probably how you'd go about indexing/updating specific chunks. You might also need a tree structure (e.g. hash the entire document or categorize them first) if you want to work in the scale of 10k+ documents.

[@MrAtomUniverse]

How do you display your code that way for video , would love to do that for my work

[@pixegami]

I actually use a lot of custom tooling to generate the slides, but you can use this to do something very similar: carbon.now.sh/

[@vidfan1967]

I want to verify an existing PowerPoint document against a PDF, e.g. with an updated law text. I want to find out, whether my statements on each PPT slide are still true. Challenge: each slide contains one or more statements, which should be verified against the PDF, for example you have 6 or 10 bullet points on one slide. To use RAG I cannot use them all for the query as they might be quite diverse and would knowledge from the PDF, that is not specifically matching any one point but all of them together. Also: the context on the slide should be considered together with each statement, e.g. the title of the slide, an intro text above the bullet point list, or the upper level information for a statement that sits in a sub-structure of bullet points. I guess I would somehow need to split the statements in the PPT in logical chunks, preserving the context. Is there a python function I could use? Or can this be done with AI (e.g. few shot) after the slide text has been extracted? If this is of wider interest, I would appreciate a video on this 🙂

[@pixegami]

Thanks for sharing your use case. Yup, this is definitely something you need to solve during the "chunking" phase of the process. For example, there are some experimental "chunking" functions from Langchain you could try: python.langchain.com/docs/modules/data_connection/document_transformers/semantic-chunker/ Also, you could "bake" in custom information to each chunked document itself. E.g. something like (each variable name is made up): chunk.metadata["page_content"] = title_of_doc_str + context_str + actual_page_text_str

[@gowthamkrishna6283]

can we do hashing on the page content to only update the modified? any one can share any link to do the same, thanks!

[@pixegami]

Yup, that's the strategy I'd probably use to detect sections of pages being updated too. I don't have a video on it yet, but it's a great idea I'll add to my list of potential tutorials!

[@johnharryduavis3414]

Im wondering if you can deploy it in huggingface so I can use it for my mobile app?

[@pixegami]

I'm not sure, I haven't used HuggingFace much self. But for sure I know it's definitely possible to deploy it as an API using some of the other cloud platforms (AWS, Azure, etc).

[@rude_people_die_young]

Great work 🎉❤

[@pixegami]

Thank you!

[@AstigsiPhilip]

Hi, this python rag can handle 70,000 pdf index files? thank you for your response.

[@devmit2071]

How do you do this NOT running it locally? i.e. using the AWS cloud for pretty much everything (PDF in vector database, Langchain, Bedrock etc...)

[@pixegami]

You'd have to change all the LLM functions to be cloud based (e.g. AWS Bedrock or OpenAI), wrap the app in an API (like FastAPI) and Docker, and deploy it to the cloud (probably as a Lambda function). I'm working on a video about that now, so stay tuned :)

[@devmit2071]

@@pixegami Thanks. I'll drop you an email with some ideas

[@rahulsharmaah]

can we get an api of this and apply in our application

[@pixegami]

Yup! That's going to be the plan for my next video (hosting a RAG app in the cloud)

[@P4jMepR]

Would this solution be suitable for evaluating the best candidates from a 100 resumes for example? How would You approach such a project?

[@pixegami]

I can't really say if it's a good solution for your use case. I'm sure some companies use LLMs as an initial filter for things like candidates/resumes. But for such a high-judgement, important decision, it's best to really validate it against human output.

[@beatsofbinary]

How to add a memory? For example if I want to ask follow up questions (contextual)?

[@pixegami]

Good question. I haven't explored this myself yet, but many LLM libraries (including Langchain) seem to have a concept of a "memory" module: python.langchain.com/v0.1/docs/modules/memory/ That'd probably be where I'd start.

[@beatsofbinary]

@@pixegami Thanks! I've seen another tutorial about that. The guy who created the script collected the chat history in a python list and added it to the prompt. Basically the LLM was given the questions and answers and was asked to reformulate that to a history. That was added then to the prompt something like: "Please answer the human's question based on this chat History"

[@pixegami]

@@beatsofbinary Ah yup, even if you use a memory module, at the end of the day all LLMs are stateless so you pretty much always have to basically just add it back into the prompt :)

[@williamroales709]

awesome! thanks ! how can i use a GPU to make local faster?

[@pixegami]

I think it's probably a setting in Ollama. Have a look at: github.com/ollama/ollama/blob/main/docs/gpu.md

[@mohsenghafari7652]

Hi dear friend . Thank you for your efforts . How to use this tutorial in PDFs at other language (for example Persian ) What will the subject ? I made many efforts and tested different models, but the results in asking questions about pdfs are not good and accurate! Thank you for the explanation

[@mohsenghafari7652]

help please🙏🙏

[@pixegami]

Answered then in a separate comment, but I think it just boils down to finding a model that works well with the language. E.g. : huggingface.co/MaralGPT/Maral-7B-alpha-1

[@mohsenghafari7652]

thanks dear 🌹

[@nvajay3829]

Brother, this is working well for PDFs, but PDFs with tables and json files it's not able to do even if i use json loader and modify the code. Can you make a video for it ? Local RAG for large json files

[@pixegami]

Ahh yes, any weird formatting inside a PDF will be challenging. I'd probably try to approach it by seeing if I can first parse it into Markdown or HTML, because I think data in those formats are a little easier to work with. If you have any examples of PDFs you'd like to parse, please feel free to share them here and I'll see if it gets interest for me to cover in a future video. Thank you!

[@rayen2313]

i have a reall problem trying to run any free LLM ( local one are too long to handle it takes about 8 mins and more) can u help me ?

[@pixegami]

Unfortunately LLMs are expensive to run, so not a lot of places will offer you a good one for free. If you can’t run a good one locally, then do you at least have a budget for an online one? They aren’t free but should still be cheap enough for learning and development.

[@rayen2313]

@@pixegami The important part of my project is that LLM should be free . But can you provide us with a vid using llama3 , it can be run not localy and also free using for example crew ai

[@ChauNguyen-tt9pz]

Hi. Thank you for your great videos here. I have learned a lot from it. I'm currently trying to run the code locally but Im running into a server problem. My error is ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it. Could you help me debug this error?

[@pixegami]

Hmm, which command/script are you running to see that error? From what little information I have, I'd say it sounds like an issue to connect to the local Ollama server. Have you started up the Ollama server successfully on your machine?

[@ChauNguyen-tt9pz]

@@pixegami Hi. Thank you for your reply. I didnt set up the Ollama server before so that's why it was giving that error.

[@ChauNguyen-tt9pz]

@@pixegami However, I'm getting this error when Im using the BedRockEmbedding function: ValueError: Error raised by inference endpoint: An error occurred (UnrecognizedClientException) when calling the InvokeModel operation: The security token included in the request is invalid. Do I need to set up some kind of AWS local server as well?

[@roxtonjhon]

How to retain context information in multi-turn conversations

[@nhtdmr]

Could you make video same project with LLAMA 3?

[@pixegami]

You can do it here! Just change the Ollama model to Llama 3: ollama.com/blog/llama3

[@ai26prasad.p25]

Hello sir , can you make one video how to create a dataset to train to a LLM model . I am getting too much data loss while I am trying to train the my data to the model , so please make a video about who to create a data for LLM and How to train it ?

[@pixegami]

Thanks for the suggestion. Fine-tuning an LLM isn't on my radar yet, but I'll add it into my list since a couple of people have already asked.

[@pm1234]

I successfully run it locally with ollama embeddings, but it pretends its answer is from the PDF while I'm 100% sure it's from elsewhere. Let me explain: My monopoly PDF rules are in French, my LLM is in French (vigostral), my question is in French (translation of the test question: How much total money does a player start with in Monopoly?), but the reply is stating $1500, an amount and a currency that are NOT in the French PDF book of rules (150'000Frs, very old rules), and it cites the sources: ['data/1d-monopoly-regle.pdf:5:1', 'data/1d-monopoly-regle.pdf:3:4' ... Asking the same question in French directly to ollama (run), without the PDF, states the actual amount (2000€). So, it makes me wonder if it really works all the time without bias (model info vs doc info), why does it cite sources while giving an answer that is not related to the sources, and how to identify what is not working in this case.

[@Yakibackk]

Maybe its related to embedding ? Which one are you using?

[@thaslim7869]

Is this work offline?

[@pm1234]

@@Yakibackk Ollama offers (only) 3 embeddings, I tested nomic and mxbai.

[@pm1234]

@@thaslim7869 ollama works locally.

[@juryel3561]

You can explicitly tell it not to use information outside of the provided documents through its instructions since its RAG you can do that.

[@Nabeel27]

How to deploy to the cloud so you can share the application?

[@pixegami]

I'm actually working on that idea for my next video, so stay tuned!

[@bachirafik8040]

Hi sir I got a aws error

[@pixegami]

What error did you get? Did you set up AWS CLI and enable the embedding models in your AWS account as well?

[@prateemnaskar1656]

I want to execute the instructions in my PDF directly, How can I do it?

[@pixegami]

You'd probably have to structure the app more like an agent, so it can look for information, but also take actions and execute on it: python.langchain.com/v0.1/docs/modules/agents/

[@deeplearner-hinglish]

why not u make a special UI by using Python's Flask?

[@user-sx3mm5ii5u]

does it deals with images inside the pdfs?

[@pixegami]

No, I don't think. You probably need some additional logic to parse/describe the images into text first (maybe using a multi-modal model to do that).

[@user-sx3mm5ii5u]

@@pixegami o you have any idea? some pdfs have text based content..some pdfs may have scanned medical reports etc..

[@prateeksaxena7808]

Hi, pls let us know how to deploy this?

[@pixegami]

This will be the topic of my next video, so stay tuned!

[@gonzalocueva6631]

hardware requirements?

[@pixegami]

This will probably depend on the model. For example: ollama.com/library/llama2 7b models generally require at least 8GB of RAM 13b models generally require at least 16GB of RAM 70b models generally require at least 64GB of RAM

[@Yakibackk]

We want llama3

[@mohsenghafari7652]

we can replace model with lama3

[@pixegami]

Sure! Just update your Ollama server to download and run Llama3: ollama.com/library/llama3

[@jack_s]

Is python can connect with llama 3?

[@Nuggiesoftruth]

@@pixegamican you make a server with access to a NAS system local o llama pulling info from it?