Nvidia's Free RAG Chatbot supports documents and youtube videos (Zero Coding - Chat With RTX)

[@darkmatter8508]

For the model "Mistral 7B int4" Something interesting I noticed was what the model was or was not allowed to say. For example: I "trained" it my own dataset that was a single text file that simply said "cat" a hundred times. I asked the model what a dog was. It did not know. I asked the model what a horse was: It did not know. I asked the model what a cat was: and it gave an in detail explanation of what a cat was. My conclusion is this. When the model trains off of our training data, it seems to supplant gaps of information with its default AI model, despite not being explicit that it is doing so. I want to test the capabilities and advantages / disadvantages in the coming days, and share my results.

[@airoundtable]

That is an interesting experiment @darkmatter8508. That phenomenon is known as "Catastrophic Forgetting" and it is a common behavior in LLMs. Fine tuning has two key aspects: 1. The amount and quality of the data that you pass to the model 2. Number of epochs and learning rate These two can affect the model's performance drastically. I have a video on fine-tuning where I dive deeper in these two aspects. But, I am curious to know the results of your further experiments if you share it with us!

[@hadi-yeg]

Very interesting video thank you Farzad! Looking forward to your works.

[@airoundtable]

Thanks Hadi!

[@godfreyogbeide2340]

Your tutorials are amazing! They've been incredibly helpful. I have a request: could you create a tutorial on building a chatbot using Node.js, React, and Next.js that can upload data to a Vector database and interact with it? I believe a tutorial on this topic would be incredibly valuable and interesting.

[@airoundtable]

Thank you very much! I am glad to hear you liked the videos! Sure, I will keep it in mind. I am now working on a few interesting projects that I am going to upload on KZbin soon, I will add the RAG chatbot using Node.js and React to the list as well.

[@navanshukhare]

Interesting and informative video, Considering the plethora of LLMs and tools popping up, it's a matter of saturation in architecture and speed. Interestingly all the RAG support PDF files, but not one supports Databases. What is your view?

[@airoundtable]

True. Current RAG techniques struggles with semi-structured data (tables, etc.) while they are performing very well on plain text. I believe as we move on, more and more data types will be covered by RAG systems including databases. However, I don't see databases to be the priority right now since at least we have varioes ways to query databases while pdfs and docx files are being piled in companies without a proper way of having access to their contents. My prediction fo future: 1. LLMs with much large context length which will significantly affect RAG systems 2. RAG techniques that can support all sort of data including semi-structured documents 3. Multimodal RAG

[@jxm8944]

Thanks for sharing! When I downloaded and installed, there was no Llama2 13B INT4 to choose from? There were only Chat With RTX 0.2, Mistral 7B INT4 1.0. Because my graphics card is NVIDIA GeForce GTX 4060 8g, is it possible that the video memory is too small? Thanks.

[@airoundtable]

You're welcome! If during the installation it didn't even give you the option for selecting the 13B model it might be due to your GPU VRAM yes. but you can check your GPU VRAM. 4060 has 8 GB and 16 GB versions as far as I remember. Double check your GPU VRAM www.makeuseof.com/how-check-how-much-vram-you-have/

[@jxm8944]

My NVIDIA GeForce GTX 4060 just has 8 GB ! Thanks. @@airoundtable

[@omicron8530]

Can we learn C++ or other programming languages on Chat with RTX, is it worth it? Like better than chat gpt 4 in learning programming languages , what do you say?

[@airoundtable]

GPT4 is a beast compare to this chatbot. This chatbot is just a software that NVIDIA released to get everyone access to a free RAG chatbot. But when it comes to real chatbot performances even GPT3.5 (ChatGPT) is much more powerful than this one. So, for learning coding GPT4 and even ChatGPT are much more superior than this chatbot.

[@omicron8530]

Make a video on comparison and which is better chat with RTX vs RAG?

[@airoundtable]

Hi @omicron8530. Thanks for the suggestion! If you mean Chat With RTX vs a customized RAG chatbot, I can compare it right here for you: Chat RTX pros: 1. Beautiful user interface (personal opinion) 2. Easy to install 3. Fasst inference time 4. Multiple functionalities (I love the fact that it also supports chatting with youtube videos) Chat With RTX cons: 1. Limited with the choices of LLMs which leads to the second on below 2. It is not as powerful and as accurate as RAG systems designed for example with GPT models (and for industrial use case, most probably a powerful chatbot is needed) 3. Does not have memory 4. Cannot be served to multiple users by hosting it on a single GPU (it is only for personal use) 5. Due to the size of its LLMs (7B and 13B), hallucination is a serious problem. Overall, it is a great personal tool to have if someone needs a quick Q&A with documents. But if you are looking for accurate results and a chatbot that can serve multiple users for example, designing a customized RAG chatbot would be the way to go. I hope this helps!

[@omicron8530]

I have rtx 3060 12288 mb (12gb), so can i run 13B one?

[@airoundtable]

As far as I realized based on the comments, NVIDIA automatically analyze your system and suggests the models based on the system specifications. So, in case you see the 13b parameter model during the installation process, there is a chance that you can. But theoretically, I think 12 GB is not enough for a 13B parameter model. It needs at least 13GB of VRAM based on my experience. (In the video I showed that it is using 12.1 GB of VRAM on my PC.) So, give it a try and please share the results with us too.

[@Araphex]

Llama isn't included with the download. Only Mistral.

[@airoundtable]

My guess is when you want to download and install the software it automatically detects your GPU and modify the installation process based on available GPU VRAM. Would you please let me know what GPU you are using and how much GPU VRAM you have?

[@Araphex]

@@airoundtable 4090 24 GB, i9 CPU, 64 GB RAM.

[@adriangpuiu]

i dont lnow, i installed it but i cant pass any youtube video URL and also when it;s displaying the reffrence doc dosent make it linkable

[@airoundtable]

For the KZbin URL I had the same problem in the beginning. It does not work if you copy and paste the URL from your browser. Below the KZbin video click share and copy the link from there and use that one in the Chatbot. For the reference part, I am not sure. I never faced that issue. Do you have a default PDF loader on your PC?

[@adriangpuiu]

@@airoundtable no, i ment, the youtube url option is not even visible , i only have the doc folder one and thats it. in regards t pdf loader i am not sure what you mean

[@airoundtable]

@@adriangpuiu hmmm. That is weird.. I am not sure why KZbin option is not even available for you. With regard to the PDF links, my blind guess was that maybe the chatbot is getting confused on how to open it but now that I am thinking that doesn't make sense actually.. weird.. Not sure why it is happening. I installed this chatbot on 3 different PCs and never faced these problems. Please let me know in case you found the solution

[@adriangpuiu]

@@airoundtable ill build my own rag in the end , already tried 3 times on the main pc. question .what do you use to open up them pdfs ?

[@airoundtable]

@@adriangpuiu well designing your own RAG project would be the best and most accurate strategy. I use Pypdf library but mainly through Langchain. It is the library that langchain uses in its backend. It works well on PDFs. I explained all the steps that I found working well in practice in RAG-GPT video in case you would want to check it out in more detail.

[@nathan_sweet]

Why do you pronounce your 'Th' sounds as 'D' sounds? Example: The word is 'this' not 'dis'. Also, much of your information is incomplete, and expects a certain knowledge level of all of your viewers. Example: You said "this chatboard is only for users with access to Series 30 or 40"- 30 or 40 series of what? Cats? Cars? Nvidia GPUs?

[@airoundtable]

Thanks for your feedback, @nathan_sweet. English isn't my first language, so I'm always working on getting better. About the series, I meant NVIDIA GPUs like series 30 and 40. In the video, I show a table with all the specs needed for this chatbot.

[@wesmaldonadogama]

Come on Nathan!😅 what about just say thank you?