Your NVIDIA GPU can power a free ChatGPT (Chat with RTX)

Chat with RTX is a ChatGPT-style application that uses Retrieval Augmented Generation (RAG) to allow you to chat with your own data on your own local NVIDIA GPU/PC.
Links mentioned:
Download Chat with RTX - www.nvidia.com/en-au/ai-on-rt...
NVIDIA RAG blog post - developer.nvidia.com/blog/rag...
My RAG tutorial (code from scratch) - • Local Retrieval Augmen...
Sign up to NVIDIA GTC 2024 - nvda.ws/3GUZygQ
PDF links:
Attention is all you need paper - arxiv.org/abs/1706.03762
Embeddings paper - vickiboykis.com/what_are_embe...
Math for ML book - mml-book.github.io/book/mml-b...
Charlie Walks (my novel) - www.charliewalks.com
Nutrition Textbook - pressbooks.oer.hawaii.edu/hum...
Other links:
Download Nutrify (my startup) - nutrify.app
Learn AI/ML (beginner-friendly course) - dbourke.link/ZTMMLcourse
Learn TensorFlow - dbourke.link/ZTMTFcourse
Learn PyTorch - dbourke.link/ZTMPyTorch
AI/ML courses/books I recommend - www.mrdbourke.com/ml-resources/
Read my novel Charlie Walks - www.charliewalks.com
Connect elsewhere:
Web - dbourke.link/web
X/Twitter - / mrdbourke
Twitch - / mrdbourke
ArXiv channel (past streams) - dbourke.link/archive-channel
Get email updates on my work - dbourke.link/newsletter
Timestamps:
0:00 - Intro
0:17 - Chat with RTX requirements
054 - What is Chat with RTX?
1:51 - Downloading Chat with RTX
2:06 - Installing Chat with RTX
2:09 - Make sure to upgrade your NVIDIA Graphics Driver
2:41 - Chat with RTX setup steps
3:04 - What are INT4 models?
3:25 - Opening Chat with RTX
3:48 - Chat with RTX running in the browser
4:33 - Allowing Chat with RTX to work with your own custom data
5:11 - Turning your own custom data into embeddings
6:43 - Question 1: What are embeddings?
7:18 - Question 2: What is the attention mechanism?
7:51 - Question 3: What is matrix multiplication?
8:31 - Question 4: What is Charlie's secret project? (asking a novel)
9:06 - Question 5: What are the macronutrients? And their functions?
10:07 - Outro/NVIDIA GTC
#AI #NVIDIA #GTC24

[@rahmatullahbarkat1002]

very detailed use case. thankyou

[@astrocoastalprocessor]

🎉 Thank you! This is just the kind of video I've been hoping to find

[@mrdbourke]

Good timing!

[@astrocoastalprocessor]

@@mrdbourke It worked flawlessly

[@gorkemocmenn]

dude.great job

[@ibriarmiroi]

Thank you very much as always. Though out of the context of tis video, i was wondering what would be the replacement for Tensorflow examination Certificate since it would be discontinues on 31st of may this year ? and also , would be possible to make video about the available Certificates online ? since your expertise opinion on each Certificate would be Gold ! thank you

[@amitkumarchejara6042]

This is awesome, but I don't own a GPU yet, still learning!!

[@MK-ce7im]

best youtuber out there

Do you know whether is it possible to have something similar to DALL-E with this tool?

[@FernandoLXIX]

Don't load a huge dataset. I uploaded every college digital file I had and the thing went crazy. After a few hours of it trying to "digest" the data I killed the program and had to reinstall because there doesn't seem to be a way of telling it to stop reading your data. I'll try again with a way samaller dataset as you did in the video.

[@gordonthomson7533]

What GPU are you running? I’m awaiting 4080 super and hoping to squeeze a load of value via this…

[@FernandoLXIX]

Don't. After some testing I can tell you it's quite useless. The answers it provides are very bad and short. It doesn't care that much about the data you provide it with. Bare in mind it's a "demo". Maybe in a few months or years it becomes decent. For now, just get chatGPT 4 instead@@gordonthomson7533

[@fabiosoave6622]

Thank you for the very comprehensive tutorial! I'm interested in creating an interactive chat BOT to respond to Facebook messages related to medical topics. What I'm asking is, given the issues some have reported regarding the management of very large datasets of scientific articles, how can we obtain a system that is both comprehensive and effective? How can we organize the material questions correctly to make the most of the program's capabilities? Thank you.

[@erentas7391]

you are back at last.

[@mrdbourke]

yo yo yo!

[@MK-ce7im]

this used to work fine but after changing my folder path for the docs, I can no longer open it. I wish I can change it somehow without opening it

[@dexterroy]

My GPU is an older RTX 2070. After downloading this nvidia package, I realised it's not going to run on my GPU. I created a similar chat bot that can do Q&A using a pdf file of your choice. I'm using llama-2-7B there. I had to use Google Colab to get 16GB GPU. But man, it's so slowww.. @Daniel Can you please do a video on pdf chat using langchain and one of those free LLM models?

[@justusbatekerd]

Did you have any errors when you installed the software? I have had numerous errors with the sentence-transformers not being installed. Can anyone help with this?

[@brendendas]

Did you find a fix yet? I had the same problem which was addressed on a forum and the solution solved the issue for me.

[@kumarvuppala5286]

what is best hardware required ?

[@animetransition1030]

Yeah make an app like this

[@doctorfunhome7842]

Quadro T600 can run it ?

[@AnshumanKumar007]

Will this on my RTX 3060 ? I have a laptop. Wish I bought the RTX 3090 back then, I could have afforded it. Still can but now I am hearing of layoffs in my org, so I am gonna have to save up in case I get axed. EDIT: It doesn't seem it can. I wish my 3060 had like 16 gbs of VRAM :(((((

[@gordonthomson7533]

That’s odd, minimum is 8gb

[@bigdhav]

I have a 4070, but i cant get it to work due to some strange dependency.

[@mrdbourke]

Which one is that? Have you installed NVIDIA drivers?

[@ybaran]

😮

[@mira8950]

can it run on rtx2050 ?

[@pabloarvilla8012]

no

[@mrdbourke]

You could try, however, the system requirements do say to use a 30XX or 40XX series GPU