How to install and first impressions of NVIDIA Chat with RTX AI Chatbot

NVIDIA Chat with RTX is like ChatGPT except that it uses your local computer and video card, an NVIDIA RTX video card, instead of OpenAI cloud servers. It allows you to use your own files with it to perform chat queries on them.
I am attempting to use Chat with RTX to analyze a folder of score sheets in pdf format from my kids hockey games to get some stats such as total goals scored in a season, top goal scorer, most penalties etc.
0:23 What my goal is.
1:13 Beginning Installation of NVIDIA Chat with RTX
2:10 Installation Problems
2:28 Installation issue solved
3:09 First launch of the AI Chatbot
3:49 First question
4:03 Basic math questions
4:15 Basic American history
4:27 Changing the AI model
5:20 Loading my data into the AI
7:05 Refreshing the dataset
7:20 First sign of trouble
9:25 There is hope
11:00 Initial conclusions
11:20 Even more hope!
11:55 This AI chatbot is asking ME to do the work LOL!
12:30 So close but so far..
13:15 Plan going forward

[@sticks7857]

Do you know if these models can be updated? Or do they really expect us to download their 40gb of datasets with slight tweaks for every update they do?

[@leksicon]

That's a great question and I'm looking forward to their next update to see how they do it. Personally I would like to see if its possible to insert an even larger data set into this software.

[@sticks7857]

@@leksicon I can totally see myself getting carried away with downloading more of them to increase the complexity of my own personal AI to the point where I've turned my spare room into a server room to house it all and have Dr. Know from the movie A.I. When it comes to storage though does the write speed of the drives matter at all? Like would it perform better on an SSD compared to an HDD? Probably a silly question but I just got this m2 drive and I'd rather not have these huge datasets hogging unnecessary space if I could just move it over to a cheap external one.

[@user-is5om2fe2l]

You can put many open source llm ,你可以自己编译符合格式的开源模型放置到Chat With RTX

[@leksicon]

@user-is5om2fe2L I will explore this and try it out, thx

[@leksicon]

@sticks7857 I run it on a rather average ssd but yeah expect better performance with a higher spec drive

[@IndieAuthorX]

I failed to install on the first few tries, but I think I lacked hard drive space. After a few tries and retries it finally just worked. My early queries are very much lacking in quality. I would like to know more about how I can train the chatbot with data on my PC because the potential is really interesting. I have nearly a million words in personal writing to sort through and use as training material. Also, I would like to query the AI regarding manuals and PDFs I upload so I can quickly answer questions on various things I am learning without paging through the manuals ad naseum. Eventually would like to query a story bible for a RPG game I would be running or a book I am writing. Exciting, enticing, tons of potential, but it is still out of reach.

[@samcheng8751]

Can it run with a Windows virtual machine in AWS or Azure?

[@mokgatlha]

Of interest is to see or know the limitations on the number of documents it can have access to, as well as the typical size of such documents. As in this video, the tendency to limit to about 4, and yet it seems to be able to reference others other than the 4, makes one wonder if it is the typical setup of the models or just a bug of some sort

[@c6jones720]

The other day I gave chat RTX a youtube video transcript about the apple vision pro and asked for a TLDR about it. It returned that the Apple vision pro was an electric bike. - I think this LLM needs a little work, but its free at least.

[@leksicon]

lol

[@TheAwesomeOdyssey]

where did you get your wallpaper for the desktop?

[@leksicon]

it's a 48:9 aspect ratio wallpaper, I can't find it for the life of me rn though

[@ershadzolfi1359]

after installing v 0.2, can I update it to some upper version in itself? or is there any major update?

[@leksicon]

I believe the first public release is version 0.2 and that is still the current version.

[@TheExceptionalState]

Thanks for doing this video. I would love to have the functionality RTX promises. You saved me from wasting my time with this, atleast for the time being. Strange conclusion: It is cool. Based on your own description my intelligence would call clunky and unreliable as a source of information and analysis. I do hope it improves.

[@khoahocgia7671]

It would be very interesting if you use it to retrieve reference data at a low level; if you want accuracy, you have to refine the data file from PDF to txt format and manually adjust it. Secondly, the level of answer output also depends on the original language model, especially those language models that have added strict filters and censorship, which are highly valued in terms of community security but inadvertently hinder our data extraction, thus prompting creators to develop their own uncensored language models. My point is that providing accurate data is still better through self-training, I know it takes time, but there is no other way.

[@densan99]

After I specified the new path pointing to my PDFs, I had to wait for the chat to digest the data. I don't have the fastest computer, and PDFs are long. The cmd windows you showed at the beginning list the progress.

[@gabrielnilo6101]

depending how the PDF is build, could be messy... would rather copy everything to .txt and use it.

[@sersomeone9704]

​That or just OCR the pdf

[@daviddegaye164]

The problem might be your files format: pdf. Other LLMs had trouble too, so I'd ask them to specifically use an OCR for the task. I don't know if the same is possible with ChatRTX yet.

[@lightplayz8020]

the zip file i got downloaded (twice now) from the official is missing the setup program why is that?

[@leksicon]

perhaps your anti virus software blocked the .exe? try another extraction method and temp disable virus scan might work

[@lightplayz8020]

@@leksicon thx, i will

[@lightplayz8020]

@@leksicon it didnt work, it still downloads incomplete

[@user-is5om2fe2l]

thanks

[@user-su3fk3ju5i]

I want to set up fixed port access and be able to access the web from other computers.

[@leksicon]

sounds interesting did you make any progress?

[@user-su3fk3ju5i]

@@leksicon No progress at present,And the ai conversation experience is not so good.

[@christopheralvarez1090]

Do I delete the extracted file or the extracted file after install ?

[@leksicon]

yes you can delete what you downloaded after installing

[@leeishere7448]

I only have the 7b but not the 13b lama. Why is that?

[@sticks7857]

It failed to install on the first few installations for me until I cleared up more space, I think it has to decompress very large files that doubles the amount of space it will inevitably take up until it deletes the compressed version afterwards, so if you don't have space free it won't finish.

[@user-is5om2fe2l]

这个我们遇到过是显卡，现存不够，4070Ti 12G 和你问题一样，4090 24G 就可以安装llama2 13b

[@leksicon]

yes make sure you have ~200gb free before running the installer

[@SaadNFlimz]

@@sticks7857what GPU are u using

[@SaadNFlimz]

Same issue. I even have copied llama2 still not in the list

[@gabrielnilo6101]

I don't actually use .pdf files because they are messy and it's hard to maintain... I would convert it to .doc or .txt Now let me talk about this APP... It's a DEMO and it's based on a open source repository in github, I bet nvidia only changed the layout and optimized the AI for RTX GPUs. I would never use it as it is... there are many problems with it and many memory leaks...

[@skidmarks4360]

Man I wish I got to the part of your video when you said that it was 60gb and not 100gb before I uninstalled Call of Duty to make room.

[@paulsalazar1345]

I just installed it after many many tries, but when the chat promps up in my browser I cant see the KZbin URL option, and I dont know why this AI is like a gaming NPC lol it just keep replying stuff about games when I am asking simple questions

[@RichCinema]

Problem is when dataset is local folder the assistant will not access the internet and when in model mode will not access your data as there is no vector embedding like AnythingLLM. I created a symlink file and replaced the dataset folder in the root directory. Will see how it flows.

[@dxgbonbon3545]

Update please

[@DhiaOuerghemmi]

I tried to install chat with RTX but the problem is a part of the setup will install (last time 15G instead of 35 G)and didn't work at all on my pc

[@leksicon]

do you have a nvidia rtx card? you need about 120gb free space on the drive with your windows install

[@DhiaOuerghemmi]

@@leksicon actually i dont have now and I have more than 120gb free space

[@mikufx3787]

After I downloaded the file I try to extract the zip file but I was unable to do it, as it was showing that it is a corrupted file so please help me out

[@mikufx3787]

I tried it 3,4 times but was unable

[@leksicon]

make sure you have a lot of space free when installing like 200gb

[@Jascensionvoid]

I keep getting this odd error when it's trying to scan my dataset - [02/23/2024-19:42:28] could not convert string to float: '100.-146' : FloatObject (b'100.-146') invalid; use 0.0 instead[02/23/2024-19:42:28] could not convert string to float: '101.-13' : FloatObject (b'101.-13') invalid;

[@leksicon]

🤔

[@Jascensionvoid]

Ikr! It happens in the terminal window right before any embedding takes place. So if I upload say, 6 PDFs I may get this error 1000 times and then it may only "index" 3 of them. So I think it has something to do in regards to it having trouble reading or scanning certain PDFs? No clue here either@onI also had it not seeing all the pdf I uploaded

[@SquintyGears]

honestly for what you're trying to do, no LLM would work. it's just not what they do, you would be more successful asking an LLM with code good responses to make a python scripts that parses and organizes the data the way you want it. it's all there in the pdf files but an LLM isn't good at going through all of them methodically to do a sum or any kind of math on the whole compilation. it would be very suitable at searching and citing all those documents for a datapoint of information. when you ask these kinds of questions from the online LLMs they tie themselves onto a scripting environment they launch and put their code suggestion in to run it. the LLM part and the file reader part aren't doing that. granted the scripting is not the difficult part of the problem so with a bit of time and some helpful documentation I am convinced people will have bolted it onto "chat with RTX" in no time and enhanced the overall usefulness. the really neat part I think is that the citation is very easily checkable to verify the answers so it can be a massive time saver.

[@leksicon]

hey thank you great answer and feedback.. I will go down this path you are suggesting to create python scripts tailored to parsing these files and sorting the data into what I’m looking to get out of it. thanks again

[@Yozza121]

I get an error saying 'Chat with RTX requires minimum of 8 GB of GPU memory to function' I have 16GB of VRAM so not sure why it fails

[@akindeabdulai1109]

i experience same on laptop

[@monkshee]

how do you havbe the llama option?

[@user-is5om2fe2l]

显存大于12G, 比如4090 24G，you'll find llama installed by default

[@monkshee]

@@user-is5om2fe2l ah i see

[@altayagalarzade]

bro what is song name?

[@leksicon]

hey it's 'EBB' by 'Density & Time'

[@Rishabhshahhh]

will it support my rtx 3060 6gb vram 16gb ram laptop

[@petarstoev4848]

I was asking myself the same question, it doesn't allow to be installed , but is there any other way?

[@leksicon]

I think the requirement is for 8gb VRAM.. perhaps there may be a way to assign Virtual vram to your system from your regular ram but it probably wouldn't be worth the effort

[@petarstoev4848]

Not worth, because its still bad?@@leksicon

[@xmdani4165]

Did you find any answers? I really want to download it but i too have a rtx 3060 laptop​@@petarstoev4848

[@AviShpayer]

I wasn't able to install at first, got a message of "failed to install". I figured it was the Anti-Virus software, turned it off and sure is, it started installing.

[@trilobty]

how do i run it again after i close the windows?

[@leksicon]

it should be in your Start menu, click the 'NVIDIA Chat with RTX' app and it will relaunch. It takes a while but it works for me.

[@Des7p]

@@leksiconIts a far way from being industry standard. The app is so large

[@kirayoshikage90]

Its showing a llama error

[@Dayzo97]

maybe with copilot?

[@mislavmikuss3731]

Can it make an image ?

[@Amogus_robloxian]

IDK

[@leksicon]

nope

[@thegooddoctor6719]

I installed it/ran it. Impression - Frak'n Large 36 GB. Very Fast. However - and the However(s) are big. Very Biased - i.e. - it filters the content you ask for (i.e. it scrubs the content with a liberal brush). Also - the replied content is short and the searching capability is not all that great. It misses a lot of content that is obviously in the material provided. In other words - Don't bother.....

[@leksicon]

waiting on the next version

[@thegooddoctor6719]

@@leksicon I have my doubts. Other open source RAGs are slower, but results of a search are By Far Better !!!!!

[@Unkn0wnUs3rname]

@@thegooddoctor6719which ones you talking about? I tried localgpt and with some tweaks it works well

[@wubinmatthew]

我安装一个试试

[@sanjaymohandubey1532]

will it work with my rtx 2050 laptop?

[@leksicon]

not unmodified, that laptop has 4gb vram this program needs 8gb vram

[@braudhadoch3432]

HAs a LLM up to August 2023 from web

[@Vansh1304]

"failed to install" (4070ti)

[@leksicon]

did you install into the default folder?

[@Vansh1304]

​@@leksiconyup. default folder. plus I do not have an antivirus. plus I have enought storage on my C drive (800GB). plus, I could'nt find any tutorial to fix that.

[@leksicon]

did you try clicking the box that says something along the lines (away from my pc) to wipe previous profiles during the install?

[@JustinLiu0721]

I have the same question. Do you fix it now?😟

[@7aggat]

same Card Same Issue

[@keyfull6908]

unzip error

[@leksicon]

re-check download and try freeing up more space on your drive

[@crabbypaddy5549]

As a fictional author, Chat with RTX is almost useless to me when talking to my document/book. The llm do not understand the difference between fiction and reality and apply ethical and moral guidelines at a comicla level. as it protested to deal with some information regarding a vampire as they are harmful and immoral, I asked it to provide me with and example of how a fictional vampire could kill a fictional person ethically end morally. it did not but told me to write it more inclusive and ethical version of events. lol. useless. It can not be reasoned with. lol. Nvidia could indirectly censor/hamper creative people by applying those guidelines to fictional works. I found another llm and software better suited for artist and authors, which is a shame since I really like Nvidia products.

[@Saraswati-gc6bo]

32.6 GB💀💀

[@leeishere7448]

gradio.exceptions.Error: 'Invalid session Edit:It's not compatible with the opera browser

[@User-3U4D0SsPjQy28]

stopped at the cmd env config...., anyone know how to restart the config?

[@3aved]

Try reinstalling it to the same folder, using the installer zip file from Nvidia.

[@User-3U4D0SsPjQy28]

@@3aved Much appreciated⭐⭐⭐