100% Offline ChatGPT Alternative?

In this video I show I was able to install an open source Large Language Model (LLM) called h2oGPT on my local computer for 100% private, 100% local chat with a GPT.
Links
* h2o website: h2o.ai/
* h2oGPT UI: falcon.h2o.ai/
* h2oGPT GM-UI : gpt-gm.h2o.ai/
* h2oGPT github repo: github.com/h2oai/h2ogpt
* h2o Discord: / discord
Timeline:
00:00 100% Local Private GPT
01:01 Try h2oGPT Now
02:03 h2oGPT Github and Paper
03:11 Model Parameters
04:18 Falcon Foundational Models
06:34 Cloning the h2oGPT Repo
07:30 Installing Requirements
09:48 Running CLI
11:13 Running h2oGPT UI
12:20 Linking to Local Files
14:14 Why Open Source LLMs?
Links to my stuff:
* KZbin: youtube.com/@robmulla?sub_con...
* Discord: / discord
* Twitch: / medallionstallion_
* Twitter: / rob_mulla
* Kaggle: www.kaggle.com/robikscube

[@AmazingArends]

Wow, I have a lot of saved documents, articles, and even e-books on my computer. The idea of my own local chatbot being able to reference all of this and carrying on a conversation with me about it is almost like having a friend that shares all my own interests and has read all the same things that I've read! Amazing how the technology is advancing! I can't wait for this!

[@uberdan08]

In its current state, i think you will be underwhelmed by its performance unless you have a pretty powerful GPU

[@Raylightsen]

​@@lynngeek5191In simple English, what are you even talking about dude?

[@fluktuition]

@@Raylightsen I think what he meant is that his GPU is not good enough, since he couldn't use the 8k token version.

[@fluktuition]

@@lynngeek5191 You said the desktop version is limited to 2000 but that is not true, you have the option for 8000, however you need a gpu that can handle it(like a RTX 4090, or RTX A6000, or a quadro rtx 8000 card..)

[@user-qw1zz2lk6g]

@@Raylightsen ok dude, here it is for you : He's trying to say that this shit ain't free homie. Apprently far from it according to @lynngeeks5191

[@chillcopyrightfreemusic]

This is amazing and very well put together! You have one of my favorite channels on all of KZbin and I’ve been able to follow what you teach. It’s an honor to learn from you! Thank you!!

[@robmulla]

Wow. Thanks for such kind words. I appreciate the positive feedback.

[@mb345]

Great video. I hope this gets a lot of views because it is relevant to so many different use cases that need to protect source data. Love the demo of how easy it is load vectorize your own local docs with Langchain.

[@robmulla]

Thanks for the feedback! Glad you liked it. Please share it anywhere you think other people might like it.

[@Robert-vj6up]

Fantastic tutorial and superb framework! Congratulations for you and the H2O team! 🔥🔥🔥

[@LaHoraMaker]

I didn't knew that you were working for h2o, but I am happy for you all. You're doing a great work making open source LLM more accesible and friendly!

[@robmulla]

Thanks! I just started there. Still will make my normal youtube content but this open source project is exactly the sort of stuff I would normally cover. Glad you liked the video.

[@lionelarucy4735]

I’ve always liked H2O, I used to use their deep learning framework a lot. Will definitely check this out.

[@sylver369]

@@robmulla You work for this?... Sad.

[@user-gq8gy1wm1n]

@@sylver369 get off his back

[@Aaronius_Maximus]

​@@sylver369if you worked for anyone "better" then why are you here? 🤔

[@friendfortheartists]

As many searches I've done on KZbin, you're Channel came up today. I'm really impressed. Great Job!

[@TranslatorsTech]

Mighty nice video, very useful! Giving local context is interesting. The question about roller coasters was a clever way to demo the feature. Thanks! 😊

[@pancakeflux]

This was great! I’m in the process of setting up langchain locally with openllm as a backend but to think I’ll try this as a next step. Thanks for sharing!

[@robmulla]

Glad you enjoyed the video! Thanks for the feedback.

[@surfernorm6360]

This is one of the best discussions of building an AI locally I have seen Bravo!! BTW the tutorial is excellent. its clearly enunciated the print ia veru big and readable for old foggies like me and he goes slow enough to follow and describes and shows what he is doing so noobs like me can follow,. also don't forget the transcription button gives details about every minute and a half . Very welll done anybody who is patient will like this. thank you Rob Mulla

[@fredimachadonet]

Awesome content! I noticed the audio dropouts from time to time. I had a similar issue this week when recording some videos myself, and the culprit for me was the Nvidia Noise Removal filter in OBS. I changed it back to RNNoise and it worked like a charm. Don't know if yours is related, but if it helps, then happy days! Cheers!

[@tenkaichi412]

Amazing! Thanks for the detailed guide. Will definitely be using this for future projects!

[@nash......]

This is awesome! Definitely going down this rabbit hole

[@Monkey-Epic]

Awesome!!! Here I was losing hope about AI / GPT being more transparent about biases getting trained "baked" into popular chatbots already & the lack of real privacy users have about what they discuss "there is no privacy". And blammo you guys already had this out in just a few months. Super cool!! Thanks to all who went in on this!

[@Charlesbabbage2209]

The last thing they want is AI “noticing” patterns in modern western society.

[@henrikgripenberg]

This was a really transformative experience and I really appreciate that you did this video!

[@doughughes257]

Thank you so much. This is a clear guide for us to begin experimenting with our vision for a new application, and the last 4 minutes is a great executive summary for me to show to my management.

[@johnzacharakis1404]

YES PLEASE make another video where you sey up all of these in a cloud environment instead of local. Excellent video, thank you very much

[@onandjvjxiou]

I would also like to see another video from you about setting up all a cloud environment. Thanks for sharing your knowledge.

[@kshitizshah7482]

Great video! I'm a fan of H20. Really impressed with the driverless performance. Helps me benchmark my own code! Gonna try this out later this week, thanks Rob!

[@marioschroers7318]

Just checked the thing out, as soon as it was mentioned (luckily, this video was suggested to me by KZbin). Being a tech enthusiast and a translator, I ended up spending an hour discussing the technical dimensions of my profession. Loved this! I will definitely look into this in more detail later. The thing is also pleasantly polite. 😊

[@TylerMacClane]

Hello Rob, I liked your video very much. I wanted to suggest that you consider making a video on how a translator or voice-to-text transformation can become a tool for everyone based on an open language model. It would be an interesting topic to explore and could benefit many people. Thank you for your great content!

[@pirateben]

there are plugins for this fine an open source one and use that

[@andysPARK]

@@pirateben what's it called Ben? Linkey pls.. :)

[@musp3r]

Look for "Faster WhisperAI", maybe it could help you in creating transcriptions and translations from audio-to-text, I've had great success in using it to transcribe youtube videos and create subtitles for them.

[@voiceofreason5893]

Fascinating stuff. So important to figure out ways of using these tools in a way that allows us to retain some privacy. Subscribed.

[@user-lk4xm6vg1w]

you got the like only for just including the last part 14:15 and after. the whole video is decently good. keep up the good work,the last part of info is really what eh people should get in their heads. BRAVO!!! ευχαριστω που το επες.

[@irfanshaikh262]

5:30 yes rob yes. Please. It will be all round approach if you start teaching python on a cloud environment. Much awaited and thanks for everything ur channels has offered me till date. Explicitly love your YT SHORTS.

[@eliotwilk]

Great work. I was looking for a tutorial like this for a long time.

[@anazi]

Amazing. Thanks and we cant wait for the cloud model.

[@siva_geddada]

I love the Content, even though no one doesn't know about this, Very very useful content we are expecting a cloud version demo also. Thank You

[@humanist3]

Excellent content, especially the LangChain part, thank you!

[@logicnetpro]

Very informative video ion how to create your own private chat bot and have it learn from your context. Genius! I look forward to further development.

[@Krath1988]

Thanks for the Video I really appreciated and support the open source community! .

[@dimitriskapetanios294]

Rob, the video is Awsome! Great content as usual 🤩 Would love to watch a version utilizing a spinned up instance from a cloud provider too ( for those of us without a gpu 😊)

[@robmulla]

Thanks for watching! Will def look into a video with steps to setup on the cloud.

[@deucebigs9860]

@@robmulla definitely interested in the cloud provider video too

[@AnimousArchangel]

The best explanation so far. I've experience using GPT4All, self hosted whisper, Wav2lip, stable diffusion and also tried few others that I failed to run succesfully. The AI community is growing so fast and is fascinating. I'm using RTX3060 12GB and the speed is okay for chatbot use case but for realtime AI game engine character response it is slow to get response. I recently get a hand of RTX3080 10GB and in this video I see you are using RTX 3080TI which has 10240 CUDA vs mine 8960. It is first time i see that you can use additional cards which in your case GTX1080 vs mine GTX1060 to run the LLM. Very informative video!

[@hottincup]

You should refit the model using Lora to get a smaller size, more narrow for in game usage, that way it's more optimize

[@CollosalTrollge]

Would AMD cards work or is it a headache?

[@AnimousArchangel]

@@CollosalTrollge currently they are using cuda technology which require nvidia cards.

[@AnimousArchangel]

@@hottincup good suggestion, i will try.

[@ScandalUK]

If you’re trying to shoe-horn a full fledged LLM to power some NPCs then you’re doing it wrong. All that you need is basic chat personalities and the ability to train based on in-game events, this requires very little processing power!

[@shukurabdul7796]

you got me subscribe !!! wow thanks for explaining in step by step for anyof your content keep it up

[@kevinmctarsney36]

Could you do a video on the “fine tuning” you talk about near the end? I like the privacy attribute of running open source locally and the fine tuning would be really beneficial.

[@MacarthurPark]

Thanks for all your efforts teaching brotha

[@peterhu3362]

I like this! U just earned yourself a new sub!! Would be interesting to know how to fine tune or train the installed LLMs

[@davida199]

This is extremely helpful, Thank you for sharing this.

[@harrisonleong4283]

Excellent stuff, thank you. I just followed your instruction, plus the README in the GIT, and spin up an instance on a Cloud VM, as my notebook has not GPU. It is fun... and I wish you can further teach us how to do the langchain things when we have a lot of documents that we want to feed them in. Thank you once again.

[@jannekallio5047]

been using Chatbots to write a tabletop GPG campaign for my friends, but having the main story in separate files has been a problem. If I can use the material I already have as my own training material it might be way easier! This chatbot might be exactly what I need! Cool, I will give it a go!

[@XX-yu4zz]

update?

[@brunaoleme]

Amazing video. Thank you so much. I also would to love to see a video about the setup in a cloud provider.

[@brianmcmurry5915]

Great video, your voice and pace is perfect. Thank you

[@syedshahab8471]

You explained everything very clearly, Thanks

[@IItiaII]

Very interesting, I stood it up on a VPS with 10 CPUs, it's painfully slow but it works!

[@Mark_Rober]

Dude, you are not even being biased THIS IS THE BEST INVENTION EVER!!! Open source??? AND it runs locally???? even without the file access feature this would've been the coolest piece of software I've ever encountered!

[@yourockst0ne]

Great work, thanks for the video 🎉

[@neuflostudios]

Yes Rob ,, would love to see your implementation in the cloud space for these models.

[@lionellvp_]

Thank you very much, finally a reasonably reasonable documentation. The topic of Cuda is also somehow a single cramp. I wanted to realize the whole thing as a Docker to keep the system more scaled, just cramp.

[@BryanEnsign]

This is exactutly what i was looking for to develop a model for internal use at my compnay. Thank you!

[@tacom6]

any specific use-case?

[@BryanEnsign]

@@tacom6 I write Inspection and test Plans, inspection test reports and standard operating procedure for industrial engineering and industrial construction. For instance if we need to write a procedure and checks for the correct way to flush HDPE or terminate and electrical panel etc. Currently I can paste SOPs or our checklists and ask if it was missed anything, ask about new ideas to add or new things entirely. It's great at asking for ASTM codes instead of looking it up or buying the PDF. I'm using Claude currently and perplexity. My company does everything internal it doesn't want the data hosted with another company. I'd like to make something for us to use internally. I believe using AI language models has sped up our procedure drafting and checksheet drafting about 40% so far. It's been game changing. But, I'm using 3d party sites and I have to scrub out client details, names etc. If I had an in house model I could let it retain all client or confidential data and others could ask requests against it also. I have a bot running that I've made through Poe using Claude as the base but I can't make it public for colleagues to use.

[@tacom6]

@@BryanEnsign sounds fantastic. my interest is similar but with focus on cybersecurity. thanks for sharing!

[@BryanEnsign]

@@tacom6 That's awesome. So many possibilities. Luck to you brother!

[@EckhartAurelius]

Very helpful video. Thank you!

[@cerebralshunt2677]

Yes, as requested, I am letting you know that I am interested in any of the potential future videos you mention in this video! You are giving gold away for free!

[@kearneyIT]

I will be trying this, can't wait!!

[@smartstudy286]

Awesome knowledgeable video this is so useful. Keep making videos like this

[@BradleyKieser]

Absolutely brilliant. Thank you.

[@MrNemo-un2fn]

this is the first video ive noticed that highlighted the actual subscribe button. it looked really clean.

[@heck0216]

This is cool and brilliant. Great tutorial

[@incometunnel7168]

Nice work buddy. Keep it up

[@jamesrico]

Great content, I enjoyed your video.

[@erikbahen8693]

Nice! It's working.. I'm excited

[@Theineluctable_SOME_CANT]

WONDERFUL! Thank you.

[@neoblackcyptron]

Wow an OpenSource GPT model. This is freaking awesome. I am working on building some AI products this is a life saver. I am excited to play with this big time. Throwing in some Vector Memory Databases to add context on top of this and I can get my first AI product out real soon. I can easily build some text to speech and Computer Vision models of my own on tensorflow to get something big to happen. Man Christmas has really come early for me this year.

[@Coecoo]

It does have many limitations such as requiring fairly beefy hardware (or get really restrictive questions) and a ton of storage space.

[@neoblackcyptron]

@@Coecoo do you think nvidia rtx 4090 can handle this? When you say beefy hardware what do you have in mind?

[@malsmith3806]

Thank you!! I’ve been stumped on building a model to generate an entire script for a Seinfeld episode based on my personal scores of each episode and I think this video just unlocked it for me!!

[@olivierdulac]

You are now master of your domain!

[@kompila]

That was good presentation. Thanks !

[@MeatCatCheesyBlaster]

This is really cool. I just installed it and tried it. actually runs pretty fast on my CPU

[@spacegirlkes]

How did you get it to work with your CPU? I keep getting token limitations on the answers. Did you follow the documentation?

[@dhrubojyotimukherjee4615]

Thank you very much for providing the beautiful resource..

[@user-jj2mo5sl7p]

great i have learn so much about how use the open source ai and ai modules ,i am glad to do my self to build the project on my local computer ,the read PDF ability is so good i well try it!

[@gregoryknight2928]

Great video. Thanks!

[@vsmcreatives3625]

Great Content Rob. I like this one more, Can you please make a video of how to customize this chatbot Look and features.

[@HaraldEngels]

Brilliant. Thank you for sharing. I am now looking for a faster machine to reproduce what you have demonstrated 🙂

[@user-ib9uu3wk5e]

Great job, thanks!👍

[@larabassabah202]

Keep good work. I'm very interested

[@SkateTube]

I am interested more about how this can turn into features for 3d model making, music making, 3d/2d game making or even software programming. Some tests of what can it generate and other stuff could be nice.

[@stevendonaldson1216]

Extremely excellent explanations

[@adriangabriel3219]

Hi Rob, great video. Could you make one on how to set it up using Azure and also how to use it for training a custom dataset (I would be especially interested in your recommendations for training the Falcon 40B)

[@major-skull-gaming]

thanks for the information Rob i just subscribed to you too

[@Listen_bros]

Man, you're just awesome ❤

[@bentomo]

Finally a video that gives me a "hello world" for an attainable local gpt alike chat bot. Now I can actually LEARN how to fine tune a model.

[@deltavthrust]

Great work. Thank you

[@kgoblin5084]

On the why you would want this question: I actually think 1 of the most compelling answers is "because it works without the internet". Most of the interesting potential applications I can think of for an LLM are not copacetic with mandated internet access... eg. using it for dynamic NPC dialogue in a video game, people don't like always-on connections for very good reasons.

[@c3rb3ru5d3d53c]

One suggestion here, this would be more popular for everyone if there was an installer like GPT4All has, as those who have no command-line experience can still use it.

[@deeplearning7097]

Thank you very much. It would be great to see something on spinning up higher memory remote GPUs.

[@NextLevelCode]

Cool video. Does h2o have plans to make a one click installer any time soon? Those are available for stable diffusion and makes setup so much nicer :) Especially when you just wanna use it and not work on developing the ai.

[@NextLevelCode]

@TheTen-PointFormatStyle-di1ez anything can have an automated installer made. It’s just if they want to invest the time or not. Nothing is so special about what they are doing

[@thefrub]

These language models are getting improved so fast, by the time you have it installed and working there's 3 better ones

[@onlyvitalnetworks2994]

Amazing work!

[@robmulla]

Thank you! Cheers!

[@0AThijs]

This works great on a collab notebook!

[@Monalexander799]

This is awesome I installed it on my windows 10 machine I just need to figure out how to optimize it like you did for your linux box

[@vimukthirandika872]

Nice tutorial. I hope we need more something like this. Btw are those gpus compatible and working fine?, since they are different generations...

[@richardblanchard2743]

Very clear explanation of the program. Great video. I wonder how people create these open source programs and still can put food on the table. They must have day jobs.

[@Ti-JAC]

Thanks, Rob good info will try out on me win 11

[@electrobification]

THIS IS A GAME CHANGER!! FOSS FOR THE WIN!

[@KBProduction]

Really enjoy the video

[@Martin-kr5nx]

Just came across your channel and I love your content. Are there any open source models we can use that have a longer context like Anthropic that we can run locally?

[@963seeker]

It would be an awesome to know how to train a model on a cloud provider, and which to choose from. Great video as always..

[@johnbobbypringle]

Hi, I'm relatively new to your channel but i watch your videos whenever they show up in my feed. You mentioned demonstrating a cloud based gpu instance to run this model and I would be interested in watching a video about this.

[@OthmanAlikhan]

Thanks for the video =)

[@sonicesb]

Amazing video! Thanks

[@robmulla]

Glad you liked it! Thanks for watching.

[@WvlfDarkfire]

Crickin SCARY! I love it! Hail our new overlords!

[@fahnub]

Thanks for the video.

[@alrabef]

Rob, awesome video. Thanks for sharing! . Could you please show the comouter and the requirements?

[@omerEX001]

Hey, great video! I would be very interested in a video showing how to run LLM model on cloud instance such as Amazon EC2 or similar.

[@aketo8082]

Thank you! Nice. What is the difference to GPT4ALL? Where are the strengths and weaknesses? Can h2oGPT understand associations? Or can it be trained accordingly? Will there be more videos with how-to’s for basics, using your own files, training and corrections?

[@Nono464.]

Hi Rob, Yes please show a walkthrough of you deploying this to an EC2 instance.