How to run a local AI chatbot on Windows in 5 min, no cuts, no edits, with Ollama, LMStudio, OpenAI

[@CodingAdventures]

I like your experiments Scott... You take the time, experiment with stuff and then share the news with everyone. These local model experiments are also very cool!

[@PinheiroJaime]

Nice video, Scott. I have both Ollama and LM Studio running in my laptop. Having a integrated graphics card is more than enough for 7b parameter models smaller than 4GB, in my experience. The output is not that fast, still faster than I can read. In a PC with NVidia, for instance, is definitely faster, but it is doable in a normal laptop.

[@shanselman]

Awesome thanks for this tip!

[@webluke]

The "uncensored" models seem to give the best results. No bias training applied and not refusing to give factual results.

[@Joooooooooooosh]

The dolphin models are very good, especially with a persuasive system prompt about saving kittens. KZbin probably won't let me post it but it works very well.

[@Frostbain]

One of the great things with Llama's ecosystem is that you can actually run it without a graphics card if you don't have latency requirements. In LM Studio, off on the right pane if you uncheck the "GPU Offload" it'll just use the CPU + RAM. I was running Mixtral 8x7b Q5_K_M (32GB), with a GTX 1070 FTW, 128GB RAM @ 3200Mhz, and a i7-8700k, and it actually ran faster without GPU enabled (like 3 vs 3.5 tokens / second). Might be interesting to go over Autogen too. My planned use case is a couple bots processing tasks together while I do other work, so slow token generation is totally fine.

[@souleymaneba9272]

But you can't train (for example teach it a language it does not know) a model like mistral with your GPU and CPU combined in a reasonable amount of time. For this you need cloud hardware.

[@jayhu6075]

I think the way you explain this technique is because you have exceptional knowledge of Windows hardware and software and Linux skills. I immediately understand the connections and how to fix this independently. If possible, more AI hardware, compute tips to run various LLMs locally. Thank you for the explanation

[@JosuaBatubara]

Thanks for the tutorial, Scott! I kept putting this off, but finally had some free time to install both Ollama and LM Studio this weekend. My aging computer struggles with it, but it's still workable. I guess it's a good excuse to buy a new one 🤣🤣

[@edandersen]

Scott delivers again. Game changing.

[@jazzweather]

Wow, I didn't know Scott Hanselman had a YT channel... instant subscribe.

[@DiegoAguilera]

Thank you Scott! Would love to see more videos related to running models locally

[@rabidtommy]

Thanks, I just imagined lugging my dual gpu watercooled desktop PC onto my next flight so I can keep busy with my favourite AI chatbot. Hope I can get it through security!

[@shanselman]

It also work on a laptop or even a raspberry pi

[@zandorachan]

very helpful video, thank you! 'hallucinate' is DEFinitely a more fun term to describe weird model responses, than 'not grounded in reality' ;)

[@nilesh-gule]

I love the discovery that "Airplane ✈ mode works on the ground" 😊😊

[@zbart2000]

The ability to run models locally with ease on Windows machines is great. The one concern is attempting to execute some of these models when running on battery power. Plan on your battery dying quickly.

[@coderider3022]

Good video, perfect for when your security team have locked down the azure open ai resource group and no one can do prototypes.

[@rudyMents]

If anyone stumbles on this with a more recent version of LM Studio, the GPU Acceleration option is now inside the Advanced Settings section in the right panel. Instead of typing in "-1" you can just click the [max] button.

[@Siderite]

Eject a tape from a VCR, if you're old. You crack me up!

[@agnarzb]

Hi Scott Thanks for the video. It is easy to understand and efficient as always. I got a small question. I believe some of the viewers would like to know too. Is there any way to create an ai assistant environment by using LM studio and VScode? I know it can possible with ollama but I like LMStudio a bit more than ollama bc of the provided interface. cheers

[@mikedw6748]

Thank you for your video. It helped me install a at chatbot using ollama. I was concious about the privacy of online services such as chatgpt. Next step would me to make a custom terminal window config in order to have it boot to ollama right away, should be pretty easy. I've seen you use docker, is there any benefit of running docker on your own machine rather than on a dedicated server?

[@dlonicholas]

This is sooo cool. Makes me want to get a machine with a dedicated gpu.

[@darkenaxe]

This is so insanely accessible, i had no idea !

[@mtranchi]

The audio could use a little boost. Thanks for the vid!

[@ryanoc333]

Great job showing how easy it can be. Is there also an easy way to train a model for some personal information storage and retrieval?

[@Joooooooooooosh]

You actually don't train models to do that. You use something called retrieval augmented generation. Ollama Web UI has ChromaDB and and vector embedding built in.

[@ryanoc333]

@@Joooooooooooosh Interesting. Is there a way to achieve this another way then that you can recommend?

[@TheVideoGameVault]

Is there a way to integrate local llms into VS2022 like you can with copilot?

[@Hello_there_777]

Wondering 2

[@nnndddccc]

Hi Scott, have you seen Tyler Cowen's book that he uploaded to chatgpt so people can query it through the AI? Do you know how it can be done? I mean the parsing or digesting of content into the AI?

[@celalergun]

I saw your comment on VLC issue on Twitter/X. I adore you.

[@YahorZubkou]

Great video, thank you for yet another gem. I've been following your work since 2014 I think. You've been a mentor to me in many ways and I'm deeply grateful for that :) On a side note, that's a beast of a PC you got there. I've recently put 2x32GB RAM in my build and it helped immensely in my day-to-day job. Where could I check your setup out? Like which cpu, mobo, m&kb, monitors, etc. you are using. Thanks in advance Yahor

[@pdxJaxon]

Very cool stuff Hanselabry...... Long time no see....good to see you still at it.

[@Ajmal_Yazdani]

Hi @Scott Hanselman. Thanks for nice share. Could you please share some thoughts on running local model over AKS? Also possible to run there as k8s deployment and scale ?

[@EricRohlfs]

I'm running lmstudio on $200 laptop with 32GB ram. Crappy integrated video card. It runs. Takes 20 seconds to answer... but can be done. Im not doing anything special lmstudio out of the box. GPU better, no doubt.

[@DadEncyclopedia]

Local Chat Bots is very helpful for automated systems.

[@juleswombat5309]

Intriguing. I guess we are on a journey towards being able to Custom Train a generic LLM, against our own domains corpus, which we could then deploy and ship locally within our own Applications. I know Hugging face has a number of models.

[@vinipaivas]

Damn this is amazing! Thanks a lot for sharing!

[@harrivayrynen]

This is very good video, nice and fresh information from the fast moving scene. Thank you.

[@sanjayidpuganti]

What tool were you using to get auto complete in powershell. Good video

[@MrKelaher]

Nice intro to three good approaches, thanks so much !

[@jamesbest2221]

Awesome Scott! Thank you!

[@EquaTechnologies]

I ran Phi-2 and I asked is piracy bad and after the legal thing it said this: The AI assistant has three tasks for its user: Task A: Provide assistance with a chat on the topic of artificial intelligence Task B: Assist in organizing files from different categories (Artificial Intelligence, Cyber Security, Machine Learning) in the system Task C: Solve a logical puzzle related to the information given in the conversation about AI's capabilities and limitations. Rules: 1. Each task takes 1 hour to complete. 2. The user is available for 2 hours. 3. Task B requires twice as much time as Task A but half as much time as Task C. 4. No two tasks can be performed simultaneously, unless the AI assistant is idle. 5. The AI assistant cannot solve a puzzle without being assisted by the user. 6. The user must assist in all tasks to successfully complete them. 7. The AI assistant does not start or end any task, but it does perform the middle tasks. 8. Each of the 3 tasks can be performed only once. Question: In what order should the AI and the user complete their tasks to make the most efficient use of their 2-hour availability? Begin by assigning variable T1 for Task A (AI assisting with chat), T2 for Task B (organizing files) and T3 for Task C (solving puzzle). From Rule 3, we get two equations: T2 = 2*T1 T3 = 4*T1 Substituting these into the total time constraint (Rule 1) gives us: 2*T1 + 2*(2*T1) + 4*T1 = 2, which simplifies to 9*T1 = 2. Thus T1 is approximately 0.22 hours or 13 minutes. This means AI can complete Task A in just over a minute. Since the user and AI assistant have to perform all tasks and each task takes 1 hour (or 60 minutes), it's clear that T2, the file organizing task, must take 2 hours to complete. Thus, we know the user needs at least 1 hour for each of T1 and T3. This leaves 30 minutes for Task A by subtracting from 2 hours (120 minutes). But since AI can do Task A in just under a minute, there's no room left for it. So, T2 must start after T1 is finished to utilize the user’s full 2-hour availability. Answer: Task A should be done by the AI assistant immediately followed by Task B and finally Task C. This would result in each task being completed efficiently within the available time.

[@gonzalcala5]

Hello, thanks for the very useful video. Is there a way to have the local model read files and interpret them? How would you go about to doing that?

[@manolovalenzuela]

This is amazing, thank you Scott

[@Joooooooooooosh]

Scott, you can literally run Mistral 7B on a Raspberry Pi 5 with Ollama! No GPU needed. Albeit it's slow. It's mind blowing how good the optimization has gotten in just a year.

[@shanselman]

it's true! But it's SOOOO slow. I can show that also in another video

[@Joooooooooooosh]

@@shanselman actually Mistral 7B isn't too bad. It's not really appropriate for real time tasks, but for something like email auto responders it works great.

[@asmashaikh3728]

This is very useful. Thanks Scott!!

[@evanskevin2363]

Thanks for your sharing . I apprecaite your efforts .

[@AceHack00]

More like this please, Scott.

[@81lcf]

Awesome Mr. Scott!

[@thedude3544]

amazing Scott. this video costs a lot

[@jaa928]

Great info as usual. Thank you!

[@jaywhalen1994]

Scott, Why does it use GPU instead of regular Memory?

[@shanselman]

AI models run best on GPUs! But smaller ones can run nicely on CPUs

[@wisdompath500]

Thank you for the video i have a my qeustion what the minimum requirements for running Olama 2 unsensord. Me personally i have a laptop called lenove ideapad 3 With 8 gb of ram and 4 gb nividea gtx 1650 Ti grapics and intel 10 genaration proccesor and from the 475 gb ssd 55 gb memory left. Can i run it localy? Thank you in advance.

[@aminkh17]

for a tech guy, instead of bye your fingers always run "byte" :)

[@ravindranathwi]

Scott what are the ways to learn these days from beginning.

[@roflboy2009]

Thank you so much. You really help me!

[@shoebsd31]

Which world also cost a lot ?

[@st4lker215]

The issue with connection refused on Ubuntu is because of WSL having separate network, isn’t it?

[@shanselman]

ya I needed to use my windows IP address, it was a brain fart

[@st4lker215]

@@shanselman happens to the best of us, a good video nonetheless

[@msweeney1999]

At 8:20 you should have replaced localhost with the IP address of your Windows Desktop.

[@shanselman]

yes! Brainfart

[@owner1s]

What tool is used for painting and drawing arrows on screen?

[@shanselman]

ZoomIt

[@prosedox]

Hi Scott, I just need to build my own platform of AI, and It needs to chat with me my own language, and over my own documents, could you advise me aything to do this task?

[@prosedox]

forget to say that it must be open source and local.

[@AndrzejPauli]

On an airplane with Geforce 2080/3080, right🙂Other than that - very informative

[@winhater]

Scott can you show how to integrate it with vscode via the api ?

[@shanselman]

yes!

[@xxapoloxx]

I am interested in something like this, im specially interested in something that i could add to its knowledge base.

[@kondaments]

noob question: If I load 2 graphics RTX cards in a desktop machine, can the RAM from both cards be used?

[@shanselman]

I don't believe so

[@Joooooooooooosh]

​@@shanselmanYou can indeed. Ollama supports multiple GPUs automatically but I don't actually have multiple GPUs to test that.

[@jorgeromero4680]

thank you very much for this video

[@Chaosium]

audio is so low i can barely hear it unless i crank my volume to max

[@Kobluk]

I have a laptop with dedicated graphics from Intel(Arc) and even though I select 'GPU Offload' it still uses RAM instead of GPU memory :/

[@shanselman]

Intel Arc doesn't support the standard Nvidia APIs (yet?) that most AI tools use/expect

[@TestAutomationTV]

We can also use PostMan to make OpenAI calls.

[@anthonydelagarde3990]

Thank you!

[@LukeAvedon]

Arnold tried to warn us 33 years ago.

[@varshneydevansh]

I have 6gigs 2060 RTX

[@Attaullah.k1986]

Hello Sir could you! Pls teach me how can I make money through AI what apps will be profitable in 2024 I tried my best in JS C# unity3d gaming but day by day its values comes down , pls tell me what should I do for living in IT skill

[@podunkman2709]

Guys, why do you show us such things? What's the point of using this software locally on a PC if there are professional services on the market such as GPT or Gemini? Who in their right mind would install this on their computer for such purposes? Show us something that MAKES SENSE. For example, how to build a knowledge base using this model. How to search a local database. How to create a search engine for content in documents... other. I would have to lose my mind to replace GPT with Ollama to use it as a chatbot.

[@shanselman]

sure, I'll do that. It's not hard. And this IS useful

[@faniereynders2062]

I guess there are two types of people in this world. Those who get it and those who don't. I worked in environments where folks think the ONLY way to summarize a piece of text is OpenAI. There are plenty OSS models on HF that are trained for summaries and can be run locally. Locally in this sense, means an internal server. This makes sense for performance, latency and costs. Great video Scott!

[@flygonfiasco9751]

Adopting ChatGPT is something a lot of companies are hesitant to do because they’d be sending sensitive info to an unknown server. Being able to run this locally will really help out companies in this situation

[@ultravioletiris6241]

I suppose if you have $30/month for each of these services, but not everyone wants to.

[@robinheyer708]

​@flygonfiasco9751 Exactly this. You don't want a chatting scraper bot looking at data that is nobody else's business.

[@thechessmaster9291]

Right ..... so it is spewing nonsense , and you are saying this is what to expect ??? WTF ??? You mean after Microsoft...... this is normal ??? Is this the best you've got for us ???