Ollama - Local Models on your machine

Site: www.ollama.ai/
My Links:
Twitter - / sam_witteveen
Linkedin - / samwitteveen
Github:
github.com/samwit/langchain-t... (updated)
github.com/samwit/llm-tutorials
Timestamps
00:00 intro
01:07 What is Ollama
01:30 Ollama Models
02:19 Installing Ollama
03:13 Running Ollama
06:17 Customizing Ollama

[@chautauquatrail]

I just did the ollama install yesterday, you are awesome for being able to produce these so quickly.

[@kenchang3456]

Thanks Sam, very interesting. It's amazing how fast the whole LLM ecosystem is moving.

[@VincentVonDudler]

Thanks, Ollama.

[@DanielSpringer]

Definitely has a docker vibe. I like it!

[@paulmiller591]

Thanks Sam Great video. These are some of the best videos on AI tools. I need to master this for my work and your approach to communication really works for me. Cheers. Keep up all things Langchain, please.

[@user-ut8ts5gv2g]

nice video. I managed to create a customized model by watching this.

[@nexuslux]

Nice and to the point video. Appreciate it!

[@kp_kovilakam]

Thank you for the introduction!

[@sonurocks341]

Great demo ! Thank you !!

[@sandrocavali9810]

Excellent intro

[@riflebird4842]

Thanks for the video, keep it up

[@richardchiodo2200]

I picked up a 12gb 3060 from my local microcenter for a pretty good price, and am now running Ollama with Open WebUI for the frontend, and they have a community repository for prompts and modelfiles. The biggest hurdle was passing the gpu through my proxmox host to my vm.

[@sitrakaforler8696]

LLama2 unsencored was quite surprising for me x) By the way. THANK YOU FOR yOUR VIDEO Every time i need to use Ollama I'm using your video to be sure of the command "Ollama run" hahah

[@willi1978]

it looks like the uncensored versions are a lot better. then it is not always giving a paragraph why it can't do what you ask it to do

[@jeffsteyn7174]

Its not just about been techincal. Its also about been productive. Do you want to spend your time building something useful or time trying to figure out how a badly maintained and documented piece of software works.

[@theh1ve]

Another great flag this Sam. I would be interested to see this running so you can make API calls.. currently using Text Gen Web UI as a server and this looks like it would be a good alternative.

[@jidun9478]

Nice, thank you. It runs so much faster than Text Gen Web UI! I wish they'd make it easier for you to add a custom choice of models though (that is a real draw back).

[@samwitteveenai]

I have a video coming that shows how to do exactly that. Its actually pretty easy.

[@nembalage]

super helpful.

[@twobob]

nice one thanks

[@tusharbokade8378]

Interesting!

[@photorealm]

Ollama for windows is out and available for download. I am testing it and it works fabulous but its very slow for me on windows. I don't think its using my nvidia GPU and can't seem to find a way to hook the GPU in under windows. But just got started, I love the fact that it is serving to a local http port as well as command line.

[@FreakyStyleytobby]

Fantastic video Sam, thank you! Ollama looks great but the big, 70B models still remain beyond the reach of typical RAM. Do you know of any way, (be it API or other) to get access to Llama70B and be able to run arbitrary tokens on the model? There are some APIs like TogetherAI but they only let you to run the endpoints like /prediction, not much more.

[@Leonid.Shamis]

Great video, as usual :) I have been using Ollama on Linux and it has been working great. I know that Ollama can be used via API but I was wondering whether its API is compatible with OpenAI API and can be used as a replacement for OpenAI API inside LangChain. Looking forward to more videos about Ollama. Thank you.

[@IanScrivener]

There are dedicated langchain and LlamaIndex connectors for Ollama. Ollama is different to OpenAI’s.. better IMO

[@VaibhavPatil-rx7pc]

Thanks

[@xdasdaasdasd4787]

awesome! I was hoping to use a custom model but didnt fully understand :(

[@mohamed_elmardi]

For windows users they can use WSL

[@fontende]

Win 10 was last for me, I found perfect Linux of PikaOS, an Ubuntu without ubuntu snaps or other crap.

[@franciscojlobaton]

Please, more. Más por favorrrr

[@mbottambotta]

Thank you Sam for posting this video. Very accessible, clearly explained. Question: what I could not see is if ollama enables you to choose the model size. I.e., whether you want llama2 7b, 13b or 70b for example.

[@brando2818]

3:37 You can, specify with ollama run llama2-uncensored Just go to the models page, then click one. It'll tell u the command if you're using cli

[@samwitteveenai]

Yes you can pick this take a look on the models page.

[@iainattwater1747]

I used the docker container just released and it works in windows.

[@VijayChintapandu]

can you provide the docker container link. from where did you downloaded

[@AlphaSynapse]

ollama is now available for windows (windows 10 or above)

[@zacharymacaroni7649]

good video :)

[@MarcellodeSales]

It seems like it's Docker :D Same feeling.. Ollama will captalize one CloudNative Software Engineers

[@Knowledge_Nuggies]

I'd be interested to learn how to build a RAG system or local LLM agent with tools like Ollama, LM Studio, LangChain etc.

[@AndyAinsworth]

LM Studio for Windows and Mac is a great way to achieve the same with a lot less setup! Also has a great internal model browser which suggests what models might run on your machine.

[@AndyAinsworth]

It can also run as an API with a click in the UI. Definitely been the easiest way for me to test out a load of different LLMs locally, nice user interface with history and markdown support.

[@alx8439]

Lm studio is a proprietary software. God only knows what else it is doing on your PC - gathering and sending out your data while you are sleeping, mining bitcoins, using your PC as exit node for TOR, keylogging everything you type - you can only guess

[@IanScrivener]

Agree, LM Studio is great. Can be run in OpenAI API mode.. which replicates OpenAI's API format.. and so can be easily used with langchain, LlamaIndex etc

[@AndyAinsworth]

@@IanScrivener Yeah, I'm hoping to get it setup to use the API via LM Studio with Microsoft AutoGen which provides a multi agent work flow with a code Interpreter.

[@scitechtalktv9742]

@@AndyAinsworththis is what I want to do also! Have you had any progress and success with this?

[@attilavass6935]

What are the pros and cons of using such "local" Ollama models on Colab Pro with 2 TB of Drive?

[@morespinach9832]

If we get these locally in our cloud, is there a best practice to keep them updated?

[@liji8672]

Hi, Sam, good video. my little question is that if you llama2 model ran on your cpu?

[@samwitteveenai]

Pretty sure it was running on Metal and using the Apple Silicon GPUs. It is certainly a quanitized model though, which helps .

[@IanScrivener]

You CAN run any llama.cop tool on CPU… though it is MUCH slower than GPU. MacOS Metal GPU is surprisingly fast…

[@user-di5tl3iv8l]

thanks for the video! how can I make ollama run the 13gb tar file i download locally?

[@Shawn-lk2ze]

I'm new to this topic and I just binged your videos. How does this compare to vLLM from your previous video? I get ollama is more user-friendly, but I'm more curious about the performance?

[@samwitteveenai]

vLLM is for more for serving full resolution models in the cloud and Ollama is for doing. vLLM shines when you have some strong GPUs to use etc.

[@Shawn-lk2ze]

@@samwitteveenaiGot it! Thanks!

[@BionicAnimations]

Hi. How do I uninstall one of them off my MacBook Pro? I am using it in terminal.

[@samwitteveenai]

ollama rm llama3 If you just type ollama in the command line you should be able to see all the commands

[@ghrasko]

In fact, it was quite easy to install ollama on Windows 10 using Windows Subsystem for Linux (WSL). In a Windows command prompt: wsl --install -d Ubuntu (this downloads and runs the Ubuntu distribution giving a Linux prompt) ollama pull llama2:13b (this downloads the selected model) ollama run llama2:13b (this runs the selected model) At this point you can white user test that will be sent to the model. This did not work for me, the keyboard input is not correctly directed to the application. This is possibly a compatibility issue with this Linux emulation. But I could fully use the downloaded models from simple Python programs directly or through Langchain.

[@nelavallisivasai8740]

It's taking more time for me to get responses from local model not as fast as yours. Can you please tell me what processor you are using? What are minimum hardware requirements to run LLM models to get faster responses?

[@abhijitkadalli6435]

feels like docker

[@alx8439]

There are bunch of alike tools (simple to use for non technical ppl). The most prominent is gpt4all. Yeah from the guys who fine tuned first llama back in march/April on their own handcrafted datasets. These guys from ollama were definitely inspired by docker, based on the syntax and architecture :)

[@technovangelist]

a few of the maintainers were early Docker employees

[@guanjwcn]

Thanks, Sam. Do you know what tricks ollama uses to make it run so smoothly locally?

[@samwitteveenai]

They are using quantized models and on macOS they are using metal etc.

[@IanScrivener]

Ollama uses llama.cpp under the hood… the fastest LLM Inference Engine. Period. Many other apps also use llama.cpp: Kobold, Oogabooga, etc Many other apps use Python inside … easier to build much much slower performance.

[@samwitteveenai]

@@IanScrivener They have the llama.cpp running on Metal on macs right. It feels like it is more than just on cpu etc. honestly haven't looked under the hood much it.

[@ronelgem23]

Does it require VRAM or just regular RAM?

[@user-ex4mf6ky6x]

Hi I am using ollama for past 2months, yes its giving the good results but what i need is it is possible to set the configuration file for ollama like setting the parameters for ollama to get a most accurate results can you make one video about how to set the custom parameters.

[@XiOh]

when is the windows version coming out? O.o

[@GrecoFPV]

Can we give this power to N8n ? Connect our local ollama with our selfhosted N8n ?

[@pensiveintrovert4318]

Any idea how to load a model that is already on my disk?

[@samyio4256]

Also another question. Do you really run this on a mac mini? If so, how much ram does your machine have?

[@samwitteveenai]

32gb of RAM

[@wendten2]

"Its Llama for those who dont have technical skills" .. the PC version is currently only available on Linux... xD

[@Thelgren00]

Can i use this to install ai town..default method was too complex for me

[@rookandpawn]

I'm coming from text-generation-webui, how can i use that model folder for ollama?

[@NoidoDev]

# New Software In the past: It only runs in WIndows, but maybe in a few years it will be available on MacOS and one day but probably never on Linux. Today: At the moment it supports MacOS and Linux, but apparently Windows support is coming soon as well.

[@samyio4256]

If the used model talks to an api, how it this a local usage? Id like to know where the prompt data goes to? Will it go to a database and the Model loads it after? Or is the model hosted seperatly in a monitored env? My basic question is, who gets the data from the input prompt?

[@samwitteveenai]

the data is only on your machine. It is all running locally. It can run an api on your machine and you can then expose that if you want to use it from somewhere else. If you are just using it on your machine all data stays on your machine.

[@samyio4256]

@@samwitteveenai wow! Thats a complete game changer! Thanks! ill sub, insane content!

[@Ryan-yj4sd]

Can you run fine tuned models?

[@IanScrivener]

Yes, and your own Loras…😊

[@Gerald-iz7mv]

what port does the webserver on? can i set that port?

[@merselfares8965]

would a i3 11gen with 8 ram and 630uhd graphics card be enough ?

[@samwitteveenai]

honestly not sure. It will probably run but you may get very slow tokens per second

[@abobunus]

how to make your own language model? for example I want to take some texts and force AI to use only this text to answer my questions

[@volkanazer9997]

Let me know when you've got it figured out. I'm curious about this as well.

[@kevinehsani3358]

I am sure windows users can probably install it under wsl

[@samwitteveenai]

I was wondering about this. i asked one of my staff to give it a quick try it but he couldn't get it working.

[@VijayChintapandu]

My system is very slow when I am running Ollama. My system is mac M2. Is this issue?

[@samwitteveenai]

depends which model you are trying to run. The video was done on a M2 Mac Mini

[@SharePointMaster]

@@samwitteveenai ohh thanks for the reply. mine also same Mac air with M2 chip but it was slow. I will check

[@foolcj9999]

can u make a video of ollama interaction using voice input. and it replies back like whisper

[@samwitteveenai]

Interesting Idea!

[@HitopFaded]

I’m trying to run it in a python environment if possible to build on top of it

[@samwitteveenai]

I have another vid there on Ollama's Python SDK etc

[@HitopFaded]

@@samwitteveenaithanks I’ll check it out

[@stanTrX]

can i upload and work with documents with ollama?

[@samwitteveenai]

yes you will need to code it to do a custom RAG

[@stanTrX]

@@samwitteveenai thanks good man but whats custom RAG?

[@kunalr_ai]

Why this new model

[@LITTLEFREDOX2]

windows version is here

[@DaeOh]

Would you consider not referring to models like Llama and Mistral as "open-source?" It sets a precedent. "Freeware," maybe?

[@alx8439]

It's a good question how we should refer to such models. It's not 100% Foss compliant because of the restrictions which come into place if you have like 700 millions of users, if my memory serves me well. But this is more like restriction for a couple of companies like ms, Google, tiktok. Who cares about them? Or am I missing something bigger?

[@spirobel2.0]

Mistral is completely open

[@clray123]

Do not mix Llama and Mistral together. Mistral has a truly open license, Llama is the Facebook/Meta poison.

[@DaeOh]

It's not open-source because you can't reproduce it without the source (training data)... Just making the equivalent of binaries available for commercial use doesn't make something "open-source..."

[@fontende]

I run only locally and cloud services are anyway blocked to our region (quite many people don't have access to such, more than 2 billions, China+dozen others countries, mostly by political non-scentific logic). And hardware allow such, thankfully to China which recycle servers and bringing on market quite a secret chips from Intel, like Xeon 22 cores, which was never released outside enterprise, it costs only 150 bucks. My motherboard Asrock X99 Extreme4 become defacto standard in China for such socket, also 150 bucks, can be filled with 256gb Ram, i've bought mine in 2020th during Gpt2, which was impossible to run locally in it's max size 1558millions, there wasn't any of current tools, i was able to run by terminal 774millions on Gpu and it's was a mess of text.

[@anispinner]

Obama

[@antonpictures]

~ % ollama pull 01-ai/Yi-VL-6B pulling manifest Error: pull model manifest: file does not exist

[@astronosmage3722]

would say oobabooga is still the way to go

[@alx8439]

Yeah. Or h20 gpt / llm studio