Getting Started on Ollama

Here is everything you need to know about getting started with Ollama. It's not hard, but sometimes the first steps can be daunting.
Be sure to sign up to my monthly newsletter at technovangelist.com/newsletter
And if interested in supporting me, sign up for my patreon at / technovangelist

[@enmingwang6332]

What a great tutorial, clear, concise and informative!!!

[@sdaiwepm]

Thank you for such a helpful explanation. I wish more tech explainers and presenters were this clear and structured.

[@milorad9301]

Thank you, Matt! Please create more videos like this; they're really clear and simple.

[@wirreswuergen]

Thank you, Matt! Your videos are awesome and already helped me a lot :)

[@zerotheory941]

If you can make a video about crew AI explaining it as simply as you did here, you'd be my hero.

[@edwardrhodes4403]

And also Autogen and other agents like Devika and how to integrate them

[@sidnewby7111]

Im so happy there's someone in the mix who actually has a career in AI/Dev. Seriously, really enjoying your content. Dont listen to any of these jerks.

[@vicnent75]

thank you for you job Matt.

[@liammcmullen4497]

Great Overview Matt, your a star!

[@richardurwin]

Thank you for the video

[@incrastic6437]

Excellent introduction. Thanks for the help

[@JoaoKruschewsky]

Hello from Brazil. I really liked your content ! thanks

[@qewolf]

Verry cool, thank you 🙏

[@Mike-vj8do]

AMAZING!

[@Drkayb]

Excellent summary, thanks alot.

[@hotbird3]

You're a very smart person 👍👊

[@JeppeGybergyoutube]

Nice video

[@user-wr4yl7tx3w]

Great content

[@ec_gadgets]

You explained it perfectly, thank you

[@technovangelist]

Glad it was helpful!

[@bens4446]

Thanks! Just downloaded Ollama and was feeling a bit lost. Would really appreciate some guidance on integrating speech recognition and text to speech into the chatbot. But just about anything you say will probably be useful. Please keep 'em coming!

[@juanjesusligero391]

Thank you so much for your tutorials! :D I would like to suggest an idea for a future video that I would be really interested in watching: a more detailed exploration of the various models (such as the instruct/base/etc. ones you've mentioned). Again, thank you very much! You rock! ^^/

[@SergiySev]

Great video, thank you for Ollama introduction! is there a way to add my own data to the model or shrink model to a particular topic? for example TailwindCSS, there is a sorce code, docs, library of the project, is there any way to train model to be able generate layouts and components based on a provided data?

[@ftlbaby]

Thanks for this! I just setup Ollama with wizard-vicuna-uncensored:30b-q8_0. Do you know what's different in the fp16 models?

[@K600K300]

thank you

[@nholmes86]

I successful run Ollama with llama 3 on Mac OS M1 8G, it runs better when you close other apps .

[@sebington-ai]

Hi Matt, do you know what determines the length of a model's answer? How does the model 'know' when to stop? Is it hard coded into the model or is it controlled by Ollama? Thanks

[@emil8367]

thanks for sharing, prune is something what I missed but very useful due to the fact of downloading large files and loosing them after each restart, was very annoying. I see ollama didn't documented it well or maybe I overlooked it

[@nicosilva4750]

Do the models return Markdown ...like lists? extended Markdown ...like tables and LaTeX? I have written my own desktop client that I put on all our machines to use OpenAI and their API (cheaper than $20 * 5/month). So I would like to have a network server for my home to run a local model. Can I set it up there and have everyone use it, or would there be performance issues? ...what about simultaneous usage?

[@dmbrv]

thanks

[@ValentinPletzer]

Thanks. I really learned a lot by watching your videos. I recently ran into an issue when writing a new template model for few shot learning. Most of the times it responds like expected but sometimes it responds to my prompt and then also inserts it's own command by adding [INST] some other prompt … and also answers it. I probably made some mistake but I cannot figure it out. That's why I would love to see you make a video on templates (if this isn't too much to ask).

[@jahbini]

I second that request!

[@anshulsingh8326]

Subed ❤️ If only you taught maths too

[@samsquamsh78]

I like your videos, always spot on and pedagogical! Why did you leave ollama?

[@technovangelist]

If we find ourselves in the same room I’ll talk about it there.

[@thepassionatecoder5404]

Do I need to know match, statistics, etc... apart from programming?

[@RobCowie]

Does it phone "home" at all, or is the model I use locally, assuming the machine is connected to the Internet, shared publicly at all, and is it secure?

[@technovangelist]

It doesn’t reach out anywhere unless you write a program to have it do something like that.

[@mshonle]

Here’s a video request: can you do one on LMSys’s SGLang? Particularly using constrained decoding?

[@tecnopadre]

Sometimes your level is so high some others too simple. Cheers.

[@Delchursing]

Great video. The costs are a bit unclear to me. Would a local ollama/llm be free to use?

[@technovangelist]

What costs? You have to own a computer with a gpu. That’s it

[@bens4446]

FYI- My llama2 install is working reasonably fast without a GPU, just a ryzan 5600G CPU, which has some rudimentary graphics capacity built into it.

[@abhijeetkumar8044]

Please create videos on how to fine tune these models 🙏

[@PoGGiE06]

Thanks Matt, why does everyone use Mistral rather than Mixtral?

[@technovangelist]

Too slow

[@blackwinegum]

I just dont get any sort of CLI when i install Ollama , the app just shows "view logs" and "Quit Ollama"

[@technovangelist]

so when you run ollama at the command line you don't see anything?

[@blackwinegum]

@@technovangelist I think i've figured it out, i think my firewall was blocking something, thanks for replying.

[@stebansb]

great content, a telegram group would be great!

[@technovangelist]

Telegram??? I think I used it once at an Idan Raichel concert but never since. What’s special about a telegram group?

[@stebansb]

@@technovangelist the other option being Discord, I feel is simpler, cleaner user interface, yet very powerful; popular with business and a slightly more mature cohort. The other option is Discord, slower more complex, popular among gamers. Either way, will be cool to have something to build a community beyond KZbin.

[@AwesomeCanadianHomes]

I have a feeling Duncan Trussell is a subscriber : )

[@alexsnow2993]

Hello! My video card is an RX580. Is there a way to make it work?

[@alexsnow2993]

Using the rx580, it will be slow? Or not work at all?

[@technovangelist]

I don’t see it on the compatibility list. github.com/ollama/ollama/blob/main/docs/gpu.md

[@technovangelist]

Just won’t work at all. I think ollama requires the newer amd drivers and amd didn’t make it backwards compatible with older cards.

[@alexsnow2993]

Thanks for the info! I can't get another V-card at the moment, and using the CPU it is a no go. Is there any version or any other AI out there, that can be configured locally?

[@technovangelist]

Everything I know of is going to need a decent recent gpu.

[@mrrohitjadhav470]

it would have been great to know how to install other models not mentioned in Ollama library with specific type of Low Vram and GGUF

[@technovangelist]

check out kzbin.info/www/bejne/ZqDYZmSiYrJ_edE

[@mrrohitjadhav470]

@@technovangelist Thanks a lot❤

[@userou-ig1ze]

I wish there was an easier way to fill in template text, and parsing pdfs. I've seen the 'function calling' video's, but somehow it's still eluding me how do get this done as easily as possible (e.g. sending a pdf over api in a curl request from another machine, and rename it sensibly/according to content)

[@technovangelist]

The biggest problem there is the pdf. You can’t easily get to the contents of the pdf. The text. It’s often jumbled up. PDF is the worst format you can use if you want the text and to do something with it. That’s also one of the benefits of pdf. It obfuscates the source text so folks can’t do anything with the text.

[@userou-ig1ze]

​@@technovangelistthanks for the reply! I used pdf2text but it was not exactly perfectly successful. I wonder how ollama frontends (e.g. webgui or webui) solve this for their RAG? Gave me hope that there is a good way of doing it 🎉

[@user-du2jz9wx6k]

Ollama runs very well on a M3 Max

[@axeljohannes3464]

Wait do I need to download anything or not? You say "Now the model should be downloaded, so you can run it with ollama run mistral" Why would it be downloaded? I just installed the Ollama software. Does it download all the models automatically? This seems very unclear

[@technovangelist]

I think you must have skipped around a bit. I very clearly said to install and then run ollama pull to download the model. Then while downloading talked about what’s going on. Then the model is downloaded and you can run it. When you downloaded the model you only downloaded that model. Why download anything? Because you want to run it.

[@axeljohannes3464]

Thanks! I got it to work@@technovangelist

[@axeljohannes3464]

I think what confused me is the term pull, and what that actually meant. So when you got to the point of speaking about downloaded, I was like "Hey, no one said anything about downloading anything"

[@makesnosense6304]

9:40 To have the same result you just need the same input, seed (and other parameters), no? Reason it's different every time is because the seed is random for every request, right? The seed used (and other parameters) make the result different because it takes a different path in the weight model.

[@technovangelist]

Using the same seed and temp doesn’t always guarantee the same result

[@makesnosense6304]

@@technovangelist Ah, because temp is a percentage randomness scale of sort.

[@makesnosense6304]

@@technovangelist What if temp is 0? Or 1?

[@technovangelist]

It’s not guaranteed

[@thiagoassisfernandes]

arch and nix are system-d distros

[@technovangelist]

Doh!

[@briannezhad1804]

Can Ollama be used in prod on Linux server?

[@technovangelist]

absolutely. lots of folks are doing just that.

[@briannezhad1804]

@@technovangelistWow, that is amazing. I would appreciate it if you could provide documents that would guide me in deploying a model for production use and "Function Calling." Ollama is an excellent tool for a startup to keep costs down and avoid OpenAI usage costs.

[@briannezhad1804]

It also gives us flexibility to keep our data in-house.

[@briannezhad1804]

​@@technovangelist This is awesome!. Do you have any reference deploy successfully into prod? We are trying to avoid OpenAI and are looking for open-source AI models with "Function calling."

[@technovangelist]

all the docs in the github repo. but it’s a pretty simple app without many dependencies. I don't know of any guidance though.

[@viniciussilvano4177]

Please, do more compatibility with GPUs. I have rx580. My processor is crying hehe

[@technovangelist]

That’s a request for AMD to add support to those older lower end cards I think.

[@viniciussilvano4177]

@@technovangelist Is there any way I can use a library that allows me to do this. Or is it actually something that depends on AMD? I'm really impressed with what using Ollama as an API has added to my projects. I would like to find some way to speed up processing without having to spend money, at least for now.

[@technovangelist]

But amd support requires a certain level of the drivers which amd only has working for newer cards. I think the only option is to buy a more recent card. The 580 is 5 years old.

[@florentflote]

[@sergey_a]

thanks for the informative video. some examples should be displayed in a video, rather than spoken, for example, to show how to use environment variables

[@technovangelist]

There are a number of videos pointed out throughout the video that provide all the examples

[@robert_kotula]

Booted up Ollama with the llama2 model and my M1 MBP just froze 💀

[@technovangelist]

That is bizarre…as in you would be the first person that has happened to. Running macOS I assume. Installed using the installer? What else was running? So you installed and then opened a terminal and run ollama run llama2 and then nothing? Probably easiest to solve on the discord.

[@robert_kotula]

@@technovangelist I’ll join the discord channel and try to troubleshoot. I’ve had a couple of tabs open in Safari and one tab in Firefox Developer addition, nothing else. Will need to dig into the performance stats on the laptop.

[@mcawesome4150]

you should have more views and subscribers

[@technovangelist]

Thanks. Both are accelerating quickly. But feel free to share. I like to say I am working on my first million subscribers. Only 985,000 short.

[@jyashi1]

First

[@technovangelist]

First what.

[@technovangelist]

About 6 hours late to be first comment

[@rude_people_die_young]

My model file refuses to create an awkward silence at the end of its output 😡