Run a Local, Private LLM By Downloading Just 1 File - ChatGPT-like Bot on Your PC!

Llamafile lets you run a large language model (LLM) from a single file. It makes LLMs more accessible to both developers and end users. It runs on six OSes on ARM64, AMD64, and Intel x86-64. Just download one file and give it a try!
---
Llamafile - github.com/Moz...
Twitter: / garyexplains
Instagram: / garyexplains
#garyexplains #llamafile

[@bazoo513]

It is utterly amazing how much of a model can fit in 4GB, together with multi-platform runtime! Excellent work, Llama people, and thanks, Gary!

[@GregRutkowski]

Your presentation is the first, I have ever watch, where the chat worked as stated.Many others present chat app's, but fail in the detail of model types, e.g. I run Windows 10 and Ubuntu-Mate. Which by tthe way is the system I'm using to post this comment.

[@pbdivyesh]

Thanks so much, I will run this on my windows machine and apple macbook! Just waiting for Snapdragon X Elite based windows computers too :)

[@natjes6017]

This was very cool! I´m gonna test this on my M1 Pro 😁 Thanx a bunch!

[@GaryExplains]

Have fun!

[@leledumbo]

I just heard about this cosmopolitan in a programming forum, it's quite interesting for a true cross platform executable format, similar to a fat binary but different.

[@FlorinArjocu]

A real hit would be to have something like this and then give it access to the internet, so it could search and get up to date information.

[@mandelafoggie9359]

Great idea

[@jameswhitaker4357]

Representation Fine Tuning ReFt model would work here, idk about implementation tho. There’s probably a GitHub repo out there with this implementation already. Thanks for the lightbulb 💡!

[@edwardglynn3197]

Amazingly easy! HP Chromebook dragonfly in Linux with no issues. Just worked! First thing that EVER just worked in Linux on Chromebook

[@JoelJosephReji]

Pretty cool!

[@asamirid]

neat impressive project i see .. thank you for the instructions and the demo.

[@mohammedakbarg6843]

Thanks Gary for sharing this with us!!! 😀 Really cool

[@user-xy5jl4df3u]

Hello, is it possible to use Llamafile to "chat" with files (pdf, doc, etc) in a local directory? Thanks.

[@BillHawkins0318]

I do hope someone answers your question soon

[@ArianeQube]

Any advantage to this than just installing LMStudio and running essentially any model on the huggingface repository?

[@TechMeldOfficial]

I am currently downloading it, yet to test it !

[@cjacobm]

Excellent video! Based on this local installation, is it possible to make API requests, in the same way as chatGPT?

[@GaryExplains]

Yes, absolutely. Best option is to use LMstudio. I have a video about that. It has a built-in server.

[@cjacobm]

@@GaryExplains Thanks! I'll look for the video.

[@hilltophorrormoviereviews9628]

This looks really interesting and I'm gonna try it out. I have two basic questions: 1. Is there such a thing as a context window on locally installed LMMs and if so what is this one's? 2. Can you upload a file (txt, etc) to it that it can read?

[@antonydecosta6262]

Thanks Gary..just tried and it works pretty well. Is there a way to train this Llm further?

[@hamesparde9888]

The training stage is generally much more computationally intensive than just using the model that results from the training.

[@skrollreaper]

@@hamesparde9888 lol nice answer mate. the short answer is yes though

[@Dhruv_s12]

Anyone please give me simple instructions that how to use this LLM with GPU as well.

[@GaryExplains]

The details are on repo page. Basically you need Visual Studio installed and the CUDA SDK. If that is something beyond your skill set then try LMstudio instead.

[@FilmFactry]

What is the use case? Why use this than Bing/Bard/Claude? What can were we have more control?

[@GaryExplains]

As I often say to people, with the highest respect, if you can't see the use case then it isn't for you. This is true of any tech and equally true for me.

[@stevejones4010]

How to make it learn, or learn mode?

[@PaulSpades]

Well, I've tried it. It's rather stupid (not that much you can cram into a 4gb model), but it runs fine and seems to give reasonable responses to anything not too technical or programing related. What's the source of the training data? I'm not at all familiar with these open source llm projects.

[@r0galik]

Interesting, I wonder about running it (natively) on Android - which is Linux under the hood...

[@mohammedakbarg6843]

Waiting for your next video about Open Interpreter and this one file local chat GPT integration 🙂. How come you didn't talk about that possibility (which I think will be happening next) in this video 🤔

[@iamseyi4real]

Can you run it on Android tablet??

[@hanspeter24]

maybe when rooted

[@iamseyi4real]

@@hanspeter24 I thought as much, I already have a rooted android device. Just wondering how to go about it

[@12polizei24wegvonhier]

Easy to install but non responsive on a raspi 5 with latest raspi OS. I probably need a Google GPU to make it more practical or be more patient

[@send2gl]

Guessing then with a suitable link it could be incorporated into a web page, further guessing there are better ways of doing that than a 4GB file though.

[@DamianMontero]

Tried it on a 2020 Dell XPS 13 running ubuntu 23.10 (not WSL) with a i7-1185G7 with no Nvideo or AMD GPU and it was easy to run (just chmod and we're done) and it ran and replies as fast as ChatGPT (almost instantly) and seems to work just fine. It was SUPER easy and seems a LOT faster than ollama's default LLM and much easier to use. I don't mind ollama and with it's API I can do a lot with it,and this of course has an API (that's what the HTML front end hits) and so I have to try this on my current project. Amazing find! Thank you @garyexplains

[@test40323]

Huh, very interesting. What are the limitations offline? On another topic, has anyone apply LLM to detect computer viruses? I wonder how well it will work on reading xray images to detect cancers?

[@GaryExplains]

What do you mean by limitations offline, exactly? It functions the same offline and offline because it doesn't use any online resources at all.

[@test40323]

@@GaryExplains , not using online resource is even more interesting as you can ask it anything.

[@GaryExplains]

Not quite, that depends on how the model was trained.

[@Riggzilla]

Is there a guide to run the llamafile with your GPU? I'm using GPU passthrough on a VM.

[@mikebakkeyt]

Impressed it works but oh my - how confidently wrong it is.... I gave it a picture of a Vulcan from a 60's airshow and it told me it was a restored WWII aircraft with propellors and that there were 14 people around it. In actual fact there were approaching 100 in view but it gave an absolute and definitive number. I also gave it a picture of Chloe Grace Moritz and it assured me the her name was Emma Watson. The issue isn't that incorrect info is provided (I didn't expect it to even try tbh) but that is is presented as a fact. As this train isn't going to get stopped, I just hope it drastically improves how it presents what it does..

[@GaryExplains]

What you describe is a common complaint about every LLM, not just LLaVA.

[@GaryExplains]

PS. The nature of LLMs mean they have no concept of correct or incorrect. In fact they have no concept of anything, just what letters (tokens) should be output based on the tokens in the input.

[@Makkenhoff]

Going to give it a try; interested in how it works.

[@kiiikoooPT]

Anyone knows if it runs on android with termux? lamacpp has a make option for android so it would be nice to be able to run it on android, I still didn't get into compiling for android with android studio, but in the lamacpp repo it says you can, so I wonder if they also made that binary for android in this file. If anyone know the answer please say something, Otherwise I will download and test it ;) Thanks in advance.

[@GaryExplains]

Try Sherpa, I haven't personally, but it is in Google Play. Repo here github.com/Bip-Rep/sherpa

[@GaryExplains]

BTW, it took me about 30 seconds to find it by googling "llama.cpp apk"

[@kiiikoooPT]

Thanks for the reply, but it does not work, only with bin files, no gguf support, and I was wondering about this one, not another one where you need to configure and download models anyway, but thanks ;) @@GaryExplains

[@BillHawkins0318]

I have gigantic documents I would love to train this thing on

[@BrendonHolt]

Well, not fully working for me. The web formatting is imbedded in the bot's reply.... User: hello Llama: sections=1&replytouser=1&template=hi\_there&style=2048&theme=default

[@arsalanganjeh198]

Can I use my GPU to run it?

[@samfisher3336]

So we use cloud-based streaming services to listen to music but we run LLM locally!

[@GaryExplains]

Obviously there is a big difference between those two. Everything you type and all the images you upload and all the data you feed to a cloud based LLM gets sent from your computer to a company like Open AI/Google/Microsoft. For music you just pick a song.

[@samfisher3336]

@@GaryExplains We gave up our privacy the moment we bought a smart device, even With the most primitive NPU. We're exposed and we got to live with that.

[@GaryExplains]

I disagree. I have plenty of data that isn't in the cloud and isn't exposed to the cloud. It is important to control what is in the cloud and what isn't. This is another tool for doing that. There are other reasons for using a local LLM, like research etc, but this was the one I picked due to your comparison with streaming music.

[@zipnone1996]

neat

[@J-manoo7]

Can I just download this on my raspberry pi ?

[@GaryExplains]

Yes

[@MatthijsVanDijck]

Is this fully offline? Or does it search for information on the Internet?

[@GaryExplains]

Fully offline.

[@felipe367]

Who are the people behind this llm ? Open ai?

[@xeon2k8]

it's written in the github link

[@GaryExplains]

No not Open AI, this is an open source LLM, see llava-vl.github.io/

[@FlorinArjocu]

OpenAI is not "open" in the normal "open source" meaning. It is a company like any other.

[@PaulSpades]

@@FlorinArjocu correction: they used to be a non-profit opensource software organization. and then they flipped to a for profit company. surprisingly, this is happening more and more these past years (opensource projects turning commercial) like mysql, couchdb, audacity.

[@felipe367]

@@FlorinArjocu correct ,they soon became “unopen”

[@NeutronStar9]

Can anyone suggest how to build an app for android using AI?

[@ahmedzedan9326]

Does this mean this is offline ?

[@GaryExplains]

Yes.

[@bipropaul5409]

Virus?

[@GaryExplains]

Bacteria?

[@xeon2k8]

the file: 5 petabytes xDD