I Ran ChatGPT on a Raspberry Pi Locally!

[@Joooooooooooosh]

Except you didn't.

[@rheymanda1074]

This is like having the title of your video being "How to install Windows 11 on Raspberry Pi" and then proceeding to explain how to install Raspberry Pi OS

[@szmonszmon]

So, you don't know what you run? "I ran ChatGPT". I skipped some parts of video to see what you really have and I saw llama and alpaca. I was really curious where did you found the ChatGPT source code... No, llama, alpaca and others are not the same as ChatGPT. They don't understand other languages than english and they have issues with other code languages than python. So, im some circumstances they are similar to ChatGPT, but only in certain use cases...

[@とふこ]

Of course, you can't run gpt3 locally, only smaller models

[@aaronjennings8385]

Still useful?

[@poogle9368]

I know its dumb because its still an impressive feat that a generative ai can run on that cheap of a computer so why lie? Well we know why.

[@TheRealUsername]

But better models like Mistral 7b gives almost the same performance as ChatGPT for the same efficiency as Llama or Alpaca, it is difficult to see the difference with ChatGPT when you run an efficient high ranked model from Huggingface leaderboard.

[@blacksailstudio]

I quite assure you that there are local LLMs you can find on hugging face that understand other languages including German

[@Vypertech]

2040: i created a new universe using phone parts

[@PovertyHelping]

Please teach how. 💐✨

[@Vypertech]

@BigiyePhilipo no

[@aaronjennings8385]

Quantization, in plain English, is a process of representing something in a simplified or discrete form. It involves reducing the complexity or precision of something to make it easier to work with or understand. Think of it like taking a detailed painting and converting it into a pixelated image. Instead of having many different shades and colors, the pixelated image uses a limited number of colors or pixels to represent the overall image. This simplification makes it easier to store, transmit, or process the image. In the context of data or numbers, quantization involves reducing the number of possible values or levels that can be used to represent a measurement or a quantity. For example, instead of representing a measurement with infinite decimal places, quantization rounds it to a specific level of precision, such as rounding a decimal to the nearest whole number or a certain number of decimal places. Quantization is commonly used in various fields, including digital signal processing, image and video compression, and data storage. It allows for more efficient use of resources, faster computations, and simpler representations, while still preserving the essential information or characteristics of the original data.

[@WilliamCamp-n9x]

AI wrote this

[@Anne_one]

Chatgpt ah response

[@longpool-u5s]

So true. Many models have heavy requirement to run, like 16 GB of RAM, but depending on your use case you can get away with a lot less. I got surprising results using a vector database and Llama 2 even with 8 GB of RAM and 4 CPUs. In Supawiki (disclosure: built by me) I am using a bit more than that, and the results are impressive. Exciting stuff indeed.

[@fredygerman_]

did you run that without a GPU?

[@longpool-u5s]

Yes, without GPUs. Ollama can run entirely on CPUs. It uses all you got and it is a bit slow, but works.@@fredygerman_

[@heian17]

​@@fredygerman_ You can run Llama 2 without GPU, by using llama.cpp

[@Here_is_Waldo]

​​ @fredygerman_ If you use something like KoboldCPP you can certainly run an LLM without a GPU, but it's very slow at a few minutes wait time per sentence. If you have say 6gb of VRAM then you can usually run a 7B-Q4 LLM reasonably quickly at about typing speed. Keep in mind that you do want proper Video RAM on a dedicated GPU, not CPU/System RAM. Even an old nVidia 1070 would work reasonably well. (NVidia GPUs are far better than Radeon at the moment for running LLMs.)

[@DiscontinuedRBASIDOWK]

Great video, a little misleading to call it ChatGPT considering the power of ChatGPT compared to this much smaller model but still a great video. Well done.

[@DataSlayerMedia]

Fair enough! But Llama is competitive with GPT 3.5!

[@zappy9880]

@@DataSlayerMedialol no its not. you used a 7b model on top of that you used llama 1 and not llama 2. right now, the only model comparable to gpt 3.5 is falcon 180b and even then it still falls behind gpt in terms of coding capabilities.

[@stevewall7044]

​@@zappy9880dolphin 7b 2.2 is pretty good.

[@abhiprojectz2995]

​@@zappy9880Obviously he did that for lots of views, don't you understand ?

Guy just using a little clickbait to kickstart his channel. No shame in that... well, maybe just a smidge.

[@3DComputing]

Thank you, I knew it was small I didnt realise just how small. 10/10 short sweet concise.

[@StephenBrown88]

Your passion for this stuff is magnetic!

[@wood6454]

That is impressive. I'm able to run 7B Q6 models on my old pc with an RX 580 and small language models like Phi 2 runs faster that I can read. I believe the future of LLMs is gonna be local instead of cloud due to privacy as you said.

[@PovertyHelping]

Thanks so much for this favorite opportunities. We really loving your online classes dear. 💖🏭

[@MorrWorm8]

Yo, I took a Pi 4 (8gb) with argon 40 case that has the m2 put a (1TB) SSD added Ubuntu and I love it! Loads fast responsive. I have my M1 MacBook, my Mac mini & a older HP running windows. Now I have a Linux desktop. I know I could have ran a VM. I enjoy bare metal. Great video. Liked & subscribed

[@UncleDavid]

telling us ur life story is crazy

[@OblivifrekTV]

Wonder if it would work better with a Raspberry Pi Cluster

[@braveonder]

For sure. Use your modern laptop with ubuntu thats all. But use at least 8 gig ram

[@gn7026]

I'm eagerly waiting to see the video on running the model on the Jetson Nano!

[@Atmatan]

Would've been where I started, personally..

[@polloman15]

I'm the only one that feels a little bit of nostalgia realizing that the world where I grew up in is already gone? When my parents were young, they had these room sized computers. My mother used to be a typewriter secretary. My father used to be a mechanic back when cars were carbureted at around high school. When I was a child, maybe 4 years old, I remember my father had a thiccc IBM laptop from work. Our first digital camera had only 256MB of memory. Today we're running AI models in a computer a little bit bigger than a wallet. I can only imagine what is waiting for us in a couple of years. Life's good :)

[@Here_is_Waldo]

This must have been the way people felt watching the car replace the horse and carriage. Our childhood world hasn't gone, it's just in a museum now.

[@DisgruntledPigumon]

Your parents had room sized computers when they were young? 🤔

[@swingtag1041]

Love the background music. What is it?

[@Diogenes20111]

Thank you so much for this educational video!

[@georgeshafik3281]

Great video, simple steps to follow everything worked the first time. It was slow and I used identical hardware to yours. Really interested in using a larger Lama model with Nvidia Jetson👍

[@whatisrokosbasilisk80]

Use smallers models then

[@alexiscolonfpv3534]

hi i have a error on my PI main: seed = 1712111644 llama_model_load: loading model from 'ggml-alpaca-7b-q4.bin' - please wait ... llama_model_load: ggml ctx size = 6065.34 MB Segmentation fault can you help thanks

[@Matt-es1wn]

You wont even need a computer, you won't even need electricity

[@onghiem]

can you integrate USB Coral AI accelerator to make this RPI faster, or could you run this on a PI cluster?

[@davidtorrens8647]

Yes please me too like to know that.

[@WebTable]

Looks like it needs more learning. Lots of answers seem to be wrong and relationships between values are also wrong. It listed Iceland's total area as 30,759 sq km (12,286 sq mi). First, I think Icelanders would sad to hear that they lost about 70,000 sq km but the conversion between sq km and sq mi is not even right (off by a little). The problem gets much worse with greenland whose total area is apparently 836,319 sq mi (217,090 sq km). The actual area of greenland is about 2.1 million sq km but even with an incorrect answer, the sq km value is smaller than the sq mi value. Apparently not trained well in D&D but it did give me an accurate synopsis of the MacGyver tv show.

[@DontTreadOnMyLiberty]

Could this be used to be trained to search a local pdf library? I have seen people make cyberdecks with Wikipedia and other preparedness related PDF documents. It would be incredible to not have to read a whole document, but rather put a question into a chat box and it search for specific information from said PDF libraries!

[@abzs5811]

He’s living lavish 🤙🏽

[@jacquesb5248]

pondering training my own model

[@legend_6483]

Nice tutorial it works perfectly on my pi

[@DataSlayerMedia]

Any interesting conversations? 👀

[@legend_6483]

@DataSlayerMedia lol, not really since the cpu was on fire, and the speed was very slow, but I liked the concept

[@rawsomeone1]

@@legend_6483 🤣😂😅

[@NicksonNg]

I tried running on a pi 5,and its still not very usable even tho theres performance boost

[@armisis]

Hmmm we need a way to cluster this in a parellel processing raspberry pi 4 and 5 cluster.

[@scottbruce5376]

I worked through this on ESXi 8 Ubuntu VM and had no problems. What's the next step? Web Interface? I have a docker that connects to LLA online, would I setup API next to connect to next?

[@mrguiltyfool]

how do you get tts to read out the chatbot output

[@vishwanathasardeshpande7092]

What happens when you add a hailo chip will it boost the performance?

[@Mauroplcr]

Hi, nice tuto, but dosent work to i have only 4gb of ram llama_model_load: ggml ctx size = 6065.34 MB Segmentation fault (core dumped)

[@BogdanOlteanu-profile]

I don't know what's scarier. The fact that if you give the community a finger they will take the entire hand and they do their best to optimize what others couldn't and run an AI model on a board like raspberry pi or the fact that it's possible :D

[@OVERLOARD949494]

I am using big LLMs on a bunch of Tesla p40s but since the cooling options are pretty loud and it consumes a lot of energy I wonder if I get better inference with a coralAI TPU on a raspberry pi than using llms on the pi without anything else. Also, would it make sense to build a pi cluster, each fitted with a coral ai tpu via pcie port?

[@monisatan]

It doesnt seem logical for now, but soon we will be able to do things close to what we want.

[@Michael-iw3ek]

Do you carry a boombox with you wherever you go so that you could play stupid background music whenever you talk?

[@SeriouslyNonya]

Would this run more cleanly on a stronger device with lots more RAM available, or is it more limited by the base model? for example, if I ran your pirate speak example with the same setup on, say a dual xeon server with 256 GB of RAM and ssd raid, would it have a chance to actually perform more properly?

[@ScorgeRudess]

Im not a hater, actually this is pretty awesome, but you are not running ChatGPT... its a very very very quantized model of a LLM but, its very awesome to run it on a rb pi

[@matteoricci9129]

Fish baiting to the max

[@sh4dyweather11]

I’m wanting to do this with a pi 5 and add a screen/camera and give him some wheel so journey with me. You should try it as well maybe on one of those racing drones, that would be sickkk.

[@getzybaggins]

at the end, do you mean vram or virtualized memory? awesome video trying this later

[@JinKee]

make it pass the butter

[@wagnergriffin670]

How come when I run the chat file I get a permission denied. I tried “sudo chmod +x chat” and then ran it but still have the same problem

[@_iseeyou_luca7529]

Does it work on 32 Bit Systems?

[@نفيس_الشيخ]

It's work without internet?

[@gerardniks3636]

only 50gb vram for gpt4? where am I going wrong, I'm making a smaller model and I'm already at 1tb ram

[@khangvutien2538]

Great video. Thanks. Is it on purpose that you use the word “ChatGPT” in the title ? I don’t know whether “ChatGPT” trademarked but it seems to me that you didn’t use the OpenAI LLM, but LLaMA 2?

[@roryleitner1532]

How can I train a chat ai on a specific very large body of text from a person from the past to bring them back to life? What would the possibilities of that chat be like?

[@oglothenerd]

OMG!!! Facebook actually did something good for open source!?

[@queerzard]

If i download an LLM and run it offline on my raspi.. Does that mean I have most the worlds knowledge packed into 4GB always accessible?

[@DataSlayerMedia]

Yes, you would have the broad strokes with probably some inaccuracies. But this isn’t really remarkable considering Wikipedia (the text) is ~10 Gigabytes.

[@OriginalAceXD]

now my question is can i run it on Asus Tinker board RK3288

[@nihilsaboo6142]

Will the performance be boosted with a Google Coral Accelerator?

[@brianbecking1]

I would also love to know this.

[@shiftednrifted]

did you run chatgpt? or did you run one of them broke ass local llms that lose the thread on conversations close to immediately, run out of tokens way too fast to be useful for most workloads, take forever to infer even on the highest end consumer hardware, and otherwise don't even slightly compare to chatgpt. its cool you ran it on a raspberry pi, but it is NOT comparable.

[@sweettoko995]

Why cant people just take a chill pill and appreciate what this guy did, hes done more than most of us ever would.

[@drealph90]

That's a dick move trying to charge money for the text version of the tutorial

[@agustinbmed]

I’m wondering if you can use it to train say your files stored in a hdd and let it do its gpt part on it? Like ask it if you have a document that has x or y content

[@chrisarmstrong2721]

Fantastic, when do you think it will be able to also do images like the latest update so gpt4 now natively pulls from Dalle!

[@sqribe]

anyone else getting a segmentation fault(core dumped) error? (i am running the raspi4 4gb.) everything compiled w/out error, but when i run ./chat i get the segmentation fault (core dumped)

[@dalivanwyngarden3204]

The download link in your bio is not working anymore unfortunately. Can you provide a new one?

[@Tom_Neverwinter]

hmm, I wonder how some of the blokes models would do

[@indieartsmidwest4042]

I'm so close but ran into a segmentation fault while trying to run the program🤷‍♂

[@gasmonkey1000]

Silly question but would the similar method also work with other gpts like GPT 4chan? Thanks and agod bless ya

[@DarthCrumbie]

Would it be possible to use this or similar setup to replace Google Home or Amazon Echo? Ever since the story broke about the person that got their amazon account suspended by an amazon delivery driver I've wanted to find a way to isolate my smart home from the internet.

[@DataSlayerMedia]

Should be! Look in ESPHome, there’s a whole community around this.

[@xevilstar]

did you know that you can install the system directly on the ssd and boot from usb ? I use nvme disks on my pi puppies :no ssd card needed :)

[@mandelafoggie9359]

If the LLM could connect to the internet, may get better responses.

[@1234kdy]

Could you make one on a zima board with a GPU in it's PCI port Maybe using a GPT build from hugging face? or better?

[@seanwieland9763]

1:20 AI wrote sentences saying that AI is the future. Not today, Skynet!

[@GOKULAPPADURAIK]

Thank you so much and its working, i have tried my kind of stuffs for getting this output but no repo made me this much easy and working properly, the prompting was very smooth, and the tokens producing was very low .... i dont know why

[@7reflection7]

How well can it handle python coding?

[@MisiSzucs]

Good video, but the title is misleading. A Llama-1-7b with only just 4 bit is really far from ChatGPT. ChatGPT has 175b parameters compared to 7b parameters in 4 bit. I would say ChatGPT passible outperforms this local Llm by 200-500% in every task.

[@slightlyarrogant]

You could build it on Coral dev board it would probably be faster and cheap as well

[@darthwater999]

>trusting google

[@thatguy1306]

How did you get sound with the text output of the program

[@Muffiz_]

it was edited in, ssh dusnt transfer sound

[@MrPenguino76]

@@Muffiz_ What TTS did you use for the voce reading?

[@JLXT7]

Can i use this in a Pi cluster?

[@JarppaGuru]

yes if its build for cluster its not magic will raspberrypi desktop work on cluster. no. bcoz its not build for cluster. think 100 RPI and run desktop and its faster than your desktop pc if you make python script ans make it work cluster. then that script work on cluster. all beta testing make script work. when its done you allready know answer. you not script anymore hahaha

[@zekeriyaatilgan521]

Does it support different languages? Or is it just English?

[@corey_deroche]

Clickbait, Chat GPT does not run locally and a Raspberry Pi is not even close to being capable of supporting it if it could.

[@DeonCarr-d4b]

Subbed. That was great.

[@DataSlayerMedia]

Welcome aboard!

[@Jim_the_Hermit]

I'll wait for voice recognition Chat GPT on a chip

[@phillangstrom8693]

I don't see the point of a low power AI that's offline whats the point if it Don't know how to answer a question or be able to find the info to help with coding since that's all I ever use AI for

[@vanhetgoor]

PIS is not the plural of Pi. A wise American Philosopher ones said: Don't eat the yellow snow!" Keep that in mind.

[@alexanderyang126]

Hello Elon, I think this project could be a useful tool for families, like a mini Wikipedia. Would it be possible to add an audio function based on the work that has already been done? I mean, speak directly to the Pi and give the answer back.

[@tschmidhuber]

And where exactly is now ChatGPT running on your Raspberry Pi?

[@chnebleluzern]

will it finally tell you how to build illegal stuff

[@W00ge]

can you please stop randomly flashing words on the screen. we can hear you

[@sweettoko995]

Its for emphasis, thats alll

[@Atmatan]

Cute proof but ultimately rather meaningless. Everyone knew it was only a matter of time until the models were quantized for lower end hardware. The real miracle is in maintaining benchmark performance and speeds on low end hardware. Good luck.

[@Blooper1980]

Sssoooooo... Not GPT!

[@OVERLOARD949494]

50gb vram? I have 60.

[@braveonder]

install linux in a laptop and run model as a server without internet. Nice video. Thank you.

[@augustuslxiii]

Why would that scare me? It's a chatbot.

[@ldandco]

Clickbait announcement: Thats not ChatGPT

[@godofdream9112]

now we are talking..

[@thisisthanish]

The thing is this guy is running this on raspberry pi 4 and the new raspberry pi 5 is 2.5x times faster just think how fast the ai would be

[@YannMetalhead]

Great video.

[@makoado6010]

its compared to openia liek ford t model compared to tesla plaid.

[@jan5504]

nice been planning to build my personal ai for linux commands so I don't have to memorize all of it.

[@pablocarballeira4443]

You are running in a pi with 8gb,try to run in 2gb and no luck

[@hand-eye4517]

its not very accurate to say "ubuntu is more tried and true" when ubuntu is just riced debian with skethcy cannonical flower touches

[@matteoricci9129]

Dude you are using alpaca, not chat gpt there is a difference

[@clipbastler]

The single most significant innovation in history is certainly NOT the internet. AI very much aligned with contemporary human overconfidence...

[@supercurioTube]

Thumb down for clickbait with an obvious lie in the current title: "I Ran ChatGPT on a Raspberry Pi Locally!"

[@vorant94]

Talking about privacy and using warp terminal… ok

[@matteoricci9129]

And you are cutting waiting time

[@Cyber_Gas]

Fedora is better than ubuntu