Raspberry Pi 16Gb - what this means for running LLM - Deep Dive

[@algobit6011]

Hi everyone, greetings from Ottawa, Canada! I ran ollama from a docker container on my M1 macbookpro. I waited for up to a minute for responses, then the words came out at a snails pace. This seemed just wrong, as i’ve seen other people get better responses from a pi4. I installed ollama directly on the macbook, and wow! Almost instant response, words spew out faster than i can read as the text scrolls up the page. Bottom line: dont use docker. Set up a venv and install ollama directly in the virtual environment. and enjoy the results.

[@kevinmcaleer28]

I’ll give this a try and test the results

[@stephenvalente3296]

I think we've finally hit the price ceiling where sales will stall in the sense of buying the 16Gb version just because it's new. I was probably a couple of button presses from buying the board and the accessories this morning until I got another opinion from a Jeff Geerling video. Pretty much confirmed my doubts, and guess a lot of people would be better off buying an N100/N150 based mini PC over the Pi 5 16GB instead for the £140-ish it was going to cost me.

[@Larimuss]

Is it a dev board though? I’m not sure why anyone would buy a pi other than for programming and robotics, ai dev etc.

[@piuser-4-500-zero]

I'm on an 8GB pi 500 running phi3.5 in terminal and getting much faster responses for that question (around 30 sec). Not sure if the model's been tweaked or your container and web interface are slowing it there. Btw, the 'one sentence' is not always 'understood' by phi3.5, I often get two, three, or several sentences each with its own fact (not entirely surprised as I've not found phi to be very good, e.g. the tiny (0.8GB) granite3-moe model from IBM gives perfect responses to that question in around 2 seconds on the same pi 500 with 8GB)

[@garydwatson]

Or you could just install ollama directly and not have to waste the resources on a docker container (not to mention possibly being resource capped by the container runtime) I think the ollama install is a single command that they have on their website.

[@AleksSem1]

@garydwatson docker almost doesn't have overhead. Seems author doesn't configure docker properly (according docker compose file) to allow GPU using for ollama.

[@CyberDunk2077]

I never thought there would ever be a 16gb raspberry pi, I don’t need another pi 5, but I kind of want one of these.

[@nosuchthing8]

What they NEED is a raspberry pi 6 with 16gb of nvidia vram

[@ray-charc3131]

Sorry, i had Rpi5 8gb, I definitely won't spend that another amount of money to buy the 16gb one. I just spent around $20 to upgrade my old laptops ram to have 16gb, since it has an old nivida graphic card, it runs locally the small LLM models such as llama2, mistral..quite well, I give up to run them with pi.

[@EntenKoeniq-nz6wd]

i saw the 16GB version today and bought it ... tomorrow i own the 4gb, 8gb and 16gb for no reason ... i have a problem

[@PaulGrayUK]

There are fewer Raspberry models than Pokemon - you chose wisely.

[@jameslawrie7517]

Ha, got you beat I bought two for future projects just like the rest of my PI 1 thru 5...

[@wikwayer]

@EntenKoeniq-nz6wddepends what are you planning to do with IT?

[@Lp-ze1tg]

You are still missing a 2gb version. 😁

[@jameslawrie7517]

@EntenKoeniq-nz6wd naw, all my pre 4s are 1gig or 512mb.

[@PaulGrayUK]

Now they released 16GB, we now have to moan that there is no 32G model. 😁

[@garydwatson]

I know they'll never do this, but I would totally be into them making a more expensive computer that has more cores and more memory and uses a lower nanometer production process... The price point for such a computer would be more, but I would gladly pay it. The idea being a nice happy medium between something like a system 76 computer which last I checked are kinda spendy... And the current offerings by raspberry pi.. Not at all sure how many people like me there are in the world though. (to say it though I will definitely be picking up the 16 gig model)

[@Unrealizedlosses]

At this price, plus modules like the AI hat you're getting close to the new Nvidia Jetson Nano Orin, which is significantly more powerful. If I'm going hardcore AI, probably not a rpi project for me.

[@mysoundiicon8473]

for ollama on raspberry pi without the docker and also for using AI kit 13/26 you need to choose the right image in the shity Pi "APP" pick [raspberry pi OS Other] pick "Raspberry pi OS lite (64-bits) (compatible with 3/4/400/5) " so if you are having problems try that one... and it might just be me but updating time before apt-get update and all that can help " sudo date -s 'YYYY-MM-dd hh:mm:ss' " i hate it the apps so much it does not say what does what. and i wasted so much time trying to get things to work.. then i pick another OS image and it works right away..

[@Larimuss]

Why not test second prompt? Also I wouldn’t expect the speed to be any different at all to a 8GB model. Interesting test though. Maybe me realise that 16gb is a bit useless for Ai because even a 7b model will just be too slow. So might as was just use 2-3b guff models. But the actually inference test is after the model is loaded. I guess that would be for the AI chip expansion though alongside the 16gb.

[@Waveshare_Ruan]

Are everyone's expectations of 16gb generally that low?

[@NoxmilesDe]

Why? For what? This doesn't make an sense

[@JarppaGuru]

nothing. you need 2000 GPU xD