Using the same temperature and sampling settings. Give it a try ;)

:)

The iPhone 13 Pro, with 6GB RAM, can comfortably run smaller 3B models like Qwen 2.5/Llama 3.2/Gemma 2 2B. For best performance, we recommend devices with 8GB RAM, such as the iPhone 15 Pro or iPhone 16 to run larger models like Qwen 2.5 7B or Llama 3.1 8B. Local LLM inference is computationally intensive, so your device may get warm depending on usage. Models need to be downloaded separately after installing the app (app includes base model pre-installed). The app processes everything locally and models don’t have internet access

@PrivateLLM thanks for clarifying

Happy new year!

Sorry to hear you're having trouble. This could be happening for several reasons. Are you using a corporate network or VPN that might block Hugging Face? Also, what device are you using (Mac/iPhone/iPad)? Is your internet connection stable? Let us know these details so we can help troubleshoot the download issue.

@@PrivateLLM running on an iPhone SE 2020. Deleted it, redownloaded, uninstalled reinstalled did everything but it’s not working

@@czr5014 Running LLMs locally has its pros and cons. One of the cons is that if you have an older device, your experience is pretty limited in terms of the models you can use. The iPhone SE 2020 has 3GB of RAM, which restricts the models it can support effectively. For reference, devices with 6GB of RAM, such as the iPhone 13 Pro or newer, perform much better. On the latest models, like the iPhone 15 Pro with 8GB of RAM, you can even run larger models such as 7B or 8B with ease. That said, the base model that comes preloaded with the app should still work on your device. Could you confirm if that is functioning as expected? If not, please let me know, and we’ll investigate further to resolve the issue. If the app isn’t meeting your needs, you can find instructions for requesting a refund in our FAQ here: privatellm.app/refund. Refunds are processed directly through the App Store.

Also, could you confirm if your device has enough available storage?

@ your FAQ pages doesn’t include your hardware requirements bullet. Also, 3 gb is not big enough to run llama even though it’s like half a gig?

Thanks for the feedback. We will be uploading more videos soon :)

Uploaded a new video of uncensored Llama 3.3 running on on M4 Max Mac: kzbin.info/www/bejne/rancXn2dlJd3sLM

Sorry to hear about the trouble! It sounds like the crash may be related to memory limitations. Local AI processing can be memory-intensive, especially with models like Gemma2 2B. Could you please try closing any memory-intensive background apps and give it another shot? You can also check your device’s available memory at the bottom of the Help screen within the app. Alternatively, you might want to try the Llama 3.2 1B model, which is lighter on memory and could work more smoothly on your device. Let us know if the issue persists, and we’ll be happy to assist further!

Yes maybe thats the problem. I noticed that my phone feells burning using that model. Maybe consider optimize or delete that model until it can be fixed

​@@SimplemanOfficial-u7u Thanks for the feedback! You raise a good point about device heating. While some users prefer having access to larger models like Gemma2 2B despite the performance trade-offs, we understand the concerns about device stress. For a smoother experience on the iPhone 13, we recommend trying our lighter models like Llama 3.2 1B or Qwen 2.5 0.5B. These are specifically optimized for older devices like yours and should run much cooler.

@@tak4272 We’re working on adding an HTTP API. We’ve always supported extension through macOS shortcuts which llama.cpp wrappers lack. Also, Ollama has additional features that we’ll never be able match: Slow inference, and low quality RTN quantized models.

You can check this out: privatellm.app/blog/run-local-gpt-on-ios-complete-guide The article is slightly dated, and we need to revise it with new model recommendations. We will do it soon.

Either one should be fine, as long as you have 48GB of RAM. This demo was run on an M2 Max Mac Studio, but an M4 Mac Mini should work as well.

@PrivateLLM 2 grand for a 48 GB system , I would be interested in base model I wonder if it could run faster on newest llama

@@christiancrow Base model with 16Gb of RAM can easily run Gemma 2 9B, Llama 3.1 8B, Qwen 2.5 14B. Checkout the model list on our website for a full list of models along with RAM requirements. privatellm.app/en#models

You are so welcome

Not really

privatellm.app

The problem is that they are limited by the amount of parameters the model has, the higher the more “intelligent and capable” model when it comes to training. But the higher the parameter is the more resources it needs since LLMs are resource hungry, especially when it comes to RAM storage.

Please try again. We've added newer models. Llama 3.2 1B and Qwen 2.5 1.5B should work well even on older devices.

It's on our roadmap, yes!

We use an auto-tuning and compilation based approach from mlc-llm and Apache TVM for LLM inference. This means that inference pipeline is optimized to extract the best possible performance from the underlying hardware for each model architecture.

We'll support it if you manage to get macOS Ventura running on it. :)

and next it can wipe your lazy ass too?

Have you tried using it along with Text to Speech action in Apple Shortcut?

You can learn more about temperature and top-p settings in this article: privatellm.app/blog/run-local-gpt-on-ios-complete-guide

iPad 9th Gen has 3 GB RAM, which means you will be limited to small models like Llama 3.2 1B or Qwen 2.5 1.5B

The difference in functionality between iOS and macOS regarding background processing stems primarily from Apple's hardware usage policies. On iOS, Apple restricts background execution of tasks that require intensive GPU usage. This limitation is enforced to preserve battery life and maintain system performance. According to Apple's guidelines, apps attempting to run a Metal kernel in the background will be terminated immediately to prevent unauthorized resource use. For Private LLM, while we can run operations in the background on macOS leveraging the GPU, iOS versions are constrained to CPU processing when the app is not in the foreground. Running Private LLM's AI-driven tasks on the CPU is technically possible, but it would be significantly slower-more than 10 times slower compared to GPU processing. This slow performance would not provide the seamless, efficient user experience we strive for. We are hopeful that future updates to iOS might offer more flexibility in how background processes can utilize system resources, including potential GPU access for apps like Private LLM. Until then, we continue to optimize our iOS app within the current constraints to ensure you get the best possible performance without compromising the health of your device or the efficiency of your applications. For more technical details, you can refer to Apple's official documentation on preparing your Metal app to run in the background: developer.apple.com/documentation/metal/gpu_devices_and_work_submission/preparing_your_metal_app_to_run_in_the_background

Merci ! Je suis ravi que vous aimiez notre application, je pense que nous pouvons le faire. Nous travaillons actuellement à ajouter plus de modèles au logiciel pour macOS. En particulier, un nouveau modèle de 7B de la startup française Mistral et quelques modèles supplémentaires de 13B. De plus, essayez notre application iOS, qui est téléchargeable gratuitement depuis que vous avez déjà l'application pour macOS.

Corrected the typo-it's iPhone 14 Pro Max. Thank you for bringing this to our attention.

Private LLM