Running Uncensored and Open Source LLMs on Your Local Machine

General AI models, like ChatGPT, Claude AI, or Gemini, have a broader scope, and their answers to questions can be correspondingly imaginative. But many vendors don't want to be held responsible for awkward answers, so they add "guard rails" to limit the responses. Those limitations often restrict the models so much as to make them unable to answer reasonable questions.
In this talk, we'll discuss the Ollama system, which allows you to download and run open-source models on your local hardware. That means you can try out so-called "uncensored" models, with limited guard rails. What’s more, because everything is running locally, no private or proprietary information is shared over the Internet. Ollama also exposes the models through a tiny web server, so you can access the service programmatically.
We'll look at how to do all of that, and how to use the newest Java features, like sealed interfaces, records, and pattern matching, to access AI models on your own hardware.
Speaker: Kenneth Kousen
Ken Kousen is a Java Champion, JavaOne Rock Star, and Grails Rock Star. He is the author of the Pragmatic Library books Mockito Made Clear and Help Your Boss Help You, the O'Reilly books Kotlin Cookbook, Modern Java Recipes, and Gradle Recipes for Android, and the Manning book Making Java Groovy. He’s a regular speaker on the No Fluff, Just Stuff conference tour, and has given presentations at conferences around the world. Beyond this, he is also the creator of the Tales from the jar side newsletter and companion KZbin channel.
His academic background includes BS degrees in Mechanical Engineering and Mathematics from MIT, an MA and PhD in Aerospace Engineering from Princeton, and an MS in Computer Science from RPI. He is currently serving as the president of the Connecticut-based Kousen IT, Inc.
Uncensored Java AI Google Slides - docs.google.com/presentation/...
#programming #machinelearning #llms #java #intellijidea
Join us:
Website: jb.gg/website
Blog: jb.gg/blog
Twitter: / intellijidea
Facebook: / intellijidea
Download IntelliJ IDEA: jb.gg/download-intellij-idea
Top 15 IntelliJ IDEA shortcuts: jb.gg/Top15IntelliJIDEAShortcuts

[@michaelrios4163]

This guy is hilarious! Looking forward to finding more content from him :D

[@AlexanderAk]

Main point: Intellij still can't use local LLMs as AI assistant

[@benjaminfraeyman]

Interesting topic 😊

[@emudojo]

what might be cheaper, running something like ollama on a private GPU instance, or just pay for one with an endpoint already, specially if we also want to store conversation history and having custom information for the model, like in a chat agent withcompany context

[@AlexanderEngler]

I'm looking forward to APUs having beefier "AI Engines". Having a small, light und energy-saving machine somewhere in the network that is running the LLMs would be perfect. RAM is also way cheaper for APUs than buying a bigger GPU with more vRAM...and you can extend the RAM size by just buying bigger or more sticks. Let's hope some small solutions will be released soon that are capable of what I described.

[@billwaterson9492]

You'd think (((big tech))) would be all over it. Where are muh competitive markets?