I strongly suggest investing in as much CPU RAM as you can. Yes, VRAM is important, but now that we can split between CPU and GPU, RAM has become way more important than just a year ago. If you really want to use large LLMs you'll want like 64GB+. With 48GB I don't think you're going to enjoy 70B models very much and 120B will be pretty slow if you can host them at all. On my machine that has 32GB RAM and 12GB VRAM, I feel that models in the 20B range are about as high as I can go without them being too slow for my taste. I didn't mention this in the video but I actually have two machines, the smaller one I mentioned and the bigger one with 64GB RAM and 16GB VRAM. Even the bigger machine doesn't like 120B models very well and 70B is about as big as I can stand on it. I don't have a tutorial for setting up local. I've considered it, and now that I've become familiar with a few different LLM backends I feel a bit more confident discussing the topic. You're not the first to ask for a guide on hosting LLMs and I feel a noob guide is long overdue. If you're interested in just my tutorials, use my tutorial playlist. My channel is a mix of things but I've split them into categories for people interested in just the technical discussions (which I completely understand).

It would take a long time to make a full video about it, but I did try it out today. I went back and forth between 2501 and 2409 for a while to try and get a direct comparison, and honestly I didn't see a huge difference. I felt like 2501 was sometimes a bit more descriptive. According to the UGI leaderboard, it's all a little better and it has a higher willingness score. I'm a bit surprised that you found it to be way worse. I think if I did a blind comparison I'd have a hard time picking out which one is which. Any issues I had like speaking out of turn or not adhering to the prompt were present in both models. Maybe some of your settings are making it act weird or there's a difference in the quantization quality? If the settings are the same between the two, I'd expect them to act similarly. I specifically tried the bartowski Mistral-Small-24B-Instruct-2501-Q6_K_L.gguf quant and the equivalent for 2409 if that matters.

I don't want my real voice out there, and I wouldn't even have this youtube channel if it weren't for voice cloning. It's me speaking into the microphone and transforming my voice with speech to speech, I've only used TTS quite recently in a couple videos, and for the most part I prefer to still narrate things myself.

lol I skipped over the clip where you mention that the first time I watched the video. Still, you should mention it's a distill either in title or description

I can update the description to be more clear about it being a distillation. Running the full model is a little out of reach for my machine, unfortunately, and I focus on models I can host myself.

Traditionally, you can feed it into RVC and train a voice model. More recently, there are utilities that require just a 15 second sample and can do instant voice cloning. I have a couple videos that talk about this, but MaskGCT and Amphion Vevo are two examples.

@@ChameleonAI many thx for getting back to me...i'll give it a shot!

I've been out of the game a little bit. My old favorite was Chronomaid Storytelling 13B, but it's a bit old. I'm still evaluating Cydonia-v1.3-Magnum-v4-22B but I like it so far. If you like qwen or anything else that's pretty censored, check out a concept called abliteration. It's a process of uncensoring a model. I've not personally messed with abliterated versions but it seems to be pretty popular. One good way of looking for models is the UGI leaderboard, that's how I find out about most of them.

Those are the people who trained the models, like it or not.

I'd say the latest XL models are pretty competitive with midjourney, and XL has fairly modest system requirements. SD3.5 is the best and biggest but it's lacking the user support and history of XL.

Thanks for the kind words. 2025 should be interesting, I'm keeping my eye on SD 3.5 and alternatives like Flux but still waiting on stuff to shake out.

I do want to cover 3.5 at some point and get into the non SD models as well, so yeah. I do have to do my homework on all that stuff though and see what shakes out in 2025.

The prompts are automatically created. I tried explaining it at 4:15 but it's difficult to convey. What it generates entirely depends on the LLM you're using, so that's why it's important to look at it in the cmd window to see what it's really outputting. The style prefix is only some text that's tacked on to the beginning of whatever the LLM outputs, so that's not very important. The important part is the prompt template, that's how SillyTavern asks for a prompt from the LLM. Basically it's the same as if you manually used the prompt template as your text input, then copy-pasted its response into SD. Now if it's creating blue vomit, that's probably a SD setting or something to do with the model (missing vae, wrong sampler, not enough steps, model incompatibility with SD forge, it could be a lot of different things).

​@@ChameleonAI oooh i get it now, you need an LLM like oobabooga? i think that's where it went wrong for me cause i dont have that. I tried to make this work solely by using sillytavern, stable diffusion forge and koboldcpp unofficial google colabs (to run cloudflare kobold api's) so thats why i was confused why the prompts for the SDforge appeared simply from the templates passed raw lol

Amphion is working on Text to Music but it's not ready yet. I'm not aware of any other open source solutions but if I find one I'll try it out.

I have another video where I compare a few different TTS solutions including F5 but I wouldn't say it's in-depth. Interestingly enough, I have the opposite problem as you where I haven't had a lot of success running F5 locally but I can run MaskGCT. I believe MaskGCT sounds generally better but takes longer and uses much more VRAM. I think most fans of F5 think that the quality is acceptable for the speed and resources that it uses. As for providing the project as a zip, it's not my project, it's from Amphion, and zipping the fully installed project with dependencies wouldn't work anyway due to cross-platform issues and how virtual environments work (I've tried. Maybe a docker container or something would work but you can't just zip it up). You're not the only one experiencing the espeak problem, though, and there are several open issues on the Amphion github about it, for example: github.com/open-mmlab/Amphion/issues/323

@@ChameleonAI Unfortunately, my problem isn't related to the RuntimeError: failed to load voice 'ja'. In my case, it says that espeak was not installed on the system, even though I already installed it. I tried both the 32-bit and 64-bit versions, but it's still not working. I also added all possible paths to the environment variables, but nothing seems to work. Maybe there's something wrong with my PC, not the code of the program itself. I don't know. PS: I am using another AI project that requires espeak to be installed on my system. I was able to use it without any issues, so I believe the problem isn't related to not installing espeak properly.

@@LoveMapV4 Yeah, looking around on the issues, some people are saying that the espeak errors aren't really about espeak so there's something to that. Unfortunately since it's not an error I can replicate on my system, the only advice I have is to check out the github issues page or simply wait and hope Amphion fixes it.

You mean my "real" voice? I always use a voice changer, so in a sense my real voice doesn't exist.

Good to know that a tutorial made in the SD 1.5 days is still relevant today.

i agree Well done

Probably over-annunciating because he was doing a reading. I do that myself sometimes.

There is a TTS option but this is primarily voice-to-voice, so it's a different application. Unless I missed something, F5 doesn't have an sts mode.

@@ChameleonAI Oh, sorry. I thought it was TTS. It makes sense now. Thanks

Only recordings. I didn't see a way of feeding in a stream real-time with the code they have right now, though it may be theoretically possible because it takes less time to run the conversion than the duration of the clip (on my machine anyway).

Great idea. I just published github.com/chameleon-ai/vevo-pinokio

@ChameleonAI thanks boss 🙏

Life is pain, I wasn't into AI stuff when I bought my rig. So much for python being cross-platform. lmao.

Yeah licensing for SD3 was a big part of why it died on the vine and why people were eager to adopt Flux. Though they have since updated their terms and now I'm seeing some early fine-tunes for 3.5 that look promising.

Since I made this video, SD 3.5 came out and is pretty competitive with flux. Stability AI also has partnered with Amazon, so maybe they've gotten past their financial troubles. As of now, there is still no SD4 announced, but I guess we'll see what they do in 2025.