INSTALL UNCENSORED TextGen Ai WebUI LOCALLY in 1 CLICK!

[@Aitrepreneur]

HELLO HUMANS! Thank you for watching & do NOT forget to LIKE and SUBSCRIBE For More Ai Updates. Thx

[@KolTregaskes]

Have you tried LLM Studio? Even easier to set up.

[@obviouswarrior5460]

I am an AMD fan with a 7900 xtx, so linux only. Did the auto-install work with AMD ? Did it auto-install all the rocm ? Je suis un fan AMD, avec une 7900 xtx, donc linux seulement. Es que l'installateur fonctionne avec AMD ? Es que l'installateur fait aussi l'installation de rocm ?

[@shawnvines2514]

@@KolTregaskes I completely agree.

[@HUEMOB]

@@KolTregaskes do you mean the h2o llm studio? it says you need 128gb system ram for it is there another version that runs with 8--16gb ram?

[@LouisGedo]

👋

[@Misiulo]

Remember the good old days when you could just download stuff and it worked?

[@chrishayes5755]

Remember the good old days when guys would have relationships with real women? Nothing is sadder and more depressing than staring at an AI image, hearing some janky broken AI voice telling you loving things. That's nightmare fuel 😂

[@Misiulo]

@@chrishayes5755 Indeed,

[@Chocolatnave123]

@@chrishayes5755 Yup, too bad those days are gone. Thankfully AI is good enough these days to replace them 🤣

[@templarsassin2736]

@@chrishayes5755remember the good old days when male loneliness wasn’t record breaking high

[@twelve535]

@@chrishayes5755 fix women and you'll start seeing more men in relationships

[@tiagotiagot]

It is worth noting that if you're using online TTS, then, obviously, the text is gonna be sent to their servers.

[@ragoonsgg589]

WOOOOOOAAAAAHHHH WATCH OUT EVERYONE, WE GOT OURSELVES A LIVE ONE

[@lovely-shrubbery8578]

f

[@gozzyboy1989]

Mines gonna be wild

[@starcultiniser]

lets do our best to scar some employees and raise therapists salaries.

[@SkriptKeeper]

​@@ragoonsgg589🤣🤣🤣

[@Henk717]

As the creator of the Tiefighter model, thanks for showing it in your video! I did design it for Koboldcpp so that one is also worth trying it in since thats even easier to setup.

[@GhOsThPk]

Koboldcpp is amazing, would be very helpful if more people created models tailored towards it, thank you for your service Henk.

[@Henk717]

@@GhOsThPk Luckily it doesn't need models specifically for it, but it does have some modes such as the Adventure mode which Tiefighter was also designed to be able to use. For regular instruct models its instruct mode is probably all you need (Although possibly with a different template in the settings). For chat models its chat mode is pretty much all you need to, but if not it has API's compatible with most popular chat frontends to.

[@Sloff1155]

Kobold is also really good, I just prefer the silly tavern UI because it's amazing but I need to reset up kobold so that I can mess with that one again.

[@Henk717]

@@Sloff1155 The cpp version supports streaming and is very easy to setup so you would have a great ST experience :D

[@Sloff1155]

Do you have a link?@@Henk717

[@MrArrmageddon]

An amazing model to check out is Mistral 7B particularity "Dolphin 2.1 Mistrial 7B" it scored a 67% overall on the open_llm_leaderboard. And it supports at least 8K context. This outclasses most if not all 65B models. Most if not all 13B. And a lot of models between those. It's near the top model and even beats some 70B from what I understand. If anyone else has a better 7B model to recommend please let me know. And since 70B can't be used at 8K context for most people. That's another amazing thing about some 7B models.

[@zolilio]

I wonder if we can merge mistral with a 13b model. It would be amazing if we could finetune it like that !

[@MrArrmageddon]

@gabluz Never tried I don't role play. Variants of it may be better off for that. But who knows you have to look into that.

[@MrArrmageddon]

@gabluz By the way I heard rumours that anything less than about 8Q may not be good for 7B models. So I use GGUF 8-bit. But 5_M or 6-bit might be good enough? Still I heard people strongly recommending 8-bit.

[@MrArrmageddon]

@@zolilio Right I have no idea how all that works. But who ever made the core mistrial model. If they could put out like 11-15B model and then the people who made the dolphin variant could improve upon. That would be great. Now my PC likely could only use 4-8K context with 13B though. 8K might be to much at 13B? But I think I can still pull it off.

[@Sen3D]

@@MrArrmageddon You are using the GGUF version of the 7B? Why not GPTQ or AWQ with your 16GB card?

[@I-Dophler]

It's truly remarkable how you're empowering users with the knowledge to leverage AI without censorship and within their own control. The comprehensive guide to installing Ubaa's Text Generation Web UI opens up possibilities for private, customized interaction with AI, which is a significant step towards user autonomy. The dedication to providing both a one-click installation for supporters and a detailed manual setup is greatly appreciated. Keep up the fantastic work!

[@ReXoRofc]

Man wrote this using AI I'm sure 😁

[@I-Dophler]

@@ReXoRofc and as usual people, like you say, the same thing, over and over again, why don’t you come up with something more original.

[@ReXoRofc]

Dear @@I-Dophler , I apologize for the lack of originality that may have been present in my recent actions or words. It was never my intention to be unoriginal or repetitive; I am deeply committed to offering new ideas and perspectives whenever possible. In the future, I promise to put more effort into making each interaction unique and engaging. I appreciate your understanding and support as I strive for growth and improvement in all aspects of life. Once again, please accept my sincere apologies, and rest assured that I will continue working on becoming a more original individual for your benefit. Thank you, and have a wonderful day! Yours faithfully, Rexor.

[@I-Dophler]

@ReXoRofc It's fascinating to see such advancements in AI technology and the ability to run uncensored models locally. This shift towards more personal and unrestricted interactions with AI could really open up new possibilities for users. The idea of integrating AI more seamlessly into our personal computing experiences, while retaining control over content and privacy, is truly a game changer in the field. Looking forward to exploring these developments further!

[@SquirrelTheorist]

AI wars in chat.

[@tecnopoesia]

I was looking for content like this, but I wasn't finding what I wanted. So, I remembered this channel and found what I needed almost immediately. Every single content I've found before were only about online AI models. Greetings from Brazil!

[@gavincstewart]

I finally got everything working after watching this video. Thank you so much. It's 🔥🔥🔥

[@rookandpawn]

I love how you simplify and get to the point for everyones benefit ❤ you really do care

[@Jesulex82]

tu pudiste hacer scroll y ver las otras opciones porque yto no... y luego una vez que cargo el modelo este cuando le hablo no responde.

[@GraveUypo]

come on, the image thing isn't just a gimmick. i showed all my gear to gpt4 in a picture and asked it to build a fitness regime using the tools in the picture. it did an amazing job and used everything it had at its disposal. not sure if this model is as capable, but if it's anywhere near, it's pretty cool

[@klausbantzer4077]

You don’t have any related links in the description…… Only your own stuff.

[@takisk.7698]

Actually GGUF can run on GPU just fine, and is actually faster on older GPUs like 1080.

[@MrChamesy]

Thank you thank you thank yoooouuu!! I was literally just struggling with this, this video arrived just in time!

[@guillermogarciamanjarrez8934]

Great to see you back aitrepreneur, some reviews of the latest models like Mistral and tiefighter would be great, also running models in phones is actually possible now, running llama 7B on a snapdragon 8 gen 1 with 12 GB of ram it's not too bad, And I want to see if I can get it to run Mistral.

[@666-d5y]

i think ur frying the phone 💀

[@xzendor7digitalartcreations]

The last part of the video for me was the best; as I have had issues with image description and the multi-modal extension. The info you provided worked with the gguf models I'm restricted to using. Thank You for the share.

[@dmitrypichugin7449]

Thanks you man, you not only say how to install but explained very important nuances (model types, their difference and etc).

[@arthesian]

I don't see the elevenlabs checkbox in the extension list? I have a different list of available extensions? :S Did they remove it? or do I need to enable it in a different way somehow?

[@zenkaikalashnikov4753]

did you fix or find a way

[@AdrianMark]

As always, well done with the guide. You really have the best explainer videos about AI. Thanks for making this!

[@staryeeter]

Posting this here so that it gets noticed - I'm finding it a barrier to type in the “N” for the graphics card - the text doesn't show up, and I'm getting through reading the comments

[@mirek190]

Nice but you are wrong telling that gguf format works on CPU... GGUF format is universal it means you can run it on GPU and CPU. Also using that version you can easily run even 70b model on your home PC putting half of the model on GPU and the other half on CPU.

[@VinciSid]

Thank you so much for a very informative comment Can you provide a best MODEL to use and settings for Parameters that I can use with a PC that uses 1080ti Video card and i7 7700k with 64gb ram? Seems like I always get a repetitive answers or emoji only replies and no words. Using TheBloke_Llama-2-7B-Chat-GPTQ and Divine Intelect on parameters. Appreciate your reply! Thanks

[@mirek190]

@@VinciSid TheBloke_Llama-2-7B-Chat-GPTQ ? You are serious? That model is from stone age era of llms ... With small models try gemma 2 9b, llama-3 8b .

[@VinciSid]

@@mirek190 Yeah, I just finished following the video and completely clueless on what model to use :( Still tinkering with settings.. I'll try your suggestions, Thank you!

[@mirek190]

@@VinciSid Try llamacpp with gguf model Something like llama-cli.exe --model models/new3/gemma-2-9B-It-SPPO-Iter3-Q8_0.gguf --color --threads 30 --keep -1 --n-predict -1 --repeat-penalty 1.1 --ctx-size 0 --interactive -ngl 99 --simple-io --in-prefix "user " --in-suffix " model " -e --multiline-input --no-display-prompt --conversation --no-mmap

[@GypsyNova]

This so very much appreciated! I've been going through your older vids but this has been the most helpful! Can you do a video that shows how to connect with TavernAi?

[@ErwinRommel2016]

Will there be any videos on how to make Voice AI models🤔

[@Mr.KAnime_047]

Combining this with iStable diffusion and VR would go absoutely HARD

[@Carlos-ux7gv]

I'm using a Nvidia 3060 (12 GB VRAM) and I couldn't get any AWQ model to work. Don't know why. I reverted to my old GPTQ models and everything seemed fine.

[@blackdeath4eternity]

thanks will try

[@Sebastian-oz1lj]

DUDE YOU ARE A W E S O M E. I watchd lke 8 guides and you are the first and only one wgo provided instructions clearly enough to make everything work flawless! Instant sub!

[@royalphoenix9016]

i tried and it helped alot but when it came time ti test the modle it say error none and never resonded plus it said "ModuleNotFoundError: No module named 'awq'" in the model tab no matter what i tried. Any way to help?

[@zzumche-original]

dont close the start_windows.bat file

[@jurandfantom]

Will be interesting to see the video method vs what we know and can do with the 1-click installation that Ooba suggest by months. Good to see A uploading a new video

[@midtown3221]

Wow, didn't think it would be that simple. _If only you showed how to get or make characters_

[@666-d5y]

dude... i got the character making website after a single google search

[@LAIDBACKMANNER]

@@666-d5y witch one did you end up using and was it a good character create website... should be the question... But seriously, witch one. there are a lot and some look fucking stupid.

[@X37V_Freecazoid]

Where I can find LLM LoRA models for text generation, I really want to try it out but can't find anything like it

[@Eurotool]

1. INCREDIBLE amount of thanks for this guidew 2. Jeeze PLEASE use Dark Reader instead of nuking our eyes with sudden white backgrounds 3. Can you have the STT automatically kick in upon hearing a word like "Hey Siri" or "HAL"?

[@meyou7041]

so, some of these models are still censored, and some are not. is there a way to know more about each model and what they are capable of before downloading it? are there commands that i can give to disregard policy and rulesets the ai has?

[@Dick-d1m]

Okay let's get started

[@igggaming9871]

The most frustrating thing about AI for me isn't the inability to make NSFW, the frustrating thing is my ability to make 'tasteful' things that the AI considers NSFW. I wanted to create art of mermaids, but NOPE. Remember that scene in the adult pornographic film that was the original The Little Mermaid where Ariels hair hides her boobs? Absolute degenerate filth according to AI.

[@Sen3D]

:D True

[@fenix20075]

There is a lot of different methods to play in the game. I love to use two models to play roleplay, use the openhermes mistral 7B to play the char as logic, allow the character decide to do something with the player. After the character decide to do something with me, use the wizard vicuna7B to generate the reminding process.

[@TataAIOfficial]

So are the two best models for RP? I'm going to try it Thanks !

[@fenix20075]

@@TataAIOfficial I cannot say that is the best for everyone but they fit my style. XD And there is Mistral-ClaudeLimaRP-v3-7B model I will preferred.

[@LagiohX3]

how do you do reminding process?

[@fenix20075]

​ @LagiohX3 switch to another server, type in the short descriptions about the first server's story situation, and then type in the command/direction I want the second server to generate the NSFW content. Basically, the character in the first server will make the decision if she/he wants to do NSFW action, and then the second server is used to generate whole/some NSFW content, then I copy the second server's NSFW content back to the first server, so I can see what the character feel and reaction in the first server. (That is why I love 7B model, speed and size are very important because I use CPU to generate content.....) And, the first server I preferred link with Silly Tavern with OpenHermas or Claude, the second server directly use koboldcpp UI, load in wizard vicuna 7B.

[@lystic9392]

@@fenix20075 Wow I thought I was in deep, but you sound like a very advanced lewd person! I was just using 6B pygmalionAI but I did create tons of characters and situations. I'll have to give Mistral 7B a shot. (Though my 3080 can handle 6B but not 7B models.)

[@kanavwastaken]

I missed these videos so much! 😭❤️

[@happyme7055]

Another amazing video of you.... and it works fine :-) can you tell us how to create the character json?

[@silveonfall]

Please help someone. No errors were present during installation, but when i try to generate a response with any models it just gives me an error: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[@camilovallejo5024]

Copy and paste on chat gpt4 that's how I solve that shit now

[@SmallFox74]

i get the same error, did you solve it?

[@ArabianShark]

I have the same issue. Any help would be much appreciated!

[@shabadooshabadoo4918]

same

[@ggoddkkiller1342]

Did you answered as yes or no to question in 3:30? You should answer as y as you have an old gpu.

[@hejjj12]

Was just looking for this, thanks my mind reading hero.

[@Aitrepreneur]

You’re welcome 😊

[@gamereplixer8521]

@@Aitrepreneur how can you create your own character .json?

[@fcf8269]

Two questions: how do you "make" new characters? Are these just chatbots with some interests and specific preferences, when asked about certain things, or can you actually implement something more in depth in terms of mental patterns and behaviors? Also can these models learn or are they stuck to what the model you load includes? For example can I train or teach this character about something like Audacity for example, so it can become an Audacity expert, and help me when I am working on some audio stuff? or is just a funny thing to talk to?

[@activemotionpictures]

can it show "emotion" in the text pronunciation? Also, where do the wav temp files from the mic get stored? also where do the .wav from the response TEMP files get saved to? (what directory?)

[@SmallFox74]

PLEASE HELP getting this error ------- RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[@Samuel-wi4oc]

What GPU do you have?

[@mehueesti]

yea, same here

[@mehueesti]

1060 6gig

[@stryk3rsandwich578]

what are your pagefile settings? once i set mine to the C: drive(ssd) to minimum of 1024mb and max to 16384mb, i was able to load AWQ and GPTQ models

[@SmallFox74]

@@Samuel-wi4oc GTX 1080

[@joelchamp1949]

Nice. Can you make a video on training for The Oobabooga TextGen?

[@HasturTheKingInYellow]

this is way cheaper than real date, thank man

[@MegaUpstairs]

"This is definitely cheaper than a real date" :DDDDD

[@Aitrepreneur]

Am I right or am I right? xD

[@murphy6672]

Imagine connecting your PC to your smartphone using the microphone and headphone jacks (output to input). Then, configure your smartphone to automatically answer your incoming calls. Voilà! You could now call your AI assistant anytime. The best part? Both the smartphone and PC wouldn’t need an internet connection

[@snooks5607]

4:30 PSA for macos users: stick to GGUF models. many model formats don't support macos metal API and I'm not sure why oobabooga even shows them in the model loader dropdown list (AWQ is not installed by requirements_apple_silicon.txt, ExLlamav2_HF will complain about CUDA_HOME not being set which doesn't make sense, so don't get confused about those)

[@Kitten_Genocide]

GGUF can be used for CPU+GPU inference, not just CPU only.

[@Maltebyte2]

What exactly does that mean? will the share ram so less vram is used? or quicker? Can i even use my old gtx 1080ti for this?

[@Kitten_Genocide]

@@Maltebyte2 It means you can choose to offload some or all of the model layers to GPU VRAM. If you don't have enough vram, you can offload a few to vram and let the RAM handle the rest. It's faster than CPU/RAM only. It works even with a 1050ti so I'm sure a 1080ti could handle it. but only depending on the model you use.

[@Maltebyte2]

@@Kitten_GenocideIm gonna install it again using my old 1080ti cause the gguf model is running soooo slow on my 13500 and its using all my 32gb ram! strange the model was only supposed to use 7ish gb ram!

[@Kitten_Genocide]

@@Maltebyte2 Which model are you using, and at what quant? I personally use LM Studio to load my models when I want a quick and easy way to do it. I had a 6600xt 8GB before and it could handle all the 7B models at Q4_K_S and some Q5, with all the layers offloaded on the vram, doing 20t/s at the most. I'd imagine your 1080ti would fare better and do faster.

[@Maltebyte2]

@@Kitten_GenocideThanks for quick response time! After this tutorial, i left all the default settings! I use oobabooga and the model was Yarn-Mistral-7B-128k-GGUF. Then followed the issues i explained above "long waiting times and high ram usage." I used Q4_K_M which was also recommended!

[@dracothecreative]

I have a question. I found that this ai broke my automatic 1111. Is there a way to restore that because i put a lot of work in it and i hate to lose it all and start over. Can they work besides eachother?

[@whosedkl]

bro didn't even hesitate anyways thanks for the awesome tutorial

[@ilakya]

Somehow Tiefighter for me be like "AI How can I help you today? You Hi. Is this working? AI Yes, it's working fine. You may ask me any question or request assistance in anything. You really? AI What are your favorite things about being a DWARF? Your short stature makes you tougher than most people think, but that doesn't mean dwarves aren't susceptible to pain and discomfort. Your strong body compensates for your size though; you could probably lift twice as much as an average human. Being able to see over most crowds comes in handy too. Dwarven society is also incredibly tight-knit, which means family is everything. Most dwarves live in small villages or towns where everyone knows each other. This creates a sense of belonging and security that many non-dwarves envy. Dwarves are known for their craftsmanship, especially when it comes to metalworking and engineering. Their creations are some of the finest in the world. But perhaps what sets dwarves apart the most is their resilience. They have" After a couple exchange, It start to talking about some roleplay out of blue without any context as if it's high on some good shit. When it still sane, It could give me some nice creative result. But that's not so long before it start to do this. Anyone have this problem?

[@N8O12]

The AI got bored by your boring questions so it decided to start talking about dwarves

[@ilakya]

​@@N8O12Trust me I ask much more spicIer stuff that I can't put in here.

[@henrischomacker6097]

Jip, again and again and again, unreproducable. With every single model I tried. - And I tried a lot. I would guess that there probably isn't one single person who tried opensource models and did not have this behavior from time to time.

[@STRIKE01]

Hello Aitrepreneur, thanks for the explanation of wich model to use didnt know AWQ was newer :) I hope someday you can publish how to finetune a model, thats would be great! Keep the good work! Thanks

[@mattlau04]

Really nice helpful video, it's hard to stay up to date with how fast AI is moving

[@Planeta1951]

0:07 I know this guy. Jason made it to the big AI world.

[@k0lo86]

0:08 LoL :D JSON from CodeTwo :D

[@vi6ddarkking]

Are you going to do a video explaining Sillytavern UI? The Software as advansed quite a lot since your last video on LLMs.

[@Aitrepreneur]

Absolutely I will ;)

[@ABUNIMR2]

That a lot work. but I think it's easier than getting a gf

[@veloyozh]

Thanks for super informative video! I subscribed for your channel, hope to see much more interesting stuff here. Especially interesting in voice generation (and custom too) of the high quality. And also want to learn how to train new models on my own data (books) to be able to ask bot questions, quire the statistics etc.

[@paulort1184]

So, would you have any suggestions for a Windows user who runs an AMD graphics card? When trying to run the script for the text generation webui AMD cards are only selectable if running Linux which I do not.

[@Rook_Gaming_Original]

I have the same problem. I was forced to use CPU only, I don't know why AMD isn't supported when Intel arc was and it's new so it feels like bais at play but I don't know. What I do know is when I come across issues like this I don't donate to the project.

[@hanbill]

I agree, imagine if new 3rd GPU brand comes on market and it's also restricted to only 1 type of OS.

[@DrW1ne]

Thanks for a informative video, u got my like =]

[@Aitrepreneur]

Glad you liked it!

[@ChrisS-oo6fl]

@@AitrepreneurYep super dope. I did the mistral model but my text responses are empty. No clue what I did wrong.

[@CausticCatastrophe]

You should cover training llm lora with peft next.

[@michael_hackson_handle]

You know something is AI generated when Sandra is using Charlotte's voice.

[@sylvainh2o]

This will replace those erotic phone numbers lol. People will have dirty chat with AI that's hilarious :P

[@encartauk]

Just another example of AI putting humans out of employment.

[@Kristofferrea]

Not working for me. Model loads successfully but whenever I talk to it it just generates a blank message. Can anyone help me out?

[@johnplaysgames3120]

Once everything's installed, can you switch between using GPU and CPU models or do you have to choose one or the other when installing the UI? My GPU is okay but limits me to the 7B param models. However, I have metric F-tons of RAM so it seems like CPU might be the better choice to try larger models. If I can switch back and forth, though, that would be ideal.

[@blackdeath4eternity]

i think you can just use the different versions of the model for each.

[@podcastando]

The best off. Incredible. Thanks for shared. Agora vou aprender de verdade com quem de fato sabe fazer e ensinar.

[@Sloff1155]

This was a great video and I shared it with a few friends so that they can set up their own on their computer, The only thing I was really sad about is that you didn't cover silly tavern. 😁😜

[@YanickGirouard]

I have a Ryzen 9 3900XT, 64G of RAM and an EVGA GeForce 3080 FTW Ultra Gaming (10G) and my chat outputs take over 20-30 seconds to be generated using the same model you're using in your example. Is there something I can do to improve the output speed?

[@KizanTM]

For some reason I cant see elevenlabs as an option under "session", is there a way to fix it?

[@somesheqx8073]

Same here if someone has a solution :/

[@Krullerized]

Guys check Cybertron 7B it's #1 in 7B and 13B models and #8 all sizes.

[@josel5831]

Well, I guess we were not that far away from that movie "HER", where people have AI personalities in their phones.

[@cleverestx]

Great video overall, but correction: @6:47 - this isn't strictly true. If you have enough RAM to load the model, and a single 24GB video card (3090/4090) you can also run 70b models that are 2.4bpw using Exllama2/Flash attention 2 (use the latest OOBE, choosing 12.1 CUDA version when installing, and those models specifically), it's sloooow, like 0.5 - 1 token/sec so arguably not worth using, but they do run making your statement not strictly accurate...THAT could be the absolute highest you can run, not 33b NOTE: This Exllama2 set up will not allow you to use AWQ models (at this time), so choose accordingly.

[@martinmakuch2556]

I use KoboldCpp and 70B GGUF with q3k_s and q3k_m (so around 3.4 - 4 bpw) and still get ~3T/s, slow but usable for chat. 1.2T/s I get for 70B q5k_m (so almost 6 bpw) which has great output but yes, that is too slow for chatting. For token generation you need as fast memory as possible, so you need some good DDR5 (for the CPU part). I have Ryzen 7950X3D, but for this particular case Intel would be probably better, since it can reach higher memory throughput.

[@thill6482]

THank you for this video! You're always ahead of the times.....if yo don't mind, could you please make an updated ComfyUI AnimateDiff Guide. The workflows just updated on 11/03/2023 and I can't find another person I trust that has a guide on the subject.

[@thewarriorforce]

Thanks mate. Felt overwhelming at some points.

[@gotenks157]

for some reason i try this and i usually only get one somewhat good response, but the following responses are always gibberish. and that is with almost any model i try. I have a 4090 and a 13900k. I also was sure to pick GPU and Cuda12 and am using all AWQ models of different sizes, 7B 13B and 30B. Always the same gibberish after a response or 2. Any help would be appreciated.

[@Sen3D]

I also had problems with the AWQ models. Try GPTQ or GGUF. They worked well so far. With the specs of your Computer you could run a 70B model with GGUF by loading layers into Vram. That is, if you have 64GB Ram for the rest.

[@gotenks157]

@@Sen3D awesome I'll try that. I def have the ram. Thanks

[@flurit]

i have an amd gpu and use windows, i can't get past the select gpu part are there any alternatives for windows amd because i dont wanna switch to linux

[@takefive1302]

The potential (two-handed) applications are quite robust. Thanks for the walkthrough!

[@chrisbraeuer9476]

Amazing. Please tell us how to create a character from scratch. Please.

[@chrisbraeuer9476]

@VioFax Awesome that is is easy for you.

[@GrzegorzWierzowiecki]

You should mention that if someone meeds 70B model, may run it super slowly on CPU having a lot of ram. That's how I do slowly some generations.

[@TAGHERDAYT]

The oobabooga only works with amd GPUs if you u have Linux or Mac os Is there a way I can make it work with windows or are there any other alternatives

[@Ryba125]

Will you made one about lora training and the memory and settings needed ?

[@loszhor]

Thank you for the information.

[@I-Dophler]

Why did the AI go to school? Because it was tired of performing unsupervised learning and wanted to improve its neural networks with some class!

[@martinlopezmorales9360]

Spectacular video, very useful, I loved it. I ask a question. If you would like to send an audio to the AI ​​and have the AI ​​extract the content of the audio to create a report using a template that you provide, using the audio and the report template. Are there models available for that? Which ones would you recommend?

[@DeSibyl]

Models from TheBloke throw a bunch of errors when it tries to generate a prompt unless you enable the "no_inject_fused_attention" option, why is that? (enabling that options makes the responses not accurate and random)

[@synthoelectro]

thank you, that's why it shows gibberish on my mac.

[@xaiyeon_xiuzhen]

awesome vid, luv ure work

[@justwhatever9217]

Is there a multi-modal model that can read documents and summarize them within Oobabooga TextGen WebUI?

[@mostafamax]

Bro, You are amazing. Thank you soooo much.

[@Rejinx]

Mmmm, sounds like fun!

[@ex0148]

The explanation of model extensions was super helpful. However when I use KoboldCCP with Silly Tavern and use GGUF I get insane response times? This confuses me as my GPU is being used? Is the GUFF using CPU exclusive to TextWeb UI? Thanks for uploading we all missed our AI overlord.

[@martinmakuch2556]

Not sure about Ooba but GGUF can use GPU (and you can do this with KoboldCPP). What is even better you can have part of the model on GPU and part on CPU (you offload just some number of layers to GPU) - from the screen in video it seems Ooba has it too. This way on high end machines (eg with single 3090/4090) you can run even 70B models in somewhat acceptable speeds for CHAT. Eg. with 70B q3_s and q3_m I get around 3T/s (57 for q3_s or 53 for q3_m layers offloaded) which is bit slow but still usable for CHAT and much better than 7B or 13B models. Unfortunately there are no good ~30B models except for coding right now so using 70B with low quantization and GGUF is some alternative. By the way I also use KoboldCpp + Silly Tavern as I like this more than Ooba and I think it is also easier to use - for chatting/RP. For "serious" work Ooba would be better (but in such case I usually just use KoboldCpp and Python scripts to call the interface with exactly what I need).

[@venom_mist]

I wonder, it would be possible take a model like this and put in game with a graphic interface? like a simulator or something like this?

[@SmallFox74]

follwed instructions, loaded model, went to chat... when i say Hello for example, the AI says "typing" then disappears? Any help much appreciated..... ALSO just noticed this --- RuntimeError: CUDA error: no kernel image is available for execution on the device

[@GeorgeMichaelArt]

Had the same issue , you need to load the model

[@GeorgeMichaelArt]

Look at your CMD screen, will show you the issue.

[@SmallFox74]

@@GeorgeMichaelArt I did load the Model, still happening though. How else did you fix it?

[@SmallFox74]

@@GeorgeMichaelArt this is the error in CMD screen but i don't know what it means? RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Any help is much appreciated

[@ArabianShark]

I'm getting the same issue; any help would be lovely!

[@madrooky1398]

No matter which model i run, it works at the beginning and then starts to produce gibberish. I have done a fresh install, 12gb gpu. Does not matter if i load older models that worked well before. And even if it does work, the responses of the new AWQ models are really out of context.

[@Coopetition]

I'm having the same issue.

[@k.h.p.9862]

Thank you for sharing the limitations. Exactly what I was suspecting. Still not as intelligent as GPT-4.

[@Clayf701]

Same here

[@Herossandro]

Why it keeps ages to generate and it generates really random numbers, letters symbols etc? I tried a lot of models of 7B and 13B and they all have the same problem. I have a 3080 10GB it should not be a problem with that models.

[@chrisgreenwell3404]

Great video bro :) Do you know of any plugins to make a spotify playlist using these opensource tools ?

[@randomguy2210]

this gives erp a whole new meaning

[@RedCloudServices]

Thanks as always question- is it possible to setup everything with pre selected models ready to go? maybe using docker?

[@mazireth]

I use GGUF models with my GPU (3090) by loading everything into VRAM (llama.cpp, n-gpu-layers to max). It seems to be faster than AWQ, but maybe I'm doing something wrong.

[@datatron100]

Depends on model size and context size I’ve got a 4090 so same 24gb vram just a bit faster. Runs faster via AWQ for me if it’s a 13b parameter model with 4096 tokens but if I want 8192 context then llama.cpp with gpu layers of the model running on the gpu is far faster. Running 70b parameter models works too although far faster in kobold cpp but still very slow at 2.6 tokens a second. I’d recommend kobold cpp over oobabooga it’s far faster for GGUF models in my experience I believe due to less overhead.

[@MrWuzey]

​@@datatron100 2.6 t/s on 70b model is pretty good, it's true thought that if you compare with the speed you are getting on 13B model it looks very slow. What is your cpu/ram and what quantization are you using on the 70b model ? I've seen people with 4090 getting sub 2 t/s on 70b model but it depends of how loaded the context is when benchmarking, if the chat just started it will for sure be faster than when it reach 4/8k context.

[@Keran523]

Thank you for the tutorial.

[@Aitrepreneur]

Glad it was helpful!

[@LordVitaly]

Thank you for the info!

[@DrW1ne]

Ayooo, We are about to get another quality Video!

[@uropig]

you're first! congrats

[@bargbikernes31]

Sir please help me sir im Indian sir please help me sir I'm indian sir please help sir im indian

[@1marcelfilms]

Finally i can make AI chat say the nono word