Automatic1111 + Oobabooga = Exchange Images with your ChatBot!

A chatbot that can send and receive images? All for free? Whatever next! Works with open source models such as GPT Neo, RWKV, Pythia, etc or even with closed ones like Llama, opt etc. Includes a free, custom character!
Welcome to the power of combining Stable Diffusion with a chatbot in order to get some rather interesting results and experiences. I know I've been chatting to this bot for too long now, so it's time to share it with some nerds!
Enjoy :)
Nb. You can now just use “adventure mode” in the usual sd extension along with this character!
== Links! ==
* Bot - github.com/nerdyrodent/StableBot
* Automatic1111 Web UI - github.com/AUTOMATIC1111/stab...
* Text Generation Web UI - github.com/oobabooga/text-gen...
* How do I create an animated SD avatar? - • Create your own animat...
* Installing Anaconda for MS Windows Beginners - • Anaconda - Python Inst...
* Stable Diffusion Playlist! - kzbin.info/aero/PLj...
* Interested in adding things to your AI Art? Try these!
Dreambooth Playlist - • Stable Diffusion Dream...
* Textual Inversion Playlist - • Stable Diffusion Textu...

[@Swonkasaur]

I think showing the potentials of lora training your own text model would be pretty huge would love to see a tutorial or a further breakdown

[@DefineMeAsOne]

Same. (Adding some YT engagement for algo).

[@ArtificialCitizens]

I too am a human adding to the algorithm for KZbin

[@NerdyRodent]

Stable diffusion API power ftw!😄

[@burrowsforge3538]

Very cool idea

[@Keiinetic]

First time watching any of your content after randomly stumbling onto your channel; indeed, I’m very impressed with all the videos you have produced. The guides, tutorials, and information you provide are edifying and enlightening. They are definitive, unequivocal, and forthright without missing valuable information. The presentation and quality are astounding, unlike some videos where the majority would often ramble and divert topics to make the content longer. I look forward to diving into all your content in the next few days, starting tonight. I just thought I’d share my thoughts. Keep up the fantastic work. Liked & Subbed!

[@swannschilling474]

Combining the most powerful local interfaces in the most clever way!! Well done Mr Rodent!!! 🧐

[@thorminator]

Epic! Can't wait to see where you take this next! 😎

[@wolfai_]

This is why I love you. You added even more to what's already available 😸

[@NerdyRodent]

👍

[@servbotz]

Omg thank you!!! I kept asking around for guidance on this and no one had a clue. You're a super chad. Subscribed!

[@NerdyRodent]

Glad I could help!

[@WAR_1933]

well said.. very well said Marshall

[@theairchitect]

always awesome content ❤ thank you nerdy 😍

[@NerdyRodent]

My pleasure!

[@Jaxs_Time]

These are excellent videos, Love watching them! Keep up the good work!

[@NerdyRodent]

Glad you like them!

[@adama7752]

I know you might not read this, but Thank You. I know 'thank you' is cliche. But I really do mean it. You put it all together in an easy to get into manner, and have brought a lot of joy into my life. Getting more into what AI can really do. I've made CPU simple neural networks in c,c++. But I really enjoy your content and it is joyful and enabling. Thank you

[@NerdyRodent]

Great that you’re having fun - it is the general idea after all 🙂

[@CarpeUniversum]

So say we all.

[@coloryvr]

WOW ! Super impressive! ....I'm just about able to keep up with the rapid development of Stable Diffusion...

[@therobotocracy]

Ohhh man, you are so awesome, We need to see some Lora training in Oobabooga! Let's do this thing!

[@chris-hayes]

I've heard legends of this very secret "video description", I hope one day I may lay eyes on it.

[@NerdyRodent]

It is a mythical beast indeed 🥸

[@wakegary]

Man I want you to do my eulogy at my funeral. Your voice is just so tongue in cheek and funny. Great delivery. There are like 20 people covering all of this stuff and I won't lie, your voice and inflection hooked me. Cheers. #speciallittlelink

[@BrandosLounge]

Never a dull moment with you bud

[@NerdyRodent]

Woo, yeah! 😃

[@flonixcorn]

Amazing stuff!

[@NerdyRodent]

Yup, it’s great fun!

[@burrowsforge3538]

I’d love to see how to incorporate this into your own characters. A mad scientist character who can now show you all of their wild creations? How would “you ” this feature into already existing characters or custom ones?

[@NerdyRodent]

I’d just describe what sort of mad scientist that character is! I’ve got cowboys from the Wild West, pirates and all sorts. Aaargh, me hearties!

[@amj2048]

this is so awesome!

[@NerdyRodent]

Yup, it’s fun 😃

[@jason-sk9oi]

Tremendous! 🎉

[@ruzanmuhammedasher3263]

I absolutely would love a video about the training model. Gpt 4chan left me wanting more fine tuned 4chan like responses to play around with without the use of 4chan like I had to since 2011.

[@riggitywrckd4325]

I have a practical idea for adding in multi-character chat to textgen. If you had buttons on the side that were connected to your chosen characters you could have them talk randomly or choose who talks next making for easy picture bot and another bot that you are talking to at the same time. I've looked into how the data is saved after a chat and there are no markers in the json for which character you would be talking to but if it was possible to add them then it would be as simple as putting in extra slots on the character tab and extra buttons that generate based on character slot, and a drop down that allows for choice of random, sequential, or round robin when you press generate. The tags that are added to the json don't even have to be the character names they could just be a number connected to the character slot. Because you are just firing groups of text at the model and seeing how it responds all you have to do is mechanize what gets fired at it. Which persona has read the context and has their own context as the generating factor. Anyway ideas I will be investigating.

[@NerdyRodent]

Yes, multi character would be crazypants fun 😀

I have one, more pressing matter, regarding textgen - a less cumbersome switching for my own personalities. That could be part of the multi-character chat, where You choose which character You impersonate Yourself while letting the AI handle the rest. At this moment I am stuck with manually renaming portraits for my chats; adding descriptions of my own persona to the character card, and changing my characters name every time I have to switch. I wish that was less tedious.

[@impactframes]

Super cool

[@barwarrior4633]

On GitHub there‘s an extension to include a long term memory so that the bot does not forget important aspects of previous conversations. I can not get it to work, could you make a tutorial on how to add such a memory to the chatbot, like character AI or other chat platforms provide ? 😊

[@NerdyRodent]

The ltm extension was working, but broke recently. Shouldn’t be too long 😉

[@KarimMarbouh]

thank you dude, nice one

[@NerdyRodent]

No problem!

[@sempredic]

yes please do the training tab,nice vid

[@bx1314]

You're killing me with your humor

[@NerdyRodent]

😉

I would love to have some better explanation about "softprompts" that text generation ui has and the training feature that has been added to it.

[@karithay]

First, excellent video. Thank you for it. Second, do you know a good way to combine the SD prompts and narrative text? I've had mixed success using the conversation model in the character by putting the prompts, then , then a narrative description, but its not always consistent. Third, is there a way to hide the prompt text? By accident I discovered that with the sd_api_pictures extension, text within asterixis now just vanish. Although the text appears in one part of the log, it is removed when sent to Automatic1111 and the text channel, so that doesn't quite address the matter.

[@Kangi_Yatapi]

Great video! But I have a question: How could I make the character update its aspect in the time, like scars or different clothes in a way that it remembers the last image created? Is something about the TEXTGEN PREFIX? The only way I found to do something nice is adding "*You describe your aspect*" at the end of my sentence.

[@arri6984]

Any way to get the saved images to retain the png metadata so it could be used later to generate similar images?

[@IlRincreTeam]

Classic Nerdy Rodent.

[@TheBreezus]

Please do an indepth of the webui in general, literally no tutorial videos, of what these functions do.

[@robxsiq7744]

100% interested in the training. not much info out there and would love a video if you're up to it.

[@aegisgfx]

We're gonna need a bigger hard drive!

[@NerdyRodent]

Very much so!

[@urglik]

what is to be done with the files and folder in the extension folder. Does just the folder go into SD extension folder or is the entire extension directory rename and put in SD extension folder? Thank you

[@aidiffuser]

Thanks!

[@NerdyRodent]

Welcome! And thank you 😃

[@johnecott7429]

yes teach us how to fine tune and use the LoRAs tab! I clicked the bell icon and subscribed

[@XavierVegasVidal]

nice!

[@ObscureFactsOfficial]

Super underated channel in ai! 10/10

[@NerdyRodent]

Appreciate that 😀

[@NotHuman44]

Yes, please make a video on Lora training in oobabooga.

[@Mono_Autophobic]

How only 25k? This feels like a true treasure

[@itaicarmeli1145]

Is there a sufficiently-simple way to use what I see "Shared GPU memory?" I got 64gb RAM and 8gb VRAM, and I always wonder when would the 32gb "dedicated GPU memory" ever be used

[@christopherritter]

Thanks for the tutorial! I ended up using the smallest OPT due to the size of my RTX 3070. I'm able to receive images BUT they're very basic, they seem to stutter (e.g., monkey, monkey, monkey), and don't add much to the prompt. Plus, I'm receiving at least 25 images with every prompt. Any ideas why this might be happening?

[@NerdyRodent]

If you’re getting repetition, then you can turn the repetition penalty up

[@nic-ori]

Thanks.

[@NerdyRodent]

You're welcome!

[@hsh0t-to1rd]

Excellent, GJ! Only that I have a problem: I get an image for every single word in the prompt, not just one picture for the whole prompt. Do you know what might be the issue?

[@NerdyRodent]

You're sending the API one word at a time. Send the whole prompt at once instead :)

[@mega3533]

I am not sure why but it constantly generating the pics. How can I make sure it generate the pic only after the promp is completed?

[@NerdyRodent]

If you use the options exactly as it shows in the video, then you’ll generate the whole prompt rather than generating one word at a time

[@transfear]

Hey Nerdy, What is the name of the tool which shows your PC stats in the upper right corner?

[@NerdyRodent]

That’s conky

[@mikrobixmikrobix]

nice, but is it work with fooocusAI or ComfiUI?

[@noodldoodl1970]

Whereabouts does it save the pictures that it generates because even when I tick the box 'Keep original received images in the outputs subdir' it doesn't actually do that? I'm guessing this means the outputs folder in the SD directory?

[@noodldoodl1970]

EDIT - they're actually stored in the \oobabooga-windows\text-generation-webui\extensions\sd_api_bot\outputs folder, in case anyone else is wondering.

[@pragmaticcrystal]

Nerdy 💛💯

[@NerdyRodent]

Hey Crystal, thanks for dropping by! 💚

[@pragmaticcrystal]

@@NerdyRodent 👍

[@cleverestx]

Thanks. On the GitHUB for the character model it says, "As the script has now been updated in the original repo, you can simply use that extension in "Adventure Mode"" -- can you explain what it meant here? Are you saying we don't need to download this anymore, just enable it in the OobaBooga UI settings under Extensions and reboot and it's there already?

[@NerdyRodent]

Exactly that, yes. Simply use the existing sd extension!

[@cleverestx]

@@NerdyRodent Thanks, I was going to post that I don't even see a BOT extension at all, but I enabled sd_api_pictures and restarted, now it comes up in chat mode on the bottom. That is all there is to it? *edit, yup it works, thanks!

[@NerdyRodent]

@@cleverestx yup! As mentioned, just enable adventure more so it always generates images

[@user-vq1ix6dv6u]

There's a new version of it implemented in the extensions that comes by default with oobabooga, but it doesn't work in the same way as yours... And it's hard to make it stay in character (With a character like the one you provided). I really prefer your version but isn't working anymore and I don't find a way to make it work back with the latest update of oobabooga.

[@user-vq1ix6dv6u]

Nevermind, I've solved it... if you want to bring back this old version of the extension just replace the "chat.cai_chatbot_wrapper" at the end of the script with "chat.generate_chat_reply_wrapper"

[@tesseract3966]

@@user-vq1ix6dv6u Thank you, this fixed my issue!

[@YongHuiNgCGlas]

May I ask , my model doesn’t send photos unless I force a picture response, even if it says it’s sending a pic , there’s no pic , is there any underlying thing I have to code?

[@NerdyRodent]

Set it to “adventure mode” for continuous pictures

[@teambeeq9433]

Brilliant video, as always 😌👍I don't have the sd_api_bot extension available nor listed even. Google didn't give any results; would you know where to look for it? I have both Text Generation WebUI running in port 7860 and latest Auto1111 running in port 7861. Afaik they can't run on same port or I don't know how to set it up like that. Tyvm in advance

[@NerdyRodent]

Links are in the video description

[@Anima_moxhya]

can I do this with 8gb VRAM may be choose different model or make text gen use the CPu instead and SD use the GPU

[@NerdyRodent]

Sure! You can try with much smaller models 😀

[@ThisOrThat13]

SO where do we find the outputs of the images?

[@catthing3398]

I am interested in the training tab, I have no clue what it does but I hope you cover it.

[@NerdyRodent]

Ah, it’s quite magical 😉

[@wiqodinchan666]

undeniable charm 6:14

[@cleverestx]

Has anyone developed an initial prompt to make a strong storyline with a choose your own adventure idea for this image/text combination?

[@thegate4407]

Hello, about the train, I wonder if it will be possible to train with a model like lama?

[@NerdyRodent]

I’m sure you could do a Lora with Llama, yes

[@RaziqBrown]

Is there any guide for training anywhere?

[@KratomSyndicate]

I'm not seeing sd api bot, was this removed on newer web ui updates?

[@NerdyRodent]

You do indeed only need the character now

[@Rixxsenga]

Wow, if you could include ai voice detection and prompting by voice you could create a powerful tool for dnd players to visualize their surroundings..

[@NerdyRodent]

Try the whisper speech to text 😉

[@daveckw]

No connection could be made because the target machine actively refused it I have problem above. Do you know how to solve it?

[@NerdyRodent]

Try connecting using telnet. If that can’t connect either, you’ve likely got a firewall preventing connection between your source and destination hosts.

[@Sylfa]

At this point, someone should train a LLM with instructions of how to work with LLMs, how to install them locally, configure them for your hardware, etc. It'd be so much easier if we could just ask InstallGPT to give us the commands necessary for our specific computer and let it handle it! 😅

[@NerdyRodent]

AGI is coming! 😀

[@FSchack]

I don't see where exactly the sd_api_bot extension is? It's not in the bot link, there's only the character?

[@NerdyRodent]

Yup - as per the GitHub & video descriptions, you only need the character now 🙂

[@FSchack]

@@NerdyRodent My bad, I started webui-user.bat --api instead of webui.bat --api

[@sergentboucherie]

I'm also old, I remember what choose your own adventure books are

[@NerdyRodent]

😀

[@ianboyles2197]

recommendations for someone with a 3070 on how to divide resources

[@lookimnotracistbut5695]

Free is still a little too expensive for me. Is there any way you could lower the price so more people may enjoy your amazing work?

[@lenoillionel2342]

ive got a problem. if i say something like "show me a cat".. it generates about 10 images, 1st image "C" 2nd image "ute" 3rd image "cat" 4th image "potrait".. and it goes on and on till the prompt is finish 😅 what im i doing wrong?

[@NerdyRodent]

You’re generating one word at a time rather than the whole prompt. You can do that, but it’s best to let it not stream 😉

[@lenoillionel2342]

@@NerdyRodent so thats what the "no stream" option is for 😅.. thanks bro.. you are the best

[@hipjoeroflmto4764]

Make a video on auto gpt plz having pinecone errors with it and all the videos are already outdated

[@GlenBland]

I am interested in training custom text lora models.

[@marklandmark3269]

Please can you helpme? I receive a KeyError: 'images'

[@funfunnier3037]

This is absolutely huge, imagine all the (n)SFW images we can produce!!! Jokes aside, I kinda have an issue with Stable D, basically says it refuses to connect, I must be doing something wrong. edit: found the issue.

[@NerdyRodent]

Make sure you’re connecting to the sd api on the right port, or if it’s on another computer that you’ve not got a firewall in the way

[@vitalis]

awesome! could you please update this for sdxl?

[@NerdyRodent]

I haven’t tried it, but the api should work for Sdxl?

[@escoladetecnologia]

How to do on collab ?

[@Websteria]

I get an error everytime I use it about a keyerror:Display

[@Scorpiove]

Same here.

[@pinpointping6175]

Same...

[@pinpointping6175]

Try different configurations with chat and extension enabled. I got it to run only when not changing the interface and enabling extension at same time.

[@MustafaDarwish1]

This needs a detailed installation tutorial!

[@NerdyRodent]

Ikr! Sorry I couldn’t make it more complicated than a 1 second copy & paste 😁

[@OsakaHarker]

For some reason is sending the prompt word by word and creating a picture for each...

[@AltoidDealer]

Add the launch flag -no-stream in the .bat file

[@OsakaHarker]

@@AltoidDealer Thank you so much, that fixed it.

[@XHackManiacX]

@@AltoidDealer You are an absolute legend m8. Had the same issue and this fixed it

[@AlienAnthony]

I would love to use the text generation webgui but it's api tools are absolutely busted. I've tried so many different ways to build a app to make it run. But it also fails to work on a local network even after enabling -listen.

[@NerdyRodent]

api and listen are very different options ;)

[@AlienAnthony]

@Nerdy Rodent yeah... ik. I'm saying both are busted. Using the built in api always gives an error when initiating it through the code on the local machine. Even the example code doesn't function it just gives an error. And I can't even connect to the standard web gui through another computer even after enabling the listen functionality.

[@AlienAnthony]

update. local network. Fixed. Windows default settings are to block all incoming traffic and it refused to turn off. API still refusing all attempt though.

[@axerawr]

Is this possible in Tavern AI ?

[@NerdyRodent]

Not that I’m aware of

[@soultakerspirit3121]

My textgen crashes every time i apply your sd_api_bot extension in the interface tab. I hope this comment doesn't get deleted like the other did.

[@NerdyRodent]

Try with the latest update 😉 Also note that any comments with links will automatically be hidden by KZbin

[@soultakerspirit3121]

@@NerdyRodent lol. i do have the latest SD and textgen. And the comment that got hidden or deleted didn't have a link in it. I was only saying that i was getting errors. The error i'm getting is in the comment of your new video.

[@timeTegus]

can you make a text to speech voice cloning video :)

[@NerdyRodent]

AI Voice Cloning - Totoise TTS kzbin.info/www/bejne/gGSQm5mJZ56FfJY

[@Some1uNo]

Damn you 😅

[@NerdyRodent]

😃

[@Charlton222]

If anyone has trouble getting sd_api_bot to be seen by Oobabooga, it might be because you now need to put the extension in the Extensions folder, not the main directory. Unfortunately, I could not get the bot to hook up with SD. It just kept hanging when I asked for the image to appear. Great idea, though!

[@johnny14794]

it's exactly what am going through, have not find a way to get it to communicate with SD.

[@electrocola9765]

Just set up --api flag for stable diffusion, it will work then

[@levansegnaro4637]

Can you hook this up to vicuna?

[@NerdyRodent]

You can use any supported model you like 😄

[@levansegnaro4637]

@@NerdyRodent have you seen vicuna's capability? It's 90% as good as chatgpt. It just got released a day ot 2 ago, please do a video on it.

[@NerdyRodent]

@@levansegnaro4637 Yup - it's just another model. Download & run with this as normal. It's not open / for commercial use though

[@levansegnaro4637]

@@NerdyRodent oh that's awesome, a bit off topic but can you connect any of these gpt's to a music maker like dance diffusion or something?

[@NerdyRodent]

@@levansegnaro4637 I guess you could use stable riffusion?

[@Bartetmedia]

Are we still playing with Windows? Dude, it's 2023 where's the Mac Install?

[@NerdyRodent]

Windows? No one still uses that, do they? Linux all the way - it’s like a real Mac! 😉

[@rafaeldelrey9239]

It didnt work. error accessing gradio.shared["display"]. Does it require automatic1111 to be run with sharing enabled?

[@NerdyRodent]

Just the api is fine, though you can share it if you like!

[@rafaeldelrey9239]

@@NerdyRodent I asked about the sharing because the error I got was related to the code accessing the shared methods of gradio