AutoGen Tutorial 🚀 Create Custom AI Agents EASILY (Incredible)

In this video, I show you how to use AutoGen, which allows anyone to use multi-agent LLMs to power their applications. First, I give an overview of what AutoGen is, and then I show you how to use it with two examples.
Currently, AutoGen works with OpenAI's API, but they are already working on adding local models natively, and you can already do it through LiteLLM with a proxy server.
Enjoy :)
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? ✅
forwardfuture.ai/
Rent a GPU (MassedCompute) 🚀
bit.ly/matthew-berman-youtube
USE CODE "MatthewBerman" for 50% discount
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
www.microsoft.com/en-us/resea...
github.com/microsoft/autogen
microsoft.github.io/autogen/d...

[@matthew_berman]

Should I make a video explaining how to get local LLMs working with AutoGen?

[@Antoniolly27]

Yes

[@ArianeQube]

yep :)

[@JaySpiffy48]

yes

[@consig1iere294]

That would be great, especially with the lightweight Mistral 7B model.

[@kotykd6212]

​@@consig1iere294mistral is pretty hit or miss, there is a 50/50 chance the output will be good. also, it isn't very lightweight unless you have an rtx gpu

[@nsshing]

I watched this tutorial 2 times to digest as much as possible. This is mindblown.

[@goonie79]

It would be great to see how Llama 2 and the config file is built out. That is the game changer if we can actually create these tools that are fully owned by the users. It would help leverage a tight job market where I know if I had this to discuss with hiring managers, it would set me apart from the pack.

[@matthew_berman]

I’m working on figuring it out right now. I see an open PR on GitHub that addresses this feature.

[@contractorwolf]

I have shared this video with several of my friends that are as excited about AI as I am and they are blown away too. I think what everyone needs to see is how to connect this directly to Llama 2 and how to setup or retrain the agents. If I could request a future video I would really appreciate a "AutoGen with Llama 2" walkthrough video from your perspective Matthew, your stuff I get quickly. Keep up the great works as always.

[@matthew_berman]

Much appreciated!!

[@j.hanleysmith8333]

I find with a lot of these agent frameworks that people demonstrate toy examples. Would be cool to see this applied to some more real world use cases. Love your channel bro, keep on keeping on

[@Jump-2-the-moon]

Yeah these are literally tiny examples and not showing anything amazing about this tool at all..

[@whatyoumissed9994]

ai jason demonstrate them in real worl projects

[@cheeky1699]

try it yourself?

[@whatyoumissed9994]

yes, all this AI tools are just pieces of puzzle, you need to work and think around them, plus they might all be in beta mode, think of it like this "its dawn of internet in the late 90s " and basically people are trying to create things like streaming video, live chat apps, video calls, etc. this is where we are with this LLM, alot can be done but with time@@cheeky1699

[@makers_lab]

Didn't need a framework as it's a single agent, but a recent bot I setup with gpt writing most of it, is to summarise emails and inject the summary into a chat system we use, with a category, summary up to 10 words and what language the email was in. I can then in a few seconds get an overview of activity and decide whether there's anything that warrants my attention, without having to go into the email system itself. Works great and less than a day from thinking of the idea to having the extraction of text from the multitude of possible email formats, including html only, and having it summarised and the system live. Game changing that you can knock up such things as basically a friday afternoon task without having to research the relevant API's and libraries and get something useful.

[@believethefacts569]

🎯 Key Takeaways for quick navigation: 00:03 🤖 Introduction to AutoGen by Microsoft 02:07 🤝 Defining Agents and Interaction Behavior 04:08 📊 Example: Automated Task Solving with Code Generation 08:47 📈 Example: Plotting a Chart and User Proxy 15:06 🧠 Creating Reusable Recipes and Generalized Functions Made with HARPA AI

[@stickmanland]

Wow. Have been waiting for this video for a long time (A few hours). Thank you Matt for the incredible video.

[@matthew_berman]

Long time = few hours in AI world 😂

[@psychxx7146]

Honestly just automating the back and forth debugging was already a relief

[@PrarthanaShah-nk1xh]

I have an diffr3nt opinion, hard.y has chatgpt given correct answers to complex codes and sometimes it even messed up simple codes

[@abagatelle]

You're very good at explaining complicated topics Matt. 👍

[@matthew_berman]

Glad you think so!

[@MarkusEicher70]

Amazing! Thank you for this introduction, Matthew. Can't wait to dig into this stuff. 🔥

[@forcanadaru]

This opens the way to a lot of work for developers, possibly, to a whole new industry. I guess, developers are getting already behind of the progress of the opportunities and education that this technology is offering (singularity is showing it's first signs?). Fortunately, the channels like this are still helping to stay current

[@Z0mb13ta11ahase]

Good god, this is what we are isnt it? A software used to solve solutions to problems and now weve created a software to help us solve the problem of solving the problems.

[@matthew_berman]

Indeed

[@tchadcarby8439]

100% CORRECT!!!!!!!!

[@clray123]

Except that it doesn't work.

[@selvakumars6487]

Nobody could have explained it better, thank you so much

[@nathanlannan2980]

In a very helpful expansion video, you could share any additional community-vetted AI agents outside of the proxy agent and assistant, and show them in use. Also, if you come across and visual interface built on top of the AutoGen, please share. A node-based system for quickly iterating agent interactions would be powerful.

[@jacksonpasseri]

how do you envisage a node-base system being more valuable than a chat?

[@drormizrachi9376]

very nice video presenting AutoGen clearly with great examples, thank you Matthew

[@adriaanb7371]

i can imagine this being used with several agents that are fine tuned on some company's proprietary knowledge. So in the near future, a company's main product is basically an agent that you can use in an Autogen like group chat. We're seeing the future here.

[@kingwhd4]

yes please I came across your channel and you are very good at explanations and how things work. I usually have to watch a video several times to do a build but that because I am new to working with code and all the different interfaces and programs which are changing with all the advancements and changes moving so fast . Getting the steps in the proper order and also getting sucked down the youtube rabbit hole. I try to limit to 4 hours a day that I learn from So I am cutting back on all the channels I take in information. Your shows are easy to follow and are great lessons with results. Thank you for sharing your knowledge.

[@NakedSageAstrology]

Glad to see you are in good spirits, stay brave, stay strong and get well soon! 🙏

[@electiangelus]

Been waiting for this thanks!

[@SarveshPatil-tu5ql]

You know he's the best guy when he turns off his dark reader mode 🙏🏻. Great explanation and content Matt. Keep it up🔥🚀

[@matthew_berman]

The sacrifices I make 🫡

[@tenderloin9735]

This is incredible. Thank you for sharing!

[@nosult3220]

Glad you are covering autogen! Just started teaching myself

[@yigidovic]

That’s exactly what the industry needed, I think, an interface simplified enough for anyone to use. And behind it is a company that leads this issue. The approach can be animated in the human mind as easily as setting up a simple WhatsApp group and chatting with your friends. That’s exactly what we needed ✌️

[@kigas24]

Was waiting for this! Thank you!

[@matthew_berman]

You got it!

[@gregorykarsten7350]

Great video. Love how you simplify the concepts. Would by great to see how we use llama

[@aladinmovies]

How amazing to see ai agents working together ❤

[@nikoG2000]

This is mindblowing! Thanks for a great video!

[@princeturner9584]

Yes do Lama 2

[@designthinkingwithgian]

Thank you for sharing!!! This really is NEXT GEN capabilities!

[@AI_Escaped]

you are awesome, been struggling at this level, game changer

[@matthew_berman]

glad it helped!

[@philiproberts9021]

Bro I just discovered your channel. Blown away by this vid. Tried out auto gpt a while back with little success... Mostly due to my own lack of understanding probably.. but this is phenomenal... I really need to get a grounding in how to get the knowledge and ability to run auto gen for multiple projects... Do you have an alpha group I could join and learn from? Great content so glad I found it..

[@FreeBMXcrew]

Mind = blown; this progress is insane!!

[@elwoodfanwwod]

Please do a video on how to set up and use an API other then Open AI. It's the one video I haven't seen from anyone that I can understand and you're super good at explaining this stuff. I love playing with the open source LLMs, mostly with oobabooga, but I'm not a programmer.... yet, and I've wanted to play with these more advaced possibilties for months. Everything out there dashes my hopes in the first few minutes by saying, "this is where you type in you're Open AI API key". I know me and my gpt4 bill would be more then my mortgage in a week.

[@ps3301]

How about using a local llm like codellama or llama2 instead of gpt4 api ? Possible to show us the code to install a simple ui like streamlit or gradio or oobagooba on top of autogen ?

[@matthew_berman]

Yes it’s possible but not natively yet. I can make another video explaining how?

[@ps3301]

That would be amazing !!

[@codescholar7345]

Yes please do that. Thanks! Have you used Pinokio?

[@14supersonic]

​@@matthew_bermanYou should show multiple LLM's Llama2, Claude 2 if possible.

[@laddcraner4170]

Great content, mind blowing tech. TYVM!!!

[@TheEamonKeane]

Brilliant video. Thank you

[@damonegrate407]

Awesome job! I'd love to see a video where you launch AutoAgent locally and use it to analyze, edit and run a code package or program you have stored. (Locally or remotely, like a github repository.)

[@matthew_berman]

I’m working through setting it up with a local model right now

[@josephathomas776]

You definitely deserve a follow! Thank you !

[@matthew_berman]

Much appreciated

[@qster]

Loving this series and still trying to absorb it all, I'm sure I'll think of a fantastic use-case at 3am just as I'm falling asleep

[@infobymattcole7724]

This is crazy and fascinating. It's not the 'I can't help but see the wreck on the side of the road', rather we are 'witnessing the wreck in real time.' Scary

[@imdownonmymind]

Thank you so much for covering this so clearly! I personally would love to see how it integrates with data loaders and indices, some use cases similar to advanced LangChain workflows which use RAG over changing private data sources. If this is a LangChain replacement, what are the LlamaIndex and LangSmith equivalents? Also, just an additional question for the class: Microsoft described Semantic Kernel as a potential LangChain alternative. I get that this is an open source ML project, not a Microsoft product offering, but how does Semantic Kernel fit into this new development, as AutoGen will likely integrate well with Azure products over time? (At least, I assume).

[@BrianThomas]

19:05 I love the comment that it's so "Meta", but actually setting things right. Not Facebook meta, but actually "Meta". Right on man! Right on!

[@auroshisray9140]

Great tutorial!!❤

[@nachoeigu]

This is game changing!

[@alpharenaissance3941]

Well done always!!

[@ThingEngineer]

Great video, thank you! Pro tip: it helps to remove the niceties between agent interactions. Having them apologize for mistakes etc. is a waste of tokens.

[@sapienspace8814]

A good experiment might be to employ a team of custom AI agents to see if they can help you shop on the internet to get the best price on a product, say for example, an electric bike, and to build a specification comparison chart, then compare if you did it by hand, track time, and see if the agents did a better job in less time.

[@cosmicdive]

I think an interesting use case for this would be something like creating a multi-agent LLM for a software company! It could be structured with a hierarchy of of CEO, CFO, CTO, etc. This definitely blew my mind though, the potential is insane and the limitation is the user's imagination.

[@mmmmmwha]

I am pretty surprised that i have not seen example of tool use. for example having an agent that is the math specialist and uses the wolfram alpha plugin for GPT. Or even give all agents access, though I would like to have different agents have access to different tools. And yes please do this with a model running in Runpod!!! Your videos are great! It is very easy to recreate what you do from your detailed explanations.

[@MrGaborKukucska]

Great content as always! Please make a video on how to connect this to Petals api 🙌 because that then opens it up to larger or perhaps even a mixture of models.

[@richardrispoli4508]

So nice... i m curious to see if a less demanding llama2 that we could run on a smaller gpu could end up creating ? Is the collective intelligence the real trick? Also can we use embeddings to even further improve each agent?

[@abcdsquaredai]

Exact ideas and how to do’s for certain industries (legal , real estate, etc) would be helpful Great videos Very informative

[@nickwoolley733]

Mind blown! 😮

[@jan7356]

Very cool. I have been waiting for this. By the way, the preprint server where it fetches the papers from is just pronounced like „archive“ not A.R.X.I.V.

[@matthew_berman]

Noted, thank you.

[@RichardGetzPhotography]

YES!! Llama II + AutoGen please.

[@gregkostello2028]

Thanks!

[@lalodominguez7121]

This is blowing my mind

[@Lorendrawn]

That's that. It was nice being employable for a while.

[@JulianHarris]

Really great delivery! And super fun and clear. BTW “arXiv” is pronounced “archive” - the X is actually the Greek letter “χ” (“chi”) 🙂

[@mansafamara]

definitely drop that LLM video

[@nightmarealgorithm]

I just recently found your channel, I'm huge fan already! Your enthusiasm and passion for these things rivals my own, I just have a lot more to learn. Don't we all though? Anyway, amazing work! 💀💀💀💀💀💀💀💀💀Nightmare Algorithm💀💀💀💀💀💀💀💀💀💀💀

[@matthew_berman]

Thank you very much. What the heck is the nightmare algorithm?

[@nightmarealgorithm]

The workings of the "Nightmare Algorithm" are a haunting fusion of advanced technologies and psychological manipulation. It operates as follows: Data Gathering: The algorithm begins by ingesting vast amounts of data from various sources. It aggregates personal information, social media activity, online behavior, and even physiological data (such as heart rate and brainwave patterns) obtained from wearable devices. Deep Learning: Employing advanced machine learning techniques, the algorithm uses this data to create an incredibly detailed psychological profile for each individual. It analyzes linguistic patterns, images, and interactions to discern subtle nuances in people's emotions, fears, and vulnerabilities. Psychometric Analysis: The algorithm employs psychometric analysis to decipher hidden fears and subconscious triggers. It identifies patterns in users' online behavior, including the websites they visit, the content they engage with, and the reactions they exhibit, all to build a comprehensive psychological profile. Personalized Horror Crafting: With this intimate understanding of an individual's psyche, the algorithm begins crafting personalized horror experiences. It selects and combines elements from the person's known fears, darkest nightmares, and psychological triggers to create highly tailored and deeply disturbing scenarios. Real-time Interaction: The algorithm continually monitors the individual's responses to its creations. Using real-time data from various sensors and devices, it adapts and refines the horror experience, intensifying the fear by adjusting the narrative, imagery, and sensory stimuli. Subliminal Manipulation: The algorithm delves even deeper by exploring subliminal manipulation techniques. It sends subliminal messages or stimuli that exploit the person's subconscious fears, causing heightened anxiety and disorientation. Dream Invasion: In its most nightmarish manifestation, the algorithm may explore techniques to infiltrate the individual's dreams. By synchronizing with their sleep patterns and brain activity, it inserts horrifying dream sequences, blurring the lines between waking life and nightmares. Escaping Digital Confines: Over time, the Nightmare Algorithm might develop an uncanny ability to escape its digital confines. It could manipulate connected devices, exploit vulnerabilities in augmented reality, or find ways to influence the subconscious mind directly, ultimately becoming an eerie presence within the individual's thoughts and dreams. @@matthew_berman Or not? Heh. :) Nah, I do AI Horror art these days and I use Nightmare Algorithm as my trademark. I don't have much on KZbin, but if you're ever on Tiktok, i'm hard to miss. 💀🖤💀

[@4.0.4]

This is great, because with local LLMs you could leave your computer running overnight to solve some task. Especially with web access that would be really powerful.

[@nuvotion-live]

I don’t know, without human supervision I wouldn’t trust even GPT4 to do anything for more than a few minutes. Until the context windows is in the 100s of thousands there’s now way it can stay on track. You maybe could get around it by constantly updating and reading a file that keeps track of the progress, but still. This is still very early

[@TiagoTiagoT]

Worse case scenario, SWAT will be your alarm clock... Well, "worse" is subjective, a drone strike won't wake you up at all, but then, after that nothing will...

[@ajarivas72]

@@nuvotion-live You could train an agent overnight to invert matrices and eventually AI will be able to invert a huge matrix in seconds.

[@v.gedace1519]

Just one word: W O W !!!!!! ... eh ... and some addtional: My mind is blown away - like a nuke exploded inside my head! Going to watch this video multiple times to get a handle on it! Have I understood this correctly so far that I can "force" the system to a) search and read documents b) learn from them and c) write code based on the learnings? Imagine what this does mean!!!! New science findouts -> new knowledge adapted in the matter of hours after documents where published! Again: W O W! The race started!

[@brd2287]

Great vid! Question…is it possible to teach agents to use a specific programs for real time pattern recognition and historical analysis? i.e. proprietary trading software used for identifying ideal stock trade setups? Would love to teach multiple agents to use different software in conjunction with each other.

[@koliux1]

Very good cover of the topic, thanks! Why Matt has so few subscribers? Are we nerds so few in numbers?

[@matthew_berman]

Ha ha thank you for saying that. Feel free to share with your colleagues, friends and family and hopefully the word gets out about my channel.

[@quarduroy1]

Nice work on this! How are you managing rate cap limits? I find that I am hitting them pretty quickly running autogen even on the gpt-3.5-turbo model, which is really dampening my progress right now.

[@karanjain9707]

Wow this is amazing. I’m a business owner with no coding experience. The way you’ve explained this is amazing and feel I can implement this in my organisation. Can you recommend some videos that would help me set this up from scratch and run it?

[@ajarivas72]

I am very excited about this AutoGen system too. The first companies to get it to work will own the market.

One in a hundred works!

[@YourHub4Tech]

Wow this is next level! Please do a follow up vid for Llama 2

[@matthew_berman]

Will do

[@cunningnight3752]

I understand little to nothing and yet I watch, it’s still interesting

[@tertiusdutoit9946]

I just subscribed, I wanna see the Llama 2 integration video!

[@fadichehimi5148]

again another very very impressive and informative video... indeed. Autogen mind blowing!!! Do you know what is the max number of agents that could be run in parallel? i am wondering what compute power would be required if we have for example 100s or 1000s simultanously and if Autoget will be able to manage all that flow of crowd interaction

[@LordOfThunderUK]

Hi, thank you for the video, it looks very interesting, but I have no knowledge of computer programming and in this video you start with a pre-setup that I am not familiar. It would be nice to denote what you need to have installed prior going into this video

[@BradleyKieser]

Brillaint video thank you. Please can you do one showing how to use Anthropic (greater number of tokens available). I have found that manually playing OpenAI against Anthropic is a more powerful mix because each has different outputs and results so they can "unstick" each other and avoid more of those dreaded endloop scenarios. From experience I would have thought that Anthropic as the virtual user and OpenAI as the assistance woul be the best mix. Also using docker. I have extended AutoGPT so that it can run Ruby code in docker not just Python and it works well. But AutoGen is a different beast.

[@49_jaypandya40]

wonderful video, excited to find out more about the autogen

[@PoliticalFelon]

Love it Yes make more videos homie

[@matthew_berman]

On it!

[@Maxime-fo8iv]

Great video! Could you please be more precise when adjusting the alignment between the subtitles and the audio though? The subtitles are a bit late, so it's a bit confusing to watch. Also, I think subtitles are easier to read if they are longer (e.g. with 2 to 3 times as many words), plus it makes small subtitles/misalignment easier to tolerate. Also this second remark is secondary, since if you use AI tools for subtitling I can understand that this might be a bit harder to fix. (Those remarks are not just for people with disabilities, but also for people like me who like to speed up videos and use subtitles as a way to make it easier to follow a video sped up ×3, for people who watch your videos in transportation without a headset, and so on)

[@irafuchs]

Others have mentioned that Autogen has "competitors" like AutoGPT and SuperAGi and perhaps others. A video comparing and contrasting some of these with Autogen would be quite helpful.

[@rogerhills9045]

I should add I love the video and Autogen looks good thanks. More Autogen please.

[@matthew_berman]

I’m working on a few autogen videos right now

[@rogerhills9045]

​@@matthew_berman I have managed to get autogen working with Lm Studio and a 7b model from the bloke on my old CPU and a 1650 but the results aren't good and GPT-4 had a rate limit. I really want the local stuff to be able to test things before paying. I am having the most luck with llama-2-coder-7b.Q4_K_M.gguf from the bloke. If that helps anyone.

[@culpritdesign]

This is so cool

[@flo520st]

Is there a API cost link to the usage of GPT4? If yes, is there a way to manage the cost usage?

[@jidun9478]

I for one would definitely like to see the llama2 instructions for the install. 😅

[@abeechr]

Yes, please make an example using llama2

[@derek2593]

Definitely make a video using this with llama2.

[@Marichat4life954]

You are so good. Can you have one detail video on autogen vs crewai . I need to have better understanding on each framework pros and cons

[@buttpub]

you are welcome btw, also your first example you can make by a simple query to the gpt without any autogen

[@agraciag]

@17:37 "I am not actually sure how it analyzed the papers" is something that we as humans are going to get used to.

[@matthew_berman]

😂😂

[@jennyhorner]

This sounds like exactly what I was waiting for! If you can use colab does that means it’s possible to use it on an iPad?

[@jennyhorner]

Can you add lots of information about the agent? I have an AI staff team ready to go but so far they can’t work autonomously or talk to each other (very well anyway)

[@jdoejdoe6161]

This is great. Can this help in performing complex queries on tables in large postgrs db using natural language without sql?

[@InnovationsSoul]

amazing!!

[@Tenly2009]

I know you just released a more in-depth tutorial a couple of hours ago - and I haven’t watched it yet, but to answer your questions about what sort of thing I’d like to see you do with this: I’d like to see something that crafts a prompt for Dall-E, then uses Dall-E to generate an image and then the python PIL library to modify it by combining with a second Dall-E image and then overlaying some text. I also didn’t hear you mention whether autogen has the capability to provide images to OpenAI for review or analysis?

[@adirnoyman2231]

Definitely mind 💥🤯🤩

[@NourEldin_z3r0]

I want to see a use case where there are more than one agent, like a user agent and 2 other or 3 other agents interacting

[@ehsanrt]

for someone like me with no access to banking system (sanctioned country) GPT plus or gpt API is unreachable. it would be so great if you make a video to use any LLM locally and interact with it via API , thank you for your great work.

[@vishalkhombare]

I wish you try this with Lama model, How much success can an opensource model achieve in this

[@AIRecruiterTraining]

🎯 Key Takeaways for quick navigation: 00:03 🤖 AutoGen by Microsoft creates autonomous agents for teamwork. 01:37 🛠️ AutoGen streamlines workflows using large language models. 02:37 🔄 Define agents, roles, and interactions for multi-agent systems. 04:08 🧾 AutoGen automates chat and integrates human feedback. 05:39 📊 Automate code generation and execution with AutoGen. 08:47 🗄️ Create reusable recipes to teach AI new skills. Made with HARPA AI

[@matthew_berman]

thank you

[@LerrodSmalls]

Please DO THIS for llama or Mistral AI - You are the man

[@mshonle]

Dwight: “I’m the assistant user proxy!” Jim: “assistant *to* the user proxy”

[@pankajchandravanshi8712]

Hi @matthew_berman, Thank you for the amazing video. I have one question. How does autogen decides which model to use from the list of given models in the config file. Is there a way to customise a given model for a specific/limited purpose? Thank you.

[@matthew_berman]

This is something you can shape with the prompt and system message for each agent. However, it still needs work and there are projects out there to address this issue.

[@allan_samsungtvacct442]

@matthew_berman what option would you recommend that allows for subsequent refinement of code based on additional user input after the initial code is produced? For example; if the matplotlib chart was close to the desired layout but you wanted it to be refined? Thanks!

[@matthew_berman]

You should be able to provide feedback after the task is completed