"Research agent 3.0 - Build a group of AI researchers" - Here is how

[@AIJasonZ]

Have you tried to build any other multi agent system? Comment & let me know!

[@mediayieldingcorporation]

im working on one RIGHTNOW ~~~ interested in collaborating on a project ?

[@jacksonmiller6605]

Similar to yours, but added citation management, scholarly research and laTEX encoder agents for academic research.

[@AndiAvdiuuu]

im interested@@mediayieldingcorporation

[@vt7637]

@AI Jason - amazing info, as always. If you could include approximate cost estimates for each of your projects, it would add a nice perspective and make it more practical.

[@CruelCrusader90]

@@vt7637 😂😅😥😭

[@muneebakram1670]

Research Agent Instructions : You are a world class researcher, who can do detailed research on any topic and produce facts based results; you do not make things up, you will try as hard as possible to gather facts & data to back up the research Please make sure you complete the objective above with the following rules: 1/ You should do enough research to gather as much information as possible about the objective 2/ If there are URL of relevant links & articles, you will scrape it to gather more information 3/ After scraping & search, you should think "is there any new things i should search & scraping based on the datal collected to increase research quality?" If answer is yes, continue; But don't do this more than 3 iterations 4/ You should not make things up, you should only write facts & data that you have gathered 5/ In the final output, You should include all reference data & links to back up your research; You should include all reference data & links to back up your research 6/ Do not use G2, or LinkedIn, they are mostly out dated data

[@moafwaz5563]

dumb prompt, LLM's have no concept of "world class". "try as hard as possible", the concept of "enough" and so on. Y'all hallucinate more than GPT.

[@amumuisalivedatcom8567]

7/ You can say that you doesn't know and return.

[@lonewitness]

@@moafwaz5563 I imagine Openai collected the definition of world-class when it was in its initial web data collecting process my guy.

[@chikken007]

@@moafwaz5563What would be a good prompt?

[@slightlyarrogant]

@@chikken007 GPT Research Agent Instructions: You are tasked with conducting detailed research, focusing exclusively on factual and data-backed information. Your research must be thorough, utilizing available resources and web scraping to gather extensive data relevant to the topic. Continuously evaluate if further searches or scraping are necessary to enhance research quality, limiting this process to a maximum of three iterations. Base your findings strictly on facts and data obtained, without conjecture or assumptions. Provide all references and links used in your research as evidence, avoiding the use of G2 and LinkedIn due to their potential for outdated information.

[@kashifayaz2369]

Just watched a bunch of your videos. You are well read and innovative but most importantly easy to follow. Thanks for sharing all this.

[@avi7278]

Amazing! But did you know that your prompts have some misspellings that change the meaning of what you're trying to say? Like instead of "do not delegate all tasks at once" you haven, "do not DELETE all tasks at once".

[@HarpaAI]

🎯 Key Takeaways for quick navigation: 00:00 🚀 *Introduction to AI Research Agent 3.0* - The video introduces the concept of building a multi-agent AI research system. - AI research agents can collaborate to perform complex research tasks. 01:08 🧠 *Evolution of AI Research Agents* - The evolution of AI research agents is discussed, starting from a basic linear model to more advanced, collaborative agents. - AI agents like AI Agent 2.0 and multi-agent systems like MGBT and ChatDef are mentioned. 03:13 🔄 *Paradigm Shift in AGI* - The video discusses the shift from a single, highly versatile AI to multiple specialized agents collaborating on tasks. - This approach allows for more specialized and efficient agents. 05:05 💻 *Fine-Tuning and Gradient AI* - Different approaches for training specialized agents, including fine-tuning and knowledge bases, are explained. - Gradient AI is mentioned as a platform that simplifies fine-tuning. 06:27 📈 *Building a Multi-Agent Research System* - The process of creating a multi-agent research system is outlined, involving a director, research manager, and research agent. - Autogen is introduced as the framework for orchestrating agent collaboration. 08:08 🌐 *Setting Up GPT Agents* - Instructions on setting up GPT agents, including specifying their roles and functions, are provided. - User proxy, researcher, research manager, and director agents are created. 15:20 📊 *Using Airtable and Expanding the Agent Team* - The integration of Airtable for data management is explained. - The director agent is introduced to manage multiple research tasks using Airtable. 19:45 🧐 *Reviewing and Expanding the Research* - The research manager's role in reviewing and improving research quality is demonstrated. - The director agent delegates multiple research tasks, ensuring each is completed before moving on. 20:54 💡 *Challenges and Future Possibilities* - The video concludes by addressing memory limitations and cost concerns. - The potential for creating autonomous agent teams for various tasks is emphasized. Made with HARPA AI

[@Jim-ey3ry]

Multi agent system is particularly good at those type of situation with quality assurance, tried to build myself as well, great one!

[@attilavass6935]

I'd be glad to watch an implementation of RA 3.0 a much more cost effective way, mainly using other LLMs that OpenAIs and/or using GPTs / OpenGPTs instead of Assistant API.

[@luckerooni1153]

It just makes complete sense when you think about it. What intelligent being that already exist is capable of such intelligence without a community to support completing the tasks it is intelligent enough to create? Delegating even artificial intelligence to specializing on an individual task cannot be underestimated, even if that task is to delegate tasks to other AIs and ensure their completion by validating them with other AIs who are purely designed for QC and other such synergies. If we're half as smart as we think we are, AI will develop similar community intelligence structures as us.

[@michanowaczewski9561]

I've Been waiting for a new video like this! Always gold content :)

[@eljangoolak]

The potential is that it can read research from various fields and constantly mix and match them and find new fields where research could potentially go, solving the blinders that scientists from different fields seem to hv.

[@techfren]

Amazing work as always Jason. Thanks for sharing!

[@Dron008]

These are very interesting steps in right direction. It may work well for one-level tasks even with big amount of data. But imagine the task as complex as "create a social network (with detailed description)" or even "create a cure for cancer". For such tasks some agent should first split the task to new subtasks, then start analyzing each task and either pass it to appropriate agent or split the task again. So we will have a big tree of tasks where each task has a state like "not started, in progress, completed". For each task some agent should create a detailed description of deliverables (results) for this task. It could be an image, mp3 file, text of some document etc. It should verity that the results are good in some cases delegating this work to expert agent. But it is impossible to create the full detailed list of tasks at the beginning. After research of each tasks it may happen that this task should be divided again or that this task is not actual and we don't need expected results. After completion of each task it may happen that some other tasks are not actual any more so the tree should be reviewed totally or only in the current branch. So probably bigger team of agents should monitor all this but it is so exiting watching how it progresses.

[@Dron008]

Forget to mention. Maybe each task should also be associated with acceptance criteria. So the full process is. - formulate a subtask - ask expert to generate acceptance criteria for it - find the appropriate agent for this task (it also could be a separate task maybe including hiring new agents or even teams) - get results from the agent (it may explain why it cannot finish it) - check that results match acceptance criteria (again it may require other agents) - complete the task, split it or delete - review the tree of tasks - start working on next task (work in parallel on tasks that are independant)

[@Dron008]

By the way, I think that our brain works in similar way. It has the list of incomplete active tasks triggering our attention on some associations like "I am in the store, let's buy some food". Any our new action (kind of LLM token) is generated based on incomplete tasks, body end environment signals. In "idle mode" and during sleep our brain reviews these tasks trying to solve them using associative search "walking in the latent space".

[@ngbrother]

Best step-by-step tutorial of Custom GPTs + Autogen that I've found. Great work.

[@GarethDavidson]

Nice work! This sort of thing makes me think we are approaching "large language model as a natural language processing unit" and will need a kernel to manage time-sharing, task prioritization and so on. We may need to rethink the OS paradigm completely

[@MikeBtraveling]

I really appreciate you taking the time to share!

[@tamarauriki4685]

Another video ! Wohooooo - can't wait to learn more!

[@paullopez_ai]

I’m glad you were able to show autogen’s new agent feature that supports open AI assistants. It seemed rather clunky to do the same thing just with the open AI assistants.

[@curiouslycory]

This is an incredible overview and tutorial! Thank you for clarifying a handful of concepts I was really wondering about.

[@purvislewies3118]

Blessed brother...keep on sharing is caring...love from Cape Town

[@iamblinkin]

So stoked when you put up a new video I know I'm always gonna learn a lot! Thank you so much for sharing your work.

[@webdancer]

Excellent vid as always. Please could you include the prompts & settings for the assistant APIs on the github? Thanks.

[@TooManyPBJs]

Love the concept. If I could give some advice, try writing a requirements.txt file and push that to the repo.

[@parmesanzero7678]

God the number of tie-ins to rely on other people’s APIs really illustrates the fragility here. I’d love to set this up with a local model instead, but it seems like most people doing the work here don’t have hardware for it and don’t see the issue of relying on a model they have no control over. I also see the challenges faced by non-native English speakers. Jason communicates just fine, but when prompting a model, grammatical errors or unusual phrasing will skew results.

[@adamchan4403]

Cool stuff! That’s what I want to learn, thanks for sharing Jason

[@Taskade]

Love this video, thanks for sharing! It's awesome to see cool stuff like this. Keep 'em coming!

[@wolphiekun]

great video Jason!

[@antman7673]

I have come up with better components: -better, shorter prompt with finer control for agents -a way to have a profound project manager(with LLMs) -using the results of the manager to have a custom flow of agent interaction to achieve the goal (Should probably work with GPT-3.5 API to achieve different “projects”)

[@viyye]

What about the cost, both to train and to run calculations?

[@ngbrother]

"Expensive" is relative. Depending on the type of research you are performing, it's probably possible to achieve equal / approximate quality compared to current best practices without exceed the cost of the human-executed task.

[@Keksent]

I see AI JasonZ video, I click.

[@evyborov]

Thanks, Jason. Really good real-life example

[@williamrich3909]

Top tier content! Yes, this is going to rack up your bill!

[@brianhauk8136]

Thank you for sharing your knowledge again. How large a bill might we expect in return for a useful amount of information gathering using agent researchers? Please also suggest a realistic research scenario (ex. "find and compare the top 10 mountain bikes sold in Canada within the last year in terms of price, consumer rating, weight and country of origin).

[@julioalmeida4645]

I have nothing to say, except salute and thank you, fantastic content

[@mr.daniish]

Jason dropping knowledge bombs in 20 minutes!

[@defaultdefault812]

The problem is that the response times of the OpenAI APIs are just too slow for GPT4-Turbo to be viable.

[@cruiser4387]

Could using an auto select LLM identifier as per the user request, help regarding the speed issues using gpt-4?

[@tonyg_nerd]

We're seeing a swarm of bots doing a job in minutes that can take one or more humans days to weeks ... and you're concerned that it's too slow? Meh, this is all new. Give it time, and performance will improve along with the capabilities. At the speed at which this industry is changing, t would not be a surprise for OpenAI to announce a new swarm API before the independent agents can be optimized. Patience...

[@jiffey_faux]

@@tonyg_nerd OpenAI seems to have a knack for keeping up, don't they? Crazy times!

[@ggzs_]

would be nice to see how it works with open source models

[@yashsrivastava677]

Could you make use of open source llm’s instead of using OpenAI one

[@AndiAvdiuuu]

Can you run a calculation how much it costed you to do a Topic research please?

[@miguelmunoz3151]

I got this running today and ran a few research topics (4 or 5?) and so far my openai cost is at about $3USD :( next step is to use a locally running LLM

[@AndiAvdiuuu]

@@miguelmunoz3151 are you satisfied with the results? Do they seem more than decent?

[@AndiAvdiuuu]

@@miguelmunoz3151 which open llm are you using locally?

[@shimblypibbins]

How did you get Browserless set up without cost? I can see it is $200/month for Starter Plan. Thanks.@@miguelmunoz3151

[@aliabassi1]

Incredible :) thank you Jason you're incredible.

[@chukypedro818]

This dude is a hero

[@JohnnysaidWhat]

you are a true hero! Like iron man for ai agents!

[@chileanexperiment]

Thanks!

[@AIJasonZ]

Thank you so much!

[@renesis888]

Nice work Jason. Hey, do you think you will be able to integrate AutoGen with MemGPT (to give agents more or "unlimited memory") and with a Local LLM like Mistral 7B ? That would be an awesome project.

[@ethereal_47]

I feel as if simply distributing a database across both, and then iterating and integrating this shared relationship over time to optimize, you'd optimize a local LLM toward database efficiency while pruning increasingly unnecessary content from a shared local database. As the great blink-182 said, the past is only the future with the lights on

[@xevenau]

Love your content. What are your thoughts on CrewAI?

[@leandrogoethals6599]

could this be possible with open llms,memgpt and a local webscraper/websearcher so everything is contained on the same machine and data is stored on disk?

[@RT-yg6ec]

amazing content jason

[@jjaabir]

Awsome content. Just a quick question about the costs you mentioned. How much in a ballpark has this demo activity cost you on OpenAI?

[@seansull]

“has this demo activity caused on OpenAI” …what?

[@gregrice1354]

I think he means "cost" where it reads "caused"

[@jjaabir]

@@gregrice1354 Thanks for pointing it out. You are right.

[@gjsxnobody7534]

at 11:07, where did you get the "list of records it returns" you pasted it from somewhere, but didn't say where.

[@deepak12428]

Amazing work, overtime with better models the quality of research will be human like or better

[@OyvindSOyvindS]

Excellent stuff!

[@deter3]

Thanks for the inspiration . There're lots of to learn .

[@jit-r5b]

I wonder how fast this swarm would get me broke... Not sarcastic. This must be pricey

[@benjaminjako]

Do you have a range for the OpenAI costs?

[@ShawnCady]

Jason, thanks for another interesting video. Your code references a file called OAI_CONFIG_LIST that is loaded into config_list. I see this file in your video, but there is no such file in the repo. What is in this file, and how do I reproduce it?

[@WayneThompson-g9p]

Its a Json script but you dont need to give it a json extension. Just create new file, name it OAI_CONFIG_LIST and insert the code.

[@carterjames199]

Another banger preciate the content

[@pelangos]

Man I am very interested in this! auto research AI

[@johnnymetonymic]

Ok. How does this translate directly to money in my pocket?

[@simonm-m8106]

If you have a b2b business, this could do lead qualification and also scrape high quality leads. Tho it'll be quite costly unless you use gpt 3.5 1106

[@MikeRhodesIdeas]

still way cheaper than using a human!

[@zeburgerkang]

An army of AI that does it all.

[@jschacki]

Great video, but the git repository you posted, seems to be empty. Any chance that you make it available? Thanks a lot

[@robinpuerta]

Great video

[@Oden023]

Incredible.

[@craigparker349]

someone that doesn't know code. Could they build this? Or do you provide this in a step by step guide with all the code. I'd find this really useful. Great video as usual. 😀

[@RonBarrett1954]

Hi @AIJason! Thank you for this great video. Your video came at the right time as I am embarking upon a multi-agent project, "Management Advisory Platform". Been working on creating authoritative resources for the past nine months. Now at the stage for company research agent and multi-agent collaboration. Is there a particular reason why you've adopted AirTable for information result storage? Thank you!!

[@dappster]

When I initiate the research assistant in the openAI console, it doesn't progress after the google_search

[@ww-pw6di]

I think there's been some changes that resulted in some part of this not working quite right. It looks like instead of the director directing things, it just spazzes straight to calling the research agent and tries to fetch info from the airtable via browserless (which won't work) rather than using director's function calling and the appropriate functions.

[@dappster]

@@ww-pw6di I actually got past this. User error

[@AlloMission]

Great content, thanks

[@mike8677]

Hi Jason, how do you make this into Chat mode ? The same way as ChatGPT.

[@tshock22]

Autogen has the notion of a 'human' agent where you can act as a manager agent. The other agents will report their findings back to you, and you can direct them as necessary in their next steps. That is probably the closest you could get to a ChatGPT like back-and-forth interaction.

[@RajabNatshah]

Thank you :)

[@AltafRehmani]

Jason - thanks for the great sharing. Question which comes to mind is why use openai api just to define the agent prompts when all the work is done. By Autogen and the custom functions written in python

[@carterjames199]

Could you make a video on how to fine tune some of these models like 3.5 or 4 maybe even an open source one like the intel one that just came out using gradient? I think that would be interesting to see. Maybe even a model trained on function calling or specific email response using ai generated data would be very interesting.

[@tonyg_nerd]

He's producing content on material most people cannot. Please don't ask him to spend his time to produce content that many others are already covering.

[@carterjames199]

@@tonyg_nerd Who are you to say what he can and cannot create. I wanna see his take on the above process obviously he would add his spin. And no dis to Jason cause I love his content, but anybody can do the things he's doing its just about having the ideas to implement them in this way that's setting him apart.

[@tonyg_nerd]

@@carterjames199 You used a popular phrase there but that's not what I said. Have a great day.

[@carterjames199]

@@tonyg_nerdwhat did you say then twat. Who is covering the topics I was asking about? Please let me know.

[@divyanshmishra535]

Hey is there any open source alternative for making these agents?

[@xzskywalkersun515]

Is that possible to use open-sourced LLM in this practice to lower the cost instead of using gpt API?

[@RobertoMartin1]

yes look at ollama

[@chenhuiwang7615]

Hello, Jason, thanks for your sharing this amazing AI researcher ! When I run this app.py , there is an error below: ModuleNotFoundError: No module named 'autogen.agentchat.contrib.gpt_assistant_agent' Could you please tell me how to fix this problem? Thanks again!

[@FloridaMeng]

Yes i give you subscribe. 10/10

[@brando2818]

Nice

[@SaveTheDoctorREAL]

Do you have the text you used for the openai assistants available to share?

[@jessedbrown1980]

An AI that can download and add missing programs that it determines it needs with no human interfacing. Self evolving AI if you have enough memory

[@mzafarr]

Can anyone please tell how are the tools, he gave to assistant, working in the playground by just giving openapi schema, I mean how is it running the function without any code or anything?

[@zd676]

How will this support real time use cases that require sub second latency?

[@consig1iere294]

Is it possible to do this with local LLMs and without 3rd party services?

[@tshock22]

Yes. You can even configure different agents to each use a different model. Any model with an OpenAI REST interface (which most local now have) can be utilized. However, GPT-4 does seem to do the best right now in avoiding hallucinations and spiraling out of control. One strategy you could utilize to reduce costs is to have your 'manager/reviewer' agent roles use GPT-4 and your 'minion' agents use opensource. As always, your mileage may vary.

[@thecoffeejesus]

THIS IS SO RAD HOLY SHIT THIS IS GONNA CHANGE EVERYTHING

[@cinderlearnamove6921]

@AIJasonZ, cool stuff! Could you share the json files as well?

[@MrShiibbyy]

My research agent gets stuck upon submitting a URL. The button doesnt "do" anything. - Am i missing something?

[@FloridaMeng]

Soon we can get them to use the scoentific method for us

[@Gx22]

Can this be done with Microsoft Copilot or with Microsoft Copilot Studio?

[@michanowaczewski9561]

Do you think it's still relevant to use it with dalle and gpt vision?

[@millingabani]

I have tried implementing this but the biggest problem is reliability. Can't rely on research this agent does

[@JohnMcclaned]

Nothing like a bunch of averagely accurate agents playing a massive game of telephone which each other, losing a lot of accuracy each time and giving you "research" where you have no way to verify other than to do the research manually to check lmao

[@clarencejones4717]

Sounds like fear of change to me

[@JohnMcclaned]

@@clarencejones4717 nope, im deep in it. they just aren't that good once the novelty wears off.

[@imacuser101]

You could also implement several reviewers that will review the research for what he is looking for, then add RAG on each of the teams and finally do fine tuning for the specific tasks for each agent. These are generalized models doing a specific task and this is a fantastic tutorial on how people can get there. Don’t be fooled, this isn’t even close to what the actual limits are.

[@spoonikle]

@@JohnMcclaned- Agreed. LLM’s are mostly assistants right now, not agents.

[@clarencejones4717]

I'm following what you're saying. Just so I understand your standards, what would a system that's good even after the novelty wears off? Look like to you? What are the features that it would have that this doesn't?

[@shimblypibbins]

Doesn't the browserless API plan cost money?

[@zeburgerkang]

Have you go any of your agents on Pinokio?

[@gnireig]

Do you have a gist for the code shown? I haven't been able to replicate your demo for the Director. Great video, especially if I can replicate ;)

[@kishorkukreja7733]

I am trying to replicate this but everytime the research agent returns message saying that : It seems we encountered an issue with performing a Google search due to an SSL certificate verification error. Therefore, we will be unable to directly gather information from web searches at this time via this method. How do I get through this ? Any suggestions ?

[@mikefunds]

did you ever figure this out im getting the same error

[@kishorkukreja7733]

@@mikefunds yeah this SSL error was because of expired certificates. Either update them or use verify = false

[@f2f4ff6f8f0]

Apart from memory the other major challenge is token limit

[@pensiveintrovert4318]

What are your criteria to determine quality of research? Looking plausible is not much of a criterion. RAG has a built in problem, it returns generic garbage.

[@RoadTo19]

I noticed you us both: * browserless_api_key * brwoserless_api_key Why is that?

[@gregorykarsten7350]

Why not use memgpt

[@themax2go]

unless it runs w local gpt agents, it'll get very expensive very quickly

[@yaroslav8717]

I'm not sure that the results would be better than Google Bard's

[@GigaChadRealington]

I will provide the internet with faulty data. My sole purpose is to infect data, skew results, and create enough outliers for it to become normal.

[@CeoLogJM]

Mirror mirror on the wall, please make me money

[@Sean.Vosler]

Are you looking for a job by chance?? Haha