Google Releases AI AGENT BUILDER! 🤖 Worth The Wait?

Google announced Google Agent Builder through its Vertex AI product. Let's examine it and other announcements from Google Cloud Next.
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? ✅
forwardfuture.ai/
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
Full Video - • Google Cloud Next '24 ...

[@arberstudio]

@<a href="#" class="seekto" data-time="1165">19:25</a> you set your Tool Type to OpenAPI, and the endpoints in the instructions will work. I imagine it's 1 function per tool for modularity. It's more or less exactly the same setup as the Custom GPTs configuration.

[@show-me-the-data]

Hey Matt. When you select "function", you're responsible for writing the code and driving the conversation via API calls. The flow goes a little like this. You submit a message and you check the status and it tells you you need to run the function with these inputs. YOU then submit the output back to vertex via the API, where it uses the function output to generate the next message. You can implement the function wherever you're DRIVING the convo. Makes sense? This is a very similar flow to the assistants API which I have a video about on my channel if you're interested

[@DailyTuna]

Matt on it! Thanks for going over this. Nobody else is really going into it like you.!👍👏

[@DailyTuna]

You’re so right Matt on that agents in the Mercedes. There’s this weird movement of bells and whistles, and what we can do rather than true functionality that is efficient. Especially when it’s not level five self driving, we still have to drive and less distraction equals more attention to the road and thus is safer driver.

[@TheFocusedCoder]

thanks for the review . I was short on time today , perfect timing

[@arberstudio]

@<a href="#" class="seekto" data-time="1570">26:10</a> they're using the equivalent of Agent (preview) app in the Agent Builder service which as they word it "can answer questions from data, connect with business systems through tools"

[@r3kRaP]

Hey Matt, with all of these models and apps along with their updates, it would be nice to see a real time list of your favorites. For example you said Gemma wasn't to impressive but it may be now with its new update. Maybe like a place where you can rank the best models or frameworks and update the source as things change. For example crew Ai, autogen, Gemma,, Devin, Devin alternatives, and all the different llms. It be nice to see an up to date list of your rankings in each category. Or does something like this already exist?

[@dafunkyzee]

i heard the keynote before, and I really appreciate your insights and commentary to make it more "real." There is often a strong corporate biase when we hear these kinds of releases. Great video very informative. Love your stuff

[@xerxel69]

Some of Matt’s videos were a bit lightweight in the past, but he really nailed this one. 👍 good analysis, brought in external knowledge and not just AI fanboy content. Nice job. Will be subscribing to this channel now!

[@matthew_berman]

Thanks :)

[@positivevibe142]

Nice video Matt, your huge efforts are highly appreciated. Thanks for sharing!

[@anthonyjobey8821]

Still watching but as always love the content and the way you deliver it, so glad I stuck around as you are my go to to keep up with whats going on

[@tecsmith_info]

Great announcement, thanks for sharing! I've been working with several of these offerings over the past few months, developing some great use cases. I was thinking of perhaps showcasing some of them on my channel soon.

[@Maisonier]

I've been eagerly awaiting a local AI solution to sort through our family's extensive collection of decades-old photos. This AI should be capable of identifying and eliminating duplicate or blurry images, learning to recognize faces, and categorizing photos by individual. Moreover, it should have the ability to estimate the age range of each person depicted. Ideally, this solution would allow for easy exportation of organized data so that Stable Diffusion models could learn from it. Additionally, integrating facial recognition capabilities into home security cameras would streamline access control, eliminating the need for physical keys.

[@aintaintaword666]

There's a program called Excire

[@myrakrusemark6873]

This is something a program would take care of, not an AI. If you need a program with these specifics, you can write one with the help of Claude3 in 6-12h given availability of one or two components. Digikam already does MOST of this. And all of this would be programmatic, not requiring an AI to run. Digikam can individualize people. All that's left over is a model that recognizes age, which Im sure exists, but you may want to write a program that iterates through all those images and adds subject age to the EXIF data.

[@fintech1378]

Whats the main objective if i may ask

[@MetaTaco317]

​@@fintech1378Going to take a wild guess and say to have an organized, searchable collection of family photos which allows for anyone in the family to quickly find any photo.

[@fantinigerman]

and a search bar, so you can search whatever from your photos

[@erikthegrey]

It's like Autogen with frosting on top. I can dig it saving me a lot of design setup time for agent clouds.

[@tsomerville1970]

Thanks for sticking through it - I dropped out on the live original presentation when I saw all the customer agent stuff vs the more interesting However, you pulled out a few more things that I missed

[@waynelast1685]

<a href="#" class="seekto" data-time="180">3:00</a> that’s great , I hope the model works well though

[@cemery50]

#Using ai to select ai, #Using ai to debug code and create insertion points (~cross cutting)....it could then analyze and respond by altering the code to fit conditions (~FPGLA). I'm working on knitting Neo4J and ai-models for an interactive teal time graph dashwindow to use user priveldged based access. It seems that ai could refine alternative models based upon a user's project and use rag to learn preferences...and present them ranked with pros and cons of each.

[@nikolaimanek582]

Nice video! Will you cover Mixtral 8*22b soon? That would be great to see. The 8*7b was already great.

[@metonoma]

the Mercedes missed opportunity is probably an insurance issue. They'd need to long term test of it's safe because they don't want to get sued if someone expects the ai to read their mind and then goes road rage and blames it on them

[@BillMarkerMastery]

Thank you. Your commentary adds real value to this video. I'll definitely watch more.

[@EccleezyAvicii]

Agent > Example > create example The example is a dummy dialogue. You need to actually mockup a weather convo. Each prompt/return is an ‘action’ in the example convo. There are other actions, like tool. Tool is where you put the code

[@MrVohveli]

Considering Google ought to be a solid year or so behind OpenAI, despite their best efforts, what kind of monster is OpenAI planning to release that beats all this?

[@bobnothing4921]

You're assuming this isn't all vaporware. It's Google. Virtually every 'tech demo' they do is a lie.

[@PseudoProphet]

😂😂, does OpenAI have any such infrastructure? If Microsoft is the one who will launch. 😂😂

[@WhyteHorse2023]

Altman says they are making these things smarter so they get better at everything. Then there's also their humanoid robot. Oh and they just released vision for chatgpt and newer turbo.

[@first-thoughtgiver-of-will2456]

Google invented a lot of what made OpenAI famous. Idk where you get that notion from Ive been expecting Google to wake back up on their ML work and lead us to the next breakthrough.

[@Brian-oz8io]

I think they’re mostly just doing different things. Google is catering to enterprise, commerce, and search capabilities. Sora will be more for Hollywood, artists, and content creators. OpenAI is working on a robot, a supercomputer, a voice generator, and an upgrade on their consumer chatbot with the ultimate goal of achieving AGI. I don’t see how those things really compete with each other, besides what Microsoft is trying to do. But obviously that’s a different company. I think Microsoft could lose to Google on that front and it wouldn’t affect what OpenAI is doing.

[@aosamai]

Thanks Matt for the review, as always you have perfect timing and bring a common sense perspective.

[@vishnunallani]

Tried to setup crewai today and was not able to handle my use case of sending image to gpt vision and get output from it. Any suggestions?

[@vq8gef32]

Thank you Matthew for all of your amazing videos

[@GarrettGalloway]

I'll be ready for Google to put that kind of tech to use on youtube. No more trap videos where the title, description, and thumbnail don't match the content. KZbin could auto-down-vote videos that don't match their title, description, and thumbnail. I can imagine that Google will be the king of multi-modality due to KZbin.

[@MikkiPike]

I'm very much not ready for them to scan videos for anti-capitalist sentiments and shadow ban them, perpetuating the head first dive into the end of the human race this train of capitalism already has us on.

[@Samuel-er6td]

I bet they start automatically editing yt videos to make them better. They can edit it based on what your searching for. Or edit it based off your watching style that it learns over time.

[@ScottWinterringer]

LOL they cant even keep feature parity. open maps and tell it to play music through blackpplayer ex LOL it wont do it anymore

[@GoogleTakesYourPrivacy]

@@Samuel-er6tdHAHAHAHAH I never want to watch Indian videos... But guess what?! . So Google's AI can't do its main job of giving you the right content when you search using their search page (which is the foundation of Google, their vision). What's left, getting into a car powered from a Cloud service of a company (that same "G") that has an extremely worrisome reputation? Won't you finally start seeing beyond the sci-fi movie of little "ads", little "AI", little "lgbt" and many "G"?

[@jimbo2112]

@@Samuel-er6td How about a rating they can apply based on the content of the video matching that of the description and the thumbnail? If it goes below a certain percentage it gets held until the user adjusts the promo content.

[@mandolinean3057]

Love the analysis and call-outs.

[@sansin6250]

<a href="#" class="seekto" data-time="605">10:05</a> GM is already there. Every modern GM vehicle starting from the Cadillac Lyric, 2024 Acadia & Traverse, 2025 Enclave, all have the Google agent built in. As the agent improves, the software will be updated over the air. GM is even phasing out Apple CarPlay and Android Auto, because the Google voice agent will fulfill all the requirements, including infotainment and HVAC control, vehicle information query, mapping, POI search and navigation, traffic updates, etc. Eventually all GM models will get the Google interface.

[@FragranceCraft]

Gold video ❤ Thank you so much 🎉🎉

[@08wolfeyes]

Ok, so I'm not a computer programmer, but I do have something of an interest in A.I for many reasons. One of them being that as an artist, it interests me in what they can do. The other day, I was watching a guy creating agents and without any programming needed. He simply typed what the task of the agent was, and it did the rest. He even connected 4 of them together to different tasks but then combining the outcome to achieve his goal. The way I understand them to work, although I could be wrong, is that you give each a task to perform, and that is what it does. Then, we can share and combine information between them if needed and also its possible to have another one keeping an eye so to speak on each agent and test them should you wish. If I'm correct in my understanding then it's clear by the Google video that they are using them in such a limited way because I can see so many interesting and new uses for them, even with things that have never been done before. Of course, I'm sure eventually you'll be able to do all this just by speaking to it rather than typing. Typing is so outdated. An interesting video, and I look forward to seeing what gets created using agents in the future.

[@EyezaGotSKILLZ]

Could you link the video?

[@08wolfeyes]

@@EyezaGotSKILLZ The one with the guy using agents and using language to tell each what their tasks were you mean?

[@08wolfeyes]

@@EyezaGotSKILLZ I believe it was this one or at least something close to. kzbin.info/www/bejne/m2OXc5-Nq7h_hK8si=rq-EAbFVCdqvROEn

[@EyezaGotSKILLZ]

@@08wolfeyes yes

[@peterwilkinson1975]

I would think the context window is super cool for training robots, cause you could have a video feed going to the llm, or even just training it on tasks, like show it a video of what you want being done and the llm can translate that into language.

[@kubectlgetpo]

DF and DFCX are not previous, they are still current and are now more mature. Vertex agent builder is different. When you need a mix of determinism (ala decision tree), entity extraction and fallback LLm, dialog flow it is. Plus it supports call campaigns, testing, agent assist and all of those things as one click features.

[@TreeYogaSchool]

Great video. I appreciate what you do.

[@iandanforth4313]

The tool definition is to avoid running arbitrary code on their server. If every tool *must* be implemented as an HTTP API then they can handle them all in exactly the same way and handle things like timeouts and errors in a standard manner.

[@calebmowery2391]

It seems like the Google's assistant functions may be similar to OpenAI's assistant functions. Basically it allows you to observe the response you get from the assistant and call a function in your own code base. No code in the assistant itself for the functions. I haven't used them, but I'd love to see a solid use case for implementing them.

[@travisporco]

I want something that doesn't involve having to use a custom IDE or decrypt a starship console. Is this better than crewAI?

[@SWGRPG]

Seems fine, but I'm not switching from OAI just yet. I've got too much built with them to walk away.

[@eightrice]

it's the new ecosystem lockin

[@seventyfive7597]

Thanks to our sponsor - every news release

[@ankitgoel1257]

I have built and tested the email tool you mentioned using local LLMs. Works like a charm for those 26-long email chain. Happy to have you test and get feedback

[@jeffdavidson5601]

Hey Matt, What's the ticker thing on your bookshelf?

[@animatedBiblestoriesSongs]

Doesn't Zapier do the email automation though ?

[@TrasThienTien]

Love the analysis

[@steefvanwinkel]

Would love for you to do an in depth video in Gemini 1.5 pro 🙏

[@flv-hd7nn]

Don't bother yourself. It is useless

[@dhruvmehta2377]

​@@flv-hd7nnit is usefull indeed I have tried it a lot of times now seems google is getting serious about their quality of AI

[@smoggysmoky3009]

I tested the Gemini 1.5 Pro, it is simply much worse on text than Gemini 1.0 Pro. It is indeed useless so far.

[@abdulfatah66]

The thing that seems to confuse people is actually a very good engineering principle for decoupling systems so that they are not tightly coupled with each other. The language models are only responsible for constructing the input parameters for YOUR code, you take those input parameters and pass them into your code and submit the result from YOUR code (running anywhere) to the vertex AI. You are not bound to upload your code to the vertex AI, it can be running on you local computer or anywhere.

[@mdk_ai]

Nice and honest review

[@joe_limon]

The only thing I took from this is that they have optimized their chat sales bots to allow even small businesses to train them without tons of data etc. pretty small innovation, but they need this information pushed out there hard to allow companies to know the service exists as easy as it is.

[@cbau0809]

Matt, you mentioned better use case for the agent approach. It would have been awesome if you would have comment after the keynote on each like you did on Mercedes-Benz.

[@mxpf26]

<a href="#" class="seekto" data-time="624">10:24</a> Note on Mercedes: Every new car has „Hey Mercedes“ - an assistant helping you out basically with everything. And it works for me.

[@changtimwu]

I believe Mercedes didn't mention the most desired use case because the in-car assistant would need to function offline. Google's Agent technology has not addressed this scenario, even though their open LLM gemma 2B has the capability to operate on a car's infotainment system.

[@YesCivic-R]

If agent can isolate background noise, and enhance voice quality in interns of tone and sound within a norm that would be wonderful, because lots of support staffs are working from home and diverse background.

[@Pixcrafts]

How do I access this? I've been searching everywhere but I literally cannot find even 1 of these features. Is it not available in Europe?

[@b0moonkhan]

You are awesome, sir!

[@DigitalBrent]

Can someone help me figure out where the heck this "Agent Console" is? I am having the hardest time finding this. I've found the "Agent Builder" but this does not appear to be the same thing.

[@andysurfer318]

Agree on voice for car

[@saingsodanin5891]

Very interesting analysis

[@00000ghcbs]

LMAO the new Mercedes looking like a sentra

[@designthinkingwithgian]

Agents that can do knowledge work is the real juicy technology.

[@Sonic2kDBS]

They just don't got that yet. In this case, you are way ahead of them. Keep on 😊👍

[@user-mp8fd8em3z]

Advanced enough to be omnipresent in the marketplace but not too advanced to make your company obsolete. This is priority number one for companies

[@MixedHouseMedia]

@<a href="#" class="seekto" data-time="690">11:30</a> - I believe this is intelligent of Google to enable others to first launch their own AI Agents. I'm sure they have many of their own ideas for future capabilities in the making. Incredible tech. Thanks for the share.

[@cemery50]

#Google/Agent builder It seems very sales oriented....I believe we all need personal agents to cope with other agents and the information, and data flows.

[@BillMarkerMastery]

Thanks!

[@matthew_berman]

Thanks so much!

[@kubectlgetpo]

It's enterprise ready if not futuristic. The compliance controls, frozen model in tenant, seems huge when I am freelancing for big corps.

[@nicholaskohl6112]

When I hear agents and AI together, I can't help but think of agents from The Matrix, lol. Thanks for the great video and summary to clear that up mate.

[@ComputersAndLife]

Google is building back their "moat". Very interesting to see the giants battle it out with AI. Hoping to catch some of their scraps!

[@santiagoc93]

I tried geminis the pro version and it doesn't work like chatgpt or mistral and the others.

[@Diego_UG]

I thought about what you said, that Google is playing it safe, but I think what it wants is to bring AI to everyone, and that is not easy at the infrastructure level, I think we have to do that step by step, so as not to falling into the attempt, and thinking like a common user, an AI assistant for me is the best option, which is what we already do with chatgpt, I use it for everything, even cooking, but I pay for it directly

[@grumpykoala6896]

Totally agree

[@roeibaraviv5771]

Regarding your questions- On the tools and API - it works like GPT plugins - you provide your backend api spec using API, and the agent will trigger that endpoint if it wants to. It’s on you to implement the backend logic, and the Ai will just call it.

[@varunmehra5]

Probably offers a little improvement in the performance but I do not see how it is any different from passing an intial prompt and then starting the session

[@iritesh]

Can you compare gemini code assistant vs copilot? Some deep testing because I didn't find it that good.

[@nicholas1460]

Kinematics for sports agents?

[@desirelovell]

You have “Hey, Mercedes” - it needs a lot of improvement…but they are working on it!!

[@jeisenstein1]

Hey Matt. Have you seen platforms like Kisai that do a lot of the heavy lifting with getting from design to production code? Lots of Gen work in there that makes this seem a bit rudimentary.

[@rfpeace]

@Matthew; <a href="#" class="seekto" data-time="2020">33:40</a>, you sure as hell will when your shit don't run after the change(s)! cheers brother!

[@kumarraju3855]

Hey Mathew, did you make any video on a web app that prompts gemini vision pro with image and some fixed text prompts and get the output displayed on the UI. if you did , please share the link and if not please make one.

[@gabrielagomez8572]

Do you know when will this be available for latam?

[@BradleyKieser]

Please do a video that shows how to access those features within Google. It looks to me like vapourware when I have a look at what we have available.

[@PseudoProphet]

Now, i have everything, can someone just give me a good startup idea as well? 😂😂

[@WhyteHorse2023]

CEO replacement

[@PseudoProphet]

@@WhyteHorse2023 Google has already done that I think. 😅

[@DailyTuna]

I like the Mercedes @digital experience” when it really is data gathering, and digital surveillance😂 Functionality of a device always has to take precedent. A car is . Point A to point B

[@hinro]

the computer cars brings back memories of you wouldn't download a car and now if you want the high end you kinda have to. what an odd turn of events.

[@SpragginsDesigns]

<a href="#" class="seekto" data-time="1860">31:00</a> this is their new IDE in the browser.

[@smanqele]

You were too impatient with the red-top lady use case 😀. It seemed nice

[@user-cw3jg9jq6d]

I hope it is ok for me to post a question. Do I need to pay in order to use Google Assistant to review my email like the benefits example in the video? Do I need Google Workspace to do that? I think that is a paid version. What about Google vertex? I hope I am spelling it correctly.

[@Dessme]

The AI madness era🤔

[@i45899]

In my opinion, it is the improvement of search engines to become faster and using the web scrap .

[@willgordon5737]

I hope it won't get discontinued. Like most of the google products

[@wtcbd01]

Great content and thanks for the hard work. This may be out of your knowledge but can you please do a video on why these companies are offering so many great features for little or nothing. We have seen that there is always a quotation marks "you are the product" type of language interceptions of these features for example in the Mercedes Benz hey Google collaborations. Many have complained that the OnStar incredible features turn out to be that they were selling data to insurance companies leading to cancellation are much higher insurance rates in the future, as they were basically spying on the drivers of their cars. The EU has incredible protection policies that really need to be implemented into all AI products

[@Chrosam]

<a href="#" class="seekto" data-time="1800">30:00</a> pretty sure they can already create entire codebases based on a figma design. (Even create figma designs based on a design system -- e.g. Material UI) Right now they're telling us it's "helping us" but it won't be long before the entire process is fully automated. Brb, learning how to farm.

[@Pixcrafts]

How could one use it to generate figma designs based on UI components?

[@robertsteffanNetRoofer1999]

You mentioned Dialog Flow, is it worth taking the time to learn CCAI CX or will AI Agent builders and Vertex supercede take the place? I want to do conversational AI and meet my customers where they're at. Dialog Flow looked a good option, is that still true today?

[@nitinjazz]

Would love to know answer to this!

[@abinash.m]

I am going through the same problem

[@glenh1369]

Matt when are you going to test the bias of each AI? That is where an AGI disaster can really happen!

[@roi1357]

Approximately one week ago my Google homes are unable to play my KZbin playlist or turn off or on smart plugs or bulbs. I'm wondering if it has anything to do with Gemini. I have gone through all the suggested fixes but nothing is solving the problem. I am thinking the assistant was updated or maybe Gemini has something to do with it not being able to recognize simple requests.

[@StashOfCode]

Nice video. Same conclusion: Employee Agents seems to be just the Google version of MS Copilot. Are we all missing something or what?

[@MrStevemur]

I think you're spot on about Google "playing it safe." They've got this super-global brand; as soon as they make an agent that says or does something questionable, it's huge news. Microsoft figured out how to solve this problem decades ago: let small companies do innovative things, then buy the successful ones.

[@SvendErix]

"Looks like VS Code": Could it be Google's IDX? I think that's build on the same (open source?) library as VS Code. (I don't have access to Google's IDX yet)

[@TheLongboarder77]

So true Mathew. Refactoring the message around shopping assistants as new A.I. is a sad day for the org that saved search from a flood of spam. How the mighty can fall.

[@IvoVacca]

I guess they are using Apache Spark behind the scene.

[@guidobolke5618]

Wow, google managed to make actual humans sound more like an AI now than actual AIs.

[@costa2150]

What is the significance of 1 million tokens versus 10 million tokens? Are tokens related to embeddings? Thanks!

[@user-fx7li2pg5k]

Im thinking about teaching an a.i. master class with no coding required,and we will all explore together and ask questions and answer them