Google Releases AI AGENT BUILDER! 🤖 Worth The Wait?

Google announced Google Agent Builder through its Vertex AI product. Let's examine it and other announcements from Google Cloud Next.
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? ✅
forwardfuture.ai/
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
Full Video - • Google Cloud Next '24 ...

[@arberstudio]

@19:25 you set your Tool Type to OpenAPI, and the endpoints in the instructions will work. I imagine it's 1 function per tool for modularity. It's more or less exactly the same setup as the Custom GPTs configuration.

[@show-me-the-data]

Hey Matt. When you select "function", you're responsible for writing the code and driving the conversation via API calls. The flow goes a little like this. You submit a message and you check the status and it tells you you need to run the function with these inputs. YOU then submit the output back to vertex via the API, where it uses the function output to generate the next message. You can implement the function wherever you're DRIVING the convo. Makes sense? This is a very similar flow to the assistants API which I have a video about on my channel if you're interested

[@DailyTuna]

You’re so right Matt on that agents in the Mercedes. There’s this weird movement of bells and whistles, and what we can do rather than true functionality that is efficient. Especially when it’s not level five self driving, we still have to drive and less distraction equals more attention to the road and thus is safer driver.

[@arberstudio]

@26:10 they're using the equivalent of Agent (preview) app in the Agent Builder service which as they word it "can answer questions from data, connect with business systems through tools"

[@DailyTuna]

Matt on it! Thanks for going over this. Nobody else is really going into it like you.!👍👏

[@TheFocusedCoder]

thanks for the review . I was short on time today , perfect timing

[@MrVohveli]

Considering Google ought to be a solid year or so behind OpenAI, despite their best efforts, what kind of monster is OpenAI planning to release that beats all this?

[@bobnothing4921]

You're assuming this isn't all vaporware. It's Google. Virtually every 'tech demo' they do is a lie.

[@PseudoProphet]

😂😂, does OpenAI have any such infrastructure? If Microsoft is the one who will launch. 😂😂

[@WhyteHorse2023]

Altman says they are making these things smarter so they get better at everything. Then there's also their humanoid robot. Oh and they just released vision for chatgpt and newer turbo.

[@first-thoughtgiver-of-will2456]

Google invented a lot of what made OpenAI famous. Idk where you get that notion from Ive been expecting Google to wake back up on their ML work and lead us to the next breakthrough.

[@Brian-oz8io]

I think they’re mostly just doing different things. Google is catering to enterprise, commerce, and search capabilities. Sora will be more for Hollywood, artists, and content creators. OpenAI is working on a robot, a supercomputer, a voice generator, and an upgrade on their consumer chatbot with the ultimate goal of achieving AGI. I don’t see how those things really compete with each other, besides what Microsoft is trying to do. But obviously that’s a different company. I think Microsoft could lose to Google on that front and it wouldn’t affect what OpenAI is doing.

[@Maisonier]

I've been eagerly awaiting a local AI solution to sort through our family's extensive collection of decades-old photos. This AI should be capable of identifying and eliminating duplicate or blurry images, learning to recognize faces, and categorizing photos by individual. Moreover, it should have the ability to estimate the age range of each person depicted. Ideally, this solution would allow for easy exportation of organized data so that Stable Diffusion models could learn from it. Additionally, integrating facial recognition capabilities into home security cameras would streamline access control, eliminating the need for physical keys.

[@aintaintaword666]

There's a program called Excire

[@myrakrusemark6873]

This is something a program would take care of, not an AI. If you need a program with these specifics, you can write one with the help of Claude3 in 6-12h given availability of one or two components. Digikam already does MOST of this. And all of this would be programmatic, not requiring an AI to run. Digikam can individualize people. All that's left over is a model that recognizes age, which Im sure exists, but you may want to write a program that iterates through all those images and adds subject age to the EXIF data.

[@fintech1378]

Whats the main objective if i may ask

[@MetaTaco317]

​@@fintech1378Going to take a wild guess and say to have an organized, searchable collection of family photos which allows for anyone in the family to quickly find any photo.

[@fantinigerman]

and a search bar, so you can search whatever from your photos

[@r3kRaP]

Hey Matt, with all of these models and apps along with their updates, it would be nice to see a real time list of your favorites. For example you said Gemma wasn't to impressive but it may be now with its new update. Maybe like a place where you can rank the best models or frameworks and update the source as things change. For example crew Ai, autogen, Gemma,, Devin, Devin alternatives, and all the different llms. It be nice to see an up to date list of your rankings in each category. Or does something like this already exist?

[@GarrettGalloway]

I'll be ready for Google to put that kind of tech to use on youtube. No more trap videos where the title, description, and thumbnail don't match the content. KZbin could auto-down-vote videos that don't match their title, description, and thumbnail. I can imagine that Google will be the king of multi-modality due to KZbin.

[@MikkiPike]

I'm very much not ready for them to scan videos for anti-capitalist sentiments and shadow ban them, perpetuating the head first dive into the end of the human race this train of capitalism already has us on.

[@Samuel-er6td]

I bet they start automatically editing yt videos to make them better. They can edit it based on what your searching for. Or edit it based off your watching style that it learns over time.

[@ScottWinterringer]

LOL they cant even keep feature parity. open maps and tell it to play music through blackpplayer ex LOL it wont do it anymore

[@GoogleTakesYourPrivacy]

@@Samuel-er6tdHAHAHAHAH I never want to watch Indian videos... But guess what?! . So Google's AI can't do its main job of giving you the right content when you search using their search page (which is the foundation of Google, their vision). What's left, getting into a car powered from a Cloud service of a company (that same "G") that has an extremely worrisome reputation? Won't you finally start seeing beyond the sci-fi movie of little "ads", little "AI", little "lgbt" and many "G"?

[@jimbo2112]

@@Samuel-er6td How about a rating they can apply based on the content of the video matching that of the description and the thumbnail? If it goes below a certain percentage it gets held until the user adjusts the promo content.

[@xerxel69]

Some of Matt’s videos were a bit lightweight in the past, but he really nailed this one. 👍 good analysis, brought in external knowledge and not just AI fanboy content. Nice job. Will be subscribing to this channel now!

[@matthew_berman]

Thanks :)

[@EccleezyAvicii]

Agent > Example > create example The example is a dummy dialogue. You need to actually mockup a weather convo. Each prompt/return is an ‘action’ in the example convo. There are other actions, like tool. Tool is where you put the code

[@dafunkyzee]

i heard the keynote before, and I really appreciate your insights and commentary to make it more "real." There is often a strong corporate biase when we hear these kinds of releases. Great video very informative. Love your stuff

[@erikthegrey]

It's like Autogen with frosting on top. I can dig it saving me a lot of design setup time for agent clouds.

[@cemery50]

#Using ai to select ai, #Using ai to debug code and create insertion points (~cross cutting)....it could then analyze and respond by altering the code to fit conditions (~FPGLA). I'm working on knitting Neo4J and ai-models for an interactive teal time graph dashwindow to use user priveldged based access. It seems that ai could refine alternative models based upon a user's project and use rag to learn preferences...and present them ranked with pros and cons of each.

[@tecsmith_info]

Great announcement, thanks for sharing! I've been working with several of these offerings over the past few months, developing some great use cases. I was thinking of perhaps showcasing some of them on my channel soon.

[@tsomerville1970]

Thanks for sticking through it - I dropped out on the live original presentation when I saw all the customer agent stuff vs the more interesting However, you pulled out a few more things that I missed

[@waynelast1685]

3:00 that’s great , I hope the model works well though

[@metonoma]

the Mercedes missed opportunity is probably an insurance issue. They'd need to long term test of it's safe because they don't want to get sued if someone expects the ai to read their mind and then goes road rage and blames it on them

[@positivevibe142]

Nice video Matt, your huge efforts are highly appreciated. Thanks for sharing!

[@sansin6250]

10:05 GM is already there. Every modern GM vehicle starting from the Cadillac Lyric, 2024 Acadia & Traverse, 2025 Enclave, all have the Google agent built in. As the agent improves, the software will be updated over the air. GM is even phasing out Apple CarPlay and Android Auto, because the Google voice agent will fulfill all the requirements, including infotainment and HVAC control, vehicle information query, mapping, POI search and navigation, traffic updates, etc. Eventually all GM models will get the Google interface.

[@anthonyjobey8821]

Still watching but as always love the content and the way you deliver it, so glad I stuck around as you are my go to to keep up with whats going on

[@nikolaimanek582]

Nice video! Will you cover Mixtral 8*22b soon? That would be great to see. The 8*7b was already great.

[@iandanforth4313]

The tool definition is to avoid running arbitrary code on their server. If every tool *must* be implemented as an HTTP API then they can handle them all in exactly the same way and handle things like timeouts and errors in a standard manner.

[@peterwilkinson1975]

I would think the context window is super cool for training robots, cause you could have a video feed going to the llm, or even just training it on tasks, like show it a video of what you want being done and the llm can translate that into language.

[@08wolfeyes]

Ok, so I'm not a computer programmer, but I do have something of an interest in A.I for many reasons. One of them being that as an artist, it interests me in what they can do. The other day, I was watching a guy creating agents and without any programming needed. He simply typed what the task of the agent was, and it did the rest. He even connected 4 of them together to different tasks but then combining the outcome to achieve his goal. The way I understand them to work, although I could be wrong, is that you give each a task to perform, and that is what it does. Then, we can share and combine information between them if needed and also its possible to have another one keeping an eye so to speak on each agent and test them should you wish. If I'm correct in my understanding then it's clear by the Google video that they are using them in such a limited way because I can see so many interesting and new uses for them, even with things that have never been done before. Of course, I'm sure eventually you'll be able to do all this just by speaking to it rather than typing. Typing is so outdated. An interesting video, and I look forward to seeing what gets created using agents in the future.

[@EyezaGotSKILLZ]

Could you link the video?

[@08wolfeyes]

@@EyezaGotSKILLZ The one with the guy using agents and using language to tell each what their tasks were you mean?

[@08wolfeyes]

@@EyezaGotSKILLZ I believe it was this one or at least something close to. kzbin.info/www/bejne/m2OXc5-Nq7h_hK8si=rq-EAbFVCdqvROEn

[@EyezaGotSKILLZ]

@@08wolfeyes yes

[@kubectlgetpo]

DF and DFCX are not previous, they are still current and are now more mature. Vertex agent builder is different. When you need a mix of determinism (ala decision tree), entity extraction and fallback LLm, dialog flow it is. Plus it supports call campaigns, testing, agent assist and all of those things as one click features.

[@thefandomcritic]

Seems fine, but I'm not switching from OAI just yet. I've got too much built with them to walk away.

[@eightrice]

it's the new ecosystem lockin

[@calebmowery2391]

It seems like the Google's assistant functions may be similar to OpenAI's assistant functions. Basically it allows you to observe the response you get from the assistant and call a function in your own code base. No code in the assistant itself for the functions. I haven't used them, but I'd love to see a solid use case for implementing them.

[@BillMarkerMastery]

Thank you. Your commentary adds real value to this video. I'll definitely watch more.

[@steefvanwinkel]

Would love for you to do an in depth video in Gemini 1.5 pro 🙏

[@flv-hd7nn]

Don't bother yourself. It is useless

[@dhruvmehta2377]

​@@flv-hd7nnit is usefull indeed I have tried it a lot of times now seems google is getting serious about their quality of AI

[@smoggysmoky3009]

I tested the Gemini 1.5 Pro, it is simply much worse on text than Gemini 1.0 Pro. It is indeed useless so far.

[@mandolinean3057]

Love the analysis and call-outs.

[@joe_limon]

The only thing I took from this is that they have optimized their chat sales bots to allow even small businesses to train them without tons of data etc. pretty small innovation, but they need this information pushed out there hard to allow companies to know the service exists as easy as it is.

[@vq8gef32]

Thank you Matthew for all of your amazing videos

[@TreeYogaSchool]

Great video. I appreciate what you do.

[@cbau0809]

Matt, you mentioned better use case for the agent approach. It would have been awesome if you would have comment after the keynote on each like you did on Mercedes-Benz.

[@FragranceCraft]

Gold video ❤ Thank you so much 🎉🎉

[@abdulfatah66]

The thing that seems to confuse people is actually a very good engineering principle for decoupling systems so that they are not tightly coupled with each other. The language models are only responsible for constructing the input parameters for YOUR code, you take those input parameters and pass them into your code and submit the result from YOUR code (running anywhere) to the vertex AI. You are not bound to upload your code to the vertex AI, it can be running on you local computer or anywhere.

[@aosamai]

Thanks Matt for the review, as always you have perfect timing and bring a common sense perspective.

[@seventyfive7597]

Thanks to our sponsor - every news release

[@changtimwu]

I believe Mercedes didn't mention the most desired use case because the in-car assistant would need to function offline. Google's Agent technology has not addressed this scenario, even though their open LLM gemma 2B has the capability to operate on a car's infotainment system.

[@mxpf26]

10:24 Note on Mercedes: Every new car has „Hey Mercedes“ - an assistant helping you out basically with everything. And it works for me.

[@vishnunallani]

Tried to setup crewai today and was not able to handle my use case of sending image to gpt vision and get output from it. Any suggestions?

[@StephenGoodfellow]

When it comes to a personal AI agent, I would NEVER allow any government or corporation have tabs on it for what should be obvious reason

[@travisporco]

I want something that doesn't involve having to use a custom IDE or decrypt a starship console. Is this better than crewAI?

[@Sonic2kDBS]

They just don't got that yet. In this case, you are way ahead of them. Keep on 😊👍

[@MrStevemur]

I think you're spot on about Google "playing it safe." They've got this super-global brand; as soon as they make an agent that says or does something questionable, it's huge news. Microsoft figured out how to solve this problem decades ago: let small companies do innovative things, then buy the successful ones.

[@PseudoProphet]

Now, i have everything, can someone just give me a good startup idea as well? 😂😂

[@WhyteHorse2023]

CEO replacement

[@PseudoProphet]

@@WhyteHorse2023 Google has already done that I think. 😅

[@YesCivic-R]

If agent can isolate background noise, and enhance voice quality in interns of tone and sound within a norm that would be wonderful, because lots of support staffs are working from home and diverse background.

[@00000ghcbs]

LMAO the new Mercedes looking like a sentra

[@jeffdavidson5601]

Hey Matt, What's the ticker thing on your bookshelf?

[@TrasThienTien]

Love the analysis

[@b0moonkhan]

You are awesome, sir!

[@cemery50]

#Google/Agent builder It seems very sales oriented....I believe we all need personal agents to cope with other agents and the information, and data flows.

[@MixedHouseMedia]

@11:30 - I believe this is intelligent of Google to enable others to first launch their own AI Agents. I'm sure they have many of their own ideas for future capabilities in the making. Incredible tech. Thanks for the share.

[@user-mp8fd8em3z]

Advanced enough to be omnipresent in the marketplace but not too advanced to make your company obsolete. This is priority number one for companies

[@andysurfer318]

Agree on voice for car

[@designthinkingwithgian]

Agents that can do knowledge work is the real juicy technology.

[@Diego_UG]

I thought about what you said, that Google is playing it safe, but I think what it wants is to bring AI to everyone, and that is not easy at the infrastructure level, I think we have to do that step by step, so as not to falling into the attempt, and thinking like a common user, an AI assistant for me is the best option, which is what we already do with chatgpt, I use it for everything, even cooking, but I pay for it directly

[@jeisenstein1]

Hey Matt. Have you seen platforms like Kisai that do a lot of the heavy lifting with getting from design to production code? Lots of Gen work in there that makes this seem a bit rudimentary.

[@roeibaraviv5771]

Regarding your questions- On the tools and API - it works like GPT plugins - you provide your backend api spec using API, and the agent will trigger that endpoint if it wants to. It’s on you to implement the backend logic, and the Ai will just call it.

[@animatedBiblestoriesSongs]

Doesn't Zapier do the email automation though ?

[@mdk_ai]

Nice and honest review

[@kubectlgetpo]

It's enterprise ready if not futuristic. The compliance controls, frozen model in tenant, seems huge when I am freelancing for big corps.

[@guidobolke5618]

Wow, google managed to make actual humans sound more like an AI now than actual AIs.

[@rfpeace]

@Matthew; 33:40, you sure as hell will when your shit don't run after the change(s)! cheers brother!

[@varunmehra5]

Probably offers a little improvement in the performance but I do not see how it is any different from passing an intial prompt and then starting the session

[@ComputersAndLife]

Google is building back their "moat". Very interesting to see the giants battle it out with AI. Hoping to catch some of their scraps!

[@Sirmrmeowmeow]

I think one of the reasons we don't see strong agent use is due to limitations from inherent statelessness of inference. Current ai uses the chat history to 'seem' continuous, but do not retain unsaid thoughts/intent/goals. This causes issues in Long Term Planning, having to rely on low information density hacks like reason aloud your actions, goals, actions; but if any text is missed in the Chat History then the next inference may not be able to infer that missing context. Those internal goals, intent, reasons, thoughts do not survive inference to inform the next inference if not explicitly said aloud and current tech has no way of getting that dense representations across inference -- once the inference ends it's practically a new ai given the chat history up to that point (no previous inference's 'thoughts'); hence the discontinuity causes issues. possible solutions: ~memory-informed inferences (as a vehicle to get dense context across to next inference) ~state based system like a giant Mamba/Jamba perhaps

[@matthew_berman]

Great input, thanks

[@robertsteffanNetRoofer1999]

You mentioned Dialog Flow, is it worth taking the time to learn CCAI CX or will AI Agent builders and Vertex supercede take the place? I want to do conversational AI and meet my customers where they're at. Dialog Flow looked a good option, is that still true today?

[@nitinjazz]

Would love to know answer to this!

[@abinash.m]

I am going through the same problem

[@saingsodanin5891]

Very interesting analysis

[@smanqele]

You were too impatient with the red-top lady use case 😀. It seemed nice

[@TheLongboarder77]

So true Mathew. Refactoring the message around shopping assistants as new A.I. is a sad day for the org that saved search from a flood of spam. How the mighty can fall.

[@memocappa5495]

It’s not that Google is playing safe, but their business is the data. So they’re moving fast to work with the highest spending advertisers because they’ll have all customer spending data routed thru their ai servers.

[@desirelovell]

You have “Hey, Mercedes” - it needs a lot of improvement…but they are working on it!!

[@grumpykoala6896]

Totally agree

[@arthurrobey4945]

Verbiage and jargon. I thought that the various AI products would separate the task and send it off to specialist agents to work on, An example. Task: "Create a business plan for me to take possession of my very own 200km long O'Neil cylinder at L4." Or Task: "Destroy Einstein's Bent Nothingness conjecture an use the newly discovered physics to design and build an anti gravity craft,"

[@nicholaskohl6112]

When I hear agents and AI together, I can't help but think of agents from The Matrix, lol. Thanks for the great video and summary to clear that up mate.

[@Chrosam]

30:00 pretty sure they can already create entire codebases based on a figma design. (Even create figma designs based on a design system -- e.g. Material UI) Right now they're telling us it's "helping us" but it won't be long before the entire process is fully automated. Brb, learning how to farm.

[@Pixcrafts]

How could one use it to generate figma designs based on UI components?

[@DaveEtchells]

A Google rep talking about Google’s “trusted” search technology was pretty hilarious.

[@mgf503]

Do you not trust that they'll search the internet for you?... As they have billions of times already. I think you've misunderstood the context and use of the phrase, and blended it in with inherent mistrust in big tech companies and blurted out a popular fashionable negative comment. . That's okay, I also have a bit of mistrust in begtech, but..., your comment as an answer to a basic English language exam, would be a fail.

[@Sephaos]

@@mgf503and your response would also be a fail, shut up, no one asked you.

[@ComputersAndLife]

To be fair, search isn't the enemy. It's the advertising network of theirs that is sucking in all of the user data. If they are contracted with a company then it will be just the search engine without the ad network which would be really good.

[@tomekg101]

Thanks!

[@matthew_berman]

Thank you!!

[@bertobertoberto3]

Matt I completely agree, this whole showing was very unimpressive

[@Pixcrafts]

How do I access this? I've been searching everywhere but I literally cannot find even 1 of these features. Is it not available in Europe?

[@iritesh]

Can you compare gemini code assistant vs copilot? Some deep testing because I didn't find it that good.

[@DailyTuna]

I like the Mercedes @digital experience” when it really is data gathering, and digital surveillance😂 Functionality of a device always has to take precedent. A car is . Point A to point B

[@SecAllTheWay]

with all the AI talent at their command, why should I keep reducing the audio volume in youtube when the ads pop up, it's insanely high vis-a-vis the content audio. Would love to know the easiest way to solve this with AI. The same can be applied for music playlists on my phone.

[@SpragginsDesigns]

31:00 this is their new IDE in the browser.

[@Miguel_Molina]

I'd have to agree AND disagree with your take @10:50. True, most of these chat functions are basically interactive FAQs vs a "virtual' CS rep that can look up your account profile. But for the latter, I'd say maturing that level of capability would definitely be cutting edge -- this is a very hard task to execute for a large organization. The range of customer's issues can be significantly broad, and training a model to provide dynamic and accurate responses against a proprietary knowledge base has proved difficult. So much money would be saved to free up resources for other efforts. Then lets also think, imagine the day AI CS reps reaches maturity. Then imagine, we all get our own personal CS rep (looking at you Siri, Alexa, Google). What is the reason to call a company anymore? What is even the reason to visit their website anymore? Or download an app? Every single interaction could be done through our agents interacting with each other. A whole new world.

[@hinro]

the computer cars brings back memories of you wouldn't download a car and now if you want the high end you kinda have to. what an odd turn of events.

[@user-cw3jg9jq6d]

I hope it is ok for me to post a question. Do I need to pay in order to use Google Assistant to review my email like the benefits example in the video? Do I need Google Workspace to do that? I think that is a paid version. What about Google vertex? I hope I am spelling it correctly.

[@chloeo7767]

Well, the moral of the story is that some people will need to look for a different career. I heard the words “ customer agent” “ customer service “ a lot. Even for Mercedes-Benz, instead of optimizing the menu to make it hands free, they ignored that option. I am able to perform these searches she’s talking about via google lens and chat GPT already, no big deal.

[@kumarraju3855]

Hey Mathew, did you make any video on a web app that prompts gemini vision pro with image and some fixed text prompts and get the output displayed on the UI. if you did , please share the link and if not please make one.

[@wtcbd01]

Great content and thanks for the hard work. This may be out of your knowledge but can you please do a video on why these companies are offering so many great features for little or nothing. We have seen that there is always a quotation marks "you are the product" type of language interceptions of these features for example in the Mercedes Benz hey Google collaborations. Many have complained that the OnStar incredible features turn out to be that they were selling data to insurance companies leading to cancellation are much higher insurance rates in the future, as they were basically spying on the drivers of their cars. The EU has incredible protection policies that really need to be implemented into all AI products

[@willgordon5737]

I hope it won't get discontinued. Like most of the google products

[@StashOfCode]

Nice video. Same conclusion: Employee Agents seems to be just the Google version of MS Copilot. Are we all missing something or what?

[@BradleyKieser]

Please do a video that shows how to access those features within Google. It looks to me like vapourware when I have a look at what we have available.

[@SvendErix]

"Looks like VS Code": Could it be Google's IDX? I think that's build on the same (open source?) library as VS Code. (I don't have access to Google's IDX yet)

[@xaxfixho]

People talk about 1million context window without mentioning cost 😒 🤑👀$$$$

[@WhyteHorse2023]

That will be the real eye opener for most people. Yeah, it's costs $17/hr to run this customer service bot.

[@Dessme]

The AI madness era🤔

[@the42nd]

How Pitchai has kept his job is beyond me.