GPT4V + Puppeteer = AI agent browse web like human? 🤖

How to build an AI agent that can control web browser to complete tasks like research, order pizza or book flight tickets? Step by step tutorial
Get free hubspot research of how does Sales team use AI in 2024: offers.hubspot.com/sales-tren...
🔗 Links
- Follow me on twitter: / jasonzhou1993
- Join my AI email list: crafters.ai/
- My discord: / discord
- Github - Web AI agent: www.crafters.ai/aitools/ai-we...
⏱️ Timestamps
0:00 Intro of self operation systems
1:25 Market opportunities
4:20 Hubspot research report
5:31 How did AI control computer?
7:30 How does self operating system works?
10:00 Tutorial 1: GPT4V Web Scraper - Scrape anything
16:57 Tutorial 2: Web AI agent
22:45 Web AI agent demo
👋🏻 About Me
My name is Jason Zhou, a product designer who shares interesting AI experiments & products. Email me if you need help building AI apps! [e-mail]
#gpt4v #autogen #gpt4 #autogpt #ai #artificialintelligence #tutorial #stepbystep #openai #llm #chatgpt #largelanguagemodels #largelanguagemodel #bestaiagent #chatgpt #agentgpt #agent #babyagi

[@AIJasonZ]

What AI agent use case do you think it will unlock with browser & computer access? Comment and let me know!

[@metinEsturb]

It can now watch youtube shorts for me, so that I can focus my lifetime on watching productive videos like yours 😉

[@minutecodingtips]

Can you provide github repo for it sir?

[@ahsin.shabbir]

Find me all relevant documents in my inbox and my desktop so that I can prepare my tax filing for 2024.

[@rab0309]

would be interesting to try it out with models that are open source and not something where you'd need for example an open ai api key

[@martinchudomel7117]

maybe automated testing

[@TheHeatzz33]

reCAPTCHA: Are you a human? Web AI Agent: *click*

[@elevenchicago]

You have been making some of the best videos on this stuff since day one. Always super practical yet very valuable content Jason. Keep it up and glad to see the HubSpot sponser! Esketit

[@unconv]

Thanks for featuring my project!

[@jasonfinance]

Great job mate! Love to see all the agent possibilities

[@oshodikolapo2159]

This is a really smart and intutive process. Thanks Jason!

[@BrianMosleyUK]

This is really valuable work Jason, beautifully conceived and executed. Keep going, this is huge. 🙏👍

[@paulmartz5053]

Congrats on your work, really useful content. Keep it up!

[@sanesanyo]

Really awesome job. Keep such videos coming. 🎉🎉

[@ElijahZuBailey]

🎯 Key Takeaways for quick navigation: 00:03 🌐 *Web AI agents with direct computer access are a trending topic, allowing AI models like GPT-4V to control computers and browsers for various tasks.* 01:28 🏢 *Robotic Process Automation (RPA) is one market category that self-operating computers can impact. RPA automates repetitive tasks in enterprises.* 03:32 🖥️ *Self-operating AI agents can handle complex and non-standardized tasks, potentially reducing setup costs compared to traditional RPA solutions.* 04:27 📊 *Understanding the end-to-end workflow of specific job functions is crucial for delivering useful AI worker solutions.* 05:36 🤖 *Two common implementations for AI agents to control web browsers involve sending simplified HTML or using screenshots with annotations for better instruction.* 08:09 🖥️ *Self-operating computer frameworks like the one discussed in the video use annotations on screenshots to instruct AI agents for interactions.* 10:02 🌐 *The tutorial demonstrates how to create a web scraper using GPT-4V for extracting data from web pages, providing cleaner results than traditional scraping.* 16:59 ⚙️ *A more advanced AI web agent is discussed, capable of interacting with websites, clicking links, and performing complex web tasks.* 21:19 🌐 *The web agent script can handle URLs provided by the agent, allowing it to navigate to specific web pages.* 22:29 🔄 *The web agent continuously clicks on links to navigate, find information, and scrape new websites.* 23:11 📊 *The web agent can interact with websites to answer questions or complete tasks, making it a valuable tool for various purposes.* 23:53 📷 *The web agent can perform complex tasks like conducting a Google search, visiting Instagram accounts, and extracting information, showcasing its potential for various applications.* Made with HARPA AI

[@carterjames199]

Great video again Jason love to see it

[@akshaygoel8576]

Amazing work! Thanks for putting in so much effort to bring quality content to the viewers.

[@redamarzouk]

I've been very excited about the last video when I first saw the paper of GPT-4v and the automation applications. I do RPA full time and even though GPT-4v is not there yet in terms of what it can do on its own for form data entry and complex automation processes, I can see the use cases where it can outshine RPA when it's enhanced.

[@avi7278]

Bro, you are such a gangster. You're the only person I follow actually putting out valuable work at this level. All these AI grifters are on their six hundredth basic ass autogen demo, just zero value videos for clicks.

[@stephantual]

This exactly.

[@hereusername]

same for me! it is really hard to find valuable info about AI beside just basics

[@jinqimao4781]

great explanation, like this kind of tutorial on AI projects!

[@amandamate9117]

AutoGen now has a AgentBuilder class. Its exactly doing how it sound, building a swarm of agents.

[@DAWEAP1]

Super interesting Jason!

[@binthem7997]

This is an amazing tutorial of how to integrate GPT into daily productivity tools. Thanks man!

[@oakbots3035]

Great Jason... this is kind of innovation is mind blowing!

[@williamrich3909]

Mind blowing! Fantastic.

[@parkjunu2400]

This is so sick.. thanks man

[@paresh3915]

Great Work Jason!! would love to see you explore creating a Autogen like framework with Open source Multimodal agents, seems like it can open a lot of possibilities without the huge costs associated with using openAI's APIs for intensive tasks. Keep up the great work.!

[@EricWatyekele]

Awesome work Jason, I was thinking the same @paresh3915 - Open source Multimodal agents instead of OpenAI

[@Yakibackk]

thanks jason for this valuable work. This toutrial are advance but very valuable spec in the market. You could easliy make big money from this method.

[@EverythinTechnology]

Great video! I would love to see if this can be done with an open source model like LLaVa.

[@user-xc9tp6sx1b]

This is Amazing !

[@techxartisanstudio]

Thanks man!

[@RoadTo19]

Another great share, cheers! One request... please consider adding links in the description to the products mentioned in videos; perhaps as affiliate links for yourself. Makes for improved UX for viewers.

[@Jim-ey3ry]

Nice, was wondering how to build a chrome plugin to control web like hyperwrite, this is handy 👍 Also the hubspot research is actually good

[@carterjames199]

Same here trying to figure out how to connect the local repo to a browser plugin

[@jhoningsoft]

Thanks!

[@youwang9156]

Jason, wondering if you have some kinda plans for diving into details of finetuning open source models like llama2?

[@JoeyGartin]

You are awesome. You should have your own instructional (paid membership) website for all of this.

[@alexanderroodt5052]

Wicked sick!

[@leewsimpson1]

Hey Jason, how would you implement the ability to type into textboxes? They usually do not have text in them, so the AI cannot identify them. This feature would likely be needed to add the ability to do more stuff like order a pizza.

[@Taskade]

Absolutely beautiful! Really appreciate you sharing this insightful video. It's always great to see such valuable content. Thanks for broadening our horizons!

[@test12382]

Bro you are LOVE

[@xonack]

let's get it!

[@sanesanyo]

Any idea about how to implement scrolling in the web agent?

[@davidwylie8491]

Amazing

[@leewsimpson1]

Nice work- thank you!! Do we really need to run node? Can’t we just use python browser libraries ?

[@amandamate9117]

yo

[@jorgezamudio8457]

mind blowing

[@podcasttakes]

More of JavaScript and node please :)

[@user-ht9st4up8q]

Been looking for such approach for while now. Until i see this video , very exciting. But I also concern over the pricing of GPT-4 vision, I hope other alternative works as well like LlvAa 2 vision or CogVLM & CogAgent

[@ex3aliber]

Hey Jason amazing video as usual! Can you do one using canva to setup and automate a youtube channel..... from creation of videos using canva to scheduling the posts... do you think this will be possible?

[@wizaaeed]

It's not about automation, it's about being authentic. If you think posting unlimited auto generated videos to youtube will give you an advantage over something, you're far off the line.

[@avi7278]

LOL, you'd end up paying $50-$100 for a single crappy AI video that nobody will watch. There's cheaper way to spam your AI crap videos if you want to do that.

[@JewelEyedRanchAndRoofing]

I really appreciate your video, I am wondering if you know a good resource or if you can do a video that walks through how to do this 100% from start to finish using windows and vscode

[@HungVo-vc4no]

Crazy !!!

[@parmesanzero7678]

24:28 Having an AI agent spend tokens to check the weather when the same search in Google results in the same reminds me of that robot that is told its whole purpose is to pass the butter.

[@JoeyZero]

Just throwing this out there: If using OpenCV to drive your annotations, you can annotate coords directly on the image without too much difficulty (to give some guidance for when GPT4V is trying to recommend a location to click). Haven't done this with puppeteer, but I'm sure it can be done ;) Anybody wanna collab on some experiments in this space? 🙏 Lolol

[@JewelEyedRanchAndRoofing]

Can we combine ai with something like ui.vision? Have the ai agent learn how to use ui.vision and then it would be able to control computer and browser

[@matthewnaples5672]

Did you have trouble getting your chrome profile working with Chrome Canary? That seems to be my only sticking point.

[@techfren]

Amazing as always! I'm still trying to build my browsing ai agent from the sxsw hackathon in Sydney lmao 😅

[@KamilKaczmarekSolutions]

meaning clickolas cage? 😂😂 love that

[@techfren]

@@KamilKaczmarekSolutions haha yup!

[@prispeshnik-istini2]

Hello! Thanks for the video, and yes, it’s already February 2024, and I would like to know what’s new, what crazy projects are being implemented using this technology? what business cases are already working today? And yes, can you help me set up this process for my task? You are super! I looked at it in translation, but I realized that you are very skilled! Bravo!

[@TraveleroftheSoul7674]

how can we provide the assistant api key of gpt 4 which is not currently supporting the image input?

[@Nuninecko]

why are not all open source LLMs working on these autonomous web agents? Are there any open source, PC based, replacements of Rabbit R1?

[@new-parents-survival-diary]

can we use Gemini Pro Vision instead of OpenAI? Could you please provide the tutorials?

[@ahsin.shabbir]

What if you have an AI agent for a consumer product that handles event management tasks normally performed by the event host? For example if there is a school doing a field trip, the AI agent can answer many of the parent's queries, send updates about the trip such as weather forecasts that may effect the trip, collect certain forms, and more. Same can be applied for events like conferences, weddings, birthdays, housewarming, etc.

[@fullcrum2089]

i'm building this

[@ahsin.shabbir]

@@fullcrum2089 how far along are you?

[@amandamate9117]

build ngga build @@fullcrum2089

[@MunirJojoVerge]

Hi Jason, I've been trying to reach out by email for a professional consultation for a company. It looks like the email might have fallen through the cracks of a million others or under the spam folder. Is there any other way I can reach out to you?

[@squiddymute]

Are there any of this sort of tutorials out there that don't use an open AI subscription and actually do the job ? Is there an actual value in this if you have to pay open AI for every new page chatgpt 4 has to process??

[@morohicham2579]

One question here , can the AI agent bypass cloudflare captcha check as a humain ?

[@swetharangaraj4521]

how to build query bot ,it should query from the database like mangodb where the data are stored from the application

[@persas1683]

Hi, I do the same as you but my Chrome Canary still require login

[@alperenersan]

Is it possible to implement with Llama 2 ?

[@user-wr4yl7tx3w]

Isn’t reading the dom tree better than screen shots given how the latter needs to interpret images as opposed to structured data?

[@AIJasonZ]

Yea DOM will be more effective, however the issue is each website DOM is very different and including lots of messy data; but if you figure out an universal way to clean up DOM then it will be effective Also with DOM, it lost ability to do some img related tasks

[@angrycrypto465]

All I need to know is how I can use this to create a trading bot that can learn and update trading knowledge based on my trades, then it can trade going forward without me

[@craigparker349]

Do you thing someone with zero coding ability can build what you've just shown? If not do you build this for people to use? Great video, found it very interesting and inspiring.

[@bt1940a]

Might i know the cost per screenshot and api?

[@asifabbas8811]

Hi the github link is missing alot of files can you update it please also with the requirement.txt files.

[@squiddymute]

does this work with ollama?

[@edgarl.mardal8256]

Can someone create a chatGPT manager to oversee that the prompt you set it to do, is actually done, and fine tune the prompt by it self? I am using chatGPT, with puppeteer, to take screenshot, and respond to the chat based on the dialog, this goes ok, but, it keeps making wrong responses as it messes up understanding the prompt.

[@wingman2tuc]

Can we use LlaVA?

[@yamani3882]

Can you scrape websites without getting detected?

[@AliAbassi1]

you are ... so fking smart man.

[@LearnCode_withAI]

Can you share the repo link?

[@croncoder862]

You sound exactly like a mature Jian Yang 😆

[@normanluismadrid422]

is this less detectable than using puppeteer alone?

[@AmplifyAmbition]

How much does each run cost on this?

[@CheatLayer]

Cheat Layer Desktop has solved this problem at the desktop level using a new multi-modal model we're launching in the coming days, which enables a generalized agent that can detect UI elements directly: kzbin.info/www/bejne/f4LYc3RtgtuIl9U

[@avi7278]

I'm not convinced. The step by step setup cost is too high and it still relies on GPT4 so the cost savings aren't even there. Or am I mistaken?

[@CheatLayer]

@@avi7278 our marketing agents drive traffic that would typically cost thousands more per month on google, and the sales agents get up to 11% reply rates--better than our human reps with another service.

[@moelkblau]

Interesting use case. Could you add the OpenAI API costs as reference?

[@Humpty0Dumpty]

A simple 4-5 step task will cost you $0.24, try complex tasks and reply back the numbers, if possible

[@moelkblau]

@@Humpty0Dumpty thanks!

[@vladimirgetselevich4704]

@@Humpty0Dumpty Pretty expensive at the moment, to do anything valuable in quantities.

[@vaibhavfromtruffle]

Do people recommend building an AI agent from scratch using Puppeteer and GPT or using an existing self-operating system like OthersideAI or MultiOn?

[@ekstrajohn]

Twitter bot armies have never been easier to make. :)

[@emmcode3414]

Don't sacrifice quality for views! You have an amazing KZbin channel that could be useful and profitable for you and for your viewers. I think I am not the only one who thinks that you should open a patron account and create premium content for people looking for more developed material 😁

[@stateportSound_wav]

That’s not a bad idea. Do you feel the content has been watered down for the sake of views?

[@emmcode3414]

@@stateportSound_wav No I don't, and I don't want to. That's why I am trying to encourage the channel to open a Patreon account, to preventing ourselves for loosing quality.

[@stateportSound_wav]

@@emmcode3414 10-4

[@stateportSound_wav]

@@emmcode3414 I hope that his skills have been able to get him to a point of not relying on youtube for financial stability, but that’s his business. Jason mentioned previously working on startups, so it sounds like he’s no rookie to business, and could be well off to the point your comment could (I hope, for Jason and ourselves!) be irrelevant.

[@hereusername]

@@stateportSound_wav Many channel about AI started just posting random news about AI or making clickbait videos just to get views which is really annoying. At this point I only check this channel

[@GargolaAli]

Hi Jason, thanks for this practical video. I would like to have your thoughts on my podcast on Process Automation and mining using AI, would be interested ?

[@user-wr4yl7tx3w]

Can this be done in Python?

[@sidhuk3128]

I dont know python. is there an alternative for Js, google didnt help me

[@KamilKaczmarekSolutions]

just take this code and go with it to gpt-4 and ask for rewrite

[@OblvckTech]

Has anyone here connected WebQL to a Crew agent? If yes, I would like to know how or a tutorial that can assist me. Thank you.

[@milostean8615]

Market size unlimited

[@cas818028]

Why would you jump back and forth between JavaScript and python? Just pic one or the other this seems super ineffective and inefficient

[@JockJutManhwaRecap]

im lost

[@howtomakepplragequit]

This is insane!

[@sauljudea6537]

why no one has built an online tool with API for this process? @aijason

[@thegreatestdao]

Sky torrents

[@grubbs420]

never trust this guy you will end up hacked

[@amandamate9117]

i have the same feeling. last time he promoted some free openAI APis and also his codes are fishy. or? also his hair or face looks like is processed with some AI filter. i think he works for chinese government.

[@mrfalvo6671]

lol@@amandamate9117

[@KamilKaczmarekSolutions]

@@amandamate9117 he 100% certified chinese government agent, agreed. in fact ai agent * 💀

[@hereusername]

@@amandamate9117 you can just check his codes yourself. It is not like secret program where you don't know what data is going off from you computer