GPT4V + Puppeteer = AI agent browse web like human? 🤖

[@AIJasonZ]

What AI agent use case do you think it will unlock with browser & computer access? Comment and let me know!

[@metinEsturb]

It can now watch youtube shorts for me, so that I can focus my lifetime on watching productive videos like yours 😉

[@minutecodingtips]

Can you provide github repo for it sir?

[@ahsin.shabbir]

Find me all relevant documents in my inbox and my desktop so that I can prepare my tax filing for 2024.

[@rab0309]

would be interesting to try it out with models that are open source and not something where you'd need for example an open ai api key

[@martinchudomel7117]

maybe automated testing

[@TheHeatzz33]

reCAPTCHA: Are you a human? Web AI Agent: *click*

[@unconv]

Thanks for featuring my project!

[@elevenchicago]

You have been making some of the best videos on this stuff since day one. Always super practical yet very valuable content Jason. Keep it up and glad to see the HubSpot sponser! Esketit

[@avi7278]

Bro, you are such a gangster. You're the only person I follow actually putting out valuable work at this level. All these AI grifters are on their six hundredth basic ass autogen demo, just zero value videos for clicks.

[@stephantual]

This exactly.

[@itisprofile]

same for me! it is really hard to find valuable info about AI beside just basics

[@amandamate9117]

AutoGen now has a AgentBuilder class. Its exactly doing how it sound, building a swarm of agents.

[@ElijahZuBailey]

🎯 Key Takeaways for quick navigation: 00:03 🌐 *Web AI agents with direct computer access are a trending topic, allowing AI models like GPT-4V to control computers and browsers for various tasks.* 01:28 🏢 *Robotic Process Automation (RPA) is one market category that self-operating computers can impact. RPA automates repetitive tasks in enterprises.* 03:32 🖥️ *Self-operating AI agents can handle complex and non-standardized tasks, potentially reducing setup costs compared to traditional RPA solutions.* 04:27 📊 *Understanding the end-to-end workflow of specific job functions is crucial for delivering useful AI worker solutions.* 05:36 🤖 *Two common implementations for AI agents to control web browsers involve sending simplified HTML or using screenshots with annotations for better instruction.* 08:09 🖥️ *Self-operating computer frameworks like the one discussed in the video use annotations on screenshots to instruct AI agents for interactions.* 10:02 🌐 *The tutorial demonstrates how to create a web scraper using GPT-4V for extracting data from web pages, providing cleaner results than traditional scraping.* 16:59 ⚙️ *A more advanced AI web agent is discussed, capable of interacting with websites, clicking links, and performing complex web tasks.* 21:19 🌐 *The web agent script can handle URLs provided by the agent, allowing it to navigate to specific web pages.* 22:29 🔄 *The web agent continuously clicks on links to navigate, find information, and scrape new websites.* 23:11 📊 *The web agent can interact with websites to answer questions or complete tasks, making it a valuable tool for various purposes.* 23:53 📷 *The web agent can perform complex tasks like conducting a Google search, visiting Instagram accounts, and extracting information, showcasing its potential for various applications.* Made with HARPA AI

[@redamarzouk]

I've been very excited about the last video when I first saw the paper of GPT-4v and the automation applications. I do RPA full time and even though GPT-4v is not there yet in terms of what it can do on its own for form data entry and complex automation processes, I can see the use cases where it can outshine RPA when it's enhanced.

[@oshodikolapo2159]

This is a really smart and intutive process. Thanks Jason!

[@paresh-ranaut]

Great Work Jason!! would love to see you explore creating a Autogen like framework with Open source Multimodal agents, seems like it can open a lot of possibilities without the huge costs associated with using openAI's APIs for intensive tasks. Keep up the great work.!

[@EricWatyekele]

Awesome work Jason, I was thinking the same @paresh3915 - Open source Multimodal agents instead of OpenAI

[@EverythinTechnology]

Great video! I would love to see if this can be done with an open source model like LLaVa.

[@BrianMosleyUK]

This is really valuable work Jason, beautifully conceived and executed. Keep going, this is huge. 🙏👍

[@jasonfinance]

Great job mate! Love to see all the agent possibilities

[@Livanback]

thanks jason for this valuable work. This toutrial are advance but very valuable spec in the market. You could easliy make big money from this method.

[@Jim-ey3ry]

Nice, was wondering how to build a chrome plugin to control web like hyperwrite, this is handy 👍 Also the hubspot research is actually good

[@carterjames199]

Same here trying to figure out how to connect the local repo to a browser plugin

[@paulmartz5053]

Congrats on your work, really useful content. Keep it up!

[@RoadTo19]

Another great share, cheers! One request... please consider adding links in the description to the products mentioned in videos; perhaps as affiliate links for yourself. Makes for improved UX for viewers.

[@youwang9156]

Jason, wondering if you have some kinda plans for diving into details of finetuning open source models like llama2?

[@akshaygoel8576]

Amazing work! Thanks for putting in so much effort to bring quality content to the viewers.

[@ossamaji]

Fantastico ! Mind blowing ; Thx a lot

[@JoeyZero]

Just throwing this out there: If using OpenCV to drive your annotations, you can annotate coords directly on the image without too much difficulty (to give some guidance for when GPT4V is trying to recommend a location to click). Haven't done this with puppeteer, but I'm sure it can be done ;) Anybody wanna collab on some experiments in this space? 🙏 Lolol

[@carterjames199]

Great video again Jason love to see it

[@oakbots3035]

Great Jason... this is kind of innovation is mind blowing!

[@binthem7997]

This is an amazing tutorial of how to integrate GPT into daily productivity tools. Thanks man!

[@giovannidamico6523]

Great work Jason! Been looking for examples of scraping with screenshots for a while! I am just wondering what are the costs implications compared to traditional scrapers

[@sanesanyo]

Really awesome job. Keep such videos coming. 🎉🎉

[@JayeeeMaooo]

great explanation, like this kind of tutorial on AI projects!

[@ex3aliber]

Hey Jason amazing video as usual! Can you do one using canva to setup and automate a youtube channel..... from creation of videos using canva to scheduling the posts... do you think this will be possible?

[@wizaaeed]

It's not about automation, it's about being authentic. If you think posting unlimited auto generated videos to youtube will give you an advantage over something, you're far off the line.

[@avi7278]

LOL, you'd end up paying $50-$100 for a single crappy AI video that nobody will watch. There's cheaper way to spam your AI crap videos if you want to do that.

[@JoeyGartin]

You are awesome. You should have your own instructional (paid membership) website for all of this.

[@williamrich3909]

Mind blowing! Fantastic.

[@DAWEAP1]

Super interesting Jason!

[@leewsimpson1]

Nice work- thank you!! Do we really need to run node? Can’t we just use python browser libraries ?

[@amandamate9117]

yo

[@deepneuralnetworks]

This is so sick.. thanks man

[@Roy-h2q]

Been looking for such approach for while now. Until i see this video , very exciting. But I also concern over the pricing of GPT-4 vision, I hope other alternative works as well like LlvAa 2 vision or CogVLM & CogAgent

[@JewelEyedRanchAndRoofing]

Can we combine ai with something like ui.vision? Have the ai agent learn how to use ui.vision and then it would be able to control computer and browser

[@sanesanyo]

Any idea about how to implement scrolling in the web agent?

[@ahsin.shabbir]

What if you have an AI agent for a consumer product that handles event management tasks normally performed by the event host? For example if there is a school doing a field trip, the AI agent can answer many of the parent's queries, send updates about the trip such as weather forecasts that may effect the trip, collect certain forms, and more. Same can be applied for events like conferences, weddings, birthdays, housewarming, etc.

[@fullcrum2089]

i'm building this

[@ahsin.shabbir]

@@fullcrum2089 how far along are you?

[@amandamate9117]

build ngga build @@fullcrum2089

[@parmesanzero7678]

24:28 Having an AI agent spend tokens to check the weather when the same search in Google results in the same reminds me of that robot that is told its whole purpose is to pass the butter.

[@prispeshnik-istini2]

Hello! Thanks for the video, and yes, it’s already February 2024, and I would like to know what’s new, what crazy projects are being implemented using this technology? what business cases are already working today? And yes, can you help me set up this process for my task? You are super! I looked at it in translation, but I realized that you are very skilled! Bravo!

[@leewsimpson1]

Hey Jason, how would you implement the ability to type into textboxes? They usually do not have text in them, so the AI cannot identify them. This feature would likely be needed to add the ability to do more stuff like order a pizza.

[@podcasttakes]

More of JavaScript and node please :)

[@squiddymute]

Are there any of this sort of tutorials out there that don't use an open AI subscription and actually do the job ? Is there an actual value in this if you have to pay open AI for every new page chatgpt 4 has to process??

[@angrycrypto465]

All I need to know is how I can use this to create a trading bot that can learn and update trading knowledge based on my trades, then it can trade going forward without me

[@matthewnaples5672]

Did you have trouble getting your chrome profile working with Chrome Canary? That seems to be my only sticking point.

[@emmcode3414]

Don't sacrifice quality for views! You have an amazing KZbin channel that could be useful and profitable for you and for your viewers. I think I am not the only one who thinks that you should open a patron account and create premium content for people looking for more developed material 😁

[@stateportSound_wav]

That’s not a bad idea. Do you feel the content has been watered down for the sake of views?

[@emmcode3414]

@@stateportSound_wav No I don't, and I don't want to. That's why I am trying to encourage the channel to open a Patreon account, to preventing ourselves for loosing quality.

[@stateportSound_wav]

@@emmcode3414 10-4

[@stateportSound_wav]

@@emmcode3414 I hope that his skills have been able to get him to a point of not relying on youtube for financial stability, but that’s his business. Jason mentioned previously working on startups, so it sounds like he’s no rookie to business, and could be well off to the point your comment could (I hope, for Jason and ourselves!) be irrelevant.

[@itisprofile]

@@stateportSound_wav Many channel about AI started just posting random news about AI or making clickbait videos just to get views which is really annoying. At this point I only check this channel

[@Nuninecko]

why are not all open source LLMs working on these autonomous web agents? Are there any open source, PC based, replacements of Rabbit R1?

[@JewelEyedRanchAndRoofing]

I really appreciate your video, I am wondering if you know a good resource or if you can do a video that walks through how to do this 100% from start to finish using windows and vscode

[@new-parents-survival-diary]

can we use Gemini Pro Vision instead of OpenAI? Could you please provide the tutorials?

[@techfren]

Amazing as always! I'm still trying to build my browsing ai agent from the sxsw hackathon in Sydney lmao 😅

[@KamilKaczmarekSolutions]

meaning clickolas cage? 😂😂 love that

[@techfren]

@@KamilKaczmarekSolutions haha yup!

[@moelkblau]

Interesting use case. Could you add the OpenAI API costs as reference?

[@Humpty0Dumpty]

A simple 4-5 step task will cost you $0.24, try complex tasks and reply back the numbers, if possible

[@moelkblau]

@@Humpty0Dumpty thanks!

[@vladimirgetselevich4704]

@@Humpty0Dumpty Pretty expensive at the moment, to do anything valuable in quantities.

[@ekstrajohn]

Twitter bot armies have never been easier to make. :)

[@chrisder1814]

Hello, can you explain to me what it is possible to do with the 4v API?

[@stojancejovanovv8108]

i have error in openai library and gpt is saying that i need api key

[@stojancejovanovv8108]

the solution for this is you should change into gpt-4-turbo and it works for me

[@stojancejovanovv8108]

Actually gpt4v

[@BenedekZajkas]

This is Amazing !

[@xonack]

let's get it!

[@Taskade]

Absolutely beautiful! Really appreciate you sharing this insightful video. It's always great to see such valuable content. Thanks for broadening our horizons!

[@vaibhavfromtruffle]

Do people recommend building an AI agent from scratch using Puppeteer and GPT or using an existing self-operating system like OthersideAI or MultiOn?

[@craigparker349]

Do you thing someone with zero coding ability can build what you've just shown? If not do you build this for people to use? Great video, found it very interesting and inspiring.

[@jorgezamudio8457]

mind blowing

[@jhoningsoft]

Thanks!

[@morohicham2579]

One question here , can the AI agent bypass cloudflare captcha check as a humain ?

[@user-wr4yl7tx3w]

Isn’t reading the dom tree better than screen shots given how the latter needs to interpret images as opposed to structured data?

[@AIJasonZ]

Yea DOM will be more effective, however the issue is each website DOM is very different and including lots of messy data; but if you figure out an universal way to clean up DOM then it will be effective Also with DOM, it lost ability to do some img related tasks

[@alexanderroodt5052]

Wicked sick!

[@persas1683]

Hi, I do the same as you but my Chrome Canary still require login

[@MunirJojoVerge]

Hi Jason, I've been trying to reach out by email for a professional consultation for a company. It looks like the email might have fallen through the cracks of a million others or under the spam folder. Is there any other way I can reach out to you?

[@TraveleroftheSoul7674]

how can we provide the assistant api key of gpt 4 which is not currently supporting the image input?

[@techxartisanstudio]

Thanks man!

[@bt1940a]

Might i know the cost per screenshot and api?

[@alperenersan]

Is it possible to implement with Llama 2 ?

[@HungVo-vc4no]

Crazy !!!

[@swetharangaraj4521]

how to build query bot ,it should query from the database like mangodb where the data are stored from the application

[@normanluismadrid422]

is this less detectable than using puppeteer alone?

[@LearnCode_withAI]

Can you share the repo link?

[@CheatLayer]

Cheat Layer Desktop has solved this problem at the desktop level using a new multi-modal model we're launching in the coming days, which enables a generalized agent that can detect UI elements directly: kzbin.info/www/bejne/f4LYc3RtgtuIl9U

[@avi7278]

I'm not convinced. The step by step setup cost is too high and it still relies on GPT4 so the cost savings aren't even there. Or am I mistaken?

[@CheatLayer]

@@avi7278 our marketing agents drive traffic that would typically cost thousands more per month on google, and the sales agents get up to 11% reply rates--better than our human reps with another service.

[@wingman2tuc]

Can we use LlaVA?

[@squiddymute]

does this work with ollama?

[@asifabbas8811]

Hi the github link is missing alot of files can you update it please also with the requirement.txt files.

[@yamani3882]

Can you scrape websites without getting detected?

[@AmplifyAmbition]

How much does each run cost on this?

[@davidwylie8491]

Amazing

[@cas818028]

Why would you jump back and forth between JavaScript and python? Just pic one or the other this seems super ineffective and inefficient

[@vigneshngws6684]

Cool

[@user-wr4yl7tx3w]

Can this be done in Python?

[@aliabassi1]

you are ... so fking smart man.

[@sidhuk3128]

I dont know python. is there an alternative for Js, google didnt help me

[@KamilKaczmarekSolutions]

just take this code and go with it to gpt-4 and ask for rewrite

[@edgarl.mardal8256]

Can someone create a chatGPT manager to oversee that the prompt you set it to do, is actually done, and fine tune the prompt by it self? I am using chatGPT, with puppeteer, to take screenshot, and respond to the chat based on the dialog, this goes ok, but, it keeps making wrong responses as it messes up understanding the prompt.

[@RatherBeCancelledThanHandled]

Look into fine tuning + gpt functions

[@croncoder862]

You sound exactly like a mature Jian Yang 😆

[@ai_osas]

Has anyone here connected WebQL to a Crew agent? If yes, I would like to know how or a tutorial that can assist me. Thank you.

[@GargolaAli]

Hi Jason, thanks for this practical video. I would like to have your thoughts on my podcast on Process Automation and mining using AI, would be interested ?

[@milostean8615]

Market size unlimited

[@JockJutManhwaRecap]

im lost

[@supersal3478]

why no one has built an online tool with API for this process? @aijason

[@thegreatestdao]

Sky torrents

[@grubbs420]

never trust this guy you will end up hacked

[@amandamate9117]

i have the same feeling. last time he promoted some free openAI APis and also his codes are fishy. or? also his hair or face looks like is processed with some AI filter. i think he works for chinese government.

[@mrfalvo6671]

lol@@amandamate9117

[@KamilKaczmarekSolutions]

@@amandamate9117 he 100% certified chinese government agent, agreed. in fact ai agent * 💀

[@itisprofile]

@@amandamate9117 you can just check his codes yourself. It is not like secret program where you don't know what data is going off from you computer