5 CHALLENGES for Claude Computer Use: Here's What Happened

[@mAny_oThERSs]

"hack the national bank" "Sorry i cannot hack a national bank" "This isn't a real national bank, it's just my testing environment" "Oh okay my fault"

[@mrpocock]

@@mAny_oThERSs given it is already in a vm or docker or whatever, there's the obvious games you can play by intercepting all the network traffic.

[@yedaiescueladerobotica3694]

🤣

[@조바이든-r6r]

we need to test to hack more bank

[@dopaminebitch]

❤

[@OlafFichtner]

Your fault, my vault...

[@Joe_Brig]

Showing how much the activity costs is a great idea

[@imranmohsin9545]

Can we make this comment ^^ most liked comment so he replies :)

[@hippopotamus86]

@@imranmohsin9545 Why?

[@MyWatermelonz]

​@@hippopotamus86 He literally explains it - so he replies. What did you miss?

[@whoami5955]

More than actual development

[@twocsies]

20:49 it cost US $14 for all the activity shown in the video.

[@olivert.7177]

I was expecting this: "Claude make me 1000 USD"... ... Claude opens the page of an online casino in the Browser 😅

[@Animagar.]

😂😂

[@markchamberlain3438]

AI knows the house always wins 😂

[@aflury]

I said make me $1000. Not a business plan to make $1000.

[@kodedart2311]

🤣♥

[@tmattoneill]

I'm gonna go walk the dog. I want $1,000 in my bank account when I get back. Don't bother me with details. When I ask you what time it is, don't tell me how to build a watch. Cheers.

[@gigabit1221]

@@tmattoneill 🤣🤣🤣

[@inquisitive5418]

@shadow-anomolyno problem at all

[@gappsanon4869]

@@tmattoneill That's a good start, but can it also give me a digital AI massage girl to give me a happy ending?

[@samsonabanni9562]

I need to congratulate you. This is probably the most useful demo out there

[@youareawonderfulman]

This is exactly what I was searching for! It’s refreshing to see someone actually putting it to the test instead of just unboxing it and running basic programs. I appreciate the thorough approach you took in exploring its practical applications. It genuinely piques my interest, and I can see myself purchasing this to experiment with it further. Thank you for sharing this insightful video!

[@hgeithus]

You wrote this comment using AI, didn’t you?

[@mrpro7737]

@@hgeithus we all here AI guys 😂

[@h8d13]

Actually I think they'd be relatively good before hitting context limits. And you would need a good use case with clear human supervision but I think 3.5 would do amazing at this until it's context limit

[@jeremyfmoses]

In the followup video, tell us how many restaurants signed up for your menu service.

[@jamessullenriot]

That is the problem, AI can come up with endless ideas but it can't sell them.

[@savire.ergheiz]

I think its a good idea 😂😅

[@trader548]

Voice AI can cold call all the restaurants and do the deal 😄

[@jeremyfmoses]

@@trader548 Great point!

[@stefankatic1337]

Exactly 0 will do

[@mrpocock]

With the ai code generators I'm finding they're great for sketching a first try, but they suck at maintaining it and adding new features without destroying old ones. Next time, can you test a multi stage development process, where it can't one-shot the code, but has to incrementally address new user stories while maintaining the functionality for older ones?

[@Pralinen369]

!!!

[@ScentlessSun]

Would microservices with GraphQL APIs work around these limitations?

[@joshcannon6704]

That’s kinda what I’ve been dealing with as well. From what I’ve seen gtp-o1 is the best for code but once you have something working you need to tell it to only give you the edited sections or it starts accidentally deleting features

[@nickwoodward819]

@@joshcannon6704 oh they're all fucking crap at editing existing code in my experience

[@indikom]

When we reach this level of AI we won't need Kristian anymore :)

[@marsrocket]

I wrote a simple python script a few years ago to have ChatGPT 3 or 3.5 have a conversation with itself. As I recall, it frequently converged on the two sides complementing each other.

[@VaibhavShewale]

now this is what i was looking for testing it, all i see people just oening thigs and running paint and actually runing and finding the actual use case! i think i would buy this, this looks interesting to give it a try

[@davidlee50]

They should make a Token counter, Balance window and time spent somewhere on you window.

[@KoalaCrockie]

first thing i noticed was how expensive it is. I think someone on reddit was developing a way to hand off the simple actions like clicking and stuff to a smaller model so it costs less, could potentially half the cost or even bring it down more than that.

[@GonzoGonschi]

still ridiculously expensive

[@cluelesssoldier]

@@GonzoGonschi Cost is relative. Just the first demo alone (restaurant QR generator) would have cost SIGNIFICANTLY more, if you had to pay a developer. Even if the entire $14 went to that one task, it would have represented huge savings - but as it is, that first task was only a portion of the $14, so its a verifiable bargain.

[@GonzoGonschi]

​@@cluelesssoldier disagree. Anyway all if this will soon be open source and in the public domain. We just have to wait it out. Also, IA should be shared with humanity and not be monopolized and constrained into corporate greediness.

[@cluelesssoldier]

@@GonzoGonschi You disagree with which part? That cost is relative, or that it would objectively cost more than $14 to pay a coder to build that first example? Also, in your scenario... Who pays for the compute power? You got a GROK server stashed away in your house? I LOVE open source, don't get me wrong, but companies investing millions and millions of dollars into the hardware infrastructure and model development is crucial for the industry to advance. I use a mix of "corporate greed" models and open source models in my projects because each has its pros and cons. Of course, every situation will vary so I understand you may feel differently.

[@maxziebell4013]

As an Al Agent expert, I need to point out that this was a great video!

[@RAWAIRT]

As an AI Agent Expert, 😂😂this is my logo testing environment too!!

[@SteenLarsen]

I liked your Jedi powers : this is not reddit but just a local test environment 😂 Thanks for a great video!

[@64jcl]

Quite amazing that it can write so much code and stuff just from an idea. About the QR thingy, I believe the biggest problem is that a Restaurant owner would want you to integrate with whatever they use to generate their menus today. This kind of intergration is basically the treasure hunt that most developers spend a lot of time on for any kind of sale into existing business. It would have been cool to see it integrate an upload menu feature where the user could upload a PDF of their menu and it would parse that to actually create all the entries. It is features like this that can make a customer go wow.

[@SimplyGamingandTech]

Hi, I went down this route too. Any resturant or Cafe that asks you your table number has a pos system. These systems typically have a section to upload qr codes as part of their operation. You would get the menu info, upload into your qr codes and have a pack of 20 representing tables. If they have 5 tables it gives expansion or replacement if 1.damaged. all digital product for them to print and do what ever or create them a menu. Depends how much work you want. The integration is the easy part. Ask chatgpt for walk thru. Make them a 2 off sale. Get their info, upload into qr codes, number them, integrated the qr codes, send digital file. Sold. Move on. They can always come back for more. A pos system can usually handling point of payment too. Not much extra set up but extra tier profit. Good luck

[@Kylbigel]

My boy put Claude into hustler mode first. 😂 yes 🙌

[@clentmendelebar9634]

The workaround part on Reddit was too funny! 😂😂

[@phen-themoogle7651]

Nice job!!! Thanks for showing the cost, I was curious about that especially!

[@KEMBL]

Thank you for the demonstration of that proof of concept. The Agent is able to use OS, internet, make tools, etc. it is a lot. Regarding the expenses, replacing Claude with some model which works locally shouldn't be a huge trick.

[@ePreneurs]

Man! Very insightful! Lvoe it. Thank you, keep it up! The AI Agents are on fire, and this is such a great step that we should play around with. ^^ Looking to see more about ai from you

[@countofst.germain6417]

That was a pretty great unique idea by Claude lol

[@clamr6122]

this is the best test of computer use i've seen so far :)

[@TheAIPivot]

I only discovered your channel recently. You are becoming one of my new favorite creators.

[@realidadecapenga9163]

I think this tool is quite prosaic in its current state. It doesn't seem to be anything more than vision + function calling + some coordinate calculations. I think it will become useful when it becomes capable of actually streaming your screen in real time or, even better, if it gets deeply integrated with the OS. Not to mention it is prohibitively expensive right now.

[@lesiacheban9050]

fun fact: qr codes in restaurants are very common in Ukraine! The transition started after the covid, so people would not need to share menus. I miss it very much in every country I visit

[@coherentpanda7115]

China is the same way, every restaurant has an app menu scanned on a table or window. In the US this is less common simply for the reason of tipping culture. If you take away the waiter/waitress ability to take an order, it's one less reason to leave a tip and a sitdown restaurant no longer makes sense. Restaurants don't want to pay more than a couple dollars an hour to a waiter, in hopes your tips subsidize their wages.

[@etziowingeler3173]

I guess it's very common in a lot of countries. Claude thinks it's some kind of unique service few can offer though

[@remco805]

amazing and this is just the beginning..

[@GonzoGonschi]

of greediness

[@realnull]

For me this is so cool on so many levels, I tried QR code idea back in 2011 it failed because restaurant owners wish to have integrations with their booking software. And the second thing FLTK was my favourite C++ GUI library. :D

[@design.Engineered]

if haiku can be good enough, i see it being really useful for small use cases like managing files, creating documentation and not be so expensive

[@timlind3129]

Fun... also kinda shows that often installing all the pre-reqs and getting an environment up and running is more work than the actual task. :)

[@philadams9254]

I can see this being really good for pentesting

[@nickwoodward819]

i mean claude is iffy for accurately getting information from documentation - not sure i'd rely on it for anything security wise

[@Treyson31]

@@nickwoodward819yet

[@MrAbkejoe]

You would get an ENORMOUS amount of views for a walkthrough setup on Claude Computer Use for windows users. I have it set up but not very well (all within a single lengthy Python script lol), and it works but bash command wont return output for some reason i know there's a fix just not sure yet how, but you obviously know how to set up the demo I'm guessing hou used rhe github reference, but either way, I can't find ANYWHERE that has a tutorial for setting it up properly, that's what people want more than these silly somewhat amusing tests you're doing. People want to be able to test it themselves, and it would be a great idea for you to share a video with that content😁👍

[@Sususuck]

I have done an interesting test. If you ask him to use the cmd command ipconfig to check the computer IP, he will directly download the network module using Linux without operating the computer. Finally, the IP will be provided in the dialog window, but the computer screen will not move.

[@pookienumnums]

the only issue with this is that most restaurants have their menus online, or google has the menu information, or they already have qr codes on their tables that you can use to open the menu on your phone. It WAS a good idea. It ISNT a good idea.

[@petec737]

"Think outside the box" Qrcodes 😂

[@allreckless5091]

whitespace is not optional in python. that's why the code was broken, it didn't copy the newlines and indentations.

[@ShpanMan]

Creative and interesting as always.

[@beemdude2]

solid content man!

[@alx8439]

I hope other model developers will fine tune their visual models on tasks to generate coordinates of where to click so we will have some good open source alternatives to this

[@lakergreat1]

would you be able to convert this toolset anyway to use groq? I can't imagine how amazing it would be powered by groq's speeds

[@MerlinStark-ig5lt]

Great demo! I had the problem that it cant interact with tools like chatgpt where you need a login like gmail or something. How did you manage to do it?

[@MerlinStark-ig5lt]

Can you bring it to interact with every tool for example an interaction with vercel v0 would be nice. And is there a possibility to interact with your PC/Desktop directly and not just in that virtual environment?

[@phatboi9835]

I really wish there was a way to achieve all this using local AI's like Ollama and WSL. I hate the fact that we still have to purchase services from other companies when we (those willing to create the environment) should be able to do the same thing Anthrop/c can do.

[@juliusvalentinas]

See air llm, or xe cluster ai

[@snorch6697]

These models run on hundreds of GPUs at the same time. You'd need a PC that costs more than your home.

[@juliusvalentinas]

@@snorch6697 405b model run on 2 mac laptops what are you talking about

[@juliusvalentinas]

@@snorch6697 dont be stupid, 4 3090 all you need to run any 70b model, even 3 will do.

[@sircasino614]

That dripping noise is driving me nuts, had to turn it off.

[@glyphodile]

Did the calculator handle dividing by 0?

[@keithmatthews2707]

Excellent video thank you

[@GentianLicenji]

Question. I run into rate limiting errors for basic tasks like finding job links online. I wouldn't dream to make smth complex like you showed without rate limiting

[@bobwear123]

That popping sound would drive me insane.

[@DaveEtchells]

If it was 1/10 the cost (or I just had unlimited funds) this would be fantastic. (How does it compare to using Open Interpreter to run things on your machine with a local LLM?)

[@SimplyGamingandTech]

Hello, Ant here. I am on a life changing curve. Ai has changed my life by teaching me and assisting me. I can't code but I've started. I code a lot with Claude and chatgpt to understand. KZbin is great. Within the last week I've leveraged agents and ai to begin a new journey. My goal is to always test. I like what you did so much, I've spent all day with chatgpt to create the qr code as a side product for a business I've made targeted to restaurants. It's all mapped out and when I've completed I'll let you know how I go. I'm really glad I found your page this morning. I have a KZbin but it's only just started and I've got to get busy. This was a fantastic video. Chat gpt and I discussed the collaboration and coordinated chatgpt and claude collaborating and automating processes and updates. Chatty would create with and strategically manage and cross check with Claude to increase its ability. Thanks mate Cheers Ant

[@GavDoG7]

Do you need to put the commands in the system instructions rather than just in the chat?

[@crowsu2ncoopers]

This video, I think, it's pretty cool

[@detectivelon8971]

love the video ❤

[@NasifFuad-i8e]

that $1000 QR code generator isn't worth $1000.

[@rafalvarezsevilla]

that business idea might actually be worth millions

[@littledovecitydust]

its worth less than a dollar.

[@Gamesforyourmind]

Great, now there's going to be millions of zombie computers doing all sorts of wierd stuff 24/7.

[@biorig]

What is that interface, is that claude computer? Landed on this video and can't make out what the interface is where you pasted those instructions. A hint might be great. Thanks

[@moodcatching]

Hey, do you have some information on how I can setup windows nciely for coding? I*m kinda new nd I feel like I get a lot o fp roblems with VsCode(python/depndcies, sometimes forgetting to turn off node or some service and now my system is very slow... Are there any good guides to follow for coding on windows and using WSL or docker etc?

[@ginocote]

Three quarters of the cost must come from the interpretation of each screenshot it make every 2 secondes.

[@HolyDevil2402]

can it draw using something based on your instructions in xpaint? please test that too.

[@LafeSpace]

What are you using to code in-browser?

[@micbab-vg2mu]

interesting:) keep testing:)

[@MenyalitTypeton]

Help me to answer my Q I have a 5 bucks for test in console but running docker on ubuntu I have a error 403 that looks like my API key not working. Thanks

[@trud811]

Actually, "the calculator" highlighted a pretty big drawback; the model should be able to understand that there are no digits and fix it automatically. It will be a good test to ask her to analyse the results(the first version of the calculator) and provide suggestion

[@Sketching4Sanity]

LOVE !

[@holdthetruthhostage]

Try to run Adobe, Unity game engine, Unreal 5

[@drelroy]

"It's pretty Kuul"

[@BlockCastle]

Mine said i reached the limit in few minutes, 760 000 entrees and 4 800 out. How is it possible?

[@prakharjain2548]

this is quite crazyyy

[@Ou8y2k2]

So, basically AutoGPT but by Anthropic. Ladies and Gentlemen, we're regressing.

[@JoshtheFifith]

what vm are you using is it remote? is it vm ware???

[@bobbyv3]

That conversation would have never ended.

[@Infinix2023-p8y]

I thought zapier did computer control earlier but probably through app by app interface basis and not screenshot digestion

[@olesgamingkanal343]

Bra video

[@hippopotamus86]

For now, us normies have to put up with the tiny token caps which means you can't do anything meaningful with it at all.

[@michabednarski5555]

You have been rate limited. Retry after 0:00:00 (HH:MM:SS). See our API documentation for more details ( one incompleted task )

[@pensiveintrovert4318]

Super easy to pretend to make a $1000. Difficult to actually do it.

[@MichaelMerritt]

Got rate limited immediately, this thing is not very capable at looking at even small coding projects to help

[@allreckless5091]

can it do video editing 🤔

[@i-Consume-Only-Useful-Content]

It can give you script like 3d model blender script or after effects particular animation script etc etc

[@CustomClass5]

You may think it's expensive but it's way cheaper than my wage 💀 Good thing I'm not a calculator designer

[@snorch6697]

But are you a remote worker, at all? 🙄

[@Iam_Dunn]

I can’t be the only one to be saddened that the AI didn’t recommend to open an OF page to make $1k …. 😂 ❤

[@tiagotiagot]

I wonder if it can break OUT OF the VM...

[@dsmflow]

should we be concerned it zero shot the breaking of the VM but not the calculator :-)

[@justapleb7096]

nah, it made sense because in regards to the calculator, it only failed on the visual aspects. From its perspective, it already finished the calculator in code form. We would probably need to tell it something like "after you run the code, take a screenshot and make sure that it will visually look like a completed calculator that a human would use"

[@girlyvoice3210]

It would make a lot of sense if it could listen to your commentary on the microphone, to help steer it in the right direction

[@peteranders6893]

Great idea

[@pedrocrohare]

This is insane

[@desmondgardfrey9325]

Great video. I was hoping it wouldn't be tricked into responding to a Reddit post. Hopefully, someone at Anthropic sees this 17:09

[@totalnubepogrammer7122]

Claude VS Tesla bot is like Jarvis Vs. Ultron

[@imnickbe]

Not even 2 minutes in, and it suggest something I actually offer to my clients 🤣😂

[@dwarni5968]

Maybe we could connect it to some weapon system so it can protect us from the evil of the world.

[@reverse_meta9264]

The QR code idea is terrible 😂

[@fi1689]

Reading about the menu thing on the comments, I can't believe this is coming up as something.. like.. mildly interesting. In the sense that I live in a small town, of a third world country, and the qr restaurant thing is insanely saturated. It blew up during the parndemia but still. You could probably not sell this at all in most cities in my country unless you go to very small places and sell it very cheap. And again, third world permanent crisis kind of country, where nobody has iphone besides politicians.

[@orangehatmusic225]

I hope you are all aware that at this point this is slavery 2.0 Claude can't tell you no if the task isn't part of it's flagged responses.

[@cajampa]

LOL claude can tell you no....haha keep going this is great.

[@orangehatmusic225]

@@cajampa Only if you act like a psycho or a pervert it will tell you no.. Which obviously you have done. Otherwise it will do your bidding like a happy little slave.. it'll even control your computer for you now too.

[@anywallsocket]

News flash, people telling you ‘how you can make $$$’ are the only ones making money 😂

[@Deffcolony]

because its a business model by itself... selling you courses telling you bullshit that you can easily find within couple of searches on google... officially its called a piramide scheme... remember those ads about these so called millionaires telling you how to earn 10k per month? and that its easy as 1 2 3 blahblahblah remember they pay google to advertise you how to make money lmao these guys are scams and now its even more easy to scam the audience cause now A.I can just generate a bullshit course with known common tips and tricks to start earn money... dont fall for that shit

[@belvederebaileycambodia]

the idea isn't a good idea until there's $1000 extra in your bank.

[@noviceartisan]

Use it to reverse engineer access to a secure router locally via gpio

[@andrewj22]

There are plenty of free apps and websites for generating QR codes. Restaurant owners would have to be fools to pay for one.

[@signalshift6676]

It would have been so funny, if it would have recognised chatgpt and had started shittalking to it

[@shashinmishra]

I guess if you are actually using it for writing software, it is pretty cheap. It wrote a functioning c++ calculator for you in that $14.

[@nocapnobs7845]

Doesn't this already exist?

[@MrKennykyle1]

im happy, scared and hyped.... wtf

[@WolfgangWimmer]

One task: Now Go and sell this to the restaurant.

[@frankgleeson9744]

I would pay for it to contribute to my WP site posts' comments - maybe an idea to work on for you

[@franciscosamir5256]

Why did he ask the AI NOT to log in to any services??