Letting GPT-4 Control My Terminal (TermGPT)

Giving LLMs like GPT-4 the ability to plan and execute terminal commands.
TermGPT github: github.com/Sentdex/TermGPT/
OpenAI Chat API tutorial: github.com/Sentdex/ChatGPT-AP...
Neural Networks from Scratch book: nnfs.io
Channel membership: / @sentdex
Discord: / discord
Reddit: / sentdex
Support the content: pythonprogramming.net/support...
Twitter: / sentdex
Instagram: / sentdex
Facebook: / pythonprogramming.net
Twitch: / sentdex

[@travelbytestv]

Beginning of Skynet

[@lazydoner6552]

😨

[@madjunir]

Baby Skynet (aka GPT) may be cute now ... But wait till it grows into a Mecha Godzilla 🤖🦖 destroyer of worlds 💀☣️

[@aoeu256]

This is just a version of agentgpt haha. I got chatgpt to run code on my Jupiter notebook btw.

[@kristopherleslie8343]

Sky net doesn’t exist lol per the continuity

[@juliap.5375]

@@kristopherleslie8343 Actually real Skynet, AI which can launch nukes, exist. And even arrived before "Terminator" movie. So-called system Perimeter, it is AI which have right without any confirmation from human to launch all nukes. Initially it was invented for situation of first preemptive strike by US regime when they could to kill top political and military leaders in USSR, but with time became more complicated and intellectual (nowadays it is supercomputer, ~50th most powerful in world).

[@ChaiTimeDataScience]

You know it will be a good weekend when Harrison shares a project for us to learn from!

[@yuriborroni5490]

Sounds like a great weekend plan!

[@avi7278]

Sentdex, good luck. You're about to fall into a deep rabbit hole with this stuff. It's a fun rabbithole, though. I ended up doing this really detailed structured setup with gpt4 building out step by step shell commands to handle a user's task input, giving it "tools" each having their purpose and structured documentation for commands and their usage (arguments, flags etc). Then you inevitably try to get it self-reflect and confirm the approach is actually the best, fix its own errors, build its own reusable scripts and add them to a library, and then try to get it to choose the right existing script for the job rather than trying to write a new one for each user task. And then there are the rules, oh the rules.... do this, don't do that, "you MUST ALWAYS do it like this". Big, long lists of rules.... And then when you can't quite get it to do what you want, but it's oh so close, then you get to the human in the loop phase, where it does most of the work, but you try to work in some manual guidance into the system to get over that last hump. And finally, you come to realize while this is all very very cool, we're still at least one generation off from REALLY being able to do this thing. At least, that was my conclusion. And also, python is kind of a sh** language. Oh and there's the cost optimization trying to get it to use 3.5 for tasks that you know it's capable of because boy do thost GPT-4 api costs really start to add up after a while.

[@ilozeet]

whys python shit what better in your view? rust?

[@moshelevy3614]

Oh man, I think I'm seeing my future

[@cyberpunkalphamale]

Ai still hasn't fixed m hairline but it does wonders on command line.

[@yuriborroni5490]

Agreed, building structured setups with GPT-4 is a fun rabbit hole to go down. However, it can be challenging to get it to do exactly what you want, and there are many rules to follow. Python can also be frustrating to work with, and GPT-4 api costs can add up quickly.

[@erenoz2910]

I'm really looking forward to the day when open source models catch up to GPT4. I've read an article recently that suggests that the open source models are quickly closing the gap, although they are probably way off the mark for precision tasks like this.

[@MrMasterNorris]

this shits amazing. People think AI is new but if you saw the GAN Theft auto video, you would know how powerful this stuffs been for a while

[@richardaragon8471]

Been waiting for a video like this, thank you.

[@matthewbishop337]

Not sure if you have seen the Kor library that is a part of LangChain, but that may be a way to train the responses well too. It's main function is to pull values from NL and convert them to a structured object, but I see no reason that you couldn't do the same thing with chat messages and expected responses. I am not exactly sure what prompts he uses, but we had some good successes with it using davinci/gpt3turbo instead of GPT4. Very cool though I can't wait to try this out!! Thanks so much for sharing!!

[@nuclear_AI]

Even your outro credits are educational 🤜🤛

[@MaJetiGizzle]

Falcon-40b recently had its license changed to being royalty free and TheBloke and others on HuggingFace are working to get it to provide faster output in spite of the Quantization slow down issues. So that might be worth giving a try with self hosting on an API and/or that with a combination of other models for different kinds of tasks.

[@sentdex]

Yeah this is the model I'm trying to figure out using atm. Don't think I wanna self host it. Wish there was a pay per token/query API for it.

[@MaJetiGizzle]

@@sentdex Well, if you don’t want to deal with self hosting using a cloud hosting platform like runcloud should allow you to set up GPU based instances or serverless function calls. Combine that with something like FastAPI would allow you to make calls to an API that are relatively affordable cost-wise.

[@WistrelChianti]

I never thought of setting the behaviour via pre-filling the conversation with fake/example conversation. That's genius! I've just been using a singular system prompt (although TBF that's all I've needed so far). I've made something similar but not as extreme. Just an implementation of the chat client in the shell. I agree, its SO much faster to have everything in there rather than going off to the net/web client. I've added a bunch of other stuff like commands for controlling conversation memory (turning it off or on, clearing, tracking token count etc) as well as code syntax highlight, logging and an option to create a sub shell to start testing stuff out so you can return and keep convo context if you run into issues. Similarly, I've a bunch of command ideas I've not implemented yet and have defo encountered bugs (mainly with the syntax highlighting) but it's working most of the time. Liking your idea about grabbing stuff from the net and feeding it to it for summary etc. I suppose you could have it do a web search, pass it the results and ask it which one it wants to look at, then fetch that back to it etc. Seems like the possibilities are endless once you get going and it's always so much fun seeing what it does/how it responds.

[@aoeu256]

How about using chatgpt to gain relational constraints about the world and using chatgpt and google to mine for the constraints hmm.

[@Lavamar]

I love how you showed it editing its own code

[@dyllanusher1379]

You might be able to use the guidance library by Microsoft to explicitly format responses on your agent

[@maalonszuman491]

hey great video like always !!! little question are you gonna make a video where you take an open source LLm not trained and teach how to train it ?

[@lio1234234]

You lucky sod. Having access to the gpt-4 API 😂. Glad to see you're making good use of it though 👍, great stuff.

[@calcs001]

Working on a similar project called AutoCoder. Works well! What your doing is very possible. AutoCoder focuses on python development. To help it along, I ask for test cases, run the test cases, ensure they work. Before any of that, run a linter and AST check for syntax issues. Also tell GPT to tell me what pip packages are required, and it installs them. On any error in the multi step process, feed the errors back to GPT for resolution. If all else failed, summarize results, with the original ask, and compose it as a new ask. Rinse repeat, works well.

[@calcs001]

Also, for easy tasks, use the gtp3.5, if you hit consecutive errors, tell it to use gtp4. Then when it’s done, it just uses 3.5 again. Everything escalates if it fails to many times.

[@gnosisdg8497]

well there is the code for that then please?

[@tugamer89]

You should continue your serie about neural network from scratch in python: it had a LOT of success and it's very helpfull!

[@AusTom]

the book works just as well! :)

[@iamacoder8331]

Just buy a book and read. Its very good.

[@tugamer89]

But it's not free as videos are

[@ander300]

Part 10 of Neural Net from Scratch, about analytical derivatives??? Please bring the series back!

[@TankorSmash]

What was the issue with subprocess? You said it didnt work but was the issue the returned value was empty, or did it error out? I'd be happy to dig into this with you as a way of saying thanks for all the content you've put out!

[@ok-__]

Hey dude, I saw some of your old videos where you created bits to play video games. Could you possibly bring that type of content back? That would be sick

[@ettmargarin]

Hi! There is a model released called WizardLM/WizardCoder-15B-V1.0 that apparently now beat open source LLMs as well as bard and both versions of Claude, in HumanEval and HumanEval+ benchmark, though it is still right behind ChatGPT 3.5. It is fine tuned for coding, self hostable with ggml versions out, and apparently fast to my knowledge. So, in a lot of ways it seems like a good open source candidate model I guess?

[@tonywhite4476]

This is f'ing awesome!

[@James-ms1su]

Cool stuff. I'm sure you're already aware of it, but Auto-GPT is aiming to do something similar and is very interesting.

[@tigoes]

The issue with the "reading from console" is when your model decides to run a non ending process, like running a web server, it will not receive any output and endup waiting indefinitely. Another problem was that sometimes it chose to run "cd path" in one command and then "npm install" in another, that wont work well in one command per process scenario. I tested an approach when a model can "open_console, close_console, send_command, and read_logs" which handles this scenario with an async approach. The problem i faced was that offen the model chose to run the same command again and again even when he has all the logs present in his context and enters some kind of infinite loop of send_command, read logs... However, this might be better now with GPT4.

[@bs_blackscout]

01:30 lol that peak 😂

[@ronan4681]

I have been working on a similar idea, but i have it structured in classes which separate files for each class. i gave GPT 3 databases, a SQL database to store info, vector database to chunk reference material into, and a separate database for project management and as it modifies its needs it creates a new task in the project database. I wrote a class for internet access for research, it can download pdf which it processes and stores in the database.. Semi working, and it has created a few classes to enhance itself. Working on a supervisor AI using a different model which acts as project manager. And when finished the plan is for the two to work together? Slowly progressing

[@AlignmentLabAI]

im developing a model based on pandagpt and ehartfords samantha that both thinks its you and sentient and can see your monitor output, part of its training data involves teaching it how to use a mouse and keyboard to navigate your computer, ill be training it on landmark attention to be able to have an 'infinite' ly long context, as well as make it able to generate images using a variation of the method bark uses to generate sound so that it can use its output to store information from one inference to the next

[@GOUST3D]

I couldn't seem to get this to work I tried changing the model in the code from gpt-4 to 3 to text-davinci-002/3 didn't work either. Really dope though!!!! Could you make an interface for this?

[@masteringmui]

Any reason you’re not using langchain to implement this? I’ve started using it and really like its abstractions and mental model. I’m building an autonomous agent with it right now for work and the early results are super promising.

[@dyllanusher1379]

Check out guidance also by Microsoft

[@RyanGrissett]

I wrote a plugin for chatgpt to help with development. It can control terminal, read and write files, start servers, maintain github repos and lots more. It has full access to my laptop through the plugin.

[@paulsoh8129]

if you don't mind the possible security issue, could you share your codes. Also how did chatgpt4 do? the context length is still max 4k+, if you "add memory" how do you ensure the api give you the right codes instead of just cutting off midpoint code suggestions due to context length.

[@RyanGrissett]

@@paulsoh8129 I'll make my github repo public and share it here soon. If there is an issue with cutting off, I can just ask it to start where it left off and append to the file. I haven't ran into that issue yet though. The plugin api uses gpt-4 by default and I think they raised the context length to 8k tokens. I don't have to add memory, because chatgpt handles that internally. If it does lose context for some reason, I added descriptions in the plugin yaml file to tell it to reread the files in the project directory if it thinks it needs to. The descriptions play a huge role in how well the AI uses your routes you've given it.

[@matthieudegroot3581]

Nice!

[@ilozeet]

@@matthieudegroot3581 yes please

[@paulsoh8129]

@@RyanGrissett thx much

[@whizv1991]

im using copy and paste from gpt4 ui to write code too, the problem im facing is for long code it runs out of token, would be greate if termgpt can have pincecone etc, and alsoe import a full project file. great wor k~!

[@chriskingston1981]

Great idea, I tried your code but with model=gpt-3.5-turbo, but it gives text like I am an AI model and have no access to internet or files. How to get around this? thanks for your great work was looking for something like this to create myself but not have the skills

[@joshuabennett8249]

Gonna coble some of this into the voice assistant I made :)

[@Bleyo]

cool book

[@YEM_]

More coffee and pasta

[@spoonikle]

Pioneering

[@vertexxdev]

very cool

[@erenoz2910]

Excuse me, did you make a GPT model improve the system it is embedded in? I think this part is a bit understated here.

[@sentdex]

Yes, at my request for the type of improvement that I thought would be useful.

[@ricardocosta9336]

Incredible how much your code is similar to mine. But yours is superior. 😊

[@lorenzweb]

This is where shits gets dangerous. Was literally just waiting for this to happend.

[@live1xbet869]

I'm a big Fan

[@tbtitans21]

Maybe the subprocess.check_output needs to be decode('utf-8')? I think I remember having issues with that function too, but I forgot the solution (if I even found it)

[@cosmicray007]

Yes, you generally need to decode the output of `subprocess.check_output()` because it returns the output as bytes. If you want to work with the output as a string (which is often the case), you need to decode it from bytes to a string. UTF-8 is a common character encoding, so `decode('utf-8')` is frequently used. Here's an example of how you might use it: ```python import subprocess output = subprocess.check_output(['ls', '-l']) output = output.decode('utf-8') print(output) ``` This code runs the command `ls -l` (which lists files in the current directory in long format) and then decodes the output from bytes to a string. If you know that the command will always output text that can be decoded with UTF-8, this approach works fine. If the command might output other byte sequences (like binary data), you might need to handle that differently. As of Python 3.7, there's also a `text` parameter you can use with `subprocess.run()` that will handle the decoding for you: ```python import subprocess result = subprocess.run(['ls', '-l'], text=True, capture_output=True) print(result.stdout) ``` This code does the same thing as the previous example, but it uses `subprocess.run()` instead of `subprocess.check_output()`, and sets `text=True` to automatically decode the output from bytes to a string. The `capture_output=True` argument is necessary to capture the output of the command (it sets both `stdout` and `stderr` to `subprocess.PIPE`). This approach is generally recommended if you're using Python 3.7 or later, as `subprocess.run()` is more flexible and easier to use than `subprocess.check_output()`.

[@cosmicray007]

ChatGPT :)

[@peterkanini867]

Can you make a Jax tutorial please and probably make a comparison with TensorFlow and PyTorch.

[@cysutube]

Hey sentdex, really enjoyed the video. Is there a reason why you went ahead trying to implement yourself instead of making use of existing projects like AutoGPT? Thanks

[@arjunsahlot]

I feel like GPT-4 is so powerful that we can literally use it as an "all in one" api. For example, an API for simulating an entire OS, an API for translating languages, an API for code completion, etc.

[@yuriborroni5490]

Agreed. GPT-4's potential is immense and it has the ability to revolutionize the way we use APIs.

[@lorenzweb]

Agreed here. If not GPT 4, then some of the next models

[@davidw8668]

Expectations ≠ reality But hey we might get there

[@Smytjf11]

​@@davidw8668 Seems like it's largely an engineering challenge at the moment. I think I saw a thing about a trillion parameter model the other day. That'll be interesting

[@davidw8668]

@Jason Smyth I'm with you it's fascinating, and in any case, the models are useful. However, currently, I don't see how a larger model would resolve the reliability issues. So the engineer ing. approach to resolve this might well be sth else.

[@TheApeMachine]

Give it access to a headless browser, that way it can properly deal with javascript websites and control the thing through the developer console.

[@user-st3eg7ee5e]

I keep getting the error "GPT 4 does not exist". I tried debugging using gpt4 and it does give feedback to try using a different model but I tried changing to different models and I keep getting the same error that says the model doesn't exist. can you help?

[@tigoes]

Already tested the concept with my autonomous-dev project that allows GPT to use console, a set of fs actions as well as google search, web scraping, ask human etc... with the goal to build a software based on a given objective. I did let him takes actions without any intervention, i dont know why is everybody tripping out. Ive noticed that he offten chose to search on google for something that is certainly in his training data (like sorting algorithms) just because a human would most likely do so. It is not taking advantage of his large knowledge. GPT 3.5 was just not good enough to handle anything other than very simple project. It is good to provide example when provided with sufficient information. Not very good when its about taking decisions and tracking progress. One of the issues is most likely related to the fact the model is not fine tunes for this kind of use case.

[@JuanFlores-il4yv]

How about a demo?

[@Elzelgator]

Is there any update about open source GPT Models? As far as I see GPT4ALL-J and OPEN ASSISTANT are the best models. Am I right?

[@sentdex]

The issue with Open Assistant that I see is the best model is the llama based one, and IMO the license for llama precludes its use. At the moment, I am poking around the Falcon 40b and 7b models, they look very promising, but I haven't quite yet found that "perfect" open source model.

[@reinerheiner1148]

Its kinda difficult if you consider the amount of parameters gpt 3.5 turbo and gpt4 have, with gpt4 around 1000b. Even if the open source models are more efficient (the right amount of data / parameter ratio) I think the currently available open source models will have a hard time coming close to gpt 3.5+, in anything else than imitating it on the surface. The best we can hope for right now is probably open source llms that are good in specific use cases. What do you think?

[@Elzelgator]

@@sentdex did you test GPT4ALL-J it runs on CPU easy to train, they have 800k prompts dataset, probably we can add several prompts to it to train for our needs. I wish you reply to this!

[@jwadaow]

@@reinerheiner1148 The problem is that they get better overall by being well rounded so restricting them to training for specific use cases reduces their ability overall.

[@reinerheiner1148]

@@jwadaow and that brings back the problem of too little parameter size, which also means too little training data, and castration of the models to 4 bit mode all resulting in open source llms that best case are able to imitage gpt3.5 somewhat. Another thing is the size of the context window which exponentially increases compute, again something massively limited by consumer gpu compute ressources. I am hoping for enough innovation so that these factors can be overcome, as the field is moving so quickly. Worst case, we'll have to wait years for consumer hardware getting fast enough to power more demanding models... Or alternatively, we can rent cloud compute at a more reasonable price in the future to make it a sustainable solution

[@irfnrdh]

yupp

[@easyBob100]

I've never understood the line "If you understand say OK"...I've seen a couple of people make prompts with that at the end. Even if it says "OK", it doesn't understand. I've messed around with the Bing Chat thing and never bother to add that. I'll make it the dungeon master of a fantasy world and it will just start going. I've made it into a simple NPC that offered quests, and it was great. Never need to say "If you understand say OK" Just my .02 :)

[@sentdex]

Tends to stop divergence from the prompt. Many times you can get away without it, especially with powerful models like gpt4, but it tends to rarely hurt as well.

[@easyBob100]

@@sentdex True, it probably doesn't hurt. I wonder though, could you also add "Don't lie to me either." would that help? Just go nuts with it. :D

[@12chachachannel]

Would you trust this plot? Would you publish it?

[@suchalooser1175]

Cool

[@Mica_No]

bro continue the series of AI in pyhon pleassee

[@MichaelMerritt]

Why not autogpt or similar? Isn’t that what this does essentially but with a ton more people working on it and plugins. (My results have been pretty mixed with autogpt fwiw)

[@Smytjf11]

AutoGPT had some problems under the hood last time I looked at it. They're trying to do too much too quickly with their prompting. The descriptions of the tools needed work, too. I had toyed around with it for a bit, but not much deeper than that.

[@sb_sfo]

It'll be interesting to see what happens when ChatGPT gets its, um, hands on its own git repo and begins optimizing and improving its codebase... (No device drivers for any IoT devices, tho.)

[@Smytjf11]

I want it connected to my IoT devices. But I also want it hosted locally.

[@maythesciencebewithyou]

The most important thing for neural nets is the data they are trained on. If the data is trash, then bullshit in, bullshit out. That's why it's important to not teach them some bullshit. Which is also true for humans.

[@sb_sfo]

@@Smytjf11 If it can control hardware (at scale, of course), then it can pull a skynet on us. That's my concern.

[@Smytjf11]

@@sb_sfo Do you want me to invent even scarier scenarios for you to be afraid of?

[@sb_sfo]

@@Smytjf11 Sure, but kindly let me offer my own uncensored vision of our imminent dystopAI first and then you can have the last word? I'm guessing we'll converge pretty close to one another. :)

[@chessdad182]

Next it will want to control your checkbook. ;-)

[@Stinosko]

Hello 🎉🎉🎉

[@DarrenTarmey]

Which other projects you looking at

[@sentdex]

Mainly Falcon 40B. Any other suggestions?

[@JonasAmidu]

How do I modify the code to allow for chatgpt 3.5 architecture? Basically the free version? Editing just the name on the model doesn’t work

[@sentdex]

The proper name for GPT 3.5 via the API is: gpt-3.5-turbo That said, OpenAI doesn't offer GPT3.5 via API for free AFAIK.

[@JonasAmidu]

@@sentdex thanks for letting me know! Keep up the good work!

[@JonasAmidu]

@@sentdex but why would they allow me to get an API secret key… I think they do. I’ll run the code with the name you’ve listed. Thanks again.

[@sentdex]

@@JonasAmidu let me know if you can!

[@JonasAmidu]

@@sentdex so I have the same error which makes me think the API for gpt 4 works different to the one on gpt 3.5. I’ll need to amend your code to get it to work. This is such a cool project.

[@teleprint-me]

I tried buying your book, but it wouldn't accept my card. Maybe a bitpay option or something more universal like stripe would work. Also, on a side note, I've been working on a similar project and I've seen other projects like it. They're all unique in their own way and they're all interesting to learn from! I love your content! Keep it up!

[@sentdex]

Currently it's PayPal and whatever shopify uses as their processor. I'm not sure why your card would be declined other than for address mismatch or smth. Besides autogpt, which other similar ones like this have you seen? Would be curious to see how others are going about a concept like this

[@teleprint-me]

@sentdex I tried replying twice, but they "disappeared into the ether". My guess is they got captured by a filter.

[@sentdex]

@teleprint-me how annoying! I see this one at least. You could email me [e-mail]

[@MeltingPlastic100]

It would be interesting if somehow gpt could run the commands in a 2nd virtual enviroment and then check the results against the users description of the expected outcome. Or maybe thats kindof what its doing already now.

[@Aperekahaziq]

My concern is: we gave it too much powaa😂

[@icantfindausernamehe]

You could opensource your book -haha

[@sentdex]

It is all open source. github.com/sentdex/nnfs_book and github.com/sentdex/nnfsix

[@1J03B]

You can literally destroy the world like this fyi

[@lowkeysolo]

Hello people!

[@sentdex]

Hello!

[@princemars6746]

Don't you have to pay for gpt 4 tokens to use the api this way. I have a gpt plus subscribtion but i believe its web based only‽

[@SamyaDaleh]

Yes, the GPT Plus subscription only includes web access, not access to the API.

[@princemars6746]

Thanks for the info!

[@AuthorKScott]

No code commetns--are you AI broh?

[@NeoKailthas]

"safe" lol

[@simssim262]

well , how about you make your own language model? I mean you do have beefy hardware right

[@jeetshah8513]

Hi

[@sentdex]

Hi there

[@ivancheburan2209]

I let chatGPT write this comment. Spoiler: not really.

[@AS-bo8tb]

First!

[@dan-cj1rr]

tbh i find it cringe and peak stupidity, all the developers trying to automate their own field to become obsolete.