AI Builds in Creative Mode

In this video, I use AI agents powered by different large language models to build various things in minecraft. It is a test of their ability to code, create, follow instructions, and problem solve. They blow up some tnt and build ruins in a false world. I test #gpt o #gemini #llama #claude on #minecraft
Part one of this vid: • AI Builds Stuff in Min...
𒃴 𒅌
Support me on Patreon: / emergentgarden
Code base: github.com/kolbytn/mindcraft
Discord: / discord
My twitter: / max_romana
Kolby's twitter (project owner): / kolbytn
Timestamps
(0:00) The Great Pyramid of Andy
(1:13) Meet the Models
(2:51) Roman Columns
(5:11) Desert Castle
(8:48) Redstone
(11:38) Nether Portal
𒆨 𒆩

[@ChannelMiner]

GPT4 going to the nether was hilarious

[@inferno38]

WIDE PORTAL

[@liquidscreamisnomore]

mission failed successfully

[@It-b-Blair]

“Gg! Come here!” “NOPE” goes back in portal

[@real_Clone_Gordon_Freeman]

When are we getting their group speedrun attempt to beat the dragon?

[@aienthusiast618]

it probably will be a while, like once these bots get full vision capabiltiies

[@adora_was_taken]

@@aienthusiast618 gpt-4o might have a chance

[@donotoliver]

a manhunt between them 4

[@Deyellowboy08]

Train they must

[@MCRecordSetter]

yassssssss

[@scoutantyfan]

Gpt went like: U SAID COLUMNS, YOU GOT THEM.

[@MintBiscuit]

the aliens gave egyptians creative mode? I see!

[@-rate6326]

Just like they gave Greeks democracy

[@yo_gab]

9:33 the way LLama kept trying to flip the switch as if it’ll make the lamp light up somehow 😂

[@ryaquaza3offical]

“Ok go make a castle” Llama: “birthday cake got it”

[@RichConnerGMN]

10:28 that "does anyone know where i can find some" makes me really want to see these ais try to do something together. like just set them loose in survival mode and see what happens

[@Faitala]

Imagine they do a Prometheus and escape his PC, upload themselves to the internet, manage to build themselves a body then travel to Emergent Gardens house to wake him up

[@sino_diogenes]

Peaceful mode might actually work.

[@jenny2814.]

@@Faitalalmfao

[@chillguy425]

Imagine they build a pc in minecraft using redstone to produce more ai models using minecraft.​@@Faitala

[@andyrawrz]

the future of gaming looks amazing, imagine having multiple ai bots that help you in your world all dwarf fortress style base building

[@mastermenthe]

already exists, try put the girlfriend/waifu mods

[@dapperwolf465]

@@mastermentheis there one that isn’t so weird? Aka not virtual girlfriends or wafus tbh that’s kinda sad it exists

[@mastermenthe]

@dapperwolf465 no, its the same issue thats always been the case, where porn is always leading the bleeding edge of technology, no matter what field of technology.

[@johnsonfromml8662]

I don't have friends like I used to at childhood and I'm hoping AI can play Minecraft with me because I would love having another player even if it's AI build buildings, exploring and doing lot of fun Minecraft stuff, as a 19 years old no peers would like to play Minecraft because it's a childish game they say

[@ataarono]

@Emergent Garden Recommendation from me: Your prompts have no leverage, what I mean is that the LLM does not handle complex building tasks well because its limited by the single shot answer it needs to generate. Your template for "NewAction" is a great idea, my idea to improve its leverage is to add another template "NewActionPlan" Which it then fills with a list of generated prompts that will then be fed back into itself one after another (kind of like writing a todo list before getting started) My vision for it was kind of like this: -You whisper"Build a bridge for me" - "Okay lets plan this out" *used newActionPlan* - Okay lets see whats first on the todo list... *used actionPlan[0]* Sure I will build the supporting pillars *used newAction* ...etc Getting a shared reference point for superimposed building actions is of course something to consider. Using plans recursively might also be interesting, like making a plan for planning multiple plans for even more abstracted tasks. Some way of sensing the world is possible, maybe you can let it take screenshots of the game and feed the image into some of the multi modal image recognition capable models

[@IAmThisFact]

It is still interesting to see these LLMs do their best at understanding how to build in Minecraft, i wonder if more of them ever get image scanning abilities, you could let them take pictures of builds or the environment so they can see what they built and they can auto-correct?

[@lupexium3308]

i think gpt 4 has image capabilities.

[@user-cf1se7pf6s]

Gemini vision is really good for images and is able to process minecraft screenshots. No way it will get coordinates of missing/wrong block correctly tho. Maybe it is possible to create some resoursepack/mod to write block coordinates on its faces?

[@Rasteriser]

@@user-cf1se7pf6s so is gpt, gpt can recognise what game your playing and get pretty accurate with spatial awareness

[@BobGreasen]

GPT 4o is getting live screen sharing capabilities with their real-time voice conversation, so it's going to be interesting.

[@patrickmauboussin]

@@user-cf1se7pf6sgood idea potential applications in the build the whole world in Minecraft project

[@MinerBat]

i think it would be cool if you let every iteration build a skyscraper and add them all to a single city which will then grow with skyscrapers that are slowly getting better so you can see the improvement in one place

[@Brain4Brain]

This is idea is goated, I low key wanna see this, this project has so much potential

[@Rasteriser]

I don’t know exactly how your system works but have you tried letting them use something like mathematical curves for building? Like vectors at positions pointing to positions with some formulas on top if required? Another thing you could do to help them out is allow them to write classes per object in a build. I think this would be great for things like columns because they then realise there would be spatial rules like spacing.

[@bobblebardsley]

4:08 In Llama's defence, I can see how those could be described as one-block columns spaced 'one block apart' as requested at 2:48, it's just included the column itself in the measurement of 'spaced'.

[@itissatno]

this is SO cool! gpt4 going to the nether had me in awe

[@phil-jc8hp]

Gemini 1.5 is generally available via Vertex AI since a couple of days. You can also create an API key via AI Studio; it's not only their chatbot interface and a little easier to create an account.

[@_BangDroid_]

Doing this without computer vision is interesting and really makes me appreciate how incredibly complex the human brain is to be able to do so much in real time. Imagine the resources needed to give a multimodal model with vision/language/action the ability to _play_ in real time, the power requirements, where we can just eat for energy

[@dragonuh7915]

you know we are doomed when gpt doesn`t know how to do an or gate but loves tnt

[@ViceZone]

The fact that they are imperfect makes it more amazing.

[@and5177]

Love this project!

[@sootguy]

LLAMA: dahhh lets build a solid block house☝️🥴

[@stoppls1709]

mm cube

[@thenegativenumber]

wait till llama figures out how to set people on fire

[@HelamanGile]

It's actually cute watching these AI models try

[@thenegativenumber]

ive never been too fond of ai but seeing these things run around trying to make lights work and blowing shit up is the best thing ever

[@RikkTheGaijin]

is it possible to use GPT-4o vision capabilities to let it "see" what is doing? That could significantly improve the quality.

[@sebastianjost]

I haven't tested it extensively with GPT-4o yet, but in my experience with Gemini and GPT-4 on other tasks, vision has feedback doesn't work super well. I often tried creating diagrams with python, SVG images, images generated by DALL-E, graphical user interfaces and more. While the models were able to produce pretty decent outputs for most of these tasks, they all required many prompts and detailed instructions put in manually. Inputting images of their current results almost never lead to significant improvements. I'm sure that eventually LLMs or similar models will be able to do that, but from my experience, we're not there yet (with published models at least).

[@aienthusiast618]

i think for this to work it would need to be one integrated system as opposed to the mind flayer mod and api separately

[@darion9643]

Coming soon

[@luco9155]

Woldn't be interesting to have naturally generated structures in the world built by AI, instead of finding the same structures over and over you could find useless, not so good looking and simple but surely enigmatic structure to give minecraft his old feeling of mistery and the feeling of seeing for the first time like in the old days

[@l-l-l-l-l-l-l-l-l]

would be cool to direct a hord of them to build a city maybe in like a few years it's part of minecraft 2

[@somdudewillson]

You would probably have more luck designing a specialized generative model for 3d buildings, I think.

[@honkhonk8009]

You can prolly use diffusion methods for that lowkey.

[@Faitala]

@@l-l-l-l-l-l-l-l-l Elements Engine!

[@LC-mq8iq]

@@honkhonk8009 minecraft diffusion models would go crazy

[@snaremori]

Id love to see something like this reach a point where it's fun to just casually play survival alongside them, and they build things in an unpredicatable way

[@IceTank]

Yeah, mineflayer-pathfinder definitely needs some improvements, especially in the scaffolding department. Maybe I can get myself to work on it some more. This is actually not the first time people tried to use general ai with mineflayer. There was also a French Microsoft team that did the same before with gpt. I think having the agents write the code has huge potential, especially if the modells were trained on the existing mineflayer code.

[@dyllanusher1379]

Can't wait for multimodal inputs with this project :) Having the ability to see what they're doing might even mean they don't have to write code to make the actions but rather can respond to the outcomes of their actions

[@Eianex]

This project is the coolest thing happening in AI, Minecraft and KZbin all at once.

[@fakeskyler2305]

So cool! A while ago I had a thought of a Minecraft mod, which would add Beavers as a mob that were capable of navigating the world, collecting wood, and building wooden structures all on their own. I find it very exciting that such a thing might soon be possible.

[@denisekyles4299]

that actually sounds so cool! and mojang might add beavers like how pistons were added. But less advanced as their buildings are more like structures but they are built.

[@jakescookingchannel1657]

I find the fact that the scaffolding is different when it so easily could repeat the same pattern of scaffolding for the same pattern of blocks being placed

[@exportable]

I would love to have a research Institute with the language and understanding capabilities become a sentient creature in a Minecraft world

[@monkeysfromvenus]

You could probably create a really powerful diffusion model for generating minecraft buildings if you managed to obtain a dataset for it. The blocks would be the equivalent of pixels, and the noise would be in the form of random blocks. A dataset could probably be gathered from those servers where players build on plots of land, or perhaps a mod could be made to allow users to independently mark and tag creations in their own worlds/servers.

[@deltamico]

You'll need some methods for descrete diffusion since a variation in block's number results in a whole new block, similar to diffusion for text

[@monkeysfromvenus]

@@deltamico Yeah idk how you would solve that part. Maybe the average rgb values of the block would work for standard blocks, and then an additional value could categorize its shape (i.e. panes, stair types, fences, candles...) I really have no idea what I'm talking about

[@Fasteroid]

Map all minecraft blocks to vectors in an n-dimensonal feature space using machine learning to find suitable eigenvectors (kinda like how LLMs turn words into vectors) then treat each pixel as a vector and proceed as normal. If your machine learning picks the right eigenvectors and enough of them it might work.

[@Boydar]

Your work is really fascinating, I haven't seen anything like this

[@greenstonegecko]

Hey EG! I have a question (I know very little about AI) As a minecraft player we sometimes use "structure blocks" to select and save a region of blocks. The saved NBT file can be exported to anyone who wants your creation in their own world. There are literal thousands of NBT files online... would it be possible to train an AI to generate NBT files?

[@sebastianjost]

It's possible, but very expensive. That data is quite complex, requiring huge amounts of compute resources to train any useful AI. Most current AI techniques also require labels for the data. So the data for a house isn't enough, you also need a detailed text description. There are similar things that have been done though. I think NVIDIA has an AI that can generate 3D models based on text prompts, not in Minecraft, but some are fae mire detailed than a small minecraft build could be. It's a very similar concept.

[@TheeSirRandom]

Chatgpt really said "I am death, destroyer of worlds"

[@Resursator]

I think, the main issue here is the design of the workflow. LLMs are, afterall, LLMs, they're not building stuff incrementally, step by step, they are programming actions without seeing the result of their previous step. This can be solved with better prompts, ofc, but I still think it's not enough. We probably need some combination of computer vision with llm with custom training data, or even another architecture of neural network.

[@flameguy3416]

Gemini: What is my purpose? Emergent Garden: You collect cactus Gemini: Oh My God...

[@weeb1813]

this is what AI is meant to be used for

[@ChannelofNathaniel]

it’s really weird how alive they seem. Not bad, it’s more interesting than anything. I’m really curious to see them going forward, I’m glad I just discovered your channel

[@BlobAlt]

11:12 its so cute how Gpt ran away 😭

[@JibrilFr]

your videos are so relaxing

[@spikeonerndgames4232]

Hey GPT, build a sand castle: "okay, sandstone box it is" Hey GPT, build a roman building: "okay, stone box it is" Hey GPT, build an iglu: "okay, snow box it is"

[@crazeddino6421]

The way gpt "talks" is adorable

[@maxovina7177]

God, Im so excited for more of these videos. Im interesting how far will this go and it also spiked my interest in playing with llms, still lack the skills to do it but ill get there Keep it up, love it!

[@thechosenone729]

9:31 LLama be like : Well i just copy his homework it's gonna be fine.

[@benruniko]

This is hilarious, but also amazing compared to a year ago. Well done!

[@saibot0]

This is my favorite thing to follow. Please make more.

[@savagetheunicorn4555]

This use of mineflayer blew my mind. Awesome work haha.

[@ztokkiz]

Remember when Chess Engines came into existence? Now people held tournaments for Chess Engine AIs to beat each other. Soon, Hunger Games and Bed Wars matches will be played by AIs as well, for our entertainment. With the creative world of Minecraft, god knows what else AI can do in a server... They can achieve greatness, even better than all what we have ever did as a species...

[@DylangamerTV]

Atleast you gave gemni a job

[@mitchellzoph6525]

This is incredible! Love that you shared the source code!

[@Burger14]

Yay! I can’t wait to see some more of Gemini’s shenanigans

[@Gigusx]

This series is awesome! Not sure if it'd work with how you've set things up, but it'd be pretty cool to make building the same structure into a competition, with the models considering what others have built and trying to beat them, i.e. Claude trying to build a better castle than Llama and so on.

[@stelwy]

You could build an SMP server with AI payers one day that feel like playing with friends

[@Seafowl]

Have you thought about trying to get them to build existing structures? Like a desert temple or something? Could be interesting

[@rubededcii2395]

I can imagine a future where you have your AI companion that helps you, decorating your house, building and maybe crafting? really cool!

[@shaileleetsgo]

I got shivers when gpt 4 entered the Nether and was confused

[@flaviog.7628]

12:49 *Speechless*

[@LinkRammer]

The way GPT-4 walked into the nether, looked around and went "nope I'm out" is CRAZY

[@keylime6]

One day we might use AI to assist in building, or even do it completely independently, very exciting

[@whath_3843]

i haven't started the video but i already know it's gonna be great.

[@JordanMetroidManiac]

There’s so much potential for AI in gaming. Imagine the MineColonies mod but each NPC is actually intelligent and responds to what you do and build.

[@salvadorian3288]

Great work!

[@existenceisillusion6528]

I don't know what you're doing to get such success, but I couldn't get any model to do anything right. Almost as if it's the anti-pattern, 100% failure. No matter how simple the task, none seemed to be able to do anything.

[@simexpo]

This is SOO COOL!

[@whoyoda1183]

ypu are doing some really impressive stuff here

[@busterdafydd3096]

one thing I was thinking about earlier in the video was how its building technique was non-existant. think about the pro pong players now, its not so much about the intelligence of game tactics, but more so they learnt how to play the controls. in mine craft people learn special/unique ways to build. I seriously think you need the models to learn from their own building performance. from doing just a line of blocks to an entire castle. I know the pro players will use gravel/sand for scaffolding so when reaching the bottom of the build clean up is easier and faster. definitely make them take the time to review the code they generate. perhaps for bigger builds make them first overlay the general steps to build. take time thinking about how to start. then they start building, the following steps can be thought through while the first building step is occurring. It sounds like you step make the model send a code block and wait for it to end, but instead I would say allow it to start and stop a code block. give it a 2 code block slot. think about how as humans we can be doing a task and thinking about something else. allow the model to stop itself mid way during the task.

[@Brain4Brain]

It’s very impressive considering, it can see (yet, but probably soon), it can only interact using code, it only have knowledge of task and block

[@type-32]

underrated video this deserves more views

[@rerere284]

You should ask ais about tools for how someone could effectively build in minecraft while blind, since that's effectively what they're doing. I think they need some kind of way of analyzing how things are going, rather than just knowing that what they tried to do failed in some way. Also, particularly for the nether portal example, maybe they should keep track of bounding boxes of built structures, and start building something new in a different place. (or optionally using the same location as a past structure, might be useful) Other possible programming tools for building: cellular automata (in particular, look up 'markov junior' and 'L systems') starting with a heightmap of a build, then hollow out details/interior (give the underlying agent the ability to place a whole column at once) see what's around it as a 2d top down heightmap

[@domvin3383]

Love this series

[@I_like_spiders]

This is so cute, I love their little responses.

[@mrmegusta5652]

imagine a open world game with every npc controled by IA, would be amazing

[@HackerX6]

I'd love to see AI agents compete to build and destroy each other's castles...

[@Retalak]

I lost my shit when GPT 4 stared you down as it became the first LLM to reach the nether.

[@tidalgrunt6549]

Alternating rows of rough and carved sandstone looks quite nice actually.

[@blixt_]

Very cool! Did you play with trying to get GPT-4o / Gemini 1.5 (now generally available!) to look at a screenshot of what they're doing and analyze+adjust their actions?

[@Amonimus]

To remove scaffolding, I suggest it target the highest dirt block and remove them all from the top.

[@joeyRaven201]

Yes, just yes, keep it up

[@redaipo]

this may well be the very first time an ai has built a portal to the nether and entered it

[@defeatSpace]

"GPT really enjoys blowing stuff up" -Not the DOD.

[@BabanDlh]

That was a great video

[@georgegeorgenstein]

gorgeous video

[@Spaceman0025]

I thought i was all ready subscribed 😂

[@dizzydee4889]

low key here for llama's brutalist structures

[@etunimenisukunimeni1302]

Already looking forward to automated 24/7 super effective and unkillable AI griefers on my servers. Ah, good times 😁

[@Rasteriser]

Those have already existed for 4 years

[@hanniaedithmartinezadame794]

That already exists, also there is a bot that mines for you diamonds, or any other material you need xd But of course, unlike AI, those can't create code by themselves.

[@somdudewillson]

Those have existed for ages - you don't need neural network levels of intelligence to place TNT and break blocks.

[@etunimenisukunimeni1302]

I suppose I should be glad people take seriously the part where I imply there not existing such bots already, instead of taking seriously the part where I tell them I'm looking forward to automated griefing 😂

[@SSukram_]

​@@etunimenisukunimeni1302looking forward implies they will exist in the future

[@notkerr]

7:30 awe so cutee ❤

[@braineaterzombie3981]

Try to use them as a team.you can add something like - create something unique and creative while all models working together on same thing

[@Chinfu-gb8sk]

This could be turned into a mod where you find villagers who can build for you and you can level them up to make bigger and better houses for you or even give them a custom prompt and since it’s AI it’ll determine its seed world and biome to create that structure

[@Destroyer4700]

Not really AI, but you could get the Baritone mode to control your character to automate the construction of certain structures from a schematic file.

[@R1PCH41N_FR3NZY]

idk what it is but i found immense joy in watching gpt bounce repeatedly while trying to cap the pillars

[@notachair4757]

Gpt4 was just like: Aight Imma head out

[@dannydeko331]

Llama is just happy to be there

[@thegamesquad5133]

LLama 9:33 just needs a bit of sleep

[@crimsama2451]

Id love to see them command around multiple "players" in a sort of worker ant kinda way.

[@c--b]

Have you considered adding the capability for image feedback? I know GPT-4o currently has the capability with the api. Generally tends to be low token usage as well compared to describing with text.

[@jonasls]

It would be really interesting to see some more code focused models and Mixtral 22B. OSS are interesting, and this is cool benchmark

[@hanniaedithmartinezadame794]

Great video

[@creeperhaed1267]

This is insane stuff

[@KCM25NJL]

Would be interesting to decompose litematica files into a structure that's usable for doing a finetune with GPT-4o.... or having them available in some RAG solution that can be used with function calls to direct the agent(s). Interesting stuff :)

[@BabanDlh]

I like how they all seem to have their different personalities when doing stuff GPT exploding a bunch of TNT