CRADLE (Part 2): An AI that can play Red Dead Dedemption 2. Reflection, Memory, Task-based Planning

[@Qzariuss]

Very excited to see this advancement happening, it's been so many years of custom automation in games. LLMs will bring a true revolution to explore so much more than what was possible before

[@johntanchongmin]

It is great that they can use memory and auto skill learning for this. However, the prompts are very game-dependent and lots of game-specific things are mentioned. While this may not be general, I do feel like even for AGI-like systems, custom prompts will need to be added. Right now we perform this custom prompt addition ourselves, but it could very likely in the near future that the prompt be automatically learned

[@Constructive-ty6pl]

For "use the hammer to open the door" I feel like old Sierra adventures are sensible enough to support that logic. But old LucasArts adventures are deliberately nonsensical sometimes, so no LLM logic could figure it out. Maybe you need to use a flowervase to pour water out on the floor, then use a can opener which causes the dog to come running and slip on the water, and they crash through the door. But, LucasArts games, you cannot die or lose or become "soft locked". So an agent can spend as much time being more and more creative until they find the solution, they will never break the game or run out of tries. Also the hint guides for LucasArts are structed I think where there are levels of hints, starting with text that is truly only a hint, like in-lore too. But the third level of hint is very explicit like [use] the [kerosene] on the [statue]. Compared to Sierra games, where your character can die, or miss an opportunity to pick up a required object, and now you can't progress or even go back. When to give up and load a previous save, or how far back to load, would be a difficult state to detect I think. Maybe you just need to increase the temperature of your LLM completion, to get a more creative result? So I wonder which is harder to train for. Sierra style, where the logic is straight and probably compatible with LLM knowledge, but the actual training itself has dead ends and punishment. Or, LucasArts, where it is a safe space, but the """logic""" is totally silly. I would love to see an agent trained on LucasArts, that outputs its reasoning for each step. "I will use the Parrot on the Intercom, because parrots mimic speech, so it will say the password". Even if that is not the solution to the puzzle, it would be entertaining to see how creative the LLM is being. Oh, also point and click adventures usually are waiting on the player, no worries about response time or Vision latency! And hey maybe a person can simply crank up the cycles in DOSBox so the game plays faster and training is accelerated, I bet there are a few games where the walk animations would be fast forwarded. If CRADLE is proof that this setup can work (extracting objectives text, using cosine similiary to limit the potential actions, etc) I bet playing old Point and Click adventures would reveal right away the limits of an LLMs reasoning and planning. Is there a LucasArts benchmark? lol

[@johntanchongmin]

Part 1 here: kzbin.info/www/bejne/g3WqpHqkq7yZgck