We collected these multimodal
Posted: Sat Jan 04, 2025 3:57 am
So I also have very special emotions towards games. I love playing games myself. Stephanie Zhan What games do you play? Jim Fan I play Minecraft, I try, but I'm not a very good player. So I hope that my artificial intelligence can make up for my poor gaming skills. I've been involved in a few gaming projects in the past. The first is a project called Mind Dojo, where we're developing a platform for general-purpose agents in the game Minecraft. For the uninitiated, Minecraft is an open voxel world where you can do whatever you want. You can build all kinds of tools and adventures. We collect a lot of data from the internet, there are videos of people playing Minecraft, there are wiki pages that explain the mechanics of the game, which are multimodal documents, and there are Minecraft subforums like Reddit where a lot of people discuss the game in natural language.
datasets and trained models to play Minecraft. This colombia phone numbers project is called Mind Dojo. Later, there was another project called Voyager. The idea came after the release of GPT - because it was the best programming model at the time. So we thought,What if we did programming as an action? Based on this idea, we developed the Voyager agent, which interacts with the Minecraft world by writing code. We use an API to convert the Minecraft world into a text representation, and then let the agent write code using the mobile API. Just like human programmers, agents don’t always write code correctly the first time. So we gave it a self-reflective loop, so that if it encounters an error or does something wrong in the Minecraft world, it will get feedback and correct its programming. Once it writes a correct program, which we call a skill, it stores it in a skill library.
When the agent encounters a similar situation in the future, it doesn’t have to go through the trial and error process again and can directly call skills from the skill library. You can think of this skill library as a code base that Voyager itself interactively builds, with absolutely no human intervention. The entire codebase is developed by Voyager itself. This is the second mechanism, the skill library. The third mechanism is what we call automatic course generation. The agent knows what it knows and what it doesn't know. So it can figure out the next mission, which is neither too hard nor too easy, and then it can follow that path to discover skills, tools, and travel in the vast world of Minecraft. Because it travels a lot, we call it Voyager. So this was one of the earliest attempts by our team to build an embodied agent to interact with the virtual world.
datasets and trained models to play Minecraft. This colombia phone numbers project is called Mind Dojo. Later, there was another project called Voyager. The idea came after the release of GPT - because it was the best programming model at the time. So we thought,What if we did programming as an action? Based on this idea, we developed the Voyager agent, which interacts with the Minecraft world by writing code. We use an API to convert the Minecraft world into a text representation, and then let the agent write code using the mobile API. Just like human programmers, agents don’t always write code correctly the first time. So we gave it a self-reflective loop, so that if it encounters an error or does something wrong in the Minecraft world, it will get feedback and correct its programming. Once it writes a correct program, which we call a skill, it stores it in a skill library.
When the agent encounters a similar situation in the future, it doesn’t have to go through the trial and error process again and can directly call skills from the skill library. You can think of this skill library as a code base that Voyager itself interactively builds, with absolutely no human intervention. The entire codebase is developed by Voyager itself. This is the second mechanism, the skill library. The third mechanism is what we call automatic course generation. The agent knows what it knows and what it doesn't know. So it can figure out the next mission, which is neither too hard nor too easy, and then it can follow that path to discover skills, tools, and travel in the vast world of Minecraft. Because it travels a lot, we call it Voyager. So this was one of the earliest attempts by our team to build an embodied agent to interact with the virtual world.