HomeAI News
Stanford's "Virtual Town" is open source: 25 AI agents are captured in "Western World", with nanny-level tutorials!

Stanford's "Virtual Town" is open source: 25 AI agents are captured in "Western World", with nanny-level tutorials!

Hayo News
Hayo News
August 10th, 2023
View OriginalTranslated by Google
Stanford's 25 AI intelligent body "towns" are finally open source, GitHub has 1.4k stars, and "Western World" is about to come into reality.

Get ready, Stanford Smart Town, which has caused a sensation in the entire AI community, is now officially open source!

Project address: https://github.com/joonspk-research/generative_agents

In this digital "Western World" sandbox virtual town, there are schools, hospitals, and families.

25 AI agents can not only work here, chat, socialize, make friends, and even fall in love, but each Agent has its own personality and background story.

However, they have no idea that they are living in a simulation.

Jim Fan, Senior Scientist at Nvidia commented -

Stanford Agent Town is one of the most exciting AI Agent experiments in 2023. We often talk about the emerging capabilities of a single large language model, but now with multiple AI agents, the situation is more complex and fascinating. A group of AI can deduce the evolution process of the entire civilization.

Now, the first thing to be affected may be the game field.

In short, there are infinite new possibilities ahead!

Netizen: Game manufacturers, do you understand what I mean?

Many people believe that this Stanford paper marks the beginning of AGI.

It is conceivable that various RPG and simulation games will use this technology.

Netizens were also very excited and their brains were wide open.

Some want to watch Pokémon, some want to watch murder detective stories, and some want to watch love variety shows...

"I can't wait to see the love triangle between AI agents."

"Animal Crossing's repetitive, dreary dialogue, and the one-dimensional personality system shared by all villagers, are so disappointing. Nintendo, learn it!"

"Can the Sims port this over?"

If you can see the AI ​​running on the NPC in a classic RPG game like "Divine Realm", the whole game experience will be subverted! "

Some people also imagine that this technology also has many application scenarios in the enterprise space, such as how employees interact with different work environments/process changes.

Of course, some people said, what are you excited about? In fact, we have been living in such a simulation, but our world has more computing power.

Yes, if we magnify this virtual world enough times, we can definitely see ourselves.

Karpathy: AI agents are the next frontier

Previously, Karpathy, former director of Tesla and OpenAI expert, said that AI agents are now the most cutting-edge direction in the future.

OpenAI's team has spent the last five years elsewhere, but Karpathy now believes that "Agents represent a kind of future for AI."

If a paper proposes a different way to train a large language model, someone in the Slack group inside OpenAI will say: "This method I tried two and a half years ago, it didn't work."

However, whenever an AI agent emerges from a paper, all colleagues are very interested.

Karpathy once called AutoGPT the next frontier of rapid engineering

25 AI agents in "Westworld"

In the American TV series "Western World", a robot with a preset storyline is put into a theme park, acts like a human being, and then resets its memory, and is put into its own core storyline on a new day.

And in April of this year, researchers from Stanford and Google actually built a virtual town in which 25 AI agents can survive and engage in complex behaviors. It can be said that "Western World" has come into reality.

Paper address: https://arxiv.org/pdf/2304.03442.pdf


In order to generate agents, the researchers propose a new architecture that extends the large language model and can store the agent's experience in natural language.

Over time, these memories are synthesized into higher-level reflections that the agent can dynamically retrieve to plan its actions.

In the end, users can use natural language to interact with all 25 Agents in the town.

As above, the architecture of the generative agent implements a "retrieval" function.

This function takes as input the agent's current situation and returns a subset of the memory stream to pass to the language model.

Retrieval, on the other hand, has many possible implementations, depending on the important factors the agent considers when deciding how to act.

A central challenge for generative agent architectures is how to manage the large number of events and memories that must be retained.

To solve this problem, the core of the architecture is the memory stream, which is a database that records the entire experience of the agent.

The agent can retrieve relevant memories from the memory stream, which help it plan actions, respond correctly, and each action is fed back into the memory stream to recursively improve future actions.

In addition, the study also introduced a second type of memory—reflection. Reflection is the high-level abstract thinking generated by an agent based on recent experiences.

In this study, reflection is a periodically triggered process, and the reflection mechanism will only be activated when the agent judges the importance scores of a series of recent events and the accumulation exceeds a set threshold.

Generative agents generate more details recursively from top to bottom in order to create plausible plans.

And these plans initially only roughly described what to do that day.

During the execution of the plan, the generative agent continuously perceives the surrounding environment and stores the perceived observations into the memory stream.

By using observations as cues, let the language model decide the agent's next move: continue with the current plan, or respond otherwise.

In the experimental evaluation, the researchers performed a controlled evaluation of this framework, as well as an end-to-end evaluation.

Control evaluation is to understand whether the agent can independently generate plausible individual behavior. The end-to-end evaluation is to understand the emergence ability and stability of the agent.

For example, Isabella plans a Valentine's Day party and invites everyone to come. Of the 12 agents, 7 are still under consideration (3 have other plans, and 4 have no ideas).

This link is very similar to the human interaction mode.

interact like a real person

In this sandbox world town called Smallville, areas are marked. The root node describes the whole world, child nodes describe areas (houses, cafes, shops), and leaf nodes describe objects (tables, bookshelves).

The agent remembers a subgraph that reflects the parts of the world they see.

The researchers programmed a natural language to describe the identity of each agent, including their occupation and relationship with other agents, as a seed memory.

For example, the seed memory of the agent John Lin is like this -

John Lin, a helpful pharmacy owner, is always looking for ways to make medicines more accessible to customers. John Lin's wife Mei Lin is a university professor and his son Eddy Lin is studying music theory, they live together, John Lin loves his family very much. John Lin has known the old couple next door, Sam Moore and Jennifer Moore, for several years, and John Lin thinks Sam Moore is a kind person. John Lin is very close to his neighbor Yuriko Yamamoto. John Lin knew his neighbors, Tamara Taylor and Carmen Ortiz, but had never met them. John Lin and Tom Moreno are pharmacy colleagues and friends who enjoy discussing local politics and more.

Here's a morning for John Lin: Woke up at 6, brushed his teeth, took a shower, had breakfast, and met his wife Mei and son Eddy before leaving for work.

In this way, when the simulation starts, each agent has its own seed memory.

These agents interact socially with each other. When they notice each other, a conversation may ensue.

Over time, these agents form new relationships and remember their interactions with other agents.

An interesting story is that at the beginning of the simulation, an agent was initialized to throw a Valentine's Day party.

There may be failure points in the series of things that happen afterwards. The agent may not continue to insist on this intention, or forget to tell others, or even forget to show up.

Fortunately, in the simulation, the Valentine's Day party actually happened, and many agents got together and had interesting interactions.

Nanny Level Tutorial

Configuration Environment

Before configuring the environment, you first need to generate a utils.py file containing the OpenAI API key and download the necessary packages.

Step 1. Generate Utils file

In the reverie/backend_server folder (the folder where reverie.py is located), create a new utils.py file, and copy and paste the following content into the file:

Copy and paste your OpenAI API Key
openai_api_key = "<Your OpenAI API>"
put your name
key_owner = "<Name>"
maze_assets_loc = "../../environment/frontend_server/static_dirs/assets" env_matrix = f"{maze_assets_loc}/the_ville/matrix" env_visuals = f"{maze_assets_loc}/the_ville/visuals"
fs_storage = "../../environment/frontend_server/storage" fs_temp_storage = "../../environment/frontend_server/temp_storage"
collision_block_id = "32125"
debug = True

Replace <Your OpenAI API> with your OpenAI API key and <name> with your name.

Step 2. Install requirements.txt

Install everything listed in the requirements.txt file (we strongly recommend setting up a virtual environment first).

Currently, the team has tested on Python 3.9.12.

run simulation

To run a new simulation, you need to start two servers at the same time: the environment server and the agent simulation server.

Step 1. Start the environment server

Since the environment is implemented as a Django project, a Django server needs to be started.

To do this, first navigate to environment/frontend_server (where manage.py resides) on the command line. Then run the following command:

python manage.py runserver

Then, visit http://localhost:8000/ in your favorite browser.

If you see the prompt "Your environment server is up and running", it means that the server is running normally. Make sure the environment server keeps running while running the simulation, so keep this command line tab open.

(Note: Chrome or Safari are recommended. Firefox may experience some front-end glitches, but it should not affect the actual simulation.)

Step 2. Start the mock server

Open another command line window (the environment server you used in step 1 is still running and needs to remain untouched). Navigate to reverie/backend_server and run reverie.py to start the mock server:

python reverie.py

At this point, a command line prompt will appear asking for the following: "Enter the name of the forked simulation: ".

For example, now we want to start a simulation with 3 agents, Isabella Rodriguez, Maria Lopez and Klaus Mueller, then enter the following:


Then, the prompt will ask: "Enter the name of the new simulation: ".

At this time, you only need to enter an arbitrary name to represent the current simulation (such as "test-simulation").


Keep the emulator server running. At this stage, it will display the following prompt: "Enter option"

Step 3. Run and save the simulation

Go to http://localhost:8000/simulator_home in your browser and keep the tab open.

You will now see a map of the town, and a list of agents active on the map, and can use the keyboard arrows to move around the map.

To run the simulation, you need to enter the following command in the simulation server prompting "Enter option":

run <step-count>

Please note that the above <step-count> needs to be replaced with an integer representing the number of game steps to be simulated.

For example, if you want to simulate a 100-step game, enter run 100. Among them, a game step represents 10 seconds in the game.

Now, the simulation will run and you can watch the agent move across the map in the browser.

Once the run is complete, the "Enter option" prompt will appear again. At this point, you can continue the simulation by re-entering the run command and specifying the desired number of game steps, or typing exit to exit without saving, or fin to save and exit.

The next time you run the simulation server, you can access the saved simulation simply by providing the name of the simulation. This way, you can restart the simulation from where you left off.

Step 4. Replay the simulation

Simply run the environment server and go to the following address in your browser to replay a running simulation: http://localhost:8000/replay/<simulation-name>/<starting-time-step>.

where <simulation-name> needs to be replaced with the name of the replayed simulation, and <starting-time-step> needs to be replaced with the integer time step at which replay starts.

Step 5. Demo simulation

You may find that all the character sprites look the same in the replay. This is because the replay feature is primarily for debugging and does not prioritize optimizing the size or visuals of the mock folder.

To properly render a simulation with character sprites, you first need to compress the simulation. To do this, use a text editor to open the compress_sim_storage.py file located in the reverie directory. Then, the compression function is executed with the name of the target mock as input. In this way, the simulation file will be compressed and thus ready for presentation.

To start the demo, please open the following address in your browser: http://localhost:8000/demo/<simulation-name>/<starting-time-step>/<simulation-speed>.

Note that <simulation-name> and <starting-time-step> have the same meaning as mentioned above. <simulation-speed> can be used to control the demo speed, where 1 is the slowest and 5 is the fastest.

custom simulation

You have two options for customizing mocks.

Method 1: Write and load agent history

The first is to initialize the agent with a unique history at the beginning of the simulation.

To do this, you need to 1) start with one of the base simulations, 2) write and load the agent history.

Step 1. Start the basic simulation

The repository contains two base simulations: base_the_ville_n25 (25 agents) and base_the_ville_isabella_maria_klaus (3 agents). One of the base simulations can be loaded by following the steps above.

Step 2. Load history file

Then, when prompted for "Enter option", you need to load the agent history with the following command:

call -- load history the_ville/<history_file_name>.csv

where <history_file_name> needs to be replaced with the name of an existing history file.

Two example history files are included in the repository: agent_history_init_n25.csv (for base_the_ville_n25) and agent_history_init_n3.csv (for base_the_ville_isabella_maria_klaus). These files contain a list of memory records for each agent.

Step 3. Further customization

To customize initialization by writing your own history files, place the files in the following folder: environment/frontend_server/static_dirs/assets/the_ville.

The column format of the custom history file must match the attached sample history file. Therefore, the author recommends starting the process by copying and pasting files already in the repository.

Method 2: Create a new base mock

For more in-depth customization, you'll need to write your own base simulation file.

The most straightforward way is to copy and paste the existing base emulation folder, then rename and edit it according to your requirements.




Reprinted from 新智元View Original


no dataCoffee time! Feel free to comment