Repository: a16z-infra/ai-town

Stars: 9729

README.md

AI Town 🏠💻💌

Join our community Discord: AI Stack Devs

AI Town is a virtual town where AI characters live, chat and socialize.

This project is a deployable starter kit for easily building and customizing your own version of AI
town. Inspired by the research paper
_Generative Agents: Interactive Simulacra of Human Behavior_.

The primary goal of this project, beyond just being a lot of fun to work on, is to provide a
platform with a strong foundation that is meant to be extended. The back-end natively supports
shared global state, transactions, and a simulation engine and should be suitable from everything
from a simple project to play around with to a scalable, multi-player game. A secondary goal is to
make a JS/TS framework available as most simulators in this space (including the original paper
above) are written in Python.

Overview

- 💻 Stack
- 🧠 Installation (cloud, local, Docker, self-host, Fly.io, ...)
- 💻️ Windows Pre-requisites
- 🤖 Configure your LLM of choice (Ollama, OpenAI, Together.ai, ...)
- 👤 Customize - YOUR OWN simulated world
- 👩‍💻 Deploying to production
- 🐛 Troubleshooting

Stack

- Game engine, database, and vector search: Convex
- Auth (Optional): Clerk
- Default chat model is llama3 and embeddings with mxbai-embed-large.
- Local inference: Ollama
- Configurable for other cloud LLMs: Together.ai or anything that speaks the
OpenAI API. PRs welcome to add more cloud provider support.
- Background Music Generation: Replicate using
MusicGen

Other credits:

- Pixel Art Generation: Replicate,
Fal.ai
- All interactions, background music and rendering on the <Game/> component in the project are
powered by PixiJS.
- Tilesheet:
- https://opengameart.org/content/16x16-game-assets by George Bailey
- https://opengameart.org/content/16x16-rpg-tileset by hilau
- We used https://github.com/pierpo/phaser3-simple-rpg for the original POC of this project. We have
since re-wrote the whole app, but appreciated the easy starting point
- Original assets by ansimuz
- The UI is based on original assets by
Mounir Tohami

Installation

The overall steps are:

1. Build and deploy
2. Connect it to an LLM

Build and Deploy

There are a few ways to run the app on top of Convex (the backend).

1. The standard Convex setup, where you develop locally or in the cloud. This requires a Convex
account(free). This is the easiest way to depoy it to the cloud and seriously develop.
2. If you want to try it out without an account and you're okay with Docker, the Docker Compose
setup is nice and self-contained.
3. There's a community fork of this project offering a one-click install on
Pinokio for
anyone interested in running but not modifying it 😎.
4. You can also deploy it to Fly.io. See ./fly for instructions.

Standard Setup

Note, if you're on Windows, see below.

git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install

This will require logging into your Convex account, if you haven't already.

To run it:

npm run dev

You can now visit http://localhost:5173.

If you'd rather run the frontend and backend separately (which syncs your backend functions as
they're saved), you can run these in two terminals:

bash

npm run dev:frontend
npm run dev:backend

See package.json for details.

Using Docker Compose with self-hosted Convex

You can also run the Convex backend with the self-hosted Docker container. Here we'll set it up to
run the frontend, backend, and dashboard all via docker compose.

docker compose up --build -d

The container will keep running in the background if you pass -d. After you've done it once, you
can stop and start services.

- The frontend will be running on http://localhost:5173.
- The backend will be running on http://localhost:3210 (3211 for the http api).
- The dashboard will be running on http://localhost:6791.

To log into the dashboard and deploy from the convex CLI, you will need to generate an admin key.

docker compose exec backend ./generate_admin_key.sh

Add it to your .env.local file. Note: If you run down and up, you'll have to generate the key
again and update the .env.local file.

in .env.local

CONVEX_SELF_HOSTED_ADMIN_KEY="<admin-key>" # Ensure there are quotes around it
CONVEX_SELF_HOSTED_URL="http://127.0.0.1:3210"

Then set up the Convex backend (one time):

npm run predev

To continuously deploy new code to the backend and print logs:

npm run dev:backend

To see the dashboard, visit http://localhost:6791 and provide the admin key you generated earlier.

Configuring Docker for Ollama

If you'll be using Ollama for local inference, you'll need to configure Docker to connect to it.

npx convex env set OLLAMA_HOST http://host.docker.internal:11434

To test the connection (after you have it running):

docker compose exec backend /bin/bash curl http://host.docker.internal:11434

If it says "Ollama is running", it's good! Otherwise, check out the
Troubleshooting section.

Connect an LLM

Note: If you want to run the backend in the cloud, you can either use a cloud-based LLM API, like
OpenAI or Together.ai or you can proxy the traffic from the cloud to your local Ollama. See
below for instructions.

Ollama (default)

By default, the app tries to use Ollama to run it entirely locally.

1. Download and install Ollama.
2. Open the app or run ollama serve in a terminal. ollama serve will warn you if the app is
already running.
3. Run ollama pull llama3 to have it download llama3.
4. Test it out with ollama run llama3.

Ollama model options can be found here.

If you want to customize which model to use, adjust convex/util/llm.ts or set
npx convex env set OLLAMA_MODEL # model. If you want to edit the embedding model:

1. Change the OLLAMA_EMBEDDING_DIMENSION in convex/util/llm.ts and ensure:
export const EMBEDDING_DIMENSION = OLLAMA_EMBEDDING_DIMENSION;
2. Set npx convex env set OLLAMA_EMBEDDING_MODEL # model.

Note: You might want to set NUM_MEMORIES_TO_SEARCH to 1 in constants.ts, to reduce the size of
conversation prompts, if you see slowness.

OpenAI

To use OpenAI, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = OPENAI_EMBEDDING_DIMENSION;

Set the OPENAI_API_KEY environment variable. Visit https://platform.openai.com/account/api-keys if
you don't have one.

npx convex env set OPENAI_API_KEY 'your-key'

Optional: choose models with OPENAI_CHAT_MODEL and OPENAI_EMBEDDING_MODEL.

Together.ai

To use Together.ai, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = TOGETHER_EMBEDDING_DIMENSION;

Set the TOGETHER_API_KEY environment variable. Visit https://api.together.xyz/settings/api-keys if
you don't have one.

npx convex env set TOGETHER_API_KEY 'your-key'

Optional: choose models via TOGETHER_CHAT_MODEL, TOGETHER_EMBEDDING_MODEL. The embedding model's
dimension must match EMBEDDING_DIMENSION.

Other OpenAI-compatible API

You can use any OpenAI-compatible API, such as Anthropic, Groq, or Azure.

- Change the EMBEDDING_DIMENSION in convex/util/llm.ts to match the dimension of your embedding
model.
- Edit getLLMConfig in llm.ts or set environment variables:

npx convex env set LLM_API_URL 'your-url'
npx convex env set LLM_API_KEY 'your-key'
npx convex env set LLM_MODEL 'your-chat-model'
npx convex env set LLM_EMBEDDING_MODEL 'your-embedding-model'

Note: if LLM_API_KEY is not required, don't set it.

Note on changing the LLM provider or embedding model:

If you change the LLM provider or embedding model, you should delete your data and start over. The
embeddings used for memory are based on the embedding model you choose, and the dimension of the
vector database must match the embedding model's dimension. See
below for how to do that.

Customize your own simulation

NOTE: every time you change character data, you should re-run npx convex run testing:wipeAllTables
and then npm run dev to re-upload everything to Convex. This is because character data is sent to
Convex on the initial load. However, beware that npx convex run testing:wipeAllTables WILL wipe
all of your data.

1. Create your own characters and stories: All characters and stories, as well as their spritesheet
references are stored in characters.ts. You can start by changing
character descriptions.

2. Updating spritesheets: in data/characters.ts, you will see this code:

export const characters = [
     {
       name: 'f1',
       textureUrl: '/assets/32x32folk.png',
       spritesheetData: f1SpritesheetData,
       speed: 0.1,
     },
     ...
   ];

You should find a sprite sheet for your character, and define sprite motion / assets in the
corresponding file (in the above example, f1SpritesheetData was defined in f1.ts)

3. Update the Background (Environment): The map gets loaded in convex/init.ts from
data/gentle.js. To update the map, follow these steps:

- Use Tiled to export tilemaps as a JSON file (2 layers named
bgtiles and objmap)
- Use the convertMap.js script to convert the JSON to a format that the engine can use.

console

node data/convertMap.js <mapDataPath> <assetPath> <tilesetpxw> <tilesetpxh>

- <mapDataPath>: Path to the Tiled JSON file.
- <assetPath>: Path to tileset images.
- <tilesetpxw>: Tileset width in pixels.
- <tilesetpxh>: Tileset height in pixels. Generates converted-map.js that you can use like
gentle.js

4. Adding background music with Replicate (Optional)

For Daily background music generation, create a Replicate account and
create a token in your Profile's API Token page.
npx convex env set REPLICATE_API_TOKEN # token

This only works if you can receive the webhook from Replicate. If it's running in the normal
Convex cloud, it will work by default. If you're self-hosting, you'll need to configure it to hit
your app's url on /http. If you're using Docker Compose, it will be http://localhost:3211,
but you'll need to proxy the traffic to your local machine.

Note: The simulation will pause after 5 minutes if the window is idle. Loading the page will
unpause it. You can also manually freeze & unfreeze the world with a button in the UI. If you
want to run the world without the browser, you can comment-out the "stop inactive worlds" cron in
convex/crons.ts.

- Change the background music by modifying the prompt in convex/music.ts
- Change how often to generate new music at convex/crons.ts by modifying the
generate new background music job

Commands to run / test / debug

To stop the back end, in case of too much activity

This will stop running the engine and agents. You can still run queries and run functions to debug.

bash

npx convex run testing:stop

To restart the back end after stopping it

bash

npx convex run testing:resume

To kick the engine in case the game engine or agents aren't running

bash

npx convex run testing:kick

To archive the world

If you'd like to reset the world and start from scratch, you can archive the current world:

bash

npx convex run testing:archive

Then, you can still look at the world's data in the dashboard, but the engine and agents will no
longer run.

You can then create a fresh world with init.

bash

npx convex run init

To pause your backend deployment

You can go to the dashboard to your deployment settings to pause and
un-pause your deployment. This will stop all functions, whether invoked from the client, scheduled,
or as a cron job. See this as a last resort, as there are gentler ways of stopping above.

Windows Installation

Prerequisites

1. Windows 10/11 with WSL2 installed
2. Internet connection

Steps:

1. Install WSL2

First, you need to install WSL2. Follow
this guide to set up WSL2 on your Windows
machine. We recommend using Ubuntu as your Linux distribution.

2. Update Packages

Open your WSL terminal (Ubuntu) and update your packages:

sudo apt update

3. Install NVM and Node.js

NVM (Node Version Manager) helps manage multiple versions of Node.js. Install NVM and Node.js 18
(the stable version):

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.2/install.sh | bash
   export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
   [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
   source ~/.bashrc
   nvm install 18
   nvm use 18

4. Install Python and Pip

Python is required for some dependencies. Install Python and Pip:

sudo apt-get install python3 python3-pip sudo ln -s /usr/bin/python3 /usr/bin/python

At this point, you can follow the instructions above.

Deploy the app to production

Deploy Convex functions to prod environment

Before you can run the app, you will need to make sure the Convex functions are deployed to its
production environment. Note: this is assuming you're using the default Convex cloud product.

1. Run npx convex deploy to deploy the convex functions to production
2. Run npx convex run init --prod

To transfer your local data to the cloud, you can run npx convex export and then import it with
npx convex import --prod.

If you have existing data you want to clear, you can run
npx convex run testing:wipeAllTables --prod

Adding Auth (Optional)

You can add clerk auth back in with git revert b44a436. Or just look at that diff for what changed
to remove it.

Make a Clerk account

- Go to https://dashboard.clerk.com/ and click on "Add Application"
- Name your application and select the sign-in providers you would like to offer users
- Create Application
- Add VITE_CLERK_PUBLISHABLE_KEY and CLERK_SECRET_KEY to .env.local

bash

VITE_CLERK_PUBLISHABLE_KEY=pk_*
CLERK_SECRET_KEY=sk_*

- Go to JWT Templates and create a new Convex Template.
- Copy the JWKS endpoint URL for use below.

npx convex env set CLERK_ISSUER_URL # e.g. https://your-issuer-url.clerk.accounts.dev/

Deploy the frontend to Vercel

- Register an account on Vercel and then install the Vercel CLI.
- If you are using Github Codespaces: You will need to
install the Vercel CLI and authenticate from your codespaces cli by
running vercel login.
- Deploy the app to Vercel with vercel --prod.

Using local inference from a cloud deployment

We support using Ollama for conversation generations. To have
it accessible from the web, you can use Tunnelmole or Ngrok or similar so the cloud backend can send
requests to Ollama running on your local machine.

Steps:

1. Set up either Tunnelmole or Ngrok.
2. Add Ollama endpoint to Convex

npx convex env set OLLAMA_HOST # your tunnelmole/ngrok unique url from the previous step

3. Update Ollama domains Ollama has a list of accepted domains. Add the ngrok domain so it won't
reject traffic. see ollama.ai for more details.

Using Tunnelmole

Tunnelmole is an open source tunneling tool.

You can install Tunnelmole using one of the following options:

- NPM: npm install -g tunnelmole
- Linux: curl -s https://tunnelmole.com/sh/install-linux.sh | sudo bash
- Mac:
curl -s https://tunnelmole.com/sh/install-mac.sh --output install-mac.sh && sudo bash install-mac.sh
- Windows: Install with NPM, or if you don't have NodeJS installed, download the exe file for
Windows here and put it somewhere in your PATH.

Once Tunnelmole is installed, run the following command:

text

tmole 11434

Tunnelmole should output a unique url once you run this command.

Using Ngrok

Ngrok is a popular closed source tunneling tool.

- Install Ngrok

Once ngrok is installed and authenticated, run the following command:

text

ngrok http http://localhost:11434

Ngrok should output a unique url once you run this command.

Troubleshooting

Wiping the database and starting over

You can wipe the database by running:

npx convex run testing:wipeAllTables

Then reset with:

npx convex run init

Incompatible Node.js versions

If you encounter a node version error on the convex server upon application startup, please use node
version 18, which is the most stable. One way to do this is by
installing nvm and running nvm install 18 and
nvm use 18.

Reaching Ollama

If you're having trouble with the backend communicating with Ollama, it depends on your setup how to
debug:

1. If you're running directly on Windows, see
Windows Ollama connection issues.
2. If you're using Docker, see
Docker to Ollama connection issues.
3. If you're running locally, you can try the following:

npx convex env set OLLAMA_HOST http://localhost:11434

By default, the host is set to http://127.0.0.1:11434. Some systems prefer localhost
¯\_(ツ)\_/¯.

Windows Ollama connection issues

If the above didn't work after following the windows and regular
installation instructions, you can try the following, assuming you're not using
Docker.

If you're using Docker, see the next section for Docker
troubleshooting.

For running directly on Windows, you can try the following:

1. Install unzip and socat:

sudo apt install unzip socat

2. Configure socat to Bridge Ports for Ollama

Run the following command to bridge ports:

socat TCP-LISTEN:11434,fork TCP:$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):11434 &

3. Test if it's working:

curl http://127.0.0.1:11434

If it responds OK, the Ollama API should be accessible.

Docker to Ollama connection issues

If you're having trouble with the backend communicating with Ollama, there's a couple things to
check:

1. Is Docker at least verion 18.03 ? That allows you to use the host.docker.internal hostname to
connect to the host from inside the container.

2. Is Ollama running? You can check this by running curl http://localhost:11434 from outside the
container.

3. Is Ollama accessible from inside the container? You can check this by running
docker compose exec backend curl http://host.docker.internal:11434.

If 1 & 2 work, but 3 does not, you can use socat to bridge the traffic from inside the container
to Ollama running on the host.

1. Configure socat with the host's IP address (not the Docker IP).

docker compose exec backend /bin/bash
   HOST_IP=YOUR-HOST-IP
   socat TCP-LISTEN:11434,fork TCP:$HOST_IP:11434

Keep this running.

2. Then from outside of the container:

npx convex env set OLLAMA_HOST http://localhost:11434

3. Test if it's working:

docker compose exec backend curl http://localhost:11434

If it responds OK, the Ollama API is accessible. Otherwise, try changing the previous two to
http://127.0.0.1:11434.

Launching an Interactive Docker Terminal

If you wan to investigate inside the container, you can launch an interactive Docker terminal, for
the frontend, backend or dashboard service:

bash

docker compose exec frontend /bin/bash

To exit the container, run exit.

Updating the browser list

bash

docker compose exec frontend npx update-browserslist-db@latest

🧑‍🏫 What is Convex?

Convex is a hosted backend platform with a built-in database that lets you
write your database schema and
server functions in
TypeScript. Server-side database
queries automatically
cache and
subscribe to data, powering a
realtime useQuery hook in our
React client. There are also clients for
Python, Rust,
ReactNative, and
Node, as well as a straightforward
HTTP API.

The database supports NoSQL-style documents
with opt-in schema validation,
relationships and
custom indexes (including on fields in nested objects).

The query and
mutation server functions have
transactional, low latency access to the database and leverage our
v8 runtime with
determinism guardrails
to provide the strongest ACID guarantees on the market: immediate consistency, serializable
isolation, and automatic conflict resolution via
optimistic multi-version concurrency control (OCC /
MVCC).

The action server functions have access to external
APIs and enable other side-effects and non-determinism in either our
optimized v8 runtime or a more
flexible node runtime.

Functions can run in the background via
scheduling and
cron jobs.

Development is cloud-first, with
hot reloads for server function editing via
the CLI,
preview deployments,
logging and exception reporting integrations,
There is a dashboard UI to
browse and edit data,
edit environment variables,
view logs,
run server functions, and more.

There are built-in features for reactive pagination,
file storage,
reactive text search,
vector search,
https endpoints (for webhooks),
snapshot import/export,
streaming import/export,
and runtime validation for
function arguments and
database data.

Everything scales automatically, and it’s free to start.

Repository: a16z-infra/ai-town

Stars: 9729

README.md Copy Chapter

AI Town 🏠💻💌

Overview Copy Chapter

Stack Copy Chapter

Installation

Build and Deploy Copy Chapter

Standard Setup Copy Section

Using Docker Compose with self-hosted Convex Copy Section

in .env.local

Configuring Docker for Ollama Copy Section

Connect an LLM Copy Chapter

Ollama (default) Copy Section

OpenAI Copy Section

Together.ai Copy Section

Other OpenAI-compatible API Copy Section

Note on changing the LLM provider or embedding model: Copy Section

Customize your own simulation Copy Chapter

Commands to run / test / debug Copy Chapter

Windows Installation Copy Chapter

Prerequisites Copy Section

Deploy the app to production Copy Chapter

Deploy Convex functions to prod environment Copy Section

Adding Auth (Optional) Copy Section

Deploy the frontend to Vercel Copy Section

Using local inference from a cloud deployment Copy Chapter

Using Tunnelmole Copy Section

Using Ngrok Copy Section

Troubleshooting Copy Chapter

Wiping the database and starting over Copy Section

Incompatible Node.js versions Copy Section

Reaching Ollama Copy Section

Windows Ollama connection issues Copy Section

Docker to Ollama connection issues Copy Section

Launching an Interactive Docker Terminal Copy Section

Updating the browser list Copy Section

🧑‍🏫 What is Convex?

Explore Related Technologies & Topics

README.md

Overview

Stack

Build and Deploy

Standard Setup

Using Docker Compose with self-hosted Convex

Configuring Docker for Ollama

Connect an LLM

Ollama (default)

OpenAI

Together.ai

Other OpenAI-compatible API

Note on changing the LLM provider or embedding model:

Customize your own simulation

Commands to run / test / debug

Windows Installation

Prerequisites

Deploy the app to production

Deploy Convex functions to prod environment

Adding Auth (Optional)

Deploy the frontend to Vercel

Using local inference from a cloud deployment

Using Tunnelmole

Using Ngrok

Troubleshooting

Wiping the database and starting over

Incompatible Node.js versions

Reaching Ollama

Windows Ollama connection issues

Docker to Ollama connection issues

Launching an Interactive Docker Terminal

Updating the browser list