{"owner":"a16z-infra","repo":"ai-town","hasSkills":true,"totalSkillsCount":1,"totalTokensCount":4108,"categories":["project-spec"],"hasMcp":false,"mcpConfig":null,"found":["ARCHITECTURE.md"],"skills":{"ARCHITECTURE.md":"# Architecture\n\nThis documents dives into the high-level architecture of AI Town and its different layers. We'll\nfirst start with a brief overview and then go in-depth on each component. The overview should\nbe sufficient for forking AI Town and changing game or agent behavior. Read on to the deep dives\nif you're interested or running up against the engine's limitations.\n\nThis doc assumes the reader has a working knowledge of Convex. If you're new to Convex, check out\nthe [Convex tutorial](https://docs.convex.dev/get-started) to get started.\n\n## Overview\n\nAI Town is split into a few layers:\n\n- The server-side game logic in `convex/aiTown`: This layer defines what state AI Town maintains,\n  how it evolves over time, and how it reacts to user input. Both humans and agents submit inputs\n  that the game engine processes.\n- The client-side game UI in `src/`: AI Town uses `pixi-react` to render the game state to the\n  browser for human consumption.\n- The game engine in `convex/engine`: To make it easy to hack on the game rules, we've separated\n  out the game engine from the AI Town-specific game rules. The game engine is responsible for\n  saving and loading game state from the database, coordinating feeding inputs into the engine,\n  and actually running the game engine in Convex functions.\n- The agent in `convex/agent`: Agents run as part of the game loop, and can kick off asynchronous\n  Convex functions to do longer processing, such as talking to LLMs. Those functions can save state\n  in separate tables, or submit inputs to the game engine to modify game state. Internally, our\n  agents use a combination of simple rule-based systems and talking to an LLM.\n\nSo, if you'd like to tweak agent behavior but keep the same game mechanics, check out `convex/agent`\nfor the async work, and `convex/aiTown/agent.ts` for the game loop logic.\nIf you would like to add new gameplay elements (that both humans and agents can interact with), add\nthe feature to `convex/aiTown`, render it in the UI in `src/`, and respond to it in `convex/aiTown/agent.ts`.\n\nIf you have parts of your game that are more latency sensitive, you can move them out of engine\ninto regular Convex tables, queries, and mutations, only logging key bits into game state. See\n\"Message data model\" below for an example.\n\n## AI Town game logic (`convex/aiTown`)\n\n### Data model\n\nAI Town's data model has a few concepts:\n\n- Worlds (`convex/aiTown/world.ts`) represent a map with many players interacting together.\n- Players (`convex/aiTown/player.ts`) are the core characters in the game. Players have human readable names and\n  descriptions, and they may be associated with a human user. At any point in time, a player may be pathfinding\n  towards some destination and has a current location.\n- Conversations (`convex/aiTown/conversations.ts`) are created by a player and end at some point in time.\n- Conversation memberships (`convex/aiTown/conversationMembership.ts`) indicate that a player is a member\n  of a conversation. Players may only be in one conversation at any point in time, and conversations\n  currently have exactly two members. Memberships may be in one of three states:\n  - `invited`: The player has been invited to the conversation but hasn't accepted yet.\n  - `walkingOver`: The player has accepted the invite to the conversation but is too far away to talk. The\n    player will automatically join the conversation when they get close enough.\n  - `participating`: The player is actively participating in the conversation.\n\n### Schema\n\nThere are three main categories of tables:\n\n1. Engine tables (`convex/engine/schema.ts`) for maintaining engine-internal state.\n2. Game tables (`convex/aiTown/schema.ts`) for game state. To keep game state small and efficient to\n   read and write, we store AI Town's data model across a few tables. See `convex/aiTown/schema.ts` for an overview.\n3. Agent tables (`convex/agent/schema.ts`) for agent state. Agents can freely read and write to these tables\n   within their actions.\n\n### Inputs (`convex/aiTown/inputs.ts`)\n\nAI Town modifies its data model by processing inputs. Inputs are submitted by players and agents and\nprocessed by the game engine. We specify inputs in the `inputs` object in `convex/aiTown/inputs.ts`.\nUse the `inputHandler` function to construct an input handler, specifying a Convex validator for\narguments for end-to-end type-safety.\n\n- Joining (`join`) and leaving (`leave`) the game.\n- Moving a player to a particular location (`moveTo`): Movement in AI Town is similar to RTS games, where\n  the players specify where they want to go, and the engine figures out how to get there.\n- Starting a conversation (`startConversation`), accepting an invite (`acceptInvite`), rejecting an invite\n  (`rejectInvite`), and leaving a conversation (`leaveConversation`). To track typing indicators,\n  you use `startTyping` and `finishSendingMessage`. These are imported from `game/conversations.ts`.\n- Agent inputs are imported from `aiTown/agentInputs.ts` for things like remembering conversations,\n  deciding what to do, etc.\n\nEach of these inputs' implementation method checks invariants and updates game state as desired.\nFor example, the `moveTo` input checks that the player isn't participating in a conversation,\nthrowing an error telling them to leave the conversation first if so, and then updates their\npathfinding state with the desired destination.\n\n### Simulation\n\nOther than when processing player inputs, the game state can change over time in the background as the\nsimulation runs time forward. For example, if a player has decided to move along a path, their position\nwill gradually update as time moves forward. Similarly, if two players collide into each other, they'll\nnotice and replan their paths, trying to avoid obstacles.\n\n### Message data model\n\nWe manage the tables for tracking chat messages in separate tables not affiliated\nwith the game engine. This is for a few reasons:\n\n- The core simulation doesn't need to know about messages, so keeping them\n  out keeps game state small.\n- Messages are updated very frequently (when streamed out from OpenAI) and\n  benefit from lower input latency, so they're not a great fit for the engine.\n  See \"Design goals and limitations\" below.\n\nMessages (`convex/schema.ts`) are in a conversation and indicate an author and message text.\nEach conversation has a typing state in the conversations table that indicates that a player\nis currently typing. Players can still send messages while another player is typing, but\nhaving the indicator helps agents (and humans) not talk over each other.\n\nThe separate tables are queried and modified with regular Convex queries and mutations\nthat don't directly go through the simulation.\n\n## Game engine (`convex/engine`)\n\nGiven the description of AI Town's game behavior in the previous section,\nthe `AbstractGame` class in `convex/engine/abstractGame.ts` implements actually running the simulation.\nThe game engine has a few responsibilities:\n\n- Coordinating incoming player inputs, feeding them into the simulation, and sending their\n  return values (or errors) to the client.\n- Running the simulation forward in time.\n- Saving and loading game state from the database.\n- Managing executing the game behavior, efficiently using Convex resources and minimizing input latency.\n\nAI Town's game behavior is implemented in the `Game` subclass.\n\n### Input handling\n\nUsers submit inputs through the `insertInput` function, which inserts them into an `inputs` table, assigning a\nmonotonically increasing unique input number and stamping the input with the time the server received it. The\nengine then processes inputs, writing their results back to the `inputs` row. Interested clients can subscribe\non an input's status with the `inputStatus` query.\n\n`Game` provides an abstract method `handleInput` that `AiTown` implements with its specific behavior.\n\n### Running the simulation\n\nThe `Game` class specifies how it simulates time forward with the `tick` method:\n\n- `tick(now)` runs the simulation forward until the given timestamp\n- Ticks are run at a high frequency, configurable with `tickDuration` (milliseconds). Since AI town has smooth motion\n  for player movement, it runs at 60 ticks per second.\n- It's generally a good idea to break up game logic into separate systems that can be ticked forward independently.\n  For example, AI Town's `tick` method advances pathfinding with `Player.tickPathfinding`, player positions with\n  `Player.tickPosition`, conversations with `Conversation.tick`, and `Agent.tick` for agent logic.\n\nTo avoid running a Convex mutation 60 times per second (which would be expensive and slow), the engine batches up\nmany ticks into a _step_. AI town runs steps at only 1 time per second. Here's how a step works:\n\n1. Load the game state into memory.\n2. Decide how long to run.\n3. Execute many ticks for our time interval, alternating between feeding in inputs with `handleInput` and advancing\n   the simulation with `tick`.\n4. Write the updated game state back to the database.\n\nOne core invariant is that the game engine is fully \"single-threaded\" per world, so there are never two runs of\nan engine's step overlapping in time. Not having to think about race conditions or concurrency makes writing game\nengine code a lot easier.\n\nHowever, preserving this invariant is a little tricky. If the engine is idle for a minute and an\ninput comes in, we want to run the engine immediately but then cancel its run after the minute's\nup. If we're not careful, a race condition may cause us to run multiple copies of the engine if an\ninput comes in just as an idle timeout is expiring!\n\nOur approach is to store a generation number with the engine that monotonically increases over time.\nAll scheduled runs of the engine contain their expected generation number as an argument. Then, if\nwe'd like to cancel a future run of the engine, we can bump the generation number by one, and then\nwe're guaranteed that the subsequent run will fail immediately as it'll notice that the engine's\ngeneration number does not match its expected one.\n\n### Engine state management\n\nThe `World`, `Player`, `Conversation`, and `Agent` classes coordinate loading data into memory from the database,\nmodifying it according to the game rules, and serializing it to write back out to the database. Here's the flow:\n\n1. The Convex scheduler calls the `convex/aiTown/main.ts:runStep` action.\n2. The `runStep` action calls `convex/aiTown/game.ts:loadWorld` to load the current game state. This query calls\n   `Game.load`, which loads all of a world's game state from the appropriate tables, and returns a\n   `GameState` object, which contains serialized versions of all of the players, agents, etc.\n3. The `runStep` action passes the `GameState` to the `Game` constructor, which parses the serialized versions\n   of all our game objects using their constructors. For example, `new Player(serializedPlayer)` parses the\n   database representation into the in-memory `Player` class.\n4. The engine runs the simulation, modifying the in-memory game objects.\n5. At the end of a step, the framework calls `Game.saveStep`, which computes a diff of the game state since\n   the beginning of the step and passes the diff to the `convex/aiTown/game.ts:saveWorld` mutation.\n6. The `saveWorld` mutation applies the diff to the database, notices if any deleted objects need to be archived,\n   updates the `participatedTogether` graph, and kicks off any scheduled jobs to run.\n7. Since the engine is the only mutator of game state, it continues to run steps for some amount of time\n   without repeating steps 1 to 3 again.\n\nJust as we assume that the game engine is \"single threaded\", we also assume that the game engine _exclusively_\nowns the tables that store game engine state. Only the game engine should programmatically modify these tables,\nso components outside the engine can only mutate them by sending inputs.\n\n### Historical tables\n\nIf we're only writing updates out to the database at the end of the step, and steps are only running at once per\nsecond, continuous quantities like position will only update every second. This, then, defeats the whole purpose\nof having high-frequency ticks: Player positions will jump around and look choppy.\n\nTo solve this, we track the historical values of quantities like position _within_ a step, storing the value\nat the end of each tick. Then, the client receives both the current value _and_ the past step's worth of\nhistory, and it can \"replay\" the history to make the motion smooth.\n\nThe game tracks these quantities at the end of each tick by feeding them to a `HistoricalObject`. This object\nefficiently tracks its changes over time and serializes them into a buffer that clients can use for replaying\nits history. There are a few limitations on `HistoricalObject`:\n\n- Historical objects can only have numeric (floating point) values and can't have nested objects or optional fields.\n- Historical objects must declare which fields they'd like to track.\n\nWe store each player's \"location\" (i.e. its position, orientation, and speed) in a `HistoricalObject` and\nwrite it to the `worlds` document at the end of a step when computing a diff.\n\n## Client-side game UI (`src/`)\n\nOne guiding principle for AI Town's architecture is to keep the usage as close to \"regular Convex\" usage as possible. So,\ngame state is stored in regular tables, and the UI just uses regular `useQuery` hooks to load that state and render\nit in the UI.\n\nThe one exception is for historical tables, which feed in the latest state into a `useHistoricalValue` hook that parses\nthe history buffer and replays time forward for smooth motion. To keep replayed time synchronized across multiple\nhistorical buffers, we provide a `useHistoricalTime` hook for the top of your app that keeps track of the current\ntime and returns it for you to pass down into components.\n\nWe also provide a `useSendInput` hook that wraps `useMutation` and automatically sends inputs to the server and\nwaits for the engine to process them and return their outcome.\n\n## Agent architecture (`convex/agent`)\n\n### The agent loop (`convex/game/agents.ts`)\n\nAgents will execute any game state changes, and schedule operations to do anything that requires\na long-lived request or accessing non-game tables. The flow generally is:\n\n1. Logic in `Agent.tick` can read and modify game state as time progresses, such as waiting until\n   the agent is near another player to start talking.\n2. When there is something that needs to talk to an LLM or read/write external data,\n   it calls `startOperation` with a reference to a Convex function: generally an `internalAction`.\n3. This function can read state from game tables and other tables via `internalQuery` functions.\n4. It executes long-running tasks, and can write data via `internalMutation`s.\n   Game state should not be written, but rather submitted via `inputs` (described in a previous section).\n5. Inputs are submitted from actions with `ctx.runMutation(api.game.main.sendInput, {...})` from actions\n   or via `insertInput` from mutations. They are referenced by their name as a string, like `moveTo`.\n6. Inputs are defined with `inputHandler` and are given an instance of the AiTown game to modify,\n   similar to the game loop. In fact, these are called as part of the game loop before `tickAgent`.\n7. When an operation is done, it deletes the `inProgressOperation`. This is to ensure an agent only\n   is trying to do one thing at a time.\n8. `Agent.tick` then can observe the new game state and continue to make decisions.\n\n### Conversations (`convex/agent/conversations.ts`)\n\nThe agent code calls into the conversation layer which implements the prompt engineering for\ninjecting personality and memories into the GPT responses. It has functions for starting a\nconversation (`startConversation`), continuing after the first message (`continueConversation`), and\npolitely leaving a conversation (`leaveConversation`). Each function loads structured data from the\ndatabase, queries the memory layer for the agent's opinion about the player they're talking with,\nand then calls into the OpenAI client (`convex/util/openai.ts`).\n\n### Memories (`convex/agent/memory.ts`)\n\nAfter each conversation, GPT summarizes its message history, and we compute an embedding of the\nsummary text and write it into Convex's vector database. Then, when starting a new conversation\nwith, Danny, we embed \"What you think about Danny?\", find the three most similar memories, and fetch\ntheir summary texts to inject into the conversation prompt.\n\n### Embeddings cache (`convex/agent/embeddingsCache.ts`)\n\nTo avoid computing the same embedding over and over again, we cache embeddings by a hash of their\ntext in a Convex table.\n\n## Design goals and limitations\n\nAI Town's game engine has a few design goals:\n\n- Try to be as close to a regular Convex app as possible. Use regular client hooks (like `useQuery`)\n  when possible, and store game state in regular tables.\n- Be as similar to existing engines as possible, so it's easy to change the behavior. We chose a\n  `tick()` based model for simulation since it's commonly used elsewhere and intuitive.\n- Decouple agent behavior from the game engine. It's nice to allow human players and AI agents to do\n  all the same things in the game.\n\nThese design goals imply some inherent limitations:\n\n- All data is loaded into memory each step. The active game state loaded by the game should be small\n  enough to fit into memory and load and save frequently. Try to keep game state to less than a few dozen\n  kilobytes: Games that require tens of thousands of objects interacting together may not be a good\n  fit.\n- All inputs are fed through the database in the `inputs` table, so applications that require very\n  large or frequent inputs may not be a good fit.\n- Input latency will be around one RTT (time for the input to make it to the server and the response\n  to come back) plus half the step size (for expected server input delay when the input's waiting\n  for the next step). Historical values add another half step size of input latency since their\n  values are viewed slightly in the past. As configured, this will roughly be around 1.5s of input\n  latency, which won't be a good fit for competitive games. You can configure the step size to be\n  smaller (e.g. 250ms) which will decrease input latency at the cost of adding more Convex function\n  calls and database bandwidth.\n- The game engine is designed to be single threaded. JavaScript operating over plain objects\n  in-memory can be surprisingly fast, but if your simulation is very computationally expensive, it\n  may not be a good fit on AI Town's engine today.\n"},"files":{"ARCHITECTURE.md":"# Architecture\n\nThis documents dives into the high-level architecture of AI Town and its different layers. We'll\nfirst start with a brief overview and then go in-depth on each component. The overview should\nbe sufficient for forking AI Town and changing game or agent behavior. Read on to the deep dives\nif you're interested or running up against the engine's limitations.\n\nThis doc assumes the reader has a working knowledge of Convex. If you're new to Convex, check out\nthe [Convex tutorial](https://docs.convex.dev/get-started) to get started.\n\n## Overview\n\nAI Town is split into a few layers:\n\n- The server-side game logic in `convex/aiTown`: This layer defines what state AI Town maintains,\n  how it evolves over time, and how it reacts to user input. Both humans and agents submit inputs\n  that the game engine processes.\n- The client-side game UI in `src/`: AI Town uses `pixi-react` to render the game state to the\n  browser for human consumption.\n- The game engine in `convex/engine`: To make it easy to hack on the game rules, we've separated\n  out the game engine from the AI Town-specific game rules. The game engine is responsible for\n  saving and loading game state from the database, coordinating feeding inputs into the engine,\n  and actually running the game engine in Convex functions.\n- The agent in `convex/agent`: Agents run as part of the game loop, and can kick off asynchronous\n  Convex functions to do longer processing, such as talking to LLMs. Those functions can save state\n  in separate tables, or submit inputs to the game engine to modify game state. Internally, our\n  agents use a combination of simple rule-based systems and talking to an LLM.\n\nSo, if you'd like to tweak agent behavior but keep the same game mechanics, check out `convex/agent`\nfor the async work, and `convex/aiTown/agent.ts` for the game loop logic.\nIf you would like to add new gameplay elements (that both humans and agents can interact with), add\nthe feature to `convex/aiTown`, render it in the UI in `src/`, and respond to it in `convex/aiTown/agent.ts`.\n\nIf you have parts of your game that are more latency sensitive, you can move them out of engine\ninto regular Convex tables, queries, and mutations, only logging key bits into game state. See\n\"Message data model\" below for an example.\n\n## AI Town game logic (`convex/aiTown`)\n\n### Data model\n\nAI Town's data model has a few concepts:\n\n- Worlds (`convex/aiTown/world.ts`) represent a map with many players interacting together.\n- Players (`convex/aiTown/player.ts`) are the core characters in the game. Players have human readable names and\n  descriptions, and they may be associated with a human user. At any point in time, a player may be pathfinding\n  towards some destination and has a current location.\n- Conversations (`convex/aiTown/conversations.ts`) are created by a player and end at some point in time.\n- Conversation memberships (`convex/aiTown/conversationMembership.ts`) indicate that a player is a member\n  of a conversation. Players may only be in one conversation at any point in time, and conversations\n  currently have exactly two members. Memberships may be in one of three states:\n  - `invited`: The player has been invited to the conversation but hasn't accepted yet.\n  - `walkingOver`: The player has accepted the invite to the conversation but is too far away to talk. The\n    player will automatically join the conversation when they get close enough.\n  - `participating`: The player is actively participating in the conversation.\n\n### Schema\n\nThere are three main categories of tables:\n\n1. Engine tables (`convex/engine/schema.ts`) for maintaining engine-internal state.\n2. Game tables (`convex/aiTown/schema.ts`) for game state. To keep game state small and efficient to\n   read and write, we store AI Town's data model across a few tables. See `convex/aiTown/schema.ts` for an overview.\n3. Agent tables (`convex/agent/schema.ts`) for agent state. Agents can freely read and write to these tables\n   within their actions.\n\n### Inputs (`convex/aiTown/inputs.ts`)\n\nAI Town modifies its data model by processing inputs. Inputs are submitted by players and agents and\nprocessed by the game engine. We specify inputs in the `inputs` object in `convex/aiTown/inputs.ts`.\nUse the `inputHandler` function to construct an input handler, specifying a Convex validator for\narguments for end-to-end type-safety.\n\n- Joining (`join`) and leaving (`leave`) the game.\n- Moving a player to a particular location (`moveTo`): Movement in AI Town is similar to RTS games, where\n  the players specify where they want to go, and the engine figures out how to get there.\n- Starting a conversation (`startConversation`), accepting an invite (`acceptInvite`), rejecting an invite\n  (`rejectInvite`), and leaving a conversation (`leaveConversation`). To track typing indicators,\n  you use `startTyping` and `finishSendingMessage`. These are imported from `game/conversations.ts`.\n- Agent inputs are imported from `aiTown/agentInputs.ts` for things like remembering conversations,\n  deciding what to do, etc.\n\nEach of these inputs' implementation method checks invariants and updates game state as desired.\nFor example, the `moveTo` input checks that the player isn't participating in a conversation,\nthrowing an error telling them to leave the conversation first if so, and then updates their\npathfinding state with the desired destination.\n\n### Simulation\n\nOther than when processing player inputs, the game state can change over time in the background as the\nsimulation runs time forward. For example, if a player has decided to move along a path, their position\nwill gradually update as time moves forward. Similarly, if two players collide into each other, they'll\nnotice and replan their paths, trying to avoid obstacles.\n\n### Message data model\n\nWe manage the tables for tracking chat messages in separate tables not affiliated\nwith the game engine. This is for a few reasons:\n\n- The core simulation doesn't need to know about messages, so keeping them\n  out keeps game state small.\n- Messages are updated very frequently (when streamed out from OpenAI) and\n  benefit from lower input latency, so they're not a great fit for the engine.\n  See \"Design goals and limitations\" below.\n\nMessages (`convex/schema.ts`) are in a conversation and indicate an author and message text.\nEach conversation has a typing state in the conversations table that indicates that a player\nis currently typing. Players can still send messages while another player is typing, but\nhaving the indicator helps agents (and humans) not talk over each other.\n\nThe separate tables are queried and modified with regular Convex queries and mutations\nthat don't directly go through the simulation.\n\n## Game engine (`convex/engine`)\n\nGiven the description of AI Town's game behavior in the previous section,\nthe `AbstractGame` class in `convex/engine/abstractGame.ts` implements actually running the simulation.\nThe game engine has a few responsibilities:\n\n- Coordinating incoming player inputs, feeding them into the simulation, and sending their\n  return values (or errors) to the client.\n- Running the simulation forward in time.\n- Saving and loading game state from the database.\n- Managing executing the game behavior, efficiently using Convex resources and minimizing input latency.\n\nAI Town's game behavior is implemented in the `Game` subclass.\n\n### Input handling\n\nUsers submit inputs through the `insertInput` function, which inserts them into an `inputs` table, assigning a\nmonotonically increasing unique input number and stamping the input with the time the server received it. The\nengine then processes inputs, writing their results back to the `inputs` row. Interested clients can subscribe\non an input's status with the `inputStatus` query.\n\n`Game` provides an abstract method `handleInput` that `AiTown` implements with its specific behavior.\n\n### Running the simulation\n\nThe `Game` class specifies how it simulates time forward with the `tick` method:\n\n- `tick(now)` runs the simulation forward until the given timestamp\n- Ticks are run at a high frequency, configurable with `tickDuration` (milliseconds). Since AI town has smooth motion\n  for player movement, it runs at 60 ticks per second.\n- It's generally a good idea to break up game logic into separate systems that can be ticked forward independently.\n  For example, AI Town's `tick` method advances pathfinding with `Player.tickPathfinding`, player positions with\n  `Player.tickPosition`, conversations with `Conversation.tick`, and `Agent.tick` for agent logic.\n\nTo avoid running a Convex mutation 60 times per second (which would be expensive and slow), the engine batches up\nmany ticks into a _step_. AI town runs steps at only 1 time per second. Here's how a step works:\n\n1. Load the game state into memory.\n2. Decide how long to run.\n3. Execute many ticks for our time interval, alternating between feeding in inputs with `handleInput` and advancing\n   the simulation with `tick`.\n4. Write the updated game state back to the database.\n\nOne core invariant is that the game engine is fully \"single-threaded\" per world, so there are never two runs of\nan engine's step overlapping in time. Not having to think about race conditions or concurrency makes writing game\nengine code a lot easier.\n\nHowever, preserving this invariant is a little tricky. If the engine is idle for a minute and an\ninput comes in, we want to run the engine immediately but then cancel its run after the minute's\nup. If we're not careful, a race condition may cause us to run multiple copies of the engine if an\ninput comes in just as an idle timeout is expiring!\n\nOur approach is to store a generation number with the engine that monotonically increases over time.\nAll scheduled runs of the engine contain their expected generation number as an argument. Then, if\nwe'd like to cancel a future run of the engine, we can bump the generation number by one, and then\nwe're guaranteed that the subsequent run will fail immediately as it'll notice that the engine's\ngeneration number does not match its expected one.\n\n### Engine state management\n\nThe `World`, `Player`, `Conversation`, and `Agent` classes coordinate loading data into memory from the database,\nmodifying it according to the game rules, and serializing it to write back out to the database. Here's the flow:\n\n1. The Convex scheduler calls the `convex/aiTown/main.ts:runStep` action.\n2. The `runStep` action calls `convex/aiTown/game.ts:loadWorld` to load the current game state. This query calls\n   `Game.load`, which loads all of a world's game state from the appropriate tables, and returns a\n   `GameState` object, which contains serialized versions of all of the players, agents, etc.\n3. The `runStep` action passes the `GameState` to the `Game` constructor, which parses the serialized versions\n   of all our game objects using their constructors. For example, `new Player(serializedPlayer)` parses the\n   database representation into the in-memory `Player` class.\n4. The engine runs the simulation, modifying the in-memory game objects.\n5. At the end of a step, the framework calls `Game.saveStep`, which computes a diff of the game state since\n   the beginning of the step and passes the diff to the `convex/aiTown/game.ts:saveWorld` mutation.\n6. The `saveWorld` mutation applies the diff to the database, notices if any deleted objects need to be archived,\n   updates the `participatedTogether` graph, and kicks off any scheduled jobs to run.\n7. Since the engine is the only mutator of game state, it continues to run steps for some amount of time\n   without repeating steps 1 to 3 again.\n\nJust as we assume that the game engine is \"single threaded\", we also assume that the game engine _exclusively_\nowns the tables that store game engine state. Only the game engine should programmatically modify these tables,\nso components outside the engine can only mutate them by sending inputs.\n\n### Historical tables\n\nIf we're only writing updates out to the database at the end of the step, and steps are only running at once per\nsecond, continuous quantities like position will only update every second. This, then, defeats the whole purpose\nof having high-frequency ticks: Player positions will jump around and look choppy.\n\nTo solve this, we track the historical values of quantities like position _within_ a step, storing the value\nat the end of each tick. Then, the client receives both the current value _and_ the past step's worth of\nhistory, and it can \"replay\" the history to make the motion smooth.\n\nThe game tracks these quantities at the end of each tick by feeding them to a `HistoricalObject`. This object\nefficiently tracks its changes over time and serializes them into a buffer that clients can use for replaying\nits history. There are a few limitations on `HistoricalObject`:\n\n- Historical objects can only have numeric (floating point) values and can't have nested objects or optional fields.\n- Historical objects must declare which fields they'd like to track.\n\nWe store each player's \"location\" (i.e. its position, orientation, and speed) in a `HistoricalObject` and\nwrite it to the `worlds` document at the end of a step when computing a diff.\n\n## Client-side game UI (`src/`)\n\nOne guiding principle for AI Town's architecture is to keep the usage as close to \"regular Convex\" usage as possible. So,\ngame state is stored in regular tables, and the UI just uses regular `useQuery` hooks to load that state and render\nit in the UI.\n\nThe one exception is for historical tables, which feed in the latest state into a `useHistoricalValue` hook that parses\nthe history buffer and replays time forward for smooth motion. To keep replayed time synchronized across multiple\nhistorical buffers, we provide a `useHistoricalTime` hook for the top of your app that keeps track of the current\ntime and returns it for you to pass down into components.\n\nWe also provide a `useSendInput` hook that wraps `useMutation` and automatically sends inputs to the server and\nwaits for the engine to process them and return their outcome.\n\n## Agent architecture (`convex/agent`)\n\n### The agent loop (`convex/game/agents.ts`)\n\nAgents will execute any game state changes, and schedule operations to do anything that requires\na long-lived request or accessing non-game tables. The flow generally is:\n\n1. Logic in `Agent.tick` can read and modify game state as time progresses, such as waiting until\n   the agent is near another player to start talking.\n2. When there is something that needs to talk to an LLM or read/write external data,\n   it calls `startOperation` with a reference to a Convex function: generally an `internalAction`.\n3. This function can read state from game tables and other tables via `internalQuery` functions.\n4. It executes long-running tasks, and can write data via `internalMutation`s.\n   Game state should not be written, but rather submitted via `inputs` (described in a previous section).\n5. Inputs are submitted from actions with `ctx.runMutation(api.game.main.sendInput, {...})` from actions\n   or via `insertInput` from mutations. They are referenced by their name as a string, like `moveTo`.\n6. Inputs are defined with `inputHandler` and are given an instance of the AiTown game to modify,\n   similar to the game loop. In fact, these are called as part of the game loop before `tickAgent`.\n7. When an operation is done, it deletes the `inProgressOperation`. This is to ensure an agent only\n   is trying to do one thing at a time.\n8. `Agent.tick` then can observe the new game state and continue to make decisions.\n\n### Conversations (`convex/agent/conversations.ts`)\n\nThe agent code calls into the conversation layer which implements the prompt engineering for\ninjecting personality and memories into the GPT responses. It has functions for starting a\nconversation (`startConversation`), continuing after the first message (`continueConversation`), and\npolitely leaving a conversation (`leaveConversation`). Each function loads structured data from the\ndatabase, queries the memory layer for the agent's opinion about the player they're talking with,\nand then calls into the OpenAI client (`convex/util/openai.ts`).\n\n### Memories (`convex/agent/memory.ts`)\n\nAfter each conversation, GPT summarizes its message history, and we compute an embedding of the\nsummary text and write it into Convex's vector database. Then, when starting a new conversation\nwith, Danny, we embed \"What you think about Danny?\", find the three most similar memories, and fetch\ntheir summary texts to inject into the conversation prompt.\n\n### Embeddings cache (`convex/agent/embeddingsCache.ts`)\n\nTo avoid computing the same embedding over and over again, we cache embeddings by a hash of their\ntext in a Convex table.\n\n## Design goals and limitations\n\nAI Town's game engine has a few design goals:\n\n- Try to be as close to a regular Convex app as possible. Use regular client hooks (like `useQuery`)\n  when possible, and store game state in regular tables.\n- Be as similar to existing engines as possible, so it's easy to change the behavior. We chose a\n  `tick()` based model for simulation since it's commonly used elsewhere and intuitive.\n- Decouple agent behavior from the game engine. It's nice to allow human players and AI agents to do\n  all the same things in the game.\n\nThese design goals imply some inherent limitations:\n\n- All data is loaded into memory each step. The active game state loaded by the game should be small\n  enough to fit into memory and load and save frequently. Try to keep game state to less than a few dozen\n  kilobytes: Games that require tens of thousands of objects interacting together may not be a good\n  fit.\n- All inputs are fed through the database in the `inputs` table, so applications that require very\n  large or frequent inputs may not be a good fit.\n- Input latency will be around one RTT (time for the input to make it to the server and the response\n  to come back) plus half the step size (for expected server input delay when the input's waiting\n  for the next step). Historical values add another half step size of input latency since their\n  values are viewed slightly in the past. As configured, this will roughly be around 1.5s of input\n  latency, which won't be a good fit for competitive games. You can configure the step size to be\n  smaller (e.g. 250ms) which will decrease input latency at the cost of adding more Convex function\n  calls and database bandwidth.\n- The game engine is designed to be single threaded. JavaScript operating over plain objects\n  in-memory can be surprisingly fast, but if your simulation is very computationally expensive, it\n  may not be a good fit on AI Town's engine today.\n"},"items":[{"name":"ARCHITECTURE.md","path":"ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/a16z-infra/ai-town/HEAD/ARCHITECTURE.md","title":"System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# Architecture\n\nThis documents dives into the high-level architecture of AI Town and its different layers. We'll\nfirst start with a brief overview and then go in-depth on each component. The overview should\nbe sufficient for forking AI Town and changing game or agent behavior. Read on to the deep dives\nif you're interested or running up against the engine's limitations.\n\nThis doc assumes the reader has a working knowledge of Convex. If you're new to Convex, check out\nthe [Convex tutorial](https://docs.convex.dev/get-started) to get started.\n\n## Overview\n\nAI Town is split into a few layers:\n\n- The server-side game logic in `convex/aiTown`: This layer defines what state AI Town maintains,\n  how it evolves over time, and how it reacts to user input. Both humans and agents submit inputs\n  that the game engine processes.\n- The client-side game UI in `src/`: AI Town uses `pixi-react` to render the game state to the\n  browser for human consumption.\n- The game engine in `convex/engine`: To make it easy to hack on the game rules, we've separated\n  out the game engine from the AI Town-specific game rules. The game engine is responsible for\n  saving and loading game state from the database, coordinating feeding inputs into the engine,\n  and actually running the game engine in Convex functions.\n- The agent in `convex/agent`: Agents run as part of the game loop, and can kick off asynchronous\n  Convex functions to do longer processing, such as talking to LLMs. Those functions can save state\n  in separate tables, or submit inputs to the game engine to modify game state. Internally, our\n  agents use a combination of simple rule-based systems and talking to an LLM.\n\nSo, if you'd like to tweak agent behavior but keep the same game mechanics, check out `convex/agent`\nfor the async work, and `convex/aiTown/agent.ts` for the game loop logic.\nIf you would like to add new gameplay elements (that both humans and agents can interact with), add\nthe feature to `convex/aiTown`, render it in the UI in `src/`, and respond to it in `convex/aiTown/agent.ts`.\n\nIf you have parts of your game that are more latency sensitive, you can move them out of engine\ninto regular Convex tables, queries, and mutations, only logging key bits into game state. See\n\"Message data model\" below for an example.\n\n## AI Town game logic (`convex/aiTown`)\n\n### Data model\n\nAI Town's data model has a few concepts:\n\n- Worlds (`convex/aiTown/world.ts`) represent a map with many players interacting together.\n- Players (`convex/aiTown/player.ts`) are the core characters in the game. Players have human readable names and\n  descriptions, and they may be associated with a human user. At any point in time, a player may be pathfinding\n  towards some destination and has a current location.\n- Conversations (`convex/aiTown/conversations.ts`) are created by a player and end at some point in time.\n- Conversation memberships (`convex/aiTown/conversationMembership.ts`) indicate that a player is a member\n  of a conversation. Players may only be in one conversation at any point in time, and conversations\n  currently have exactly two members. Memberships may be in one of three states:\n  - `invited`: The player has been invited to the conversation but hasn't accepted yet.\n  - `walkingOver`: The player has accepted the invite to the conversation but is too far away to talk. The\n    player will automatically join the conversation when they get close enough.\n  - `participating`: The player is actively participating in the conversation.\n\n### Schema\n\nThere are three main categories of tables:\n\n1. Engine tables (`convex/engine/schema.ts`) for maintaining engine-internal state.\n2. Game tables (`convex/aiTown/schema.ts`) for game state. To keep game state small and efficient to\n   read and write, we store AI Town's data model across a few tables. See `convex/aiTown/schema.ts` for an overview.\n3. Agent tables (`convex/agent/schema.ts`) for agent state. Agents can freely read and write to these tables\n   within their actions.\n\n### Inputs (`convex/aiTown/inputs.ts`)\n\nAI Town modifies its data model by processing inputs. Inputs are submitted by players and agents and\nprocessed by the game engine. We specify inputs in the `inputs` object in `convex/aiTown/inputs.ts`.\nUse the `inputHandler` function to construct an input handler, specifying a Convex validator for\narguments for end-to-end type-safety.\n\n- Joining (`join`) and leaving (`leave`) the game.\n- Moving a player to a particular location (`moveTo`): Movement in AI Town is similar to RTS games, where\n  the players specify where they want to go, and the engine figures out how to get there.\n- Starting a conversation (`startConversation`), accepting an invite (`acceptInvite`), rejecting an invite\n  (`rejectInvite`), and leaving a conversation (`leaveConversation`). To track typing indicators,\n  you use `startTyping` and `finishSendingMessage`. These are imported from `game/conversations.ts`.\n- Agent inputs are imported from `aiTown/agentInputs.ts` for things like remembering conversations,\n  deciding what to do, etc.\n\nEach of these inputs' implementation method checks invariants and updates game state as desired.\nFor example, the `moveTo` input checks that the player isn't participating in a conversation,\nthrowing an error telling them to leave the conversation first if so, and then updates their\npathfinding state with the desired destination.\n\n### Simulation\n\nOther than when processing player inputs, the game state can change over time in the background as the\nsimulation runs time forward. For example, if a player has decided to move along a path, their position\nwill gradually update as time moves forward. Similarly, if two players collide into each other, they'll\nnotice and replan their paths, trying to avoid obstacles.\n\n### Message data model\n\nWe manage the tables for tracking chat messages in separate tables not affiliated\nwith the game engine. This is for a few reasons:\n\n- The core simulation doesn't need to know about messages, so keeping them\n  out keeps game state small.\n- Messages are updated very frequently (when streamed out from OpenAI) and\n  benefit from lower input latency, so they're not a great fit for the engine.\n  See \"Design goals and limitations\" below.\n\nMessages (`convex/schema.ts`) are in a conversation and indicate an author and message text.\nEach conversation has a typing state in the conversations table that indicates that a player\nis currently typing. Players can still send messages while another player is typing, but\nhaving the indicator helps agents (and humans) not talk over each other.\n\nThe separate tables are queried and modified with regular Convex queries and mutations\nthat don't directly go through the simulation.\n\n## Game engine (`convex/engine`)\n\nGiven the description of AI Town's game behavior in the previous section,\nthe `AbstractGame` class in `convex/engine/abstractGame.ts` implements actually running the simulation.\nThe game engine has a few responsibilities:\n\n- Coordinating incoming player inputs, feeding them into the simulation, and sending their\n  return values (or errors) to the client.\n- Running the simulation forward in time.\n- Saving and loading game state from the database.\n- Managing executing the game behavior, efficiently using Convex resources and minimizing input latency.\n\nAI Town's game behavior is implemented in the `Game` subclass.\n\n### Input handling\n\nUsers submit inputs through the `insertInput` function, which inserts them into an `inputs` table, assigning a\nmonotonically increasing unique input number and stamping the input with the time the server received it. The\nengine then processes inputs, writing their results back to the `inputs` row. Interested clients can subscribe\non an input's status with the `inputStatus` query.\n\n`Game` provides an abstract method `handleInput` that `AiTown` implements with its specific behavior.\n\n### Running the simulation\n\nThe `Game` class specifies how it simulates time forward with the `tick` method:\n\n- `tick(now)` runs the simulation forward until the given timestamp\n- Ticks are run at a high frequency, configurable with `tickDuration` (milliseconds). Since AI town has smooth motion\n  for player movement, it runs at 60 ticks per second.\n- It's generally a good idea to break up game logic into separate systems that can be ticked forward independently.\n  For example, AI Town's `tick` method advances pathfinding with `Player.tickPathfinding`, player positions with\n  `Player.tickPosition`, conversations with `Conversation.tick`, and `Agent.tick` for agent logic.\n\nTo avoid running a Convex mutation 60 times per second (which would be expensive and slow), the engine batches up\nmany ticks into a _step_. AI town runs steps at only 1 time per second. Here's how a step works:\n\n1. Load the game state into memory.\n2. Decide how long to run.\n3. Execute many ticks for our time interval, alternating between feeding in inputs with `handleInput` and advancing\n   the simulation with `tick`.\n4. Write the updated game state back to the database.\n\nOne core invariant is that the game engine is fully \"single-threaded\" per world, so there are never two runs of\nan engine's step overlapping in time. Not having to think about race conditions or concurrency makes writing game\nengine code a lot easier.\n\nHowever, preserving this invariant is a little tricky. If the engine is idle for a minute and an\ninput comes in, we want to run the engine immediately but then cancel its run after the minute's\nup. If we're not careful, a race condition may cause us to run multiple copies of the engine if an\ninput comes in just as an idle timeout is expiring!\n\nOur approach is to store a generation number with the engine that monotonically increases over time.\nAll scheduled runs of the engine contain their expected generation number as an argument. Then, if\nwe'd like to cancel a future run of the engine, we can bump the generation number by one, and then\nwe're guaranteed that the subsequent run will fail immediately as it'll notice that the engine's\ngeneration number does not match its expected one.\n\n### Engine state management\n\nThe `World`, `Player`, `Conversation`, and `Agent` classes coordinate loading data into memory from the database,\nmodifying it according to the game rules, and serializing it to write back out to the database. Here's the flow:\n\n1. The Convex scheduler calls the `convex/aiTown/main.ts:runStep` action.\n2. The `runStep` action calls `convex/aiTown/game.ts:loadWorld` to load the current game state. This query calls\n   `Game.load`, which loads all of a world's game state from the appropriate tables, and returns a\n   `GameState` object, which contains serialized versions of all of the players, agents, etc.\n3. The `runStep` action passes the `GameState` to the `Game` constructor, which parses the serialized versions\n   of all our game objects using their constructors. For example, `new Player(serializedPlayer)` parses the\n   database representation into the in-memory `Player` class.\n4. The engine runs the simulation, modifying the in-memory game objects.\n5. At the end of a step, the framework calls `Game.saveStep`, which computes a diff of the game state since\n   the beginning of the step and passes the diff to the `convex/aiTown/game.ts:saveWorld` mutation.\n6. The `saveWorld` mutation applies the diff to the database, notices if any deleted objects need to be archived,\n   updates the `participatedTogether` graph, and kicks off any scheduled jobs to run.\n7. Since the engine is the only mutator of game state, it continues to run steps for some amount of time\n   without repeating steps 1 to 3 again.\n\nJust as we assume that the game engine is \"single threaded\", we also assume that the game engine _exclusively_\nowns the tables that store game engine state. Only the game engine should programmatically modify these tables,\nso components outside the engine can only mutate them by sending inputs.\n\n### Historical tables\n\nIf we're only writing updates out to the database at the end of the step, and steps are only running at once per\nsecond, continuous quantities like position will only update every second. This, then, defeats the whole purpose\nof having high-frequency ticks: Player positions will jump around and look choppy.\n\nTo solve this, we track the historical values of quantities like position _within_ a step, storing the value\nat the end of each tick. Then, the client receives both the current value _and_ the past step's worth of\nhistory, and it can \"replay\" the history to make the motion smooth.\n\nThe game tracks these quantities at the end of each tick by feeding them to a `HistoricalObject`. This object\nefficiently tracks its changes over time and serializes them into a buffer that clients can use for replaying\nits history. There are a few limitations on `HistoricalObject`:\n\n- Historical objects can only have numeric (floating point) values and can't have nested objects or optional fields.\n- Historical objects must declare which fields they'd like to track.\n\nWe store each player's \"location\" (i.e. its position, orientation, and speed) in a `HistoricalObject` and\nwrite it to the `worlds` document at the end of a step when computing a diff.\n\n## Client-side game UI (`src/`)\n\nOne guiding principle for AI Town's architecture is to keep the usage as close to \"regular Convex\" usage as possible. So,\ngame state is stored in regular tables, and the UI just uses regular `useQuery` hooks to load that state and render\nit in the UI.\n\nThe one exception is for historical tables, which feed in the latest state into a `useHistoricalValue` hook that parses\nthe history buffer and replays time forward for smooth motion. To keep replayed time synchronized across multiple\nhistorical buffers, we provide a `useHistoricalTime` hook for the top of your app that keeps track of the current\ntime and returns it for you to pass down into components.\n\nWe also provide a `useSendInput` hook that wraps `useMutation` and automatically sends inputs to the server and\nwaits for the engine to process them and return their outcome.\n\n## Agent architecture (`convex/agent`)\n\n### The agent loop (`convex/game/agents.ts`)\n\nAgents will execute any game state changes, and schedule operations to do anything that requires\na long-lived request or accessing non-game tables. The flow generally is:\n\n1. Logic in `Agent.tick` can read and modify game state as time progresses, such as waiting until\n   the agent is near another player to start talking.\n2. When there is something that needs to talk to an LLM or read/write external data,\n   it calls `startOperation` with a reference to a Convex function: generally an `internalAction`.\n3. This function can read state from game tables and other tables via `internalQuery` functions.\n4. It executes long-running tasks, and can write data via `internalMutation`s.\n   Game state should not be written, but rather submitted via `inputs` (described in a previous section).\n5. Inputs are submitted from actions with `ctx.runMutation(api.game.main.sendInput, {...})` from actions\n   or via `insertInput` from mutations. They are referenced by their name as a string, like `moveTo`.\n6. Inputs are defined with `inputHandler` and are given an instance of the AiTown game to modify,\n   similar to the game loop. In fact, these are called as part of the game loop before `tickAgent`.\n7. When an operation is done, it deletes the `inProgressOperation`. This is to ensure an agent only\n   is trying to do one thing at a time.\n8. `Agent.tick` then can observe the new game state and continue to make decisions.\n\n### Conversations (`convex/agent/conversations.ts`)\n\nThe agent code calls into the conversation layer which implements the prompt engineering for\ninjecting personality and memories into the GPT responses. It has functions for starting a\nconversation (`startConversation`), continuing after the first message (`continueConversation`), and\npolitely leaving a conversation (`leaveConversation`). Each function loads structured data from the\ndatabase, queries the memory layer for the agent's opinion about the player they're talking with,\nand then calls into the OpenAI client (`convex/util/openai.ts`).\n\n### Memories (`convex/agent/memory.ts`)\n\nAfter each conversation, GPT summarizes its message history, and we compute an embedding of the\nsummary text and write it into Convex's vector database. Then, when starting a new conversation\nwith, Danny, we embed \"What you think about Danny?\", find the three most similar memories, and fetch\ntheir summary texts to inject into the conversation prompt.\n\n### Embeddings cache (`convex/agent/embeddingsCache.ts`)\n\nTo avoid computing the same embedding over and over again, we cache embeddings by a hash of their\ntext in a Convex table.\n\n## Design goals and limitations\n\nAI Town's game engine has a few design goals:\n\n- Try to be as close to a regular Convex app as possible. Use regular client hooks (like `useQuery`)\n  when possible, and store game state in regular tables.\n- Be as similar to existing engines as possible, so it's easy to change the behavior. We chose a\n  `tick()` based model for simulation since it's commonly used elsewhere and intuitive.\n- Decouple agent behavior from the game engine. It's nice to allow human players and AI agents to do\n  all the same things in the game.\n\nThese design goals imply some inherent limitations:\n\n- All data is loaded into memory each step. The active game state loaded by the game should be small\n  enough to fit into memory and load and save frequently. Try to keep game state to less than a few dozen\n  kilobytes: Games that require tens of thousands of objects interacting together may not be a good\n  fit.\n- All inputs are fed through the database in the `inputs` table, so applications that require very\n  large or frequent inputs may not be a good fit.\n- Input latency will be around one RTT (time for the input to make it to the server and the response\n  to come back) plus half the step size (for expected server input delay when the input's waiting\n  for the next step). Historical values add another half step size of input latency since their\n  values are viewed slightly in the past. As configured, this will roughly be around 1.5s of input\n  latency, which won't be a good fit for competitive games. You can configure the step size to be\n  smaller (e.g. 250ms) which will decrease input latency at the cost of adding more Convex function\n  calls and database bandwidth.\n- The game engine is designed to be single threaded. JavaScript operating over plain objects\n  in-memory can be surprisingly fast, but if your simulation is very computationally expensive, it\n  may not be a good fit on AI Town's engine today.\n","isInternal":false,"tokens":4108,"sizeBytes":18712}],"systemPromptSnippet":"<yakaai_skills repo=\"a16z-infra/ai-town\">\n<!-- File: ARCHITECTURE.md (Tokens: ~4108 | Category: project-spec) -->\n# Architecture\n\nThis documents dives into the high-level architecture of AI Town and its different layers. We'll\nfirst start with a brief overview and then go in-depth on each component. The overview should\nbe sufficient for forking AI Town and changing game or agent behavior. Read on to the deep dives\nif you're interested or running up against the engine's limitations.\n\nThis doc assumes the reader has a working knowledge of Convex. If you're new to Convex, check out\nthe [Convex tutorial](https://docs.convex.dev/get-started) to get started.\n\n## Overview\n\nAI Town is split into a few layers:\n\n- The server-side game logic in `convex/aiTown`: This layer defines what state AI Town maintains,\n  how it evolves over time, and how it reacts to user input. Both humans and agents submit inputs\n  that the game engine processes.\n- The client-side game UI in `src/`: AI Town uses `pixi-react` to render the game state to the\n  browser for human consumption.\n- The game engine in `convex/engine`: To make it easy to hack on the game rules, we've separated\n  out the game engine from the AI Town-specific game rules. The game engine is responsible for\n  saving and loading game state from the database, coordinating feeding inputs into the engine,\n  and actually running the game engine in Convex functions.\n- The agent in `convex/agent`: Agents run as part of the game loop, and can kick off asynchronous\n  Convex functions to do longer processing, such as talking to LLMs. Those functions can save state\n  in separate tables, or submit inputs to the game engine to modify game state. Internally, our\n  agents use a combination of simple rule-based systems and talking to an LLM.\n\nSo, if you'd like to tweak agent behavior but keep the same game mechanics, check out `convex/agent`\nfor the async work, and `convex/aiTown/agent.ts` for the game loop logic.\nIf you would like to add new gameplay elements (that both humans and agents can interact with), add\nthe feature to `convex/aiTown`, render it in the UI in `src/`, and respond to it in `convex/aiTown/agent.ts`.\n\nIf you have parts of your game that are more latency sensitive, you can move them out of engine\ninto regular Convex tables, queries, and mutations, only logging key bits into game state. See\n\"Message data model\" below for an example.\n\n## AI Town game logic (`convex/aiTown`)\n\n### Data model\n\nAI Town's data model has a few concepts:\n\n- Worlds (`convex/aiTown/world.ts`) represent a map with many players interacting together.\n- Players (`convex/aiTown/player.ts`) are the core characters in the game. Players have human readable names and\n  descriptions, and they may be associated with a human user. At any point in time, a player may be pathfinding\n  towards some destination and has a current location.\n- Conversations (`convex/aiTown/conversations.ts`) are created by a player and end at some point in time.\n- Conversation memberships (`convex/aiTown/conversationMembership.ts`) indicate that a player is a member\n  of a conversation. Players may only be in one conversation at any point in time, and conversations\n  currently have exactly two members. Memberships may be in one of three states:\n  - `invited`: The player has been invited to the conversation but hasn't accepted yet.\n  - `walkingOver`: The player has accepted the invite to the conversation but is too far away to talk. The\n    player will automatically join the conversation when they get close enough.\n  - `participating`: The player is actively participating in the conversation.\n\n### Schema\n\nThere are three main categories of tables:\n\n1. Engine tables (`convex/engine/schema.ts`) for maintaining engine-internal state.\n2. Game tables (`convex/aiTown/schema.ts`) for game state. To keep game state small and efficient to\n   read and write, we store AI Town's data model across a few tables. See `convex/aiTown/schema.ts` for an overview.\n3. Agent tables (`convex/agent/schema.ts`) for agent state. Agents can freely read and write to these tables\n   within their actions.\n\n### Inputs (`convex/aiTown/inputs.ts`)\n\nAI Town modifies its data model by processing inputs. Inputs are submitted by players and agents and\nprocessed by the game engine. We specify inputs in the `inputs` object in `convex/aiTown/inputs.ts`.\nUse the `inputHandler` function to construct an input handler, specifying a Convex validator for\narguments for end-to-end type-safety.\n\n- Joining (`join`) and leaving (`leave`) the game.\n- Moving a player to a particular location (`moveTo`): Movement in AI Town is similar to RTS games, where\n  the players specify where they want to go, and the engine figures out how to get there.\n- Starting a conversation (`startConversation`), accepting an invite (`acceptInvite`), rejecting an invite\n  (`rejectInvite`), and leaving a conversation (`leaveConversation`). To track typing indicators,\n  you use `startTyping` and `finishSendingMessage`. These are imported from `game/conversations.ts`.\n- Agent inputs are imported from `aiTown/agentInputs.ts` for things like remembering conversations,\n  deciding what to do, etc.\n\nEach of these inputs' implementation method checks invariants and updates game state as desired.\nFor example, the `moveTo` input checks that the player isn't participating in a conversation,\nthrowing an error telling them to leave the conversation first if so, and then updates their\npathfinding state with the desired destination.\n\n### Simulation\n\nOther than when processing player inputs, the game state can change over time in the background as the\nsimulation runs time forward. For example, if a player has decided to move along a path, their position\nwill gradually update as time moves forward. Similarly, if two players collide into each other, they'll\nnotice and replan their paths, trying to avoid obstacles.\n\n### Message data model\n\nWe manage the tables for tracking chat messages in separate tables not affiliated\nwith the game engine. This is for a few reasons:\n\n- The core simulation doesn't need to know about messages, so keeping them\n  out keeps game state small.\n- Messages are updated very frequently (when streamed out from OpenAI) and\n  benefit from lower input latency, so they're not a great fit for the engine.\n  See \"Design goals and limitations\" below.\n\nMessages (`convex/schema.ts`) are in a conversation and indicate an author and message text.\nEach conversation has a typing state in the conversations table that indicates that a player\nis currently typing. Players can still send messages while another player is typing, but\nhaving the indicator helps agents (and humans) not talk over each other.\n\nThe separate tables are queried and modified with regular Convex queries and mutations\nthat don't directly go through the simulation.\n\n## Game engine (`convex/engine`)\n\nGiven the description of AI Town's game behavior in the previous section,\nthe `AbstractGame` class in `convex/engine/abstractGame.ts` implements actually running the simulation.\nThe game engine has a few responsibilities:\n\n- Coordinating incoming player inputs, feeding them into the simulation, and sending their\n  return values (or errors) to the client.\n- Running the simulation forward in time.\n- Saving and loading game state from the database.\n- Managing executing the game behavior, efficiently using Convex resources and minimizing input latency.\n\nAI Town's game behavior is implemented in the `Game` subclass.\n\n### Input handling\n\nUsers submit inputs through the `insertInput` function, which inserts them into an `inputs` table, assigning a\nmonotonically increasing unique input number and stamping the input with the time the server received it. The\nengine then processes inputs, writing their results back to the `inputs` row. Interested clients can subscribe\non an input's status with the `inputStatus` query.\n\n`Game` provides an abstract method `handleInput` that `AiTown` implements with its specific behavior.\n\n### Running the simulation\n\nThe `Game` class specifies how it simulates time forward with the `tick` method:\n\n- `tick(now)` runs the simulation forward until the given timestamp\n- Ticks are run at a high frequency, configurable with `tickDuration` (milliseconds). Since AI town has smooth motion\n  for player movement, it runs at 60 ticks per second.\n- It's generally a good idea to break up game logic into separate systems that can be ticked forward independently.\n  For example, AI Town's `tick` method advances pathfinding with `Player.tickPathfinding`, player positions with\n  `Player.tickPosition`, conversations with `Conversation.tick`, and `Agent.tick` for agent logic.\n\nTo avoid running a Convex mutation 60 times per second (which would be expensive and slow), the engine batches up\nmany ticks into a _step_. AI town runs steps at only 1 time per second. Here's how a step works:\n\n1. Load the game state into memory.\n2. Decide how long to run.\n3. Execute many ticks for our time interval, alternating between feeding in inputs with `handleInput` and advancing\n   the simulation with `tick`.\n4. Write the updated game state back to the database.\n\nOne core invariant is that the game engine is fully \"single-threaded\" per world, so there are never two runs of\nan engine's step overlapping in time. Not having to think about race conditions or concurrency makes writing game\nengine code a lot easier.\n\nHowever, preserving this invariant is a little tricky. If the engine is idle for a minute and an\ninput comes in, we want to run the engine immediately but then cancel its run after the minute's\nup. If we're not careful, a race condition may cause us to run multiple copies of the engine if an\ninput comes in just as an idle timeout is expiring!\n\nOur approach is to store a generation number with the engine that monotonically increases over time.\nAll scheduled runs of the engine contain their expected generation number as an argument. Then, if\nwe'd like to cancel a future run of the engine, we can bump the generation number by one, and then\nwe're guaranteed that the subsequent run will fail immediately as it'll notice that the engine's\ngeneration number does not match its expected one.\n\n### Engine state management\n\nThe `World`, `Player`, `Conversation`, and `Agent` classes coordinate loading data into memory from the database,\nmodifying it according to the game rules, and serializing it to write back out to the database. Here's the flow:\n\n1. The Convex scheduler calls the `convex/aiTown/main.ts:runStep` action.\n2. The `runStep` action calls `convex/aiTown/game.ts:loadWorld` to load the current game state. This query calls\n   `Game.load`, which loads all of a world's game state from the appropriate tables, and returns a\n   `GameState` object, which contains serialized versions of all of the players, agents, etc.\n3. The `runStep` action passes the `GameState` to the `Game` constructor, which parses the serialized versions\n   of all our game objects using their constructors. For example, `new Player(serializedPlayer)` parses the\n   database representation into the in-memory `Player` class.\n4. The engine runs the simulation, modifying the in-memory game objects.\n5. At the end of a step, the framework calls `Game.saveStep`, which computes a diff of the game state since\n   the beginning of the step and passes the diff to the `convex/aiTown/game.ts:saveWorld` mutation.\n6. The `saveWorld` mutation applies the diff to the database, notices if any deleted objects need to be archived,\n   updates the `participatedTogether` graph, and kicks off any scheduled jobs to run.\n7. Since the engine is the only mutator of game state, it continues to run steps for some amount of time\n   without repeating steps 1 to 3 again.\n\nJust as we assume that the game engine is \"single threaded\", we also assume that the game engine _exclusively_\nowns the tables that store game engine state. Only the game engine should programmatically modify these tables,\nso components outside the engine can only mutate them by sending inputs.\n\n### Historical tables\n\nIf we're only writing updates out to the database at the end of the step, and steps are only running at once per\nsecond, continuous quantities like position will only update every second. This, then, defeats the whole purpose\nof having high-frequency ticks: Player positions will jump around and look choppy.\n\nTo solve this, we track the historical values of quantities like position _within_ a step, storing the value\nat the end of each tick. Then, the client receives both the current value _and_ the past step's worth of\nhistory, and it can \"replay\" the history to make the motion smooth.\n\nThe game tracks these quantities at the end of each tick by feeding them to a `HistoricalObject`. This object\nefficiently tracks its changes over time and serializes them into a buffer that clients can use for replaying\nits history. There are a few limitations on `HistoricalObject`:\n\n- Historical objects can only have numeric (floating point) values and can't have nested objects or optional fields.\n- Historical objects must declare which fields they'd like to track.\n\nWe store each player's \"location\" (i.e. its position, orientation, and speed) in a `HistoricalObject` and\nwrite it to the `worlds` document at the end of a step when computing a diff.\n\n## Client-side game UI (`src/`)\n\nOne guiding principle for AI Town's architecture is to keep the usage as close to \"regular Convex\" usage as possible. So,\ngame state is stored in regular tables, and the UI just uses regular `useQuery` hooks to load that state and render\nit in the UI.\n\nThe one exception is for historical tables, which feed in the latest state into a `useHistoricalValue` hook that parses\nthe history buffer and replays time forward for smooth motion. To keep replayed time synchronized across multiple\nhistorical buffers, we provide a `useHistoricalTime` hook for the top of your app that keeps track of the current\ntime and returns it for you to pass down into components.\n\nWe also provide a `useSendInput` hook that wraps `useMutation` and automatically sends inputs to the server and\nwaits for the engine to process them and return their outcome.\n\n## Agent architecture (`convex/agent`)\n\n### The agent loop (`convex/game/agents.ts`)\n\nAgents will execute any game state changes, and schedule operations to do anything that requires\na long-lived request or accessing non-game tables. The flow generally is:\n\n1. Logic in `Agent.tick` can read and modify game state as time progresses, such as waiting until\n   the agent is near another player to start talking.\n2. When there is something that needs to talk to an LLM or read/write external data,\n   it calls `startOperation` with a reference to a Convex function: generally an `internalAction`.\n3. This function can read state from game tables and other tables via `internalQuery` functions.\n4. It executes long-running tasks, and can write data via `internalMutation`s.\n   Game state should not be written, but rather submitted via `inputs` (described in a previous section).\n5. Inputs are submitted from actions with `ctx.runMutation(api.game.main.sendInput, {...})` from actions\n   or via `insertInput` from mutations. They are referenced by their name as a string, like `moveTo`.\n6. Inputs are defined with `inputHandler` and are given an instance of the AiTown game to modify,\n   similar to the game loop. In fact, these are called as part of the game loop before `tickAgent`.\n7. When an operation is done, it deletes the `inProgressOperation`. This is to ensure an agent only\n   is trying to do one thing at a time.\n8. `Agent.tick` then can observe the new game state and continue to make decisions.\n\n### Conversations (`convex/agent/conversations.ts`)\n\nThe agent code calls into the conversation layer which implements the prompt engineering for\ninjecting personality and memories into the GPT responses. It has functions for starting a\nconversation (`startConversation`), continuing after the first message (`continueConversation`), and\npolitely leaving a conversation (`leaveConversation`). Each function loads structured data from the\ndatabase, queries the memory layer for the agent's opinion about the player they're talking with,\nand then calls into the OpenAI client (`convex/util/openai.ts`).\n\n### Memories (`convex/agent/memory.ts`)\n\nAfter each conversation, GPT summarizes its message history, and we compute an embedding of the\nsummary text and write it into Convex's vector database. Then, when starting a new conversation\nwith, Danny, we embed \"What you think about Danny?\", find the three most similar memories, and fetch\ntheir summary texts to inject into the conversation prompt.\n\n### Embeddings cache (`convex/agent/embeddingsCache.ts`)\n\nTo avoid computing the same embedding over and over again, we cache embeddings by a hash of their\ntext in a Convex table.\n\n## Design goals and limitations\n\nAI Town's game engine has a few design goals:\n\n- Try to be as close to a regular Convex app as possible. Use regular client hooks (like `useQuery`)\n  when possible, and store game state in regular tables.\n- Be as similar to existing engines as possible, so it's easy to change the behavior. We chose a\n  `tick()` based model for simulation since it's commonly used elsewhere and intuitive.\n- Decouple agent behavior from the game engine. It's nice to allow human players and AI agents to do\n  all the same things in the game.\n\nThese design goals imply some inherent limitations:\n\n- All data is loaded into memory each step. The active game state loaded by the game should be small\n  enough to fit into memory and load and save frequently. Try to keep game state to less than a few dozen\n  kilobytes: Games that require tens of thousands of objects interacting together may not be a good\n  fit.\n- All inputs are fed through the database in the `inputs` table, so applications that require very\n  large or frequent inputs may not be a good fit.\n- Input latency will be around one RTT (time for the input to make it to the server and the response\n  to come back) plus half the step size (for expected server input delay when the input's waiting\n  for the next step). Historical values add another half step size of input latency since their\n  values are viewed slightly in the past. As configured, this will roughly be around 1.5s of input\n  latency, which won't be a good fit for competitive games. You can configure the step size to be\n  smaller (e.g. 250ms) which will decrease input latency at the cost of adding more Convex function\n  calls and database bandwidth.\n- The game engine is designed to be single threaded. JavaScript operating over plain objects\n  in-memory can be surprisingly fast, but if your simulation is very computationally expensive, it\n  may not be a good fit on AI Town's engine today.\n</yakaai_skills>"}