Repository: BasedHardware/omi

Stars: 9566

CLAUDE.md

Omi Development Guide

Behavior

- Never ask for permission to access folders, run commands, search the web, or use tools. Just do it.
- Never ask for confirmation. Just act. Make decisions autonomously and proceed without checking in.
- You have full access to the user's computer — browser, desktop, all apps. Never ask the user to do something you can do yourself (sign in, click buttons, dismiss dialogs, etc.).

Setup

Pre-commit Hook (required)

bash

ln -s -f ../../scripts/pre-commit .git/hooks/pre-commit

Mobile App

bash

cd app && bash setup.sh ios    # or: bash setup.sh android

---

Backend (Python)

Rules

- No in-function imports — all imports at module top level.
- Import hierarchy (low → high): database/ → utils/ → routers/ → main.py. Never import upward.
- Memory management — del byte arrays after processing, .clear() dicts/lists holding data.
- Async I/O — never requests.* in async (use httpx.AsyncClient pools from utils/http_client.py), never Thread().start().join() (use critical_executor/storage_executor), never time.sleep() in async (use asyncio.sleep()). Run python scripts/lint_async_blockers.py before committing.

Logging Security

Never log raw sensitive data. Use sanitize() and sanitize_pii() from utils.log_sanitizer.
- sanitize() for response.text, API responses, error bodies.
- sanitize_pii() for names, emails, user text.
- Keep UIDs, IPs, status codes visible for debugging.
- Never put raw response.text in exception messages.

Service Map

text

Shared: Firestore, Redis
backend (main.py)
  ├── ws ──► pusher (pusher/)
  ├── ──────► diarizer (diarizer/)
  ├── ──────► vad (modal/)
  └── ──────► deepgram (self-hosted or cloud)
pusher
  ├── ──────► diarizer (diarizer/)
  └── ──────► deepgram (cloud)
agent-proxy (agent-proxy/main.py)
  └── ws ──► user agent VM (private IP, port 8080)notifications-job (modal/job.py)  [cron]

Helm charts: backend/charts/{backend-listen,pusher,diarizer,vad,deepgram-self-hosted,agent-proxy}/

See service descriptions in AGENTS.md. Update both files when service boundaries change.

---

App (Flutter)

Localization

- All user-facing strings must use l10n: context.l10n.keyName instead of hardcoded strings.
- Add new keys via jq (never read full ARB files). See skill add-a-new-localization-key-l10n-arb.
- Translate all 33 locales — no English text in non-English ARB files. Use omi-add-missing-language-keys-l10n skill.
- Regenerate after changes: cd app && flutter gen-l10n

Firebase Prod Config

Never run flutterfire configure — it overwrites prod credentials. Prod config files in app/ios/Config/Prod/, app/lib/firebase_options_prod.dart, app/android/app/src/prod/.

Verifying UI Changes (agent-flutter)

After editing Flutter UI code, verify programmatically — don't just hot restart and hope.

bash

kill -SIGUSR2 $(pgrep -f "flutter run" | head -1)                # hot restart
AGENT_FLUTTER_LOG=/tmp/flutter-run.log agent-flutter connect      # reconnect after restart
agent-flutter snapshot -i                                         # see interactive widgets
agent-flutter find type button press                              # find and tap
agent-flutter fill @e5 "hello"                                    # type into textfield
agent-flutter screenshot /tmp/evidence.png                        # PR evidence

Key rules:
- Re-snapshot before every interaction (refs go stale). Use press x y as coordinate fallback.
- AGENT_FLUTTER_LOG must point to flutter run stdout (not logcat).
- find type X / find text "label" is more stable than @ref numbers.
- Add Key('descriptive_name') to new interactive widgets for find key.
- See app/e2e/SKILL.md for navigation architecture, screen map, and known flows.

---

Desktop (macOS)

Building & Running

- ./run.sh — full local dev (build + backend + tunnel + app)
- ./run.sh --yolo — quick start with prod backend, no local services
- Release builds are handled entirely by Codemagic CI (no local release script)
- Build command: xcrun swift build -c debug --package-path Desktop (the xcrun prefix is required)
- DO NOT use bare swift build, xcodebuild, or launch from build/ directly

Named Test Bundles

When testing a feature or bug fix, always create a separate named bundle:

bash

OMI_APP_NAME="omi-fix-rewind" ./run.sh

This installs to /Applications/omi-fix-rewind.app with bundle ID com.omi.omi-fix-rewind.

Rules:
- ALWAYS prefix with omi- (e.g., omi-fix-rewind, omi-6512-polling, omi-vision-test) so bundles are grouped in /Applications/
- NEVER use bare ./run.sh when testing a specific change — it overwrites "Omi Dev"
- NEVER kill or interfere with "Omi", "Omi Beta" — those are production installs
- Keep app name and bundle suffix identical (e.g., omi-search.app → com.omi.omi-search)
- Named bundles get their own permissions, auth state, and database
- After building, launch and interact programmatically to confirm it runs — don't stop at compile

Verifying UI Changes (agent-swift)

After editing Swift UI code, verify programmatically via macOS Accessibility API:

bash

agent-swift connect --bundle-id com.omi.omi-fix-rewind           # connect to named bundle
agent-swift snapshot -i                                           # interactive elements only
agent-swift click @e3                                             # CGEvent click (SwiftUI)
agent-swift press @e3                                             # AXPress (AppKit buttons)
agent-swift fill @e5 "text"                                       # type into field
agent-swift wait text "Settings"                                  # wait for text
agent-swift screenshot /tmp/evidence.png                          # PR evidence

Key rules:
- Prefer click over press for SwiftUI (CGEvent triggers NavigationLink; AXPress is AppKit only).
- Re-snapshot before every interaction (refs go stale).
- Always use snapshot -i (interactive only) — full snapshots are very verbose.
- agent-swift doctor verifies Accessibility permission.
- Dev bundle ID: com.omi.desktop-dev. Prod: com.omi.computer-macos.
- See desktop/e2e/SKILL.md for navigation architecture and known flows.

---

Computer Control (clicking, typing, screenshots)

For controlling the Mac GUI. Use the right tool for each job:

| Task | Tool | Example |
|------|------|---------|
| Click at coordinates | cliclick | cliclick c:X,Y |
| Screenshots/OCR | codriver | mcp__codriver__desktop_screenshot (scale: 0.5) |
| Native macOS app testing | agent-swift | See Desktop section above |
| Browser automation | playwright MCP | Headless, most reliable |
| Existing browser tabs | claude-in-chrome | Only when extension connected |

Workflow: screenshot (codriver) → find target → click (cliclick c:X,Y)

Rules:
- NEVER try 3+ different click tools for the same action — pick one and commit.
- codriver at scale: 0.5 → multiply coordinates by 2 before clicking.
- Prefer cliclick over automac/mac-use-mcp (coordinate bugs on multi-monitor).

---

Formatting

The pre-commit hook auto-formats, but you can run manually:

| Language | Command |
|----------|---------|
| Dart (app/) | dart format --line-length 120 <files> |
| Python (backend/) | black --line-length 120 --skip-string-normalization <files> |
| C/C++ (firmware) | clang-format -i <files> |

Files ending in .gen.dart or .g.dart are auto-generated — don't format manually.

---

Git

Rules

- Always commit to the current branch — never switch branches.
- Never push directly to main. Land changes through PRs only.
- Never squash merge PRs — use regular merge.
- Make individual commits per file, not bulk commits.
- If push fails (remote ahead): git pull --rebase && git push.
- Never push or create PRs unless explicitly asked — commit locally by default.
- Always work in a git worktree for code changes. Use EnterWorktree to isolate work.

RELEASE Command

Create branch from main, individual commits per file, push/create PR, merge without squash, switch back to main and pull.

RELEASEWITHBACKEND Command

Full RELEASE flow + gh workflow run gcp_backend.yml -f environment=prod -f branch=main.

---

Testing

Run backend/test-preflight.sh to verify environment. Run backend/test.sh (backend) or app/test.sh (app) before committing.

CI/CD

See docs/runbooks/deploy.md for deploy triggers and checks.

Logs

See docs/runbooks/logging.md for log commands.

Documentation Maintenance

- If a PR changes setup, test commands, safety rules, service boundaries, or env vars — update this file in the same PR.
- Keep AGENTS.md synced with this file. Update both in the same commit.
- For architecture/core flow/API changes — update Mintlify docs (docs/) in the same PR.
- If a PR changes audio streaming, transcription, conversation lifecycle, or listen/pusher WebSocket — update docs/doc/developer/backend/listen_pusher_pipeline.mdx.

README.md

omi

A 2nd brain you trust more than your 1st

Omi captures your screen and conversations, transcribes in real-time, generates summaries and action items, and gives you an AI chat that remembers everything you've seen and heard. Works on desktop, phone and wearables. Fully open source.

Trusted by 300,000+ professionals.

![Discord](http://discord.omi.me)&ensp;
![GitHub Repo stars](https://github.com/BasedHardware/Omi)&ensp;
![License: MIT](https://opensource.org/licenses/MIT)

Website · Docs · Discord · Twitter · DeepWiki

</div>

Quick Start

<p align="center">
<a href="https://app.omi.me">Try in Browser</a>
</p>

bash

git clone https://github.com/BasedHardware/omi.git && cd omi/desktop && ./run.sh --yolo

Builds the macOS app, connects to the cloud backend, and launches. No env files, no credentials, no local backend.

Requirements: macOS 14+, Xcode (includes Swift & code signing), Node.js

<details>
<summary>Full Installation</summary>

For local development with the full backend stack:

1. Install prerequisites

bash

xcode-select --install
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

2. Clone and configure

bash

git clone https://github.com/BasedHardware/omi.git
cd omi/desktop
cp Backend-Rust/.env.example Backend-Rust/.env

3. Build and run

bash

./run.sh

See desktop/README.md for environment variables and credential setup.

Mobile App

bash

cd app && bash setup.sh ios    # or: bash setup.sh android

</details>

<details>
<summary>How it works</summary>

text

┌─────────────────────────────────────────────────────────┐
│                      Your Devices                       │
│                                                         │
│  ┌──────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │ Omi      │  │ macOS App    │  │ Mobile App        │  │
│  │ Wearable │  │ (Swift/Rust) │  │ (Flutter)         │  │
│  └────┬─────┘  └──────┬───────┘  └────────┬──────────┘  │
│       │    BLE         │   HTTPS/WS        │             │
└───────┼────────────────┼───────────────────┼─────────────┘
        │                │                   │
        ▼                ▼                   ▼
┌─────────────────────────────────────────────────────────┐
│                    Omi Backend (Python)                  │
│                                                         │
│  ┌─────────┐  ┌──────────┐  ┌─────────┐  ┌──────────┐  │
│  │ Listen  │  │ Pusher   │  │ VAD     │  │ Diarizer │  │
│  │ (REST)  │  │ (WS)     │  │ (GPU)   │  │ (GPU)    │  │
│  └─────────┘  └──────────┘  └─────────┘  └──────────┘  │
│                                                         │
│  ┌─────────┐  ┌──────────┐  ┌─────────┐  ┌──────────┐  │
│  │ Deepgram│  │ Firestore│  │ Redis   │  │ LLMs     │  │
│  │ (STT)   │  │ (DB)     │  │ (Cache) │  │ (AI)     │  │
│  └─────────┘  └──────────┘  └─────────┘  └──────────┘  │
└─────────────────────────────────────────────────────────┘

| Component | Path | Stack |
|-----------|------|-------|
| macOS app | desktop/ | Swift, SwiftUI, Rust backend |
| Mobile app | app/ | Flutter (iOS & Android) |
| Backend API | backend/ | Python, FastAPI, Firebase |
| Firmware | omi/ | nRF, Zephyr, C |
| Omi Glass | omiGlass/ | ESP32-S3, C |
| SDKs | sdks/ | React Native, Swift, Python |
| AI Personas | web/personas-open-source/ | Next.js |

</details>

Documentation

Getting Started

- Introduction
- Quick Start Guide
- macOS App Development
- Mobile App Setup
- Backend Setup
- Contributing

Building Apps

- App Development Guide
- Example Apps — GitHub, Slack, OmiMentor
- Audio Streaming Apps
- Custom Chat Tools
- Submit to App Store

API & SDKs

- API Reference — REST endpoints for memories, conversations, action items
- Python SDK
- Swift SDK
- React Native SDK
- MCP Server — Model Context Protocol integration

Architecture

- Backend Deep Dive
- Transcription Pipeline
- Chat System
- Audio Streaming Pipeline
- BLE Protocol

Omi Hardware

!Omi

Open-source AI wearables that pair with the mobile app for 24h+ continuous capture.

- Buy Omi
- Buy Omi Glass Dev Kit — ESP32-S3, camera + audio
- Open Source Hardware Designs
- Buying Guide
- Build the Device
- Flash Firmware
- Integrate Your Wearable
- Hardware Specs

License

MIT — see LICENSE

Desktop Installers & Release Binaries Latest

Repository: BasedHardware/omi

Stars: 9566

CLAUDE.md Copy Chapter

Omi Development Guide

Behavior Copy Chapter

Setup Copy Chapter

Pre-commit Hook (required) Copy Section

Mobile App Copy Section

Backend (Python) Copy Chapter

Rules Copy Section

Logging Security Copy Section

Service Map Copy Section

App (Flutter) Copy Chapter

Localization Copy Section

Firebase Prod Config Copy Section

Verifying UI Changes (agent-flutter) Copy Section

Desktop (macOS) Copy Chapter

Building & Running Copy Section

Named Test Bundles Copy Section

Verifying UI Changes (agent-swift) Copy Section

Computer Control (clicking, typing, screenshots) Copy Chapter

Formatting Copy Chapter

Git Copy Chapter

Rules Copy Section

RELEASE Command Copy Section

RELEASEWITHBACKEND Command Copy Section

Testing Copy Chapter

CI/CD Copy Chapter

Logs Copy Chapter

Documentation Maintenance Copy Chapter

README.md Copy Chapter

omi

A 2nd brain you trust more than your 1st Copy Section

Quick Start Copy Chapter

Mobile App Copy Section

Documentation Copy Chapter

Getting Started Copy Section

Building Apps Copy Section

API & SDKs Copy Section

Architecture Copy Section

Omi Hardware Copy Chapter

License Copy Chapter

Explore Related Technologies & Topics

CLAUDE.md

Behavior

Setup

Pre-commit Hook (required)

Mobile App

Backend (Python)

Rules

Logging Security

Service Map

App (Flutter)

Localization

Firebase Prod Config

Verifying UI Changes (agent-flutter)

Desktop (macOS)

Building & Running

Named Test Bundles

Verifying UI Changes (agent-swift)

Computer Control (clicking, typing, screenshots)

Formatting

Git

Rules

RELEASE Command

RELEASEWITHBACKEND Command

Testing

CI/CD

Logs

Documentation Maintenance

README.md

A 2nd brain you trust more than your 1st

Quick Start

Mobile App

Documentation

Getting Started

Building Apps

API & SDKs

Architecture

Omi Hardware

License