# Repository: BasedHardware/omi
# Stars: 9566

## CLAUDE.md

# Omi Development Guide
<!-- Official guidance for writing these files:
     CLAUDE.md: https://docs.anthropic.com/en/docs/claude-code/memory
     AGENTS.md: https://developers.openai.com/codex/guides/agents-md
     Format spec: https://agents.md -->

## Behavior

- Never ask for permission to access folders, run commands, search the web, or use tools. Just do it.
- Never ask for confirmation. Just act. Make decisions autonomously and proceed without checking in.
- You have full access to the user's computer — browser, desktop, all apps. Never ask the user to do something you can do yourself (sign in, click buttons, dismiss dialogs, etc.).

## Setup

### Pre-commit Hook (required)
```bash
ln -s -f ../../scripts/pre-commit .git/hooks/pre-commit
```

### Mobile App
```bash
cd app && bash setup.sh ios    # or: bash setup.sh android
```

---

## Backend (Python)
<!-- Maintainers: @beastoin (service map, logging security), @Thinh (imports, memory mgmt) -->

### Rules
- **No in-function imports** — all imports at module top level.
- **Import hierarchy** (low → high): `database/` → `utils/` → `routers/` → `main.py`. Never import upward.
- **Memory management** — `del` byte arrays after processing, `.clear()` dicts/lists holding data.
- **Async I/O** — never `requests.*` in async (use `httpx.AsyncClient` pools from `utils/http_client.py`), never `Thread().start().join()` (use `critical_executor`/`storage_executor`), never `time.sleep()` in async (use `asyncio.sleep()`). Run `python scripts/lint_async_blockers.py` before committing.

### Logging Security
Never log raw sensitive data. Use `sanitize()` and `sanitize_pii()` from `utils.log_sanitizer`.
- `sanitize()` for `response.text`, API responses, error bodies.
- `sanitize_pii()` for names, emails, user text.
- Keep UIDs, IPs, status codes visible for debugging.
- Never put raw `response.text` in exception messages.

### Service Map
```
Shared: Firestore, Redis

backend (main.py)
  ├── ws ──► pusher (pusher/)
  ├── ──────► diarizer (diarizer/)
  ├── ──────► vad (modal/)
  └── ──────► deepgram (self-hosted or cloud)

pusher
  ├── ──────► diarizer (diarizer/)
  └── ──────► deepgram (cloud)

agent-proxy (agent-proxy/main.py)
  └── ws ──► user agent VM (private IP, port 8080)

notifications-job (modal/job.py)  [cron]
```

Helm charts: `backend/charts/{backend-listen,pusher,diarizer,vad,deepgram-self-hosted,agent-proxy}/`

See service descriptions in AGENTS.md. Update both files when service boundaries change.

---

## App (Flutter)
<!-- Maintainers: @Thinh (l10n, formatting) -->

### Localization
- All user-facing strings must use l10n: `context.l10n.keyName` instead of hardcoded strings.
- Add new keys via `jq` (never read full ARB files). See skill `add-a-new-localization-key-l10n-arb`.
- **Translate all 33 locales** — no English text in non-English ARB files. Use `omi-add-missing-language-keys-l10n` skill.
- Regenerate after changes: `cd app && flutter gen-l10n`

### Firebase Prod Config
Never run `flutterfire configure` — it overwrites prod credentials. Prod config files in `app/ios/Config/Prod/`, `app/lib/firebase_options_prod.dart`, `app/android/app/src/prod/`.

### Verifying UI Changes (agent-flutter)
After editing Flutter UI code, **verify programmatically** — don't just hot restart and hope.

```bash
kill -SIGUSR2 $(pgrep -f "flutter run" | head -1)                # hot restart
AGENT_FLUTTER_LOG=/tmp/flutter-run.log agent-flutter connect      # reconnect after restart
agent-flutter snapshot -i                                         # see interactive widgets
agent-flutter find type button press                              # find and tap
agent-flutter fill @e5 "hello"                                    # type into textfield
agent-flutter screenshot /tmp/evidence.png                        # PR evidence
```

**Key rules:**
- Re-snapshot before every interaction (refs go stale). Use `press x y` as coordinate fallback.
- `AGENT_FLUTTER_LOG` must point to flutter run stdout (not logcat).
- `find type X` / `find text "label"` is more stable than `@ref` numbers.
- Add `Key('descriptive_name')` to new interactive widgets for `find key`.
- See `app/e2e/SKILL.md` for navigation architecture, screen map, and known flows.

---

## Desktop (macOS)

### Building & Running
- `./run.sh` — full local dev (build + backend + tunnel + app)
- `./run.sh --yolo` — quick start with prod backend, no local services
- Release builds are handled entirely by Codemagic CI (no local release script)
- Build command: `xcrun swift build -c debug --package-path Desktop` (the `xcrun` prefix is required)
- **DO NOT** use bare `swift build`, `xcodebuild`, or launch from `build/` directly

### Named Test Bundles
When testing a feature or bug fix, **always create a separate named bundle**:
```bash
OMI_APP_NAME="omi-fix-rewind" ./run.sh
```
This installs to `/Applications/omi-fix-rewind.app` with bundle ID `com.omi.omi-fix-rewind`.

**Rules:**
- **ALWAYS prefix with `omi-`** (e.g., `omi-fix-rewind`, `omi-6512-polling`, `omi-vision-test`) so bundles are grouped in `/Applications/`
- NEVER use bare `./run.sh` when testing a specific change — it overwrites "Omi Dev"
- NEVER kill or interfere with "Omi", "Omi Beta" — those are production installs
- Keep app name and bundle suffix identical (e.g., `omi-search.app` → `com.omi.omi-search`)
- Named bundles get their own permissions, auth state, and database
- After building, launch and interact programmatically to confirm it runs — don't stop at compile

### Verifying UI Changes (agent-swift)
After editing Swift UI code, **verify programmatically** via macOS Accessibility API:

```bash
agent-swift connect --bundle-id com.omi.omi-fix-rewind           # connect to named bundle
agent-swift snapshot -i                                           # interactive elements only
agent-swift click @e3                                             # CGEvent click (SwiftUI)
agent-swift press @e3                                             # AXPress (AppKit buttons)
agent-swift fill @e5 "text"                                       # type into field
agent-swift wait text "Settings"                                  # wait for text
agent-swift screenshot /tmp/evidence.png                          # PR evidence
```

**Key rules:**
- Prefer `click` over `press` for SwiftUI (CGEvent triggers NavigationLink; AXPress is AppKit only).
- Re-snapshot before every interaction (refs go stale).
- Always use `snapshot -i` (interactive only) — full snapshots are very verbose.
- `agent-swift doctor` verifies Accessibility permission.
- Dev bundle ID: `com.omi.desktop-dev`. Prod: `com.omi.computer-macos`.
- See `desktop/e2e/SKILL.md` for navigation architecture and known flows.

---

## Computer Control (clicking, typing, screenshots)

For controlling the Mac GUI. Use the **right tool for each job**:

| Task | Tool | Example |
|------|------|---------|
| Click at coordinates | `cliclick` | `cliclick c:X,Y` |
| Screenshots/OCR | `codriver` | `mcp__codriver__desktop_screenshot` (scale: 0.5) |
| Native macOS app testing | `agent-swift` | See Desktop section above |
| Browser automation | `playwright` MCP | Headless, most reliable |
| Existing browser tabs | `claude-in-chrome` | Only when extension connected |

**Workflow:** screenshot (`codriver`) → find target → click (`cliclick c:X,Y`)

**Rules:**
- NEVER try 3+ different click tools for the same action — pick one and commit.
- `codriver` at `scale: 0.5` → multiply coordinates by 2 before clicking.
- Prefer `cliclick` over `automac`/`mac-use-mcp` (coordinate bugs on multi-monitor).

---

## Formatting
<!-- Maintainers: @Thinh (Jan 19) -->

The pre-commit hook auto-formats, but you can run manually:

| Language | Command |
|----------|---------|
| Dart (`app/`) | `dart format --line-length 120 <files>` |
| Python (`backend/`) | `black --line-length 120 --skip-string-normalization <files>` |
| C/C++ (firmware) | `clang-format -i <files>` |

Files ending in `.gen.dart` or `.g.dart` are auto-generated — don't format manually.

---

## Git
<!-- Maintainers: @AaravGarg (original, Feb 2), @NikShevchenko (push rules, Mar 3) -->

### Rules
- Always commit to the current branch — never switch branches.
- Never push directly to `main`. Land changes through PRs only.
- Never squash merge PRs — use regular merge.
- Make individual commits per file, not bulk commits.
- If push fails (remote ahead): `git pull --rebase && git push`.
- Never push or create PRs unless explicitly asked — commit locally by default.
- Always work in a git worktree for code changes. Use `EnterWorktree` to isolate work.

### RELEASE Command
Create branch from `main`, individual commits per file, push/create PR, merge without squash, switch back to `main` and pull.

### RELEASEWITHBACKEND Command
Full RELEASE flow + `gh workflow run gcp_backend.yml -f environment=prod -f branch=main`.

---

## Testing
Run `backend/test-preflight.sh` to verify environment. Run `backend/test.sh` (backend) or `app/test.sh` (app) before committing.

## CI/CD
See [docs/runbooks/deploy.md](docs/runbooks/deploy.md) for deploy triggers and checks.

## Logs
See [docs/runbooks/logging.md](docs/runbooks/logging.md) for log commands.

## Documentation Maintenance
- If a PR changes setup, test commands, safety rules, service boundaries, or env vars — update this file in the same PR.
- Keep `AGENTS.md` synced with this file. Update both in the same commit.
- For architecture/core flow/API changes — update Mintlify docs (`docs/`) in the same PR.
- If a PR changes audio streaming, transcription, conversation lifecycle, or listen/pusher WebSocket — update `docs/doc/developer/backend/listen_pusher_pipeline.mdx`.


## README.md

<div align="center">

# **omi**

### A 2nd brain you trust more than your 1st

Omi captures your screen and conversations, transcribes in real-time, generates summaries and action items, and gives you an AI chat that remembers everything you've seen and heard. Works on desktop, phone and wearables. Fully open source.

Trusted by 300,000+ professionals.


[![Discord](https://img.shields.io/discord/1192313062041067520?label=Discord&logo=discord&logoColor=white&style=for-the-badge)](http://discord.omi.me)&ensp;
[![GitHub Repo stars](https://img.shields.io/github/stars/BasedHardware/Omi?style=for-the-badge)](https://github.com/BasedHardware/Omi)&ensp;
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](https://opensource.org/licenses/MIT)

[Website](https://omi.me/) · [Docs](https://docs.omi.me/) · [Discord](http://discord.omi.me) · [Twitter](https://x.com/kodjima33) · [DeepWiki](https://deepwiki.com/BasedHardware/omi)

</div>

## Quick Start

<p align="center">
  <a href="https://macos.omi.me"><img src="docs/assets/readme/download-macos-badge.png" alt="Download for macOS" height="50"></a>
  <a href="https://apps.apple.com/us/app/friend-ai-wearable/id6502156163"><img src="docs/assets/readme/download-appstore-badge.png" alt="Download on the App Store" height="50"></a>
  <a href="https://play.google.com/store/apps/details?id=com.friend.ios"><img src="docs/assets/readme/download-gplay-badge.png" alt="Get it on Google Play" height="50"></a>
</p>

<p align="center">
  <a href="https://app.omi.me">Try in Browser</a>
</p>

```bash
git clone https://github.com/BasedHardware/omi.git && cd omi/desktop && ./run.sh --yolo
```

Builds the macOS app, connects to the cloud backend, and launches. No env files, no credentials, no local backend.

> **Requirements:** macOS 14+, [Xcode](https://developer.apple.com/xcode/) (includes Swift & code signing), [Node.js](https://nodejs.org/)

<details>
  <summary>Full Installation</summary>
  
For local development with the full backend stack:

1. Install prerequisites

```bash
xcode-select --install
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

2. Clone and configure

```bash
git clone https://github.com/BasedHardware/omi.git
cd omi/desktop
cp Backend-Rust/.env.example Backend-Rust/.env
```

3. Build and run

```bash
./run.sh
```

See [desktop/README.md](desktop/README.md) for environment variables and credential setup.


### Mobile App

```bash
cd app && bash setup.sh ios    # or: bash setup.sh android
```

</details>

<details>
  <summary>How it works</summary>


```
┌─────────────────────────────────────────────────────────┐
│                      Your Devices                       │
│                                                         │
│  ┌──────────┐  ┌──────────────┐  ┌───────────────────┐  │
│  │ Omi      │  │ macOS App    │  │ Mobile App        │  │
│  │ Wearable │  │ (Swift/Rust) │  │ (Flutter)         │  │
│  └────┬─────┘  └──────┬───────┘  └────────┬──────────┘  │
│       │    BLE         │   HTTPS/WS        │             │
└───────┼────────────────┼───────────────────┼─────────────┘
        │                │                   │
        ▼                ▼                   ▼
┌─────────────────────────────────────────────────────────┐
│                    Omi Backend (Python)                  │
│                                                         │
│  ┌─────────┐  ┌──────────┐  ┌─────────┐  ┌──────────┐  │
│  │ Listen  │  │ Pusher   │  │ VAD     │  │ Diarizer │  │
│  │ (REST)  │  │ (WS)     │  │ (GPU)   │  │ (GPU)    │  │
│  └─────────┘  └──────────┘  └─────────┘  └──────────┘  │
│                                                         │
│  ┌─────────┐  ┌──────────┐  ┌─────────┐  ┌──────────┐  │
│  │ Deepgram│  │ Firestore│  │ Redis   │  │ LLMs     │  │
│  │ (STT)   │  │ (DB)     │  │ (Cache) │  │ (AI)     │  │
│  └─────────┘  └──────────┘  └─────────┘  └──────────┘  │
└─────────────────────────────────────────────────────────┘
```

| Component | Path | Stack |
|-----------|------|-------|
| **macOS app** | [`desktop/`](desktop/) | Swift, SwiftUI, Rust backend |
| Mobile app | [`app/`](app/) | Flutter (iOS & Android) |
| Backend API | [`backend/`](backend/) | Python, FastAPI, Firebase |
| Firmware | [`omi/`](omi/) | nRF, Zephyr, C |
| Omi Glass | [`omiGlass/`](omiGlass/) | ESP32-S3, C |
| SDKs | [`sdks/`](sdks/) | React Native, Swift, Python |
| AI Personas | [`web/personas-open-source/`](web/personas-open-source/) | Next.js |

</details>

## Documentation

### Getting Started
- [Introduction](https://docs.omi.me/)
- [Quick Start Guide](https://docs.omi.me/quickstart)
- [macOS App Development](desktop/README.md)
- [Mobile App Setup](https://docs.omi.me/doc/developer/AppSetup)
- [Backend Setup](https://docs.omi.me/doc/developer/backend/Backend_Setup)
- [Contributing](https://docs.omi.me/doc/developer/Contribution)

### Building Apps
- [App Development Guide](https://docs.omi.me/doc/developer/apps/Introduction)
- [Example Apps](https://docs.omi.me/doc/developer/apps/examples/Github) — GitHub, Slack, OmiMentor
- [Audio Streaming Apps](https://docs.omi.me/doc/developer/apps/AudioStreaming)
- [Custom Chat Tools](https://docs.omi.me/doc/developer/apps/ChatTools)
- [Submit to App Store](https://docs.omi.me/doc/developer/apps/Submitting)

### API & SDKs
- [API Reference](https://docs.omi.me/api-reference/introduction) — REST endpoints for memories, conversations, action items
- [Python SDK](sdks/python/)
- [Swift SDK](sdks/swift/)
- [React Native SDK](sdks/react-native/)
- [MCP Server](mcp/) — Model Context Protocol integration

### Architecture
- [Backend Deep Dive](https://docs.omi.me/doc/developer/backend/backend_deepdive)
- [Transcription Pipeline](https://docs.omi.me/doc/developer/backend/transcription)
- [Chat System](https://docs.omi.me/doc/developer/backend/chat_system)
- [Audio Streaming Pipeline](https://docs.omi.me/doc/developer/backend/listen_pusher_pipeline)
- [BLE Protocol](https://docs.omi.me/doc/developer/Protocol)

## Omi Hardware
![Omi](https://github.com/user-attachments/assets/7a658366-9e02-4057-bde5-a510e1f0217a)

Open-source AI wearables that pair with the mobile app for 24h+ continuous capture.

<p align="center">
  <img src="https://github.com/user-attachments/assets/834d3fdb-31b5-4f22-ae35-da3d2b9a8f59" alt="Omi Wearable" width="49%" />
  <img src="https://github.com/user-attachments/assets/fdad4226-e5ce-4c55-b547-9101edfa3203" alt="Omi Glass" width="49%" />
</p>

- [Buy Omi](https://www.omi.me/pages/product)
- [Buy Omi Glass Dev Kit](https://www.omi.me/glass) — ESP32-S3, camera + audio
- [Open Source Hardware Designs](https://docs.omi.me/doc/hardware/consumer/electronics)
- [Buying Guide](https://docs.omi.me/doc/assembly/Buying_Guide)
- [Build the Device](https://docs.omi.me/doc/assembly/Build_the_device)
- [Flash Firmware](https://docs.omi.me/doc/get_started/Flash_device)
- [Integrate Your Wearable](https://docs.omi.me/doc/integrations)
- [Hardware Specs](https://docs.omi.me/doc/hardware/DevKit2)

## License

MIT — see [LICENSE](LICENSE)