# Technical Documentation: a-ghorbani/pocketpal-ai > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/a-ghorbani/pocketpal-ai) ### Overview (`README.md`)
PocketPal AI logo # PocketPal AI **A private AI assistant that runs entirely on your phone.** Chat with language models, give them a voice, and let them use tools — all on-device. No account, no cloud, no internet required. pocketpal.dev · Get the app · Leaderboard · PalsHub · Discussions
[](https://apps.apple.com/us/app/pocketpal-ai/id6502579498) [](https://play.google.com/store/apps/details?id=com.pocketpalai) [](https://github.com/a-ghorbani/pocketpal-ai/releases) [](LICENSE) [](https://github.com/a-ghorbani/pocketpal-ai/stargazers) [](https://github.com/a-ghorbani/pocketpal-ai/issues) [](https://github.com/sponsors/a-ghorbani)
--- ## Why PocketPal AI? Most AI apps are a thin window onto someone else's server — every message you type gets shipped off, logged, and analyzed somewhere you can't see. PocketPal flips that around: **the AI lives on your phone, and your conversations never leave it.** - **🔒 Private by default** — every prompt, response, and document stays on your device. Nothing is uploaded or stored on external servers. - **✈️ Works offline** — download a model once and it just works, with no connection and no account. On a plane, on a trail, anywhere. - **📱 Runs on hardware you already own** — real language models, voices, and tools, tuned to make the most of your phone's CPU, GPU, and NPU. - **🆓 Free and open source** — no subscription, no "pro" tier to unlock the AI. MIT-licensed and built in the open. > **Privacy note:** The only data that ever leaves your device is what you explicitly choose to share — benchmark results (if you opt into the leaderboard) and feedback you submit through the app. ## Contents - [Features](#features) - [Get the app](#get-the-app) - [How it works](#how-it-works) - [Using the app](#using-the-app) - [For developers](#for-developers) - [Contributing](#contributing) - [Roadmap](#roadmap) - [Community & support](#community--support) - [License](#license) ## Features - **🧠 On-device chat** — run GGUF language models (Gemma, Qwen, Phi, Llama, and more) fully offline. - **🗣️ Text-to-speech** — give your assistant a voice with on-device neural TTS (Kokoro and other engines), no cloud calls. - **🎭 Pals** — create personalized assistants with their own model, system prompt, and personality (Assistant and Roleplay types). - **🛍️ [PalsHub](https://palshub.ai/)** — discover and install community Pals, including premium ones via in-app purchase. - **🛠️ Talents & tools** — let capable Pals call built-in tools (calculator, date/time, rich HTML rendering) inside a tool-use loop. - **📥 Hugging Face integration** — search and download GGUF models, including gated ones, directly from the HF Hub with your access token. - **📊 Benchmarking** — measure tokens/sec and memory, and optionally compare on the [AI Phone Leaderboard](https://pocketpal.dev/leaderboard). - **⚡ Hardware acceleration** — CPU, GPU (Metal on iOS, OpenCL/Adreno on Android), and NPU (Qualcomm Hexagon) inference paths, with graceful fallback. - **🌍 Localized** — available in 11 languages, on phones and tablets, including full iPad support. ## Get the app | Platform | | | --- | --- | | **iOS / iPadOS** | [](https://apps.apple.com/us/app/pocketpal-ai/id6502579498) | | **Android** | [](https://play.google.com/store/apps/details?id=com.pocketpalai) | **Three steps to your first chat:** 1. **Install** PocketPal from the App Store or Google Play. 2. **Download a model** — tap the menu (☰) → **Models**, pick one that fits your phone, and download (or add one from Hugging Face). 3. **Load it and start chatting** — that's it, you're running AI fully offline. ## How it works You don't need to know any of this to use PocketPal — but if you're curious how a phone runs real AI offline, here's the short version. PocketPal is a four-layer stack, from the silicon up to the chat UI. Each layer has one job, and the dependency direction is strictly top-down — the JS app talks to native bridges, bridges talk to inference engines, engines target hardware backends.
PocketPal AI on-device stack — UI & Tool Use → Bridging → Engine → Hardware
| Layer | What runs here | | --- | --- | | **UI & Tool Use** | The React Native app (UI via React Native Paper, state via MobX, chat history in WatermelonDB). The **`AgentRunner`** drives each chat turn — streaming tokens, dispatching **Talents** (tools) when the model calls them, and feeding results back for follow-up reasoning. **Pals** are configurable personas; **PalsHub** is the in-app marketplace for sharing and buying them. | | **Bridging** | Native modules that connect JavaScript to the engines. [`llama.rn`](https://github.com/mybigday/llama.rn) bridges LLM inference over JSI; [`react-native-speech`](https://github.com/a-ghorbani/react-native-speech) and `onnxruntime-react-native` bridge text-to-speech. | | **Engine** | The inference engines. **llama.cpp** runs language models in the quantized **GGUF** format. **ONNX Runtime** runs TTS voice models in the **ONNX** format. | | **Hardware** | Where the math actually happens. PocketPal targets **CPU** (universal fallback), **GPU** (Metal on iOS, OpenCL on Qualcomm Adreno for Android), and **NPU** (Qualcomm Hexagon) — falling back gracefully and offloading partial layers when a full backend isn't available. | ## Using the app
📥 Download & load a model
1. Open the app and tap the **Menu** (☰), then go to **Models**. 2. Pick a model from the list and tap **Download**, or tap **+** to add one from Hugging Face or local storage. 3. From Hugging Face, search GGUF models and choose a quantization that fits your device's memory and storage — download now or bookmark for later. 4. After downloading, tap **Load** (or use the chevron icon left of the chat input to load right from the chat screen). Download Models
💬 Chat
1. Make sure a model is loaded. 2. Open the **Chat** page and start talking. 3. The screen stays awake during inference and deactivates when idle. 4. **Copy** a full response with the copy icon, or long-press a paragraph to copy just that. 5. **Edit** any of your messages with a long-press — the AI regenerates from your change. Hit **retry** for a fresh answer, optionally with a different model. Chat
🎭 Pals & PalsHub
Create personalized assistants: - **Assistant Pal** — pick a default model, set a system prompt (write it yourself or have the app generate one), and customize the chat input color. - **Roleplay Pal** — everything above, plus location, the AI's role, and other contextual parameters. Switch personas with the Pal picker on the chat page. Browse **[PalsHub](https://palshub.ai/)** in-app to discover community Pals, including premium ones via in-app purchase. Assistant Pal

Creating a cocktail-recipe assistant

📊 Benchmark your device
1. Open the **Benchmark** page. 2. Run performance tests to compare speed and efficiency across models. 3. Review tokens/sec and memory usage. 4. Optionally share your results to the [AI Phone Leaderboard](https://pocketpal.dev/leaderboard). Benchmark
🔑 Set up a Hugging Face token (for gated models)
1. Create an access token in your Hugging Face account ([docs](https://huggingface.co/docs/hub/en/security-tokens)). 2. In PocketPal, go to **Settings → Set Token**, paste it, and save. Token setup
💌 Send feedback
Go to **App Info → "Sharing your thoughts"**, type your feedback — feature requests, suggestions, anything — and submit. Send feedback
## For developers PocketPal is a standard React Native app. If you can build a React Native project, you can build PocketPal. ### Prerequisites - **Node.js** — version is pinned in [`.nvmrc`](.nvmrc) (currently `22.21.0`); run `nvm use` to match it. Older Node will fail the `engines` check. - **Yarn 1 (Classic)** — `packageManager` is pinned to `yarn@1.22.22`. - **Xcode** + **CocoaPods**, and **Ruby + Bundler** (for iOS / Fastlane tooling). - **Android Studio** + Android SDK/NDK. See the [React Native environment setup](https://reactnative.dev/docs/set-up-your-environment) for platform details. ### Clone, install & run ```bash git clone https://github.com/a-ghorbani/pocketpal-ai cd pocketpal-ai nvm use # match the pinned Node version yarn install # install JS dependencies (cd ios && pod install) # iOS only yarn start # Metro bundler yarn ios # build + run on iOS simulator yarn android # build + run on Android emulator ``` Core on-device chat works without any backend keys; only PalsHub/auth features need additional configuration. > **Native-change rule:** if you change `package.json`, a native module, `ios/`, `android/`, the Podfile, or `build.gradle`, re-run `pod install` and rebuild both platforms — a JS reload won't pick up native changes. ### Quality gates ```bash yarn lint # ESLint yarn typecheck # tsc --noEmit yarn test # Jest yarn l10n:validate # validate locale JSON (placeholders, integrity) yarn test:perf # render-performance tests (Reassure) yarn test:perf:compare [base] # render counts of HEAD vs. base (default: merge-base with origin/main) ``` Run `yarn lint && yarn typecheck && yarn test` before opening a PR. Pull requests also run the render-count comparison; see [render-performance tests](docs/render-perf-tests.md). Commits are validated by Commitlint ([Conventional Commits](https://www.conventionalcommits.org/)) via a Husky hook.
Repository layout
``` src/ ├── screens/ # Chat, Models, Pals, Benchmark, Settings, About, … ├── components/ # Reusable UI ├── store/ # MobX stores (Model, ChatSession, Pal, TTS, HF, Benchmark, …) ├── services/ │ ├── agent/ # AgentRunner — the chat / tool loop │ ├── talents/ # Tool engines + registries │ ├── tts/ # TTS engines (kokoro, kitten, supertonic, system) │ ├── palshub/ # PalsHub marketplace integration │ └── downloads/ # Model download manager ├── database/ # WatermelonDB schema, models, migrations ├── repositories/ # Data-access layer over the DB ├── locales/ # i18n JSON + lazy loader (index.ts is the registry) └── hooks/ api/ theme/ utils/ config/ specs/ ```
Tech stack
Versions are pinned in [`package.json`](package.json); the highlights: | Area | Choice | | --- | --- | | Framework | React Native `0.82.1`, React `19.1.1` (New Architecture) | | Language | TypeScript `5.0.4` | | UI | React Native Paper `5.14.5`, React Navigation | | State | MobX `6` (`mobx`, `mobx-react`, `mobx-persist-store`) | | Persistence | WatermelonDB (chat history), AsyncStorage (settings), Keychain (secrets) | | LLM | `llama.rn` `0.13.0-rc.7` → llama.cpp b11385 · GGUF | | TTS | `react-native-speech` `2.3.1` + `onnxruntime-react-native` `1.23.2` · ONNX | | Tooling | Yarn 1 (Classic), ESLint, Prettier, Jest, Husky + Commitlint |
Extending PocketPal
A **Talent** is a tool the model can call mid-conversation. Engines are registered in a `TalentRegistry`, exposed to the model as tool schemas; the `AgentRunner` detects a call, runs the engine, and returns the result for the next turn. | Talent | Engine | Does | | --- | --- | --- | | `calculate` | `CalculateEngine` | Arithmetic / expression evaluation | | `datetime` | `DatetimeEngine` | Current date / time | | `render_html` | `RenderHtmlEngine` | Renders model-produced HTML in chat | Good first contributions: - A new **Talent** — implement a `TalentEngine` and register it in `src/services/talents/`. - A new **TTS engine** — add it under `src/services/tts/engines/`. - A new **locale** — add a JSON file in `src/locales/` (or translate on [Weblate](https://hosted.weblate.org/projects/pocketpal-ai/)).
## Contributing Contributions are welcome — bug reports, fixes, features, translations, and docs all help. 1. Fork and branch: `git checkout -b feature/your-feature-name` 2. Make your changes; run on a device/emulator (`yarn ios` / `yarn android`). Re-run `pod install` + rebuild if you touched native code. 3. Gate locally: `yarn lint && yarn typecheck && yarn test` 4. Commit with [Conventional Commits](https://www.conventionalcommits.org/): `git commit -m "feat: add new talent"` 5. Push and open a pull request. Please read the [Contributing Guidelines](CONTRIBUTING.md) and [Code of Conduct](CODE_OF_CONDUCT.md) first. Want to translate PocketPal into your language? Join us on [Weblate](https://hosted.weblate.org/projects/pocketpal-ai/). ## Roadmap - **Tool use expansion** — grow the Talents catalog and deepen the agentic loop so Pals can do more, fully on-device. Have an idea or found a bug? [Open an issue](https://github.com/a-ghorbani/pocketpal-ai/issues/new/choose) or start a [discussion](https://github.com/a-ghorbani/pocketpal-ai/discussions). ## Community & support - 💬 **Questions & ideas** — [GitHub Discussions](https://github.com/a-ghorbani/pocketpal-ai/discussions) - 🐛 **Bugs & requests** — [GitHub Issues](https://github.com/a-ghorbani/pocketpal-ai/issues/new/choose) - 🌐 **Website** — [pocketpal.dev](https://pocketpal.dev/) - ❤️ **Support development** — PocketPal is free and ad-free; [sponsoring](https://github.com/sponsors/a-ghorbani) helps keep it that way. ## License Licensed under the [MIT License](LICENSE). ## Acknowledgements PocketPal AI stands on the shoulders of the open-source community, including: - **[llama.cpp](https://github.com/ggerganov/llama.cpp)** — efficient on-device LLM inference. - **[llama.rn](https://github.com/mybigday/llama.rn)** — llama.cpp bindings for React Native. - **[react-native-speech](https://github.com/a-ghorbani/react-native-speech)** — React Native TTS bridge powering on-device voices. - **[ONNX Runtime](https://onnxruntime.ai/)** — cross-platform inference engine powering on-device TTS. - **[React Native](https://reactnative.dev/)**, **[MobX](https://mobx.js.org/)**, **[React Native Paper](https://callstack.github.io/react-native-paper/)**, **[React Navigation](https://reactnavigation.org/)**, **[WatermelonDB](https://github.com/Nozbe/WatermelonDB)**, and many other open-source libraries that make this project possible.

Made with ❤️ for people who want AI that stays on their phone.
If PocketPal is useful to you, consider giving it a ⭐ — it helps others find the project.
--- ### Overview (`e2e/README.md`) # PocketPal E2E Tests End-to-end tests using Appium + WebDriverIO for local devices and AWS Device Farm. ## Setup ```bash cd e2e yarn install ``` ## Test Specs | Spec | What it tests | Duration | |------|---------------|----------| | `quick-smoke` | Full user journey: navigate to Models → search HuggingFace → download SmolLM2-135M → load model → chat → verify inference completes | ~50-70s/device | | `load-stress` | Download model, run multiple load/unload cycles with inference between each. Catches crash-on-reload bugs | ~5-10 min/device | | `thinking` | Loads Qwen3-0.6B (thinking model), verifies thinking toggle, thinking bubble appears, toggle off suppresses it | ~3-5 min/device | | `diagnostic` | Dumps Appium page source XML at each screen. For debugging selectors, not a real test | ~10s | | `benchmark-matrix` | Iterates {models} × {quants} × {backends} on Android, writes canonical JSON report per run. iOS runs through `yarn bench:ios` instead. Measurement infrastructure, not an automated gate. | ~25-45 min | ## Benchmark Matrix Runner Drives the in-app **BenchmarkRunnerScreen** via deep link (`pocketpal://e2e/benchmark`) — no WDIO required for ad-hoc runs. Three tiers gated by `BENCH_TIER`: | Tier | Models × quants × backends | Cells | Runtime | When | |------|----------------------------|------:|---------|------| | `smoke` (default) | 3 × 3 × 2 | 18 | ~10–15 min | Regression gate | | `focused` | 6 × 6 × 2 | ~60 | ~30–45 min | Investigation | | `full` | 11 × 8 × 2 | ~165 | ~3 hr/device | Default-tier recalibration | Model + quant rosters live in [`fixtures/benchmark-models.ts`](fixtures/benchmark-models.ts) (single source: `BENCHMARK_FULL_MODELS`; smaller tiers derived as id filters). **Android one-shot run** (assumes E2E-flavor APK is already installed on the device): ```bash # 1. Generate config + adb push to device BENCH_TIER=smoke yarn build:bench-config --push # default device BENCH_TIER=full yarn build:bench-config --push # specific device # 2. Cold-launch the runner via deep link adb shell am start -a android.intent.action.VIEW \ -d 'pocketpal://e2e/benchmark' -p com.pocketpalai.e2e # 3. Tap "Run benchmark matrix" on the screen (or via adb input tap) # 4. Wait for `bench-runner-screen-status` content-desc == "complete" # 5. Pull the report adb pull /sdcard/Android/data/com.pocketpalai.e2e/files/benchmark-report-*.json ``` Filters narrow the chosen tier (cannot widen it): ```bash BENCH_TIER=full BENCH_MODELS=qwen3.5-2b BENCH_QUANTS=q4_0,q6_k yarn build:bench-config ``` Heavy models (Phi-3.5, Phi-4-mini, Gemma-4-E2B) are last in the `full` tier; if the OS ANR-kills the app on a heavy CPU bench, partial-row data from earlier cells survives in the JSON report. ### iOS (physical iPhone) `yarn bench:ios` drives the same matrix with `xcrun devicectl` only (no Appium). It pushes `bench-config.json` into the app's `Documents`, sends the deep link to the already-running app (a link on cold launch is dropped), polls `Documents` for the new `benchmark-report-*.json` until the runner writes its terminal `outcome`, then pulls, stamps and pass-gates the report. > **`--app` replaces the App Store PocketPal.** iOS has no separate E2E bundle id, so installing the E2E build over `ai.pocketpal` wipes that app's data. Without `--app` the driver uses whatever is installed, which must be an E2E build. ```bash # 1. Build the E2E IPA (development signing) -> ios/build/PocketPal.ipa yarn ios:build:ipa # 2. Preview: writes the config, prints the cell count and the devicectl plan, touches no device BENCH_TIER=smoke yarn bench:ios --device --dry-run # 3. Install and run (udid from `xcrun devicectl list devices`) BENCH_TIER=smoke yarn bench:ios --device --app ../ios/build/PocketPal.ipa ``` Keep the iPhone unlocked and on power, and do not lock it or switch apps during a run: a suspended app stops writing rows and the run ends `failed:timeout`. The runner holds the screen awake itself, so Auto-Lock need not be changed. Reports land in `e2e/debug-output/benchmarks/` (`--out` to change). Extra env: `BENCH_MAX_WAIT_MIN` (default 60, enough for `smoke` and `focused` at the tier-table runtimes; `full` takes ~3 h on Android and is not yet timed on iOS, so run it with `BENCH_MAX_WAIT_MIN=240`), `BENCH_IOS_SETTLE_S` (15), `BENCH_IOS_START_TIMEOUT_S` (120), and `E2E_DEVICE_NAME` / `E2E_PLATFORM_VERSION` / `E2E_DEVICE_SOC` to override the stamped metadata. See `yarn bench:ios --help`. A timeout names its cause: with no new row for 15 min it blames a lock or backgrounding, and with rows still arriving it reports rows seen against expected and asks for a larger `BENCH_MAX_WAIT_MIN`. Baselines are per platform: `merge-bench-reports.ts` refuses to mix Android and iOS reports, and `benchmark-compare.ts` exits 2 on a cross-platform pair. ### Baselines (`baselines/benchmark/.json`) Committed reference data the [`benchmark-compare.ts`](scripts/benchmark-compare.ts) regression checker runs against. One file per device class. Refresh by re-running the `full` tier on that device, then merging the captured reports: ```bash npx ts-node scripts/merge-bench-reports.ts \ --input '/path/to/benchmark-report-*.json' \ --out baselines/benchmark/.json \ --device 'POCO X9 Pro Myron' \ --soc 'Snapdragon 8 Elite Gen 5 / Adreno 840' \ --commit "$(git rev-parse --short HEAD)" \ --llama-rn-version "$(node -p 'require(\"../package.json\").dependencies[\"llama.rn\"]')" \ --drop-models lfm2-1.2b ``` The merger dedupes across multiple raw reports (latest run per `model_id × quant × backend` wins), drops stale model ids, sorts deterministically, and strips the debug-only `log_signals.raw_matches` so baseline diffs stay focused on perf deltas. Memory-profile baselines live in the sibling [`baselines/memory/`](baselines/memory/) directory. ## Local Testing ### Prerequisites - Xcode configured (for iOS) - Android SDK configured (for Android) - Build the app first (see below) ### Build ```bash # iOS simulator yarn ios:build:e2e # iOS real device (IPA, requires code signing) yarn ios:build:ipa # Android E2E APK (required — prod APK has no automation bridge) yarn android:build:e2e # Installs as com.pocketpalai.e2e and coexists side-by-side with the # prod install (com.pocketpalai). E2E_BUILD=true is set automatically # so the automation bridge (src/__automation__/) ships in this APK. ``` **Flavor.** E2E targets the `e2e` flavor (`com.pocketpalai.e2e`, debuggable), which ships the automation bridge. The `prod` flavor has no bridge — specs will silently fail there. **Firebase.** `android/app/google-services.json` (gitignored) must contain client entries for both `com.pocketpalai` and `com.pocketpalai.e2e`. If the build fails with a google-services plugin error, the `.e2e` client entry is missing from your local copy. ### Unified E2E Runner All local test execution goes through a single `yarn e2e` command: ```bash # Simple smoke test on default device yarn e2e:ios --spec quick-smoke yarn e2e:android --spec quick-smoke # Test each model in isolation (one WDIO process per model) yarn e2e:ios --each-model yarn e2e:ios --each-model --models smollm2-135m,qwen3-0.6b # Crash reproduction (load-stress on a specific model) yarn e2e --platform ios --spec load-stress --models gemma-2-2b # Multi-device pipeline (iterate across devices from devices.json) yarn e2e:ios --each-device yarn e2e:ios --devices virtual-only --skip-build # Run on whatever real devices are currently plugged in yarn e2e:android --devices connected --skip-build # Full matrix: every model x every device yarn e2e:ios --each-device --each-model # Include crash-repro models in the pool yarn e2e:ios --each-model --all-models # Dry run (preview what would execute) yarn e2e --platform both --each-device --each-model --dry-run # List available models yarn e2e --list-models ``` ### Flags | Flag | Values | Default | Description | |------|--------|---------|-------------| | `--platform` | `ios`, `android`, `both` | _(required)_ | Which platform(s) to test | | `--spec` | `quick-smoke`, `load-stress`, `diagnostic`, `language`, `all` | `quick-smoke` | Which test spec to run | | `--models` | comma-separated model IDs | _(all)_ | Specific model(s) to test | | `--each-model` | _(flag)_ | off | Iterate spec once per model (isolated process) | | `--all-models` | _(flag)_ | off | Include crash-repro models in the pool | | `--devices` | `all`, `virtual-only`, `real-only`, `connected`, or comma-separated IDs | `all` | Device filter (implies `--each-device`) | | `--each-device` | _(flag)_ | off | Iterate across devices from `devices.json` | | `--mode` | `local`, `device-farm` | `local` | Execution mode (switches wdio config) | | `--skip-build` | _(flag)_ | builds by default | Skip app builds, reuse existing | | `--dry-run` | _(flag)_ | off | Print what would run without executing | | `--report-dir` | path | auto-timestamped | Override report output directory | | `--list-models` | _(flag)_ | off | List all available models and exit | ### Direct WDIO Commands For ad-hoc runs where you need to pass WDIO-specific flags, invoke WDIO directly: ```bash npx wdio run wdio.ios.local.conf.ts --spec specs/quick-smoke.spec.ts npx wdio run wdio.android.local.conf.ts --spec specs/load-stress.spec.ts ``` ### Environment Variables (WDIO Configs) Both `wdio.ios.local.conf.ts` and `wdio.android.local.conf.ts` accept these env vars with backward-compatible defaults: | Env Var | iOS Default | Android Default | Purpose | |---------|-------------|-----------------|---------| | `E2E_DEVICE_NAME` | `iPhone 17 Pro` | `emulator-5554` | Device/simulator name | | `E2E_PLATFORM_VERSION` | `26.0` | `16` | OS version | | `E2E_DEVICE_UDID` | _(none)_ | _(none)_ | Device UDID (required for real devices) | | `E2E_APP_PATH` | `../ios/build/.../PocketPal.app` | `../android/.../app-e2e-releaseE2e.apk` | Path to built app | | `E2E_APPIUM_PORT` | `4723` | `4723` | Appium server port | | `E2E_XCODE_ORG_ID` | _(none)_ | N/A | Apple Team ID (required for real iOS devices) | | `E2E_XCODE_SIGNING_ID` | `Apple Development` | N/A | Code signing identity for WDA | ### Multi-Device Setup To use `--each-device`, set up a device inventory: 1. Copy the template: ```bash cp devices.template.json devices.json ``` 2. Edit `devices.json` with your actual devices (simulators, emulators, USB-connected real devices). See `devices.template.json` for the format. **Finding device UDIDs:** ```bash # iOS xcrun xctrace list devices # Android adb devices -l ``` > `devices.json` is gitignored — each machine has its own. ### Reports Each run creates a timestamped directory under `e2e/reports/`: ``` e2e/reports/2026-02-13T16-14-12-758/ summary.json # Overall results + per-run breakdown junit-results.xml # Merged JUnit XML (for CI integration) iphone-17-pro-sim/ # Per-device subdirectory (when --each-device) smollm2-135m/ # Per-model subdirectory (when --each-model) junit-smollm2-135m.xml screenshots/ ``` ## AWS Device Farm Testing ### Prerequisites 1. AWS Account with Device Farm access 2. Create a Device Farm project 3. Set environment variables or GitHub Secrets: - `AWS_ACCESS_KEY_ID` - `AWS_SECRET_ACCESS_KEY` - `AWS_DEVICE_FARM_PROJECT_ARN` ### Run via GitHub Actions 1. Go to Actions → "E2E Tests (AWS Device Farm)" 2. Click "Run workflow" 3. Select platform (android, ios, or both) ### Run manually ```bash yarn e2e:aws --platform android --app path/to/app.apk ``` ## Project Structure ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Writing Tests ### Selectors Use `testID` and `accessibilityLabel` for reliable cross-platform selectors: ```typescript import {Selectors} from '../helpers/selectors'; // By testID await $(Selectors.byTestId('send-button')).click(); // By text (exact match) await $(Selectors.byText('Models')).click(); // By partial text await $(Selectors.byPartialText('Download')).click(); // By accessibility label await $(Selectors.byAccessibilityLabel('Chat input')).click(); ``` ### Page Objects Use page objects for common interactions: ```typescript import {ChatPage, DrawerPage, ModelsPage} from '../pages'; await ChatPage.openDrawer(); await DrawerPage.navigateToModels(); await ModelsPage.openHuggingFaceSearch(); ``` ## Cost Estimation (AWS Device Farm) | Usage | Approximate Cost | |-------|------------------| | 10 min test run, 1 device | ~$1.70 | | 10 min test run, 2 devices (iOS+Android) | ~$3.40 | | 30 runs/month, 2 devices | ~$100/month | Pricing: $0.17 per device minute ## Benchmark Matrix The `benchmark-matrix` spec is **measurement infrastructure**, not an automated gate. It iterates `{models} × {quants} × {backends}` on Android, drives the in-app Benchmark screen for each cell, and writes a canonical JSON report to `e2e/debug-output/benchmarks/benchmark--.json`. The JSON is incremental: a mid-matrix crash preserves completed rows. This spec is Android-only; iOS uses `yarn bench:ios` (see [Benchmark Matrix Runner](#benchmark-matrix-runner)). The matrix is 2 models × 8 quants × 2 backends = 32 runs at full scale; env-var filters reduce this. ### Usage ```bash # Full matrix on the currently connected Android device (~25-45 min) yarn e2e --platform android --spec benchmark-matrix --skip-build # Single cell (smoke) BENCH_MODELS=qwen3-1.7b BENCH_QUANTS=q4_0 BENCH_BACKENDS=cpu \ yarn e2e --platform android --spec benchmark-matrix --skip-build # Preseeded mode (see "Preseed workflow" below) MODELS_PRESEEDED=1 yarn e2e --platform android --spec benchmark-matrix --skip-build ``` ### Environment variables | Var | Values | Description | |-----|--------|-------------| | `BENCH_MODELS` | comma-separated model ids (lowercase) | e.g. `qwen3-1.7b,gemma-3-1b` | | `BENCH_QUANTS` | comma-separated rung labels | e.g. `q4_0,q6_k`; full set: `iq1_s,q2_k,q3_k_m,q4_0,q4_k_m,q5_k_m,q6_k,q8_0` | | `BENCH_BACKENDS` | comma-separated tiers | `cpu`, `gpu` | | `MODELS_PRESEEDED` | `1` to enable | Skip downloads; use already-pushed GGUFs on device | | `E2E_DEVICE_SOC` | free-form string | Recorded in the JSON `soc` field; not used to drive tests | ### JSON schema Top-level: ```jsonc { "version": "1.0", "device": "SM-S948U", "soc": "Snapdragon 8 Elite Gen 2", // or null "commit": "abc1234", "llama_rn_version": "0.12.0-rc.8", "platform": "android", "os_version": "16", "timestamp": "2026-04-21T…", "preseeded": false, "runs": [ /* BenchmarkRun[] */ ] } ``` Per-run (`BenchmarkRun`): ```jsonc { "model_id": "qwen3-1.7b", "quant": "q4_0", // canonical lowercase rung label "requested_backend": "cpu", // "cpu" | "gpu" "effective_backend": "cpu", // see below "pp_avg": 123.4, // tokens/s, nullable "tg_avg": 18.2, // tokens/s, nullable "wall_ms": 24571, "peak_memory_mb": 812.3, // nullable "log_signals": { // structured — see src/__automation__/logSignals.ts "opencl_init": true, "opencl_device_name": "QUALCOMM Adreno(TM) 840", "adreno_gen": "A8X", "large_buffer_enabled": true, "large_buffer_unsupported": false, "offloaded_layers": 29, "total_layers": 29, "raw_matches": [ /* up to 200 matched native-log lines, debug only */ ] }, "init_settings": { /* modelStore.contextInitParams snapshot */ }, "status": "ok", // "ok" | "skipped" | "failed" "reason": "…", // set on skipped "error": "…", // set on failed (first 500 chars) "timestamp": "2026-04-21T…" } ``` ### Interpreting `effective_backend` Derived from the structured `log_signals` payload, not regex on raw text: | Value | Meaning | |-------|---------| | `cpu` | No OpenCL init observed — pure CPU path. | | `opencl` | OpenCL initialised, all layers offloaded to GPU, no large-buffer regression. | | `cpu+opencl-partial` | OpenCL initialised but some layers ran on CPU, or `large_buffer_unsupported` triggered a fallback. | | `metal` | Weights on a Metal (`MTL*`) buffer with every layer offloaded (iOS). | | `cpu+metal-partial` | Metal buffer present but fewer layers offloaded than the model has. | | `unknown` | OpenCL initialised but layer counts absent — investigate `log_signals.raw_matches`. | A row where `requested_backend=gpu` but `effective_backend=cpu` is the canonical "silent CPU fallback" we want to catch. The comparison script flags this as a regression even when `pp_avg` / `tg_avg` numbers look fine. ### Preseed workflow (E2E flavor required) Preseeded mode skips all HuggingFace downloads and loads GGUFs that have already been pushed to the device's app-private storage. This is the fast path once you've downloaded each rung once. **Precondition: the app must be the E2E flavor.** The prod `release` APK is non-debuggable, so `adb shell run-as com.pocketpalai` and `adb push` into `/data/data/com.pocketpalai/files/…` will not work. The `e2e` flavor + `releaseE2e` buildType is debuggable by design (same Hermes/release optimizer as prod, just with `debuggable=true` flipped on). Build and install it: ```bash yarn android:build:e2e adb install -r android/app/build/outputs/apk/e2e/releaseE2e/app-e2e-releaseE2e.apk ``` On-device path (matches `ModelStore.getModelFullPath`, with the E2E flavor's applicationId): ``` /data/data/com.pocketpalai.e2e/files/models/hf///.gguf ``` Push each GGUF once: ```bash adb shell run-as com.pocketpalai.e2e mkdir -p \ files/models/hf/bartowski/Qwen_Qwen3-1.7B-GGUF # copy via /data/local/tmp to avoid run-as's stdin limitations: adb push Qwen_Qwen3-1.7B-Q4_0.gguf /data/local/tmp/ adb shell run-as com.pocketpalai.e2e sh -c \ 'cat /data/local/tmp/Qwen_Qwen3-1.7B-Q4_0.gguf > \ files/models/hf/bartowski/Qwen_Qwen3-1.7B-GGUF/Qwen_Qwen3-1.7B-Q4_0.gguf' ``` Then run with `MODELS_PRESEEDED=1`. The spec fails fast (before touching the matrix loop) with a per-file `adb push` template if anything is missing — no silent download fallback. ### Comparing two reports ```bash npx tsx e2e/scripts/benchmark-compare.ts \ path/to/baseline.json path/to/current.json ``` Flags rows where either `pp_avg` or `tg_avg` delta exceeds `|delta%| > 15` (override with `--pct N`). Additional flags, all independent of numeric deltas: - `effective_backend:->` — silent backend fallback (e.g. requested GPU but ran on CPU). - `status_regression()` — baseline row was `ok`, current row flipped to `failed`/`skipped`. - `pp_null_regression` / `tg_null_regression` — both rows claim `status:'ok'` but the current numeric metric is `null` while the baseline was numeric (catches partial native failures the screen didn't reject as failed). - Top-level `bench_protocol_mismatch` — the persisted `bench` block (`pp`/`tg`/`pl`/`nr`) differs between reports. Comparison is skipped (`pass:false`) and the CLI exits 2 because pp/tg numbers are not comparable across protocols. Reports that omit the `bench` block (e.g. legacy baselines pre-v1.1) skip the check with a stderr warning. `pass` is `true` only when no row is flagged AND `missing_in_current` is empty AND no `bench_protocol_mismatch`. Exit codes: 0 pass, 1 regression, 2 input/protocol error. ### Known limitations (v1) - `yarn bench:ios` drives physical iPhones only: `devicectl` cannot target a simulator. - On an iOS simulator the in-app runner reports GPU cells as `cpu`, so they fail `backend-mismatch`. - `yarn bench:ios --app` replaces the App Store build and wipes its data. - Hexagon NPU tier excluded (Android only; no iOS equivalent). - Preseed requires the E2E-flavor APK (see above). - Static IQ1_S rung is substituted with IQ2_M for Qwen3 1.7B and Gemma 3 1B — neither is published at IQ1_S by bartowski or lmstudio-community. The canonical rung label in the JSON remains `iq1_s` so reports are comparable when IQ1_S eventually ships. - LFM2 1.2B slot 3 is deferred: no publisher has a complete 8-quant set. --- ### Overview (`src/__automation__/README.md`) # `src/__automation__/` — E2E Automation Bridge Single home for all code that exists ONLY to support E2E automation (Appium/WebDriverIO specs). Every file in this folder MUST be dead-code-eliminated from the production bundle. ## Build-time contract The folder is gated behind the `__E2E__` global, inlined at build time by `babel-plugin-transform-define`: - `E2E_BUILD=true` in the build env → `__E2E__ = true` → everything here ships. - Everything else (default dev + prod) → `__E2E__ = false` → Metro/Hermes strips the entire gate body (and, transitively, the imports it reaches). The Android `e2e` flavor (`applicationId com.pocketpalai.e2e`, buildType `releaseE2e`) is the production-like build that sets `E2E_BUILD=true`. The default prod flavor (`com.pocketpalai`) leaves `E2E_BUILD` unset. Two independent guardrails enforce the contract: 1. **ESLint `no-restricted-imports`** — in `.eslintrc.js`, files outside this folder may not import from `src/__automation__/...` EXCEPT for the allow-listed entry points (`App.tsx`, `src/hooks/useDeepLinking.ts`). 2. **CI bundle-grep** — in `.github/workflows/ci.yml`, the `build-android` job extracts `assets/index.android.bundle` from the prod APK and fails the build if any of the automation marker strings (`AUTOMATION_BRIDGE`, `memory-snapshot-label`, `memory-snapshot-result`, `BENCH_RUN_MATRIX`, `bench-runner-screen-status`, `IAP_FAKE_STORE`, `IAP_API_BASE_OVERRIDE`) appear. The ground truth for DCE. > **Note on the static-import question.** Some readers worry that `App.tsx` > statically importing from `./src/__automation__/index.ts` (e.g. for > `` and the deep-link Drawer screen) leaks the entire > automation module graph into the prod bundle, since ES imports enter the > Metro graph regardless of whether the JSX render sites are reachable. > > The DCE contract here does NOT rest on import-site purity. It rests on the > **bundle-grep gate** in `.github/workflows/ci.yml` (the > `Verify prod APK has no automation-bridge code` step at lines 202-233). > That step extracts `assets/index.android.bundle` from the prod APK and > fails the build if any of the registered marker strings > (`AUTOMATION_BRIDGE`, `memory-snapshot-label`, `memory-snapshot-result`, > `BENCH_RUN_MATRIX`, `bench-runner-screen-status`, `IAP_FAKE_STORE`, > `IAP_API_BASE_OVERRIDE`) appear. Hermes' > constant-folding + tree-shaking on `__E2E__ === false` is what actually > removes the code; the grep is the empirical contract that proves it for > every prod build. > > When adding a new automation surface, register a new marker (a literal > string referenced from inside the runtime code, never just JSDoc — Hermes > can DCE pure-comment literals) and add it to the markers list in `ci.yml`. > Then trust the gate, not the import site. ## Adding a new adapter 1. Create `src/__automation__/adapters/FooAdapter.tsx` — a functional component that renders a hidden, accessibility-tree-friendly surface (see `MemoryAdapter.tsx` as reference). Hidden adapters are appropriate when the spec needs to instrument the regular UX flow at lifecycle moments (memory profiling is the canonical case). 2. Render the adapter inside `AutomationBridge.tsx`: ```tsx <> ``` 3. Add a test at `adapters/__tests__/FooAdapter.test.tsx` that covers the command protocol and the testID contract. 4. Update the CI bundle-grep markers in `.github/workflows/ci.yml` if your adapter introduces new protocol strings. For tasks that are synthetic harnesses (don't need to instrument the real UX flow), prefer adding a dedicated screen under `screens/` — mounted as an `__E2E__`-gated `Drawer.Screen` in `App.tsx` and reachable by deep link only. See `BenchmarkRunnerScreen.tsx` as reference. ## Current adapters | Adapter | Purpose | Commands | |---------|---------|----------| | `MemoryAdapter` | Memory profile snapshots for the `memory-profile` spec | `snap::