openscreen Documentation

Create stunning demos for free. Open-source, no subscriptions, no watermarks, and free for commercial use. An alternative to Screen Studio.

35,503 stars TypeScript TypeScript 1 file ยท ~13,195 tokens #electron#open-source#pixijs#screen-capture#screen-recorder
RAW Doc

File: README.md

[!Note]
OpenScreen is now archived and no longer maintained. For continued maintenance and development, a community-driven spin-off led by one of the core contributors is available here: https://github.com/EtienneLescot/openscreen

[!WARNING]
This started as a side project that blew up; not production grade and you'll hit bugs, but hopefully it covers what you need. This project will soon be archived.

If you don't want to pay $29/month for Screen Studio but want a version that does what most people seem to need - quick, polished product demos and walkthroughs you'd post on X, Reddit or Youtube. OpenScreen does not offer every Screen Studio feature, but covers a lot of the core functionality.

Screen Studio is an awesome product and this is definitely not a 1:1 clone. If you just want something fully free and open source, this project should cover most of your needs.

100% free for both personal and commercial use. Use it, modify it, distribute it. Please respect the License.

[!NOTE]
Software should be accessible. OpenScreen has no paid tiers, premium features, upsells, or functionality locked behind a paywall.

Core Features

  • Record a specific window, or your whole screen.
  • Record microphone and system audio.
  • Webcam overlay with picture-in-picture, drag-to-position, mirroring, and shape options.
  • Auto or manual zooms with adjustable depth, duration, easing, and pixel-precise position; auto-zoom follows your cursor as you work.
  • Custom cursor size, smoothing, and click effects, with cursor themes and post-recording path smoothing.
  • Automatic captions for voiceovers, generated on-device with no upload (works offline).
  • Wallpapers, solid colors, gradients, or your own background image.
  • Motion blur.
  • Crop, trim, and per-segment speed control on the timeline.
  • Text, arrow, and image annotations, with text animation presets.
  • Timeline snapping guides and an audio waveform to make trimming easier.
  • Customizable keyboard shortcuts.
  • Export to MP4 or GIF in multiple aspect ratios and resolutions.
  • Languages supported: Arabic, English, Spanish, French, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Turkish, Vietnamese, Simplified Chinese, and Traditional Chinese.

Installation

Download the latest installer for your platform from the GitHub Releases page.

macOS

The easiest way to install on macOS is via Homebrew:

bash
brew install --cask siddharthvaddem/openscreen/openscreen

Brew automatically picks the right build for Apple Silicon or Intel, and verifies the download against a notarized signature so Gatekeeper won't block it.

To update later: brew upgrade --cask openscreen
To uninstall: brew uninstall --cask openscreen (add --zap to also remove app data)

Manual install (if you prefer)

If you'd rather grab the .dmg directly from the Releases page and encounter Gatekeeper blocking the app, you can bypass it by running the following command in your terminal after installation:

bash
xattr -rd com.apple.quarantine /Applications/Openscreen.app

Note: Give your terminal Full Disk Access in System Settings > Privacy & Security to grant you access and then run the above command.

After running this command, proceed to System Preferences > Security & Privacy to grant the necessary permissions for "screen recording" and "accessibility". Once permissions are granted, you can launch the app.

[!NOTE]
Upgrading from an older version and hitting permission issues? If you already had OpenScreen installed and the new version won't record (Screen Recording or Accessibility keep failing even after you grant them), uninstall the old version, remove OpenScreen's existing entries under System Settings > Privacy & Security (both Screen Recording and Accessibility), then do a fresh install and grant the permissions again when prompted.

Windows

Install via winget:

bash
winget install SiddharthVaddem.OpenScreen

To update later: winget upgrade SiddharthVaddem.OpenScreen
To uninstall: winget uninstall SiddharthVaddem.OpenScreen

If you'd rather grab the .exe installer directly, download it from the Releases page.

Linux

Three packages are published to the Releases page for each version. Pick the one that matches your distro:

Debian / Ubuntu / Pop!_OS (.deb)

bash
sudo apt install ./Openscreen-Linux-latest.deb

Arch / Manjaro (.pacman)

bash
sudo pacman -U Openscreen-Linux-latest.pacman

Any distro (.AppImage)

bash
chmod +x Openscreen-Linux-*.AppImage
./Openscreen-Linux-*.AppImage

NixOS / Nix (flake)

Try without installing:

bash
nix run github:siddharthvaddem/openscreen

Install into your user profile:

bash
nix profile install github:siddharthvaddem/openscreen

For a NixOS system config (flake):

nix
{
  inputs.openscreen.url = "github:siddharthvaddem/openscreen";

  outputs = { nixpkgs, openscreen, ... }: {
    nixosConfigurations.<host> = nixpkgs.lib.nixosSystem {
      modules = [
        openscreen.nixosModules.default
        { programs.openscreen.enable = true; }
      ];
    };
  };
}

For Home Manager, use openscreen.homeManagerModules.default with the same programs.openscreen.enable = true;.

You may need to grant screen recording permissions depending on your desktop environment.

Sandbox error: If the AppImage fails to launch with a "sandbox" error, run it with --no-sandbox:

bash
./Openscreen-Linux-*.AppImage --no-sandbox

Platform differences

Everything in the editor and export is the same on macOS, Windows, and Linux: zooms, backgrounds, motion blur, crop/trim/speed, blur regions, annotations, auto-captions, projects, export, and all languages. The differences are in capture, where macOS and Windows use a native pipeline that Linux doesn't have:

  • Native recording: macOS (ScreenCaptureKit) and Windows (Windows Graphics Capture) record through a native pipeline for higher quality and clean window-level capture. Linux records through the browser pipeline instead.
  • Custom cursors: on macOS and Windows the real cursor is captured (shape, type, and clicks), which powers the cursor themes, click effects, and editable cursor overlay. On Linux only the cursor position is captured (used for auto-zoom), so those cursor options aren't available.
  • Webcam: captured natively on macOS and Windows; on Linux it's recorded through the browser, but still works as a picture-in-picture overlay.
  • System audio support varies by OS:
    • macOS: requires macOS 13+. On macOS 14.2+ you'll be prompted to grant audio capture permission. macOS 12 and below can't capture system audio (mic still works).
    • Windows: works out of the box.
    • Linux: needs PipeWire (default on Ubuntu 22.04+, Fedora 34+). Older PulseAudio-only setups may not capture system audio (mic should still work).

License

This project is licensed under the MIT License. By using this software, you agree that the authors are not liable for any issues, damages, or claims arising from its use.


File: docs/testing/macos-native-cursor.md

macOS native cursor test pipeline

This document covers manual and diagnostic testing for macOS native cursor capture โ€” the path that records real system cursor bitmaps via NSCursor.currentSystem and surfaces them through the OpenScreen editor and export pipeline.

How the macOS cursor helper works

The helper binary (openscreen-macos-cursor-helper) runs as a child process of Electron during recording. It:

  • polls NSCursor.currentSystem at the configured sample interval
  • converts each cursor image to PNG and computes a SHA-256 content hash as a stable asset id
  • emits the full base64 bitmap payload once per unique cursor shape per session; subsequent samples carry only the assetId so stdout stays small
  • tracks left-button down/up events via CGEventTap and tags each sample with interactionType
  • uses the Accessibility API to detect text and pointer affordances (link/button/input roles) when Accessibility is granted; these shapes use the bundled high-quality SVG replacements instead of the raw bitmap

Each sample line is newline-delimited JSON:

json
{ "type": "ready", "timestampMs": 1234567890, "accessibilityTrusted": true, "mouseTapReady": true }
{ "type": "sample", "timestampMs": 1234567891, "assetId": "a7472...", "asset": { "id": "a7472...", "imageDataUrl": "data:image/png;base64,...", "width": 64, "height": 64, "hotspotX": 16, "hotspotY": 16, "scaleFactor": 2.0 }, "cursorType": null, "leftButtonDown": false, "leftButtonPressed": false, "leftButtonReleased": false }
{ "type": "sample", "timestampMs": 1234567924, "assetId": "a7472...", "cursorType": null, "leftButtonDown": false, "leftButtonPressed": false, "leftButtonReleased": false }

asset is present only the first time a given assetId appears. The TypeScript session (MacNativeCursorRecordingSession) collects unique assets into a map and sets provider: "native" in the final CursorRecordingData when at least one bitmap was captured.

Build the helper

bash
npm run build:native:mac

This builds both Swift helpers (openscreen-screencapturekit-helper and openscreen-macos-cursor-helper) and copies them to:

  • electron/native/screencapturekit/build/ โ€” used by the local dev server
  • electron/native/bin/darwin-arm64/ or darwin-x64/ โ€” used by packaged builds

Requires Xcode (not just Command Line Tools). If you see a build error about missing SDK metadata, run:

bash
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
sudo xcodebuild -license accept

Smoke-test the helper directly

You can run the cursor helper standalone to inspect its raw output before involving the full app:

bash
BIN=electron/native/screencapturekit/build/openscreen-macos-cursor-helper
("$BIN" '{"sampleIntervalMs":100}' & PID=$!; sleep 2; kill $PID) | head -20

Expected first line:

json
{"type":"ready","mouseTapReady":true,"accessibilityTrusted":false,"timestampMs":...}

accessibilityTrusted: false is normal in dev/unsigned builds. It means text/pointer affordance detection is disabled; native bitmap capture still works.

Expected sample lines:

json
{"type":"sample","assetId":"a7472...","asset":{"id":"a7472...","imageDataUrl":"data:image/png;base64,...","width":64,"height":64,"hotspotX":26,"hotspotY":16,"scaleFactor":2.0},...}
{"type":"sample","assetId":"a7472...",...}

Move the cursor over a text input while the helper is running and check that a new assetId appears with a different bitmap (if Accessibility is granted โ€” see below).

Point the app at a custom helper binary

bash
export OPENSCREEN_MAC_CURSOR_HELPER_EXE=/path/to/openscreen-macos-cursor-helper
npm run dev

macOS permissions

Two separate permissions are needed:

Permission What it enables Where to grant
Screen Recording ScreenCaptureKit video capture System Settings โ†’ Privacy & Security โ†’ Screen & System Audio Recording โ†’ Electron โœ…
Accessibility text / pointer cursor type detection (affordance hints) System Settings โ†’ Privacy & Security โ†’ Accessibility โ†’ Electron โœ…

Screen Recording is required to record. Without it the recording never starts.

Accessibility is optional. Without it, cursorType will always be null and all cursors render from their captured bitmaps (no SVG substitution). This is the expected fallback and does not degrade cursor quality for non-text/pointer shapes.

After granting either permission in System Settings, fully quit and relaunch the dev server โ€” getMediaAccessStatus caches the result per-process.

Manual test checklist

P0 โ€” core bitmap capture

  • Record a short clip. Open the editor. Confirm the default arrow cursor is the real system arrow (not the bundled SVG approximation).
  • Record while hovering over a web browser. Confirm custom-CSS cursors (e.g. cursor: grab, cursor: crosshair) appear as their actual shapes.
  • Export to MP4. Confirm the cursor renders correctly in the exported video.
  • Export to GIF. Same check.

P1 โ€” affordance substitution (requires Accessibility)

  • Grant Accessibility permission and restart the app.
  • Record hovering over a text input field. Confirm the text I-beam uses the bundled SVG version (prettier than the system bitmap).
  • Record hovering over a link/button. Confirm the pointer hand uses the bundled SVG.

P1 โ€” hotspot alignment (Retina)

  • On a Retina display, record a precise click on a small button. In the editor, confirm the cursor tip aligns with the actual click point. The helper reports scaleFactor: 2.0; the renderer divides pixel dimensions and hotspot by this value to recover point sizes.

P1 โ€” click detection

  • Record several left-clicks. In the editor, confirm the click-bounce animation fires on each click.
  • Confirm interactionType: "click" and "mouseup" events are present in the recording session sidecar (cursorRecordingData inside <videoPath>.cursor.json).

P2 โ€” graceful degradation

  • Remove both build-output copies of the helper binary and start a recording. The session should succeed with provider: "none" (position-only telemetry, default arrow rendered). Restore both binaries afterward.
    bash
    ARCH=$([ "$(uname -m)" = "arm64" ] && echo darwin-arm64 || echo darwin-x64)
    mv electron/native/screencapturekit/build/openscreen-macos-cursor-helper /tmp/cursor-helper-build
    mv electron/native/bin/$ARCH/openscreen-macos-cursor-helper /tmp/cursor-helper-bin
    # ... start recording, then restore:
    mv /tmp/cursor-helper-bin electron/native/bin/$ARCH/openscreen-macos-cursor-helper
    mv /tmp/cursor-helper-build electron/native/screencapturekit/build/openscreen-macos-cursor-helper
  • Revoke Accessibility. Confirm recording still works and cursors render from bitmaps (no SVG substitution).

P2 โ€” multi-display

  • Move the cursor to a secondary display during recording. Confirm the cursor clips to the canvas edge rather than snapping invisible on fast swipes. Confirm it hides after โ‰ˆ100 ms of sustained out-of-bounds movement.

P2 โ€” long recording memory

  • Record for 3โ€“5 minutes while switching between many apps (browser, terminal, editor). The helper should not grow in memory because each iteration drains Cocoa objects via autoreleasepool. Check Activity Monitor โ†’ openscreen-macos-cursor-helper RSS stays flat after the first few seconds.

What a healthy recording looks like

Inspect the cursor sidecar file written alongside the recorded video (<videoPath>.cursor.json). For a recording saved to /tmp/rec.mp4, the sidecar is /tmp/rec.mp4.cursor.json:

json
{
  "version": 2,
  "provider": "native",
  "assets": [
    { "id": "a7472...", "platform": "darwin", "imageDataUrl": "data:image/png;base64,...", "width": 64, "height": 64, "hotspotX": 26.0, "hotspotY": 16.0, "scaleFactor": 2.0 }
  ],
  "samples": [
    { "timeMs": 0, "cx": 0.42, "cy": 0.38, "visible": true, "assetId": "a7472...", "interactionType": "move" },
    ...
  ]
}

provider: "native" and a non-empty assets array confirm bitmap capture is active. If you see provider: "none" and assets: [], the helper was not found or exited before ready.

Native macOS capture backend

The app routes macOS recordings through the ScreenCaptureKit helper (openscreen-screencapturekit-helper) when it is available, so the real system cursor is excluded from the video frame. The cursor position and bitmap are captured separately by the cursor helper and composited in the editor and export pipeline.

Current native availability rules:

  • macOS 13 (Ventura) or newer
  • openscreen-screencapturekit-helper binary is present
  • Screen Recording permission is granted

Build both helpers locally:

bash
npm run build:native:mac

For local diagnostics with a custom helper binary, use the environment override:

bash
export OPENSCREEN_MAC_CURSOR_HELPER_EXE=/path/to/openscreen-macos-cursor-helper
npm run dev

Known limitations

  • Intel (x86_64) Macs: the distributed helper is built for darwin-arm64. On Intel Macs, you need to build from source with npm run build:native:mac on the target machine.
  • Accessibility permission in unsigned/dev builds: getMediaAccessStatus("accessibility") may not reflect the toggle state for unsigned Electron in dev mode. The helper will always probe and report accessibilityTrusted in its ready event โ€” use that as the authoritative signal.
  • App-defined custom cursors (CGS layer): NSCursor.currentSystem captures the active AppKit cursor. Cursors set at the CoreGraphics/CGS layer by some games or GPU-accelerated apps may not be visible here. This is a known macOS API limitation.

File: docs/testing/windows-native-cursor.md

Windows native cursor test pipeline

This branch includes two Windows-focused diagnostics for fast iteration on native cursor capture and rendering. They are intentionally local developer tools: they create short videos and JSON reports so cursor changes can be inspected without doing a full manual record/edit/export cycle.

Native sampler diagnostic

powershell
npm run test:cursor-native:win

This script does not launch OpenScreen. It:

  • starts a Windows GetCursorInfo sampler
  • moves the real OS pointer with SetCursorPos
  • captures native cursor handles, hotspots, assets, and standard IDC_* cursor types
  • writes normalized CursorRecordingData
  • generates an abstract preview video
  • generates a real-screen preview video using screenshots of the current desktop

The output directory is printed in the command result, for example:

text
C:\Users\\AppData\Local\Temp\openscreen-cursor-native-...

Useful files:

  • report.json: sample counts, asset counts, cursor handles, and generated artifact paths
  • cursor-recording-data.json: sidecar-compatible cursor data
  • preview.webm: abstract path/asset/hotspot preview
  • real-capture-preview.webm: real desktop screenshot background with reconstructed cursor overlay
  • assets/*.png: raw cursor bitmaps captured from Windows

Environment overrides:

powershell
$env:CURSOR_TEST_DURATION_MS = "3000"
$env:CURSOR_TEST_SAMPLE_INTERVAL_MS = "16"
$env:CURSOR_TEST_SCREEN_FRAME_INTERVAL_MS = "80"
$env:CURSOR_TEST_OUTPUT_DIR = "C:\temp\openscreen-cursor-test"
npm run test:cursor-native:win

OpenScreen preview capture

powershell
npm run capture:openscreen-preview

This script launches the real Electron app, injects a fixture video plus cursor sidecar data, opens the editor, captures frames from the actual OpenScreen preview UI, and encodes them into a WebM.

By default it uses the latest cursor-recording-data.json generated by npm run test:cursor-native:win. To force a specific sidecar:

powershell
$env:CURSOR_RECORDING_DATA_PATH = "C:\path\to\cursor-recording-data.json"
npm run capture:openscreen-preview

Useful environment overrides:

powershell
$env:OPENSCREEN_PREVIEW_SKIP_BUILD = "true"
$env:OPENSCREEN_PREVIEW_FRAME_COUNT = "120"
$env:OPENSCREEN_PREVIEW_FPS = "30"
$env:OPENSCREEN_PREVIEW_OUTPUT_DIR = "C:\temp\openscreen-preview"
npm run capture:openscreen-preview

Useful files:

  • openscreen-preview.webm: video of the real OpenScreen editor preview
  • frames/*.png: captured preview frames
  • report.json: fixture paths, source sidecar, frame count, and output path

What these tests validate

Together, the scripts make it quick to inspect:

  • whether Windows cursor samples are visible and continuous
  • whether native hotspots stay anchored when scaling to 3x
  • whether standard Windows cursors are recognized via IDC_*
  • whether high-quality SVG cursor replacements follow the native hotspot
  • whether the real OpenScreen preview renders the same cursor behavior as the diagnostic pipeline

They are not a full substitute for an end-to-end manual recording pass. Before shipping cursor changes, also test a real capture session and export from the packaged app.

Known Gap

Windows native cursor Click Bounce is currently backlogged. Size, Smoothing, and Motion Blur can be validated through preview/export, but Click Bounce has not shown a visible effect in packaged-app manual testing. The current diagnostic can observe synthetic click metadata, but that is not enough to validate the real OpenScreen record -> preview -> export path.

Track the open item in docs/engineering/windows-native-recorder-roadmap.md under Native Cursor Click Bounce Is Not Visibly Applied.

Native Windows capture backend

The app now routes Windows recordings through an external WGC helper instead of Electron getDisplayMedia. This is meant to remove the coordinate and clock split that made the reconstructed cursor drift in the preview/export path.

Current native availability rules:

  • Windows 10 build 19041 or newer
  • a helper executable is available

The helper currently implements display/window video capture, system audio loopback, default microphone capture, Media Foundation webcam capture, and DirectShow fallback for selected virtual cameras such as NVIDIA Broadcast. Webcam frames are composed into the primary MP4 as a bottom-right picture-in-picture overlay, and black webcam warmup frames are ignored until the first visible frame is available.

Build OpenScreen's helper locally:

powershell
npm run build:native:win

Smoke-test the helper directly:

powershell
npm run test:wgc-helper:win
npm run test:wgc-helper:win -- --capture-cursor
npm run test:wgc-window:win
npm run test:wgc-audio:win
npm run test:wgc-mic:win
npm run test:wgc-mixed-audio:win
npm run test:wgc-webcam:win

For local diagnostics with another compatible helper, point OpenScreen at that executable:

powershell
$env:OPENSCREEN_WGC_CAPTURE_EXE = "C:\path\to\wgc-capture.exe"
npm run build-vite
npm run dev

The helper receives one JSON config argument, emits JSON lifecycle events, prints the legacy Recording started marker, accepts stop on stdin, and prints Recording stopped. Output path: . See electron/native/README.md for the exact contract and build output paths.


File: docs/engineering/macos-native-recorder-roadmap.md

macOS Native Recorder Roadmap

OpenScreen's macOS recorder should follow the same architecture boundaries as the Windows native recorder: Electron owns session orchestration and persistence, while a platform-native helper owns capture, timing, encoding, and platform-specific permissions.

This work is intentionally scoped as a macOS-only port. Windows native capture remains owned by the WGC helper, and Linux remains on the existing Electron path.

Goals

  • Capture displays and windows through ScreenCaptureKit.
  • Exclude the real system cursor during capture when using the editable OpenScreen cursor overlay.
  • Preserve the current high-quality cursor overlay path in preview and export.
  • Capture macOS system audio through ScreenCaptureKit on supported macOS versions.
  • Capture microphone audio through the same native timing domain where the OS supports it, or through an explicit companion path until it can be moved into the helper.
  • Mix system audio and microphone audio into the primary MP4 without renderer-side track assembly.
  • Capture webcam video natively and compose it into the helper-owned MP4 during the native-recording migration.
  • Keep screen video, audio, webcam, and cursor aligned to one native timing origin.
  • Package per-architecture helper binaries with macOS builds.

Non-Goals

  • Replacing the editor/export pipeline.
  • Changing Windows native capture behavior.
  • Adding Linux native capture.
  • Shipping a silent fallback from native macOS capture to Electron capture when the user explicitly requested a native-only feature.

Architecture

The renderer keeps the existing recording controls. On macOS, useScreenRecorder should eventually send a complete recording request to Electron instead of assembling display, audio, microphone, webcam, and cursor streams in the browser.

Electron owns the native recording session:

  • resolves the selected display/window source;
  • resolves output paths;
  • starts cursor telemetry capture when editable cursor mode is selected;
  • starts the ScreenCaptureKit helper process;
  • sends pause/resume/stop/cancel commands;
  • writes RecordingSession manifests;
  • reports explicit errors when a macOS-native capability is unavailable.

The helper owns macOS media capture:

  • ScreenCaptureKit display/window frames;
  • ScreenCaptureKit system audio where supported;
  • microphone capture or helper-owned companion audio capture;
  • webcam capture and initial picture-in-picture composition;
  • AVFoundation/VideoToolbox encoding and muxing;
  • stream timestamp normalization.

Helper Contract V1

The helper receives a single JSON argument:

json
{
  "schemaVersion": 1,
  "recordingId": 1234567890,
  "source": {
    "type": "display",
    "sourceId": "screen:0:0",
    "displayId": 1,
    "windowId": null,
    "bounds": { "x": 0, "y": 0, "width": 1920, "height": 1080 }
  },
  "video": {
    "fps": 60,
    "width": 1920,
    "height": 1080,
    "bitrate": 18000000,
    "hideSystemCursor": true
  },
  "audio": {
    "system": { "enabled": true },
    "microphone": {
      "enabled": true,
      "deviceId": "default",
      "deviceName": "MacBook Pro Microphone",
      "gain": 1.4
    }
  },
  "webcam": {
    "enabled": true,
    "deviceId": "default",
    "deviceName": "FaceTime HD Camera",
    "width": 1280,
    "height": 720,
    "fps": 30
  },
  "cursor": {
    "mode": "editable-overlay"
  },
  "outputs": {
    "screenPath": "/Users/me/Library/Application Support/openscreen/recordings/recording-123.mp4",
    "manifestPath": "/Users/me/Library/Application Support/openscreen/recordings/recording-123.session.json"
  }
}

The helper emits newline-delimited JSON events to stdout:

json
{ "event": "ready", "schemaVersion": 1 }
{ "event": "recording-started", "timestampMs": 1234567890 }
{ "event": "warning", "code": "microphone-unavailable", "message": "..." }
{ "event": "recording-stopped", "screenPath": "..." }
{ "event": "error", "code": "screen-permission-denied", "message": "..." }

Implementation Phases

Current PR status: macOS screen/window capture routes through the ScreenCaptureKit helper when it is available so editable-cursor recordings can hide the system cursor. The helper now writes ScreenCaptureKit system audio into the primary MP4 and attempts runtime-gated native microphone capture on macOS versions that expose ScreenCaptureKit microphone output. Webcam capture is currently an Electron-recorded sidecar attached to the same recording session; native AVFoundation webcam composition remains the target end state.

1. Native Session Boundary

  • Add a structured macOS native recording request type.
  • Add a macOS helper resolver and build script placeholders.
  • Keep the helper contract process-based, matching the Windows helper boundary.
  • Do not route production macOS recording through this helper until the helper is available and validated.

Acceptance:

  • TypeScript build passes.
  • The macOS helper path and request contract are documented and testable without affecting Windows/Linux behavior.

2. ScreenCaptureKit Display Capture

  • Implement a Swift helper using ScreenCaptureKit.
  • Select display captures by displayId.
  • Encode H.264 MP4 through AVFoundation/VideoToolbox.
  • Set showsCursor = false when editable cursor overlay mode is selected.

Acceptance:

  • Display-only recording produces a valid MP4.
  • The real cursor is not baked into editable-cursor recordings.

3. ScreenCaptureKit Window Capture

  • Resolve Electron window:* selections to ScreenCaptureKit window ids.
  • Capture SCContentFilter(desktopIndependentWindow:).
  • Handle closed/minimized/protected windows with explicit errors.
  • Keep window selection and capture source resolution in Electron/main, not the renderer.

Acceptance:

  • Capturing a normal app window works with cursor/audio/webcam disabled.
  • Unsupported windows return clear native errors.

4. System Audio

  • Enable ScreenCaptureKit system audio on supported macOS versions.
  • Keep audio format and timing owned by the helper.
  • Encode or mux AAC audio into the primary MP4.

Acceptance:

  • System-audio-only recordings produce a valid AAC track.
  • Unsupported macOS versions return an explicit capability error.

5. Microphone

  • Resolve the selected microphone device from the renderer-provided browser deviceId and user-visible label.
  • Capture microphone audio in the helper timing domain.
  • Apply OpenScreen microphone gain policy.
  • Mix system and microphone audio before final AAC output.

Acceptance:

  • Mic-only and mic-plus-system recordings produce a valid, balanced AAC track.
  • Device selection honors the selected microphone, not only the default device.

6. Webcam Composition

  • Capture the selected camera natively through AVFoundation.
  • Match browser device id first where possible, then user-visible label.
  • Compose an initial picture-in-picture overlay into the primary MP4.
  • Hide webcam output until the first usable frame to avoid black startup flashes.

Acceptance:

  • Native display/window recordings can include webcam without returning to Electron capture.
  • Selected camera is honored.

7. Runtime Controls

  • Add pause/resume commands to the helper.
  • Add cancel command that removes partial outputs.
  • Keep restart as stop-discard-start until the helper exposes a native restart operation.

Acceptance:

  • Pause/resume keeps output duration coherent.
  • Cancel leaves no stale media/session files.

8. Test Pipeline

  • npm run build:native:mac: builds Swift helper binaries on macOS.
  • npm run test:sck-helper:mac: display-only helper smoke test.
  • npm run test:sck-window:mac: window capture smoke test.
  • npm run test:sck-audio:mac: system audio smoke test when supported.
  • npm run test:sck-mic:mac: microphone smoke test.
  • npm run test:sck-webcam:mac: webcam smoke test when a webcam is available.
  • Packaging check: confirms helpers are available under electron/native/bin/darwin-${arch} in packaged builds.

SSOT Rules

  • src/lib/nativeMacRecording.ts is the renderer/main TypeScript request contract.
  • This document is the feature-level contract and phase checklist.
  • The Swift helper owns ScreenCaptureKit/AVFoundation media timing.
  • Electron owns output paths, session manifests, and selected source/device resolution.
  • Renderer code must use existing hooks/client APIs and should not bind directly to helper process details.

File: docs/engineering/windows-native-recorder-roadmap.md

Windows Native Recorder Roadmap

OpenScreen's Windows recorder should be owned by one native backend. Electron capture can remain available for non-Windows platforms and temporary developer diagnostics, but Windows production recording should not silently fall back to getDisplayMedia / MediaRecorder.

Goals

  • Capture displays and windows through Windows Graphics Capture (WGC).
  • Render the native Windows cursor as OpenScreen's high-quality scalable cursor overlay.
  • Capture system audio through WASAPI loopback.
  • Capture microphone audio through WASAPI.
  • Mix system audio and microphone audio into the primary screen recording.
  • Capture webcam video natively and compose it into the Windows helper MP4 during the native-recording migration.
  • Keep preview/export aligned because screen video, audio, webcam, and cursor share one native timing origin.
  • Keep exported MP4s Windows-friendly: H.264 video plus AAC audio. Opus-in-MP4 is not an acceptable Windows export target.
  • Package the native helper with the Windows app.

Non-Goals

  • Replacing the editor/export pipeline.
  • Replacing the editor/export pipeline. A later pass can reintroduce a separate editable native webcamVideoPath; the current Windows-native milestone prioritizes a helper-owned multi-flux MP4 with deterministic screen/audio/mic/webcam sync.
  • Adding a native fallback for macOS or Linux in this branch.

Target Architecture

The renderer keeps the existing recording controls. On Windows, useScreenRecorder sends a complete recording request to Electron and does not assemble Windows MediaStream tracks with MediaRecorder.

Electron owns the native recording session:

  • resolves the selected source;
  • resolves output paths;
  • starts cursor sampling;
  • starts the helper process;
  • sends pause/resume/stop/cancel commands;
  • writes RecordingSession manifests;
  • reports explicit errors when a Windows-native capability is unavailable.

The helper owns Windows media capture:

  • WGC screen/window frames;
  • WASAPI system loopback;
  • WASAPI microphone input;
  • Media Foundation webcam capture;
  • DirectShow webcam fallback for virtual cameras not visible to Media Foundation;
  • Media Foundation encoding/muxing;
  • stream timestamp normalization.

Helper Contract V2

The helper receives a single JSON argument:

json
{
  "schemaVersion": 2,
  "recordingId": 1234567890,
  "source": {
    "type": "display",
    "sourceId": "screen:0:0",
    "displayId": 123,
    "windowHandle": null,
    "bounds": { "x": 0, "y": 0, "width": 1920, "height": 1080 }
  },
  "video": {
    "fps": 60,
    "width": 1920,
    "height": 1080,
    "bitrate": 18000000
  },
  "audio": {
    "system": { "enabled": true },
    "microphone": { "enabled": true, "deviceId": "default", "gain": 1.4 }
  },
  "webcam": {
    "enabled": true,
    "deviceId": "default",
    "deviceName": "Camera (NVIDIA Broadcast)",
    "width": 1280,
    "height": 720,
    "fps": 30,
    "bitrate": 18000000
  },
  "outputs": {
    "screenPath": "C:\\Users\\me\\recording-123.mp4",
    "manifestPath": "C:\\Users\\me\\recording-123.session.json"
  }
}

The helper emits newline-delimited JSON events to stdout:

json
{ "event": "ready", "schemaVersion": 2 }
{ "event": "recording-started", "timestampMs": 1234567890 }
{ "event": "warning", "code": "audio-device-unavailable", "message": "..." }
{ "event": "recording-stopped", "screenPath": "..." }
{ "event": "error", "code": "unsupported-window-source", "message": "..." }

During migration, Electron also accepts the current textual helper messages so existing display-only smoke tests keep working.

Implementation Phases

1. Native Session Boundary

  • Add a structured Windows native recording request type.
  • Pass source kind, audio flags, microphone device, webcam flags, and output paths into the helper.
  • On Windows, do not silently fall back to Electron capture. If the helper is unavailable or a native feature is missing, show a clear error.
  • Keep Electron fallback only for non-Windows and optional developer diagnostics.

Acceptance:

  • Display-only recording still works.
  • Enabling an unsupported native feature returns an explicit native error instead of recording through Electron.

2. WASAPI System Audio

Status: initial implementation landed. The helper captures the default render endpoint with WASAPI loopback, passes the runtime mix format into MFEncoder, and muxes AAC audio into the primary MP4. Long-run drift correction and explicit silence insertion remain follow-up hardening work.

  • Add WasapiLoopbackCapture.
  • Capture the default render endpoint in shared loopback mode.
  • Keep WasapiLoopbackCapture responsible only for device activation, packet capture, and packet timestamps.
  • Keep MFEncoder responsible for all Media Foundation stream definitions and muxing.
  • Feed the endpoint mix format into MFEncoder as the single source of truth for audio stream shape: sample rate, channel count, bits per sample, block alignment, average bytes/sec, and subtype (PCM or Float).
  • Encode the primary screen MP4 with H.264 video and AAC audio through one IMFSinkWriter.
  • Timestamp audio from the captured frame count in 100ns units. The first implementation uses the WASAPI packet timeline; later drift correction will add explicit silence or resampling if long recordings show measurable clock skew.
  • Treat microphone mixing as a later phase. System loopback must land first without introducing renderer-side audio code.

Acceptance:

  • Screen MP4 has an AAC audio track when system audio is enabled.
  • A 5-minute recording has audio/video duration drift below one frame.

SSOT rules for this phase:

  • src/lib/nativeWindowsRecording.ts is the renderer/main TypeScript request contract.
  • docs/engineering/windows-native-recorder-roadmap.md is the feature-level contract and phase checklist.
  • WgcSession::captureWidth()/captureHeight() is the encoded screen frame size until a dedicated native scaling stage exists.
  • WasapiLoopbackCapture::inputFormat() is the runtime audio format source used by MFEncoder.
  • The renderer passes both the browser webcam deviceId and selected display label as deviceName; electron/native/wgc-capture/src/webcam_capture.* is the only place that maps those values to Media Foundation devices.
  • Electron resolves the selected label to a DirectShow filter CLSID once and passes it as webcamDirectShowClsid; the helper must not independently guess among DirectShow filters.
  • No duplicated hard-coded audio format assumptions in main.cpp.

3. WASAPI Microphone

Status: initial implementation in progress. The helper can open the default WASAPI capture endpoint, apply the OpenScreen microphone gain, encode mic-only audio, and mix system loopback plus microphone through a single queued AudioMixer timeline when both endpoints expose the same runtime format. Audio endpoints are warmed before WGC starts, the mixer drops pre-roll and begins its paced timeline on the first encoded video frame, then cuts queued tail audio on stop so the MP4 does not drift past the video. Browser deviceId to MMDevice id mapping, resampling between mismatched endpoint formats, and drift correction remain follow-up hardening work.

  • Add microphone device enumeration and stable device-id mapping.
  • Capture selected/default microphone through WASAPI.
  • Apply OpenScreen's current mic gain policy.
  • Mix microphone and system audio before AAC encoding.

Acceptance:

  • Mic-only, system-only, and mixed audio recordings produce a valid AAC track.
  • Device unplug/permission failure produces an explicit error or warning.

4. Webcam Capture

  • Add Media Foundation webcam source reader.
  • Select requested dimensions/fps or the nearest format accepted by Media Foundation.
  • Convert webcam samples to BGRA and compose them into the primary helper MP4 as an initial bottom-right picture-in-picture overlay.
  • Ignore black webcam warmup frames and keep the overlay hidden until the first visible frame is available, so virtual cameras do not flash a black picture-in-picture rectangle at recording start.
  • Keep the helper process as the SSOT for screen/window, WASAPI system audio, microphone, webcam, and mux timing.
  • Match the requested webcam through Media Foundation friendly names first, then browser device ids/symbolic links, so UI selection remains stable across Chromium and Windows native device namespaces.
  • Use the Electron-resolved DirectShow CLSID when the selected virtual camera, for example NVIDIA Broadcast, is registered for DirectShow but absent from Media Foundation enumeration.
  • Later: promote the same webcam capture source to a separate editable native webcamVideoPath if product requirements need post-recording layout edits.

Acceptance:

  • Native display/window recordings can include webcam without returning to Electron capture.
  • npm run test:wgc-webcam:win validates the helper path when a webcam is available and skips explicitly when no webcam device exists.
  • Combined webcam + system audio + microphone produces one MP4 with H.264 video and AAC audio.

5. Native Window Capture

Status: initial implementation in progress. Electron parses the window:<HWND>:... desktop source id through the shared native Windows recording contract and passes windowHandle to the helper. The helper resolves the HWND, validates it with IsWindow, and creates the WGC item with CreateForWindow(HWND). Resize/minimize/move hardening and protected-window diagnostics remain follow-up work.

  • Resolve Electron window:* selections to an HWND.
  • Use WGC CreateForWindow(HWND).
  • Handle window close, minimize, resize, DPI scaling, and monitor moves.
  • Return clear errors for unsupported protected windows.

Acceptance:

  • Capturing a normal app window works with cursor/audio/mic/webcam.
  • Window resize and movement do not corrupt the recording.

6. Runtime Controls

  • Add pause/resume commands to the helper.
  • Add cancel command that removes partial screen/webcam outputs.
  • Keep restart as stop-discard-start from Electron until the helper supports a native restart event.

Acceptance:

  • Pause/resume keeps preview duration coherent.
  • Cancel leaves no stale media/session/cursor files.

7. Test Pipeline

  • npm run test:wgc-helper:win: display-only helper smoke test.
  • npm run test:wgc-audio:win: validates AAC track presence and duration.
  • npm run test:wgc-window:win: captures a fixture window by HWND.
  • npm run test:wgc-webcam:win: validates webcam output when a webcam is available, otherwise skips explicitly.
  • Packaging check: confirms the helper is in app.asar.unpacked.
  • Export check: exported MP4s generated from native recordings keep an AAC audio track when the source has audio.
  • npm run test:wgc-mic:win: validates default-microphone capture writes an AAC track when an input endpoint is available.
  • npm run test:wgc-mixed-audio:win: validates system loopback plus microphone writes one mixed AAC track when endpoint formats are compatible.

Backlog

Native Cursor Click Bounce Is Not Visibly Applied

Status: open. Do not treat Windows native cursor Click Bounce as shipped.

Problem:

  • The cursor settings UI exposes Size, Smoothing, Motion Blur, and Click Bounce.
  • On Windows native cursor recordings, Size, Smoothing, and Motion Blur are visibly applied in preview/export.
  • Click Bounce still has no visible effect in manual packaged-app testing, even after adding click-related sample metadata.

What has already been tried:

  • Added interactionType: "click" | "mouseup" | "move" to native cursor samples.
  • Added polling-based left-button state through GetAsyncKeyState.
  • Added the GetAsyncKeyState low-bit path to catch quick clicks between samples.
  • Added a PowerShell/C# WH_MOUSE_LL mouse hook experiment and launched the sampler through a temporary .ps1 file to avoid Windows command-line length limits.
  • Updated npm run test:cursor-native:win so the diagnostic can observe a synthetic short click and emit clickSampleCount.

Current diagnosis:

  • The diagnostic can observe synthetic click events, but this has not translated into a visible Click Bounce effect in the real packaged app.
  • The test currently proves that some click metadata can be recorded, not that the full OpenScreen record -> preview -> export path displays a bounce at the expected time.
  • The current native implementation may be animating from metadata that is not present in the real recording session, may be using the wrong timestamp origin, or may be applying a scale change too subtle to notice on the DOM/native cursor path.

Next investigation when resumed:

  • Inspect the actual .cursor.json/session sidecar generated by a packaged-app manual recording and confirm whether real clicks produce interactionType: "click" at the right timeMs.
  • Add a targeted end-to-end fixture that records a known click, loads the generated project, and asserts the preview/export cursor scale changes across adjacent frames.
  • Compare the native DOM cursor path against the older PixiCursorOverlay click visual state and decide whether native cursor bounce should be a scale-only animation, an additional click ring, or a short explicit keyframe animation independent of sample cadence.
  • If event capture remains unreliable in the PowerShell sampler, move click events into a small native cursor helper instead of PowerShell/C# script injection.

Ship Criteria

  • Windows display capture works with cursor, system audio, microphone, and webcam.
  • Windows window capture works with cursor, system audio, microphone, and webcam.
  • Preview and export show no cursor position drift.
  • Preview and export show no measurable audio/video/webcam drift.
  • Windows production builds do not depend on Electron capture fallback.

File: docs/architecture/native-bridge.md

Native Bridge Architecture

Goal

Provide a single, resilient source of truth for platform-native capabilities while keeping Electron transport thin and renderer APIs unified.

Layers

  1. Native adapters
    Platform-specific providers implement stable domain interfaces such as cursor telemetry or system asset discovery.

  2. Main-process services
    Services orchestrate adapters, own runtime state, and expose domain-level operations.

  3. Unified IPC transport
    Renderer code talks to a single native-bridge:invoke channel using versioned contracts.

  4. Renderer client
    React code should consume src/native/client.ts rather than binding directly to ad hoc Electron APIs.

Principles

  • Single source of truth: runtime-native state lives in the Electron main process.
  • Capability-first: renderer can query support before attempting native behavior.
  • Versioned contracts: requests and responses are explicit and evolve predictably.
  • Resilience: every response uses a consistent result envelope with stable error codes.

Current rollout

This repository now contains the initial scaffold:

  • shared contracts in src/native/contracts.ts
  • renderer SDK in src/native/client.ts
  • main-process state store in electron/native-bridge/store.ts
  • cursor telemetry adapter in electron/native-bridge/cursor/telemetryCursorAdapter.ts
  • domain services in electron/native-bridge/services/*
  • unified handler registration in electron/ipc/nativeBridge.ts

The legacy window.electronAPI surface still exists for backward compatibility. New native-facing features should prefer the unified bridge client.


File: electron/native/README.md

Native capture helpers

macOS

macOS native recording will use a ScreenCaptureKit helper with the same process boundary as the Windows WGC helper:

  1. Electron resolves the selected source, output paths, and user-selected devices.
  2. The helper receives one structured JSON request.
  3. The helper owns ScreenCaptureKit/AVFoundation capture, timing, encoding, and muxing.
  4. Electron persists the resulting media/session manifest and reports helper errors explicitly.

Helper locations:

  1. OPENSCREEN_SCK_CAPTURE_EXE, for local development and diagnostics.
  2. electron/native/screencapturekit/build/openscreen-screencapturekit-helper, for locally built Swift output.
  3. electron/native/bin/darwin-arm64/openscreen-screencapturekit-helper or electron/native/bin/darwin-x64/openscreen-screencapturekit-helper, for packaged prebuilt helpers.

The macOS cursor-shape helper is resolved from OPENSCREEN_MAC_CURSOR_HELPER_EXE first, then the matching openscreen-macos-cursor-helper binary in the same local build and packaged electron/native/bin/darwin-${arch} directories.

Build the macOS helper with:

bash
npm run build:native:mac

On non-macOS hosts this command exits successfully and does not affect Windows/Linux development. On macOS it builds the Swift package at electron/native/screencapturekit, writes the development binaries to electron/native/screencapturekit/build, and copies redistributable binaries to electron/native/bin/darwin-${arch}.

The current helper implementation supports display/window ScreenCaptureKit video capture, cursor exclusion through SCStreamConfiguration.showsCursor, H.264 encoding, MP4 muxing, and ScreenCaptureKit system audio. It also attempts native ScreenCaptureKit microphone capture when the running macOS version exposes that capability. Webcam recording currently stays as an Electron sidecar and is attached to the same recording session after the native screen capture stops.

Electron exposes is-native-mac-capture-available for capability probing. It resolves the same helper locations listed above and reports missing-helper until a Swift helper binary is present. When available, macOS recording routes screen/window capture through the native helper so editable cursor recordings do not bake the system cursor into the video. Cursor positions are sampled in Electron; when the cursor helper is available and Accessibility is granted, samples are also tagged with link/text cursor hints such as pointer.

See docs/engineering/macos-native-recorder-roadmap.md for the contract, rollout phases, and SSOT rules.

Windows

Windows native recording is resolved from one of these locations:

  1. OPENSCREEN_WGC_CAPTURE_EXE, for local development and diagnostics.
  2. electron/native/wgc-capture/build/wgc-capture.exe, for a locally built Ninja helper.
  3. electron/native/wgc-capture/build/Release/wgc-capture.exe, for a locally built multi-config helper.
  4. electron/native/bin/win32-x64/wgc-capture.exe or electron/native/bin/win32-arm64/wgc-capture.exe, for packaged prebuilt helpers.

Build the Windows helper with:

powershell
npm run build:native:win

The build writes the CMake output to electron/native/wgc-capture/build/wgc-capture.exe and copies the redistributable binary to electron/native/bin/win32-x64/wgc-capture.exe.

The helper contract is process-based: the app starts the process with one JSON argument and sends commands on stdin. stop\n finalizes the recording. During migration the helper prints both newline-delimited JSON events and the legacy text messages Recording started / Recording stopped. Output path: .

Current V2 JSON shape:

json
{
  "schemaVersion": 2,
  "recordingId": 123,
  "sourceType": "display",
  "sourceId": "screen:0:0",
  "displayId": 1,
  "windowHandle": null,
  "outputPath": "C:\\path\\recording-123.mp4",
  "videoWidth": 1920,
  "videoHeight": 1080,
  "fps": 60,
  "captureSystemAudio": false,
  "captureMic": false,
  "microphoneDeviceId": "default",
  "microphoneDeviceName": "Microphone (NVIDIA Broadcast)",
  "microphoneGain": 1.4,
  "webcamEnabled": true,
  "webcamDeviceId": "default",
  "webcamDeviceName": "Camera (NVIDIA Broadcast)",
  "webcamWidth": 1280,
  "webcamHeight": 720,
  "webcamFps": 30,
  "outputs": {
    "screenPath": "C:\\path\\recording-123.mp4"
  }
}

The current helper implementation supports display/window video capture, system audio loopback, selected-microphone capture, Media Foundation webcam capture, and a DirectShow webcam fallback for virtual cameras that are not exposed through Media Foundation. Webcam frames are currently composed into the primary MP4 as a bottom-right picture-in-picture overlay. Browser deviceId values do not always map to Media Foundation symbolic links or WASAPI endpoint IDs, so the renderer passes both browser IDs and user-visible device names. For microphones, the helper tries the requested WASAPI endpoint ID first, then resolves an active capture endpoint by microphoneDeviceName, then falls back to the default endpoint. For webcams, Electron resolves a matching DirectShow filter CLSID for the selected label; the helper uses Media Foundation first, then that exact DirectShow filter when the requested camera is absent from Media Foundation.

Smoke-test the helper with:

powershell
npm run test:wgc-helper:win
npm run test:wgc-window:win
npm run test:wgc-audio:win
npm run test:wgc-mic:win
npm run test:wgc-mixed-audio:win
npm run test:wgc-webcam:win

To validate a specific native webcam manually:

powershell
$env:OPENSCREEN_WGC_TEST_WEBCAM_DEVICE_NAME = "NVIDIA Broadcast"
npm run test:wgc-webcam:win
Remove-Item Env:OPENSCREEN_WGC_TEST_WEBCAM_DEVICE_NAME

To validate a specific native microphone manually:

powershell
$env:OPENSCREEN_WGC_TEST_MICROPHONE_DEVICE_NAME = "Microphone (NVIDIA Broadcast)"
npm run test:wgc-mic:win
Remove-Item Env:OPENSCREEN_WGC_TEST_MICROPHONE_DEVICE_NAME