File: README.md
[!Note]
OpenScreen is now archived and no longer maintained. For continued maintenance and development, a community-driven spin-off led by one of the core contributors is available here: https://github.com/EtienneLescot/openscreen
[!WARNING]
This started as a side project that blew up; not production grade and you'll hit bugs, but hopefully it covers what you need. This project will soon be archived.
If you don't want to pay $29/month for Screen Studio but want a version that does what most people seem to need - quick, polished product demos and walkthroughs you'd post on X, Reddit or Youtube. OpenScreen does not offer every Screen Studio feature, but covers a lot of the core functionality.
Screen Studio is an awesome product and this is definitely not a 1:1 clone. If you just want something fully free and open source, this project should cover most of your needs.
100% free for both personal and commercial use. Use it, modify it, distribute it. Please respect the License.
[!NOTE]
Software should be accessible. OpenScreen has no paid tiers, premium features, upsells, or functionality locked behind a paywall.
Core Features
- Record a specific window, or your whole screen.
- Record microphone and system audio.
- Webcam overlay with picture-in-picture, drag-to-position, mirroring, and shape options.
- Auto or manual zooms with adjustable depth, duration, easing, and pixel-precise position; auto-zoom follows your cursor as you work.
- Custom cursor size, smoothing, and click effects, with cursor themes and post-recording path smoothing.
- Automatic captions for voiceovers, generated on-device with no upload (works offline).
- Wallpapers, solid colors, gradients, or your own background image.
- Motion blur.
- Crop, trim, and per-segment speed control on the timeline.
- Text, arrow, and image annotations, with text animation presets.
- Timeline snapping guides and an audio waveform to make trimming easier.
- Customizable keyboard shortcuts.
- Export to MP4 or GIF in multiple aspect ratios and resolutions.
- Languages supported: Arabic, English, Spanish, French, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Turkish, Vietnamese, Simplified Chinese, and Traditional Chinese.
Installation
Download the latest installer for your platform from the GitHub Releases page.
macOS
The easiest way to install on macOS is via Homebrew:
brew install --cask siddharthvaddem/openscreen/openscreenBrew automatically picks the right build for Apple Silicon or Intel, and verifies the download against a notarized signature so Gatekeeper won't block it.
To update later: brew upgrade --cask openscreen
To uninstall: brew uninstall --cask openscreen (add --zap to also remove app data)
Manual install (if you prefer)
If you'd rather grab the .dmg directly from the Releases page and encounter Gatekeeper blocking the app, you can bypass it by running the following command in your terminal after installation:
xattr -rd com.apple.quarantine /Applications/Openscreen.appNote: Give your terminal Full Disk Access in System Settings > Privacy & Security to grant you access and then run the above command.
After running this command, proceed to System Preferences > Security & Privacy to grant the necessary permissions for "screen recording" and "accessibility". Once permissions are granted, you can launch the app.
[!NOTE]
Upgrading from an older version and hitting permission issues? If you already had OpenScreen installed and the new version won't record (Screen Recording or Accessibility keep failing even after you grant them), uninstall the old version, remove OpenScreen's existing entries under System Settings > Privacy & Security (both Screen Recording and Accessibility), then do a fresh install and grant the permissions again when prompted.
Windows
Install via winget:
winget install SiddharthVaddem.OpenScreenTo update later: winget upgrade SiddharthVaddem.OpenScreen
To uninstall: winget uninstall SiddharthVaddem.OpenScreen
If you'd rather grab the .exe installer directly, download it from the Releases page.
Linux
Three packages are published to the Releases page for each version. Pick the one that matches your distro:
Debian / Ubuntu / Pop!_OS (.deb)
sudo apt install ./Openscreen-Linux-latest.debArch / Manjaro (.pacman)
sudo pacman -U Openscreen-Linux-latest.pacmanAny distro (.AppImage)
chmod +x Openscreen-Linux-*.AppImage
./Openscreen-Linux-*.AppImageNixOS / Nix (flake)
Try without installing:
nix run github:siddharthvaddem/openscreenInstall into your user profile:
nix profile install github:siddharthvaddem/openscreenFor a NixOS system config (flake):
{
inputs.openscreen.url = "github:siddharthvaddem/openscreen";
outputs = { nixpkgs, openscreen, ... }: {
nixosConfigurations.<host> = nixpkgs.lib.nixosSystem {
modules = [
openscreen.nixosModules.default
{ programs.openscreen.enable = true; }
];
};
};
}For Home Manager, use openscreen.homeManagerModules.default with the same programs.openscreen.enable = true;.
You may need to grant screen recording permissions depending on your desktop environment.
Sandbox error: If the AppImage fails to launch with a "sandbox" error, run it with --no-sandbox:
./Openscreen-Linux-*.AppImage --no-sandboxPlatform differences
Everything in the editor and export is the same on macOS, Windows, and Linux: zooms, backgrounds, motion blur, crop/trim/speed, blur regions, annotations, auto-captions, projects, export, and all languages. The differences are in capture, where macOS and Windows use a native pipeline that Linux doesn't have:
- Native recording: macOS (ScreenCaptureKit) and Windows (Windows Graphics Capture) record through a native pipeline for higher quality and clean window-level capture. Linux records through the browser pipeline instead.
- Custom cursors: on macOS and Windows the real cursor is captured (shape, type, and clicks), which powers the cursor themes, click effects, and editable cursor overlay. On Linux only the cursor position is captured (used for auto-zoom), so those cursor options aren't available.
- Webcam: captured natively on macOS and Windows; on Linux it's recorded through the browser, but still works as a picture-in-picture overlay.
- System audio support varies by OS:
- macOS: requires macOS 13+. On macOS 14.2+ you'll be prompted to grant audio capture permission. macOS 12 and below can't capture system audio (mic still works).
- Windows: works out of the box.
- Linux: needs PipeWire (default on Ubuntu 22.04+, Fedora 34+). Older PulseAudio-only setups may not capture system audio (mic should still work).
License
This project is licensed under the MIT License. By using this software, you agree that the authors are not liable for any issues, damages, or claims arising from its use.
File: docs/testing/macos-native-cursor.md
macOS native cursor test pipeline
This document covers manual and diagnostic testing for macOS native cursor capture โ the path that records real system cursor bitmaps via NSCursor.currentSystem and surfaces them through the OpenScreen editor and export pipeline.
How the macOS cursor helper works
The helper binary (openscreen-macos-cursor-helper) runs as a child process of Electron during recording. It:
- polls
NSCursor.currentSystemat the configured sample interval - converts each cursor image to PNG and computes a SHA-256 content hash as a stable asset id
- emits the full base64 bitmap payload once per unique cursor shape per session; subsequent samples carry only the
assetIdso stdout stays small - tracks left-button down/up events via
CGEventTapand tags each sample withinteractionType - uses the Accessibility API to detect
textandpointeraffordances (link/button/input roles) when Accessibility is granted; these shapes use the bundled high-quality SVG replacements instead of the raw bitmap
Each sample line is newline-delimited JSON:
{ "type": "ready", "timestampMs": 1234567890, "accessibilityTrusted": true, "mouseTapReady": true }
{ "type": "sample", "timestampMs": 1234567891, "assetId": "a7472...", "asset": { "id": "a7472...", "imageDataUrl": "data:image/png;base64,...", "width": 64, "height": 64, "hotspotX": 16, "hotspotY": 16, "scaleFactor": 2.0 }, "cursorType": null, "leftButtonDown": false, "leftButtonPressed": false, "leftButtonReleased": false }
{ "type": "sample", "timestampMs": 1234567924, "assetId": "a7472...", "cursorType": null, "leftButtonDown": false, "leftButtonPressed": false, "leftButtonReleased": false }asset is present only the first time a given assetId appears. The TypeScript session (MacNativeCursorRecordingSession) collects unique assets into a map and sets provider: "native" in the final CursorRecordingData when at least one bitmap was captured.
Build the helper
npm run build:native:macThis builds both Swift helpers (openscreen-screencapturekit-helper and openscreen-macos-cursor-helper) and copies them to:
electron/native/screencapturekit/build/โ used by the local dev serverelectron/native/bin/darwin-arm64/ordarwin-x64/โ used by packaged builds
Requires Xcode (not just Command Line Tools). If you see a build error about missing SDK metadata, run:
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
sudo xcodebuild -license acceptSmoke-test the helper directly
You can run the cursor helper standalone to inspect its raw output before involving the full app:
BIN=electron/native/screencapturekit/build/openscreen-macos-cursor-helper
("$BIN" '{"sampleIntervalMs":100}' & PID=$!; sleep 2; kill $PID) | head -20Expected first line:
{"type":"ready","mouseTapReady":true,"accessibilityTrusted":false,"timestampMs":...}accessibilityTrusted: false is normal in dev/unsigned builds. It means text/pointer affordance detection is disabled; native bitmap capture still works.
Expected sample lines:
{"type":"sample","assetId":"a7472...","asset":{"id":"a7472...","imageDataUrl":"data:image/png;base64,...","width":64,"height":64,"hotspotX":26,"hotspotY":16,"scaleFactor":2.0},...}
{"type":"sample","assetId":"a7472...",...}Move the cursor over a text input while the helper is running and check that a new assetId appears with a different bitmap (if Accessibility is granted โ see below).
Point the app at a custom helper binary
export OPENSCREEN_MAC_CURSOR_HELPER_EXE=/path/to/openscreen-macos-cursor-helper
npm run devmacOS permissions
Two separate permissions are needed:
| Permission | What it enables | Where to grant |
|---|---|---|
| Screen Recording | ScreenCaptureKit video capture | System Settings โ Privacy & Security โ Screen & System Audio Recording โ Electron โ |
| Accessibility | text / pointer cursor type detection (affordance hints) |
System Settings โ Privacy & Security โ Accessibility โ Electron โ |
Screen Recording is required to record. Without it the recording never starts.
Accessibility is optional. Without it, cursorType will always be null and all cursors render from their captured bitmaps (no SVG substitution). This is the expected fallback and does not degrade cursor quality for non-text/pointer shapes.
After granting either permission in System Settings, fully quit and relaunch the dev server โ getMediaAccessStatus caches the result per-process.
Manual test checklist
P0 โ core bitmap capture
- Record a short clip. Open the editor. Confirm the default arrow cursor is the real system arrow (not the bundled SVG approximation).
- Record while hovering over a web browser. Confirm custom-CSS cursors (e.g.
cursor: grab,cursor: crosshair) appear as their actual shapes. - Export to MP4. Confirm the cursor renders correctly in the exported video.
- Export to GIF. Same check.
P1 โ affordance substitution (requires Accessibility)
- Grant Accessibility permission and restart the app.
- Record hovering over a text input field. Confirm the text I-beam uses the bundled SVG version (prettier than the system bitmap).
- Record hovering over a link/button. Confirm the pointer hand uses the bundled SVG.
P1 โ hotspot alignment (Retina)
- On a Retina display, record a precise click on a small button. In the editor, confirm the cursor tip aligns with the actual click point. The helper reports
scaleFactor: 2.0; the renderer divides pixel dimensions and hotspot by this value to recover point sizes.
P1 โ click detection
- Record several left-clicks. In the editor, confirm the click-bounce animation fires on each click.
- Confirm
interactionType: "click"and"mouseup"events are present in the recording session sidecar (cursorRecordingDatainside<videoPath>.cursor.json).
P2 โ graceful degradation
- Remove both build-output copies of the helper binary and start a recording. The session should succeed with
provider: "none"(position-only telemetry, default arrow rendered). Restore both binaries afterward.bashARCH=$([ "$(uname -m)" = "arm64" ] && echo darwin-arm64 || echo darwin-x64) mv electron/native/screencapturekit/build/openscreen-macos-cursor-helper /tmp/cursor-helper-build mv electron/native/bin/$ARCH/openscreen-macos-cursor-helper /tmp/cursor-helper-bin # ... start recording, then restore: mv /tmp/cursor-helper-bin electron/native/bin/$ARCH/openscreen-macos-cursor-helper mv /tmp/cursor-helper-build electron/native/screencapturekit/build/openscreen-macos-cursor-helper - Revoke Accessibility. Confirm recording still works and cursors render from bitmaps (no SVG substitution).
P2 โ multi-display
- Move the cursor to a secondary display during recording. Confirm the cursor clips to the canvas edge rather than snapping invisible on fast swipes. Confirm it hides after โ100 ms of sustained out-of-bounds movement.
P2 โ long recording memory
- Record for 3โ5 minutes while switching between many apps (browser, terminal, editor). The helper should not grow in memory because each iteration drains Cocoa objects via
autoreleasepool. CheckActivity Monitorโopenscreen-macos-cursor-helperRSS stays flat after the first few seconds.
What a healthy recording looks like
Inspect the cursor sidecar file written alongside the recorded video (<videoPath>.cursor.json). For a recording saved to /tmp/rec.mp4, the sidecar is /tmp/rec.mp4.cursor.json:
{
"version": 2,
"provider": "native",
"assets": [
{ "id": "a7472...", "platform": "darwin", "imageDataUrl": "data:image/png;base64,...", "width": 64, "height": 64, "hotspotX": 26.0, "hotspotY": 16.0, "scaleFactor": 2.0 }
],
"samples": [
{ "timeMs": 0, "cx": 0.42, "cy": 0.38, "visible": true, "assetId": "a7472...", "interactionType": "move" },
...
]
}provider: "native" and a non-empty assets array confirm bitmap capture is active. If you see provider: "none" and assets: [], the helper was not found or exited before ready.
Native macOS capture backend
The app routes macOS recordings through the ScreenCaptureKit helper (openscreen-screencapturekit-helper) when it is available, so the real system cursor is excluded from the video frame. The cursor position and bitmap are captured separately by the cursor helper and composited in the editor and export pipeline.
Current native availability rules:
- macOS 13 (Ventura) or newer
openscreen-screencapturekit-helperbinary is present- Screen Recording permission is granted
Build both helpers locally:
npm run build:native:macFor local diagnostics with a custom helper binary, use the environment override:
export OPENSCREEN_MAC_CURSOR_HELPER_EXE=/path/to/openscreen-macos-cursor-helper
npm run devKnown limitations
- Intel (x86_64) Macs: the distributed helper is built for
darwin-arm64. On Intel Macs, you need to build from source withnpm run build:native:macon the target machine. - Accessibility permission in unsigned/dev builds:
getMediaAccessStatus("accessibility")may not reflect the toggle state for unsigned Electron in dev mode. The helper will always probe and reportaccessibilityTrustedin itsreadyevent โ use that as the authoritative signal. - App-defined custom cursors (CGS layer):
NSCursor.currentSystemcaptures the active AppKit cursor. Cursors set at the CoreGraphics/CGS layer by some games or GPU-accelerated apps may not be visible here. This is a known macOS API limitation.
File: docs/testing/windows-native-cursor.md
Windows native cursor test pipeline
This branch includes two Windows-focused diagnostics for fast iteration on native cursor capture and rendering. They are intentionally local developer tools: they create short videos and JSON reports so cursor changes can be inspected without doing a full manual record/edit/export cycle.
Native sampler diagnostic
npm run test:cursor-native:winThis script does not launch OpenScreen. It:
- starts a Windows
GetCursorInfosampler - moves the real OS pointer with
SetCursorPos - captures native cursor handles, hotspots, assets, and standard
IDC_*cursor types - writes normalized
CursorRecordingData - generates an abstract preview video
- generates a real-screen preview video using screenshots of the current desktop
The output directory is printed in the command result, for example:
C:\Users\\AppData\Local\Temp\openscreen-cursor-native-...Useful files:
report.json: sample counts, asset counts, cursor handles, and generated artifact pathscursor-recording-data.json: sidecar-compatible cursor datapreview.webm: abstract path/asset/hotspot previewreal-capture-preview.webm: real desktop screenshot background with reconstructed cursor overlayassets/*.png: raw cursor bitmaps captured from Windows
Environment overrides:
$env:CURSOR_TEST_DURATION_MS = "3000"
$env:CURSOR_TEST_SAMPLE_INTERVAL_MS = "16"
$env:CURSOR_TEST_SCREEN_FRAME_INTERVAL_MS = "80"
$env:CURSOR_TEST_OUTPUT_DIR = "C:\temp\openscreen-cursor-test"
npm run test:cursor-native:winOpenScreen preview capture
npm run capture:openscreen-previewThis script launches the real Electron app, injects a fixture video plus cursor sidecar data, opens the editor, captures frames from the actual OpenScreen preview UI, and encodes them into a WebM.
By default it uses the latest cursor-recording-data.json generated by npm run test:cursor-native:win. To force a specific sidecar:
$env:CURSOR_RECORDING_DATA_PATH = "C:\path\to\cursor-recording-data.json"
npm run capture:openscreen-previewUseful environment overrides:
$env:OPENSCREEN_PREVIEW_SKIP_BUILD = "true"
$env:OPENSCREEN_PREVIEW_FRAME_COUNT = "120"
$env:OPENSCREEN_PREVIEW_FPS = "30"
$env:OPENSCREEN_PREVIEW_OUTPUT_DIR = "C:\temp\openscreen-preview"
npm run capture:openscreen-previewUseful files:
openscreen-preview.webm: video of the real OpenScreen editor previewframes/*.png: captured preview framesreport.json: fixture paths, source sidecar, frame count, and output path
What these tests validate
Together, the scripts make it quick to inspect:
- whether Windows cursor samples are visible and continuous
- whether native hotspots stay anchored when scaling to
3x - whether standard Windows cursors are recognized via
IDC_* - whether high-quality SVG cursor replacements follow the native hotspot
- whether the real OpenScreen preview renders the same cursor behavior as the diagnostic pipeline
They are not a full substitute for an end-to-end manual recording pass. Before shipping cursor changes, also test a real capture session and export from the packaged app.
Known Gap
Windows native cursor Click Bounce is currently backlogged. Size, Smoothing, and Motion Blur can be validated through preview/export, but Click Bounce has not shown a visible effect in packaged-app manual testing. The current diagnostic can observe synthetic click metadata, but that is not enough to validate the real OpenScreen record -> preview -> export path.
Track the open item in docs/engineering/windows-native-recorder-roadmap.md under Native Cursor Click Bounce Is Not Visibly Applied.
Native Windows capture backend
The app now routes Windows recordings through an external WGC helper instead of Electron getDisplayMedia. This is meant to remove the coordinate and clock split that made the reconstructed cursor drift in the preview/export path.
Current native availability rules:
- Windows 10 build 19041 or newer
- a helper executable is available
The helper currently implements display/window video capture, system audio loopback, default microphone capture, Media Foundation webcam capture, and DirectShow fallback for selected virtual cameras such as NVIDIA Broadcast. Webcam frames are composed into the primary MP4 as a bottom-right picture-in-picture overlay, and black webcam warmup frames are ignored until the first visible frame is available.
Build OpenScreen's helper locally:
npm run build:native:winSmoke-test the helper directly:
npm run test:wgc-helper:win
npm run test:wgc-helper:win -- --capture-cursor
npm run test:wgc-window:win
npm run test:wgc-audio:win
npm run test:wgc-mic:win
npm run test:wgc-mixed-audio:win
npm run test:wgc-webcam:winFor local diagnostics with another compatible helper, point OpenScreen at that executable:
$env:OPENSCREEN_WGC_CAPTURE_EXE = "C:\path\to\wgc-capture.exe"
npm run build-vite
npm run devThe helper receives one JSON config argument, emits JSON lifecycle events, prints the legacy Recording started marker, accepts stop on stdin, and prints Recording stopped. Output path: . See electron/native/README.md for the exact contract and build output paths.
File: docs/engineering/macos-native-recorder-roadmap.md
macOS Native Recorder Roadmap
OpenScreen's macOS recorder should follow the same architecture boundaries as the Windows native recorder: Electron owns session orchestration and persistence, while a platform-native helper owns capture, timing, encoding, and platform-specific permissions.
This work is intentionally scoped as a macOS-only port. Windows native capture remains owned by the WGC helper, and Linux remains on the existing Electron path.
Goals
- Capture displays and windows through ScreenCaptureKit.
- Exclude the real system cursor during capture when using the editable OpenScreen cursor overlay.
- Preserve the current high-quality cursor overlay path in preview and export.
- Capture macOS system audio through ScreenCaptureKit on supported macOS versions.
- Capture microphone audio through the same native timing domain where the OS supports it, or through an explicit companion path until it can be moved into the helper.
- Mix system audio and microphone audio into the primary MP4 without renderer-side track assembly.
- Capture webcam video natively and compose it into the helper-owned MP4 during the native-recording migration.
- Keep screen video, audio, webcam, and cursor aligned to one native timing origin.
- Package per-architecture helper binaries with macOS builds.
Non-Goals
- Replacing the editor/export pipeline.
- Changing Windows native capture behavior.
- Adding Linux native capture.
- Shipping a silent fallback from native macOS capture to Electron capture when the user explicitly requested a native-only feature.
Architecture
The renderer keeps the existing recording controls. On macOS, useScreenRecorder should eventually send a complete recording request to Electron instead of assembling display, audio, microphone, webcam, and cursor streams in the browser.
Electron owns the native recording session:
- resolves the selected display/window source;
- resolves output paths;
- starts cursor telemetry capture when editable cursor mode is selected;
- starts the ScreenCaptureKit helper process;
- sends pause/resume/stop/cancel commands;
- writes
RecordingSessionmanifests; - reports explicit errors when a macOS-native capability is unavailable.
The helper owns macOS media capture:
- ScreenCaptureKit display/window frames;
- ScreenCaptureKit system audio where supported;
- microphone capture or helper-owned companion audio capture;
- webcam capture and initial picture-in-picture composition;
- AVFoundation/VideoToolbox encoding and muxing;
- stream timestamp normalization.
Helper Contract V1
The helper receives a single JSON argument:
{
"schemaVersion": 1,
"recordingId": 1234567890,
"source": {
"type": "display",
"sourceId": "screen:0:0",
"displayId": 1,
"windowId": null,
"bounds": { "x": 0, "y": 0, "width": 1920, "height": 1080 }
},
"video": {
"fps": 60,
"width": 1920,
"height": 1080,
"bitrate": 18000000,
"hideSystemCursor": true
},
"audio": {
"system": { "enabled": true },
"microphone": {
"enabled": true,
"deviceId": "default",
"deviceName": "MacBook Pro Microphone",
"gain": 1.4
}
},
"webcam": {
"enabled": true,
"deviceId": "default",
"deviceName": "FaceTime HD Camera",
"width": 1280,
"height": 720,
"fps": 30
},
"cursor": {
"mode": "editable-overlay"
},
"outputs": {
"screenPath": "/Users/me/Library/Application Support/openscreen/recordings/recording-123.mp4",
"manifestPath": "/Users/me/Library/Application Support/openscreen/recordings/recording-123.session.json"
}
}The helper emits newline-delimited JSON events to stdout:
{ "event": "ready", "schemaVersion": 1 }
{ "event": "recording-started", "timestampMs": 1234567890 }
{ "event": "warning", "code": "microphone-unavailable", "message": "..." }
{ "event": "recording-stopped", "screenPath": "..." }
{ "event": "error", "code": "screen-permission-denied", "message": "..." }Implementation Phases
Current PR status: macOS screen/window capture routes through the ScreenCaptureKit helper when it is available so editable-cursor recordings can hide the system cursor. The helper now writes ScreenCaptureKit system audio into the primary MP4 and attempts runtime-gated native microphone capture on macOS versions that expose ScreenCaptureKit microphone output. Webcam capture is currently an Electron-recorded sidecar attached to the same recording session; native AVFoundation webcam composition remains the target end state.
1. Native Session Boundary
- Add a structured macOS native recording request type.
- Add a macOS helper resolver and build script placeholders.
- Keep the helper contract process-based, matching the Windows helper boundary.
- Do not route production macOS recording through this helper until the helper is available and validated.
Acceptance:
- TypeScript build passes.
- The macOS helper path and request contract are documented and testable without affecting Windows/Linux behavior.
2. ScreenCaptureKit Display Capture
- Implement a Swift helper using ScreenCaptureKit.
- Select display captures by
displayId. - Encode H.264 MP4 through AVFoundation/VideoToolbox.
- Set
showsCursor = falsewhen editable cursor overlay mode is selected.
Acceptance:
- Display-only recording produces a valid MP4.
- The real cursor is not baked into editable-cursor recordings.
3. ScreenCaptureKit Window Capture
- Resolve Electron
window:*selections to ScreenCaptureKit window ids. - Capture
SCContentFilter(desktopIndependentWindow:). - Handle closed/minimized/protected windows with explicit errors.
- Keep window selection and capture source resolution in Electron/main, not the renderer.
Acceptance:
- Capturing a normal app window works with cursor/audio/webcam disabled.
- Unsupported windows return clear native errors.
4. System Audio
- Enable ScreenCaptureKit system audio on supported macOS versions.
- Keep audio format and timing owned by the helper.
- Encode or mux AAC audio into the primary MP4.
Acceptance:
- System-audio-only recordings produce a valid AAC track.
- Unsupported macOS versions return an explicit capability error.
5. Microphone
- Resolve the selected microphone device from the renderer-provided browser
deviceIdand user-visible label. - Capture microphone audio in the helper timing domain.
- Apply OpenScreen microphone gain policy.
- Mix system and microphone audio before final AAC output.
Acceptance:
- Mic-only and mic-plus-system recordings produce a valid, balanced AAC track.
- Device selection honors the selected microphone, not only the default device.
6. Webcam Composition
- Capture the selected camera natively through AVFoundation.
- Match browser device id first where possible, then user-visible label.
- Compose an initial picture-in-picture overlay into the primary MP4.
- Hide webcam output until the first usable frame to avoid black startup flashes.
Acceptance:
- Native display/window recordings can include webcam without returning to Electron capture.
- Selected camera is honored.
7. Runtime Controls
- Add pause/resume commands to the helper.
- Add cancel command that removes partial outputs.
- Keep restart as stop-discard-start until the helper exposes a native restart operation.
Acceptance:
- Pause/resume keeps output duration coherent.
- Cancel leaves no stale media/session files.
8. Test Pipeline
npm run build:native:mac: builds Swift helper binaries on macOS.npm run test:sck-helper:mac: display-only helper smoke test.npm run test:sck-window:mac: window capture smoke test.npm run test:sck-audio:mac: system audio smoke test when supported.npm run test:sck-mic:mac: microphone smoke test.npm run test:sck-webcam:mac: webcam smoke test when a webcam is available.- Packaging check: confirms helpers are available under
electron/native/bin/darwin-${arch}in packaged builds.
SSOT Rules
src/lib/nativeMacRecording.tsis the renderer/main TypeScript request contract.- This document is the feature-level contract and phase checklist.
- The Swift helper owns ScreenCaptureKit/AVFoundation media timing.
- Electron owns output paths, session manifests, and selected source/device resolution.
- Renderer code must use existing hooks/client APIs and should not bind directly to helper process details.
File: docs/engineering/windows-native-recorder-roadmap.md
Windows Native Recorder Roadmap
OpenScreen's Windows recorder should be owned by one native backend. Electron capture can remain available for non-Windows platforms and temporary developer diagnostics, but Windows production recording should not silently fall back to getDisplayMedia / MediaRecorder.
Goals
- Capture displays and windows through Windows Graphics Capture (WGC).
- Render the native Windows cursor as OpenScreen's high-quality scalable cursor overlay.
- Capture system audio through WASAPI loopback.
- Capture microphone audio through WASAPI.
- Mix system audio and microphone audio into the primary screen recording.
- Capture webcam video natively and compose it into the Windows helper MP4 during the native-recording migration.
- Keep preview/export aligned because screen video, audio, webcam, and cursor share one native timing origin.
- Keep exported MP4s Windows-friendly: H.264 video plus AAC audio. Opus-in-MP4 is not an acceptable Windows export target.
- Package the native helper with the Windows app.
Non-Goals
- Replacing the editor/export pipeline.
- Replacing the editor/export pipeline. A later pass can reintroduce a separate editable native
webcamVideoPath; the current Windows-native milestone prioritizes a helper-owned multi-flux MP4 with deterministic screen/audio/mic/webcam sync. - Adding a native fallback for macOS or Linux in this branch.
Target Architecture
The renderer keeps the existing recording controls. On Windows, useScreenRecorder sends a complete recording request to Electron and does not assemble Windows MediaStream tracks with MediaRecorder.
Electron owns the native recording session:
- resolves the selected source;
- resolves output paths;
- starts cursor sampling;
- starts the helper process;
- sends pause/resume/stop/cancel commands;
- writes
RecordingSessionmanifests; - reports explicit errors when a Windows-native capability is unavailable.
The helper owns Windows media capture:
- WGC screen/window frames;
- WASAPI system loopback;
- WASAPI microphone input;
- Media Foundation webcam capture;
- DirectShow webcam fallback for virtual cameras not visible to Media Foundation;
- Media Foundation encoding/muxing;
- stream timestamp normalization.
Helper Contract V2
The helper receives a single JSON argument:
{
"schemaVersion": 2,
"recordingId": 1234567890,
"source": {
"type": "display",
"sourceId": "screen:0:0",
"displayId": 123,
"windowHandle": null,
"bounds": { "x": 0, "y": 0, "width": 1920, "height": 1080 }
},
"video": {
"fps": 60,
"width": 1920,
"height": 1080,
"bitrate": 18000000
},
"audio": {
"system": { "enabled": true },
"microphone": { "enabled": true, "deviceId": "default", "gain": 1.4 }
},
"webcam": {
"enabled": true,
"deviceId": "default",
"deviceName": "Camera (NVIDIA Broadcast)",
"width": 1280,
"height": 720,
"fps": 30,
"bitrate": 18000000
},
"outputs": {
"screenPath": "C:\\Users\\me\\recording-123.mp4",
"manifestPath": "C:\\Users\\me\\recording-123.session.json"
}
}The helper emits newline-delimited JSON events to stdout:
{ "event": "ready", "schemaVersion": 2 }
{ "event": "recording-started", "timestampMs": 1234567890 }
{ "event": "warning", "code": "audio-device-unavailable", "message": "..." }
{ "event": "recording-stopped", "screenPath": "..." }
{ "event": "error", "code": "unsupported-window-source", "message": "..." }During migration, Electron also accepts the current textual helper messages so existing display-only smoke tests keep working.
Implementation Phases
1. Native Session Boundary
- Add a structured Windows native recording request type.
- Pass source kind, audio flags, microphone device, webcam flags, and output paths into the helper.
- On Windows, do not silently fall back to Electron capture. If the helper is unavailable or a native feature is missing, show a clear error.
- Keep Electron fallback only for non-Windows and optional developer diagnostics.
Acceptance:
- Display-only recording still works.
- Enabling an unsupported native feature returns an explicit native error instead of recording through Electron.
2. WASAPI System Audio
Status: initial implementation landed. The helper captures the default render endpoint with WASAPI loopback, passes the runtime mix format into MFEncoder, and muxes AAC audio into the primary MP4. Long-run drift correction and explicit silence insertion remain follow-up hardening work.
- Add
WasapiLoopbackCapture. - Capture the default render endpoint in shared loopback mode.
- Keep
WasapiLoopbackCaptureresponsible only for device activation, packet capture, and packet timestamps. - Keep
MFEncoderresponsible for all Media Foundation stream definitions and muxing. - Feed the endpoint mix format into
MFEncoderas the single source of truth for audio stream shape: sample rate, channel count, bits per sample, block alignment, average bytes/sec, and subtype (PCMorFloat). - Encode the primary screen MP4 with H.264 video and AAC audio through one
IMFSinkWriter. - Timestamp audio from the captured frame count in 100ns units. The first implementation uses the WASAPI packet timeline; later drift correction will add explicit silence or resampling if long recordings show measurable clock skew.
- Treat microphone mixing as a later phase. System loopback must land first without introducing renderer-side audio code.
Acceptance:
- Screen MP4 has an AAC audio track when system audio is enabled.
- A 5-minute recording has audio/video duration drift below one frame.
SSOT rules for this phase:
src/lib/nativeWindowsRecording.tsis the renderer/main TypeScript request contract.docs/engineering/windows-native-recorder-roadmap.mdis the feature-level contract and phase checklist.WgcSession::captureWidth()/captureHeight()is the encoded screen frame size until a dedicated native scaling stage exists.WasapiLoopbackCapture::inputFormat()is the runtime audio format source used byMFEncoder.- The renderer passes both the browser webcam
deviceIdand selected display label asdeviceName;electron/native/wgc-capture/src/webcam_capture.*is the only place that maps those values to Media Foundation devices. - Electron resolves the selected label to a DirectShow filter CLSID once and passes it as
webcamDirectShowClsid; the helper must not independently guess among DirectShow filters. - No duplicated hard-coded audio format assumptions in
main.cpp.
3. WASAPI Microphone
Status: initial implementation in progress. The helper can open the default WASAPI capture endpoint, apply the OpenScreen microphone gain, encode mic-only audio, and mix system loopback plus microphone through a single queued AudioMixer timeline when both endpoints expose the same runtime format. Audio endpoints are warmed before WGC starts, the mixer drops pre-roll and begins its paced timeline on the first encoded video frame, then cuts queued tail audio on stop so the MP4 does not drift past the video. Browser deviceId to MMDevice id mapping, resampling between mismatched endpoint formats, and drift correction remain follow-up hardening work.
- Add microphone device enumeration and stable device-id mapping.
- Capture selected/default microphone through WASAPI.
- Apply OpenScreen's current mic gain policy.
- Mix microphone and system audio before AAC encoding.
Acceptance:
- Mic-only, system-only, and mixed audio recordings produce a valid AAC track.
- Device unplug/permission failure produces an explicit error or warning.
4. Webcam Capture
- Add Media Foundation webcam source reader.
- Select requested dimensions/fps or the nearest format accepted by Media Foundation.
- Convert webcam samples to BGRA and compose them into the primary helper MP4 as an initial bottom-right picture-in-picture overlay.
- Ignore black webcam warmup frames and keep the overlay hidden until the first visible frame is available, so virtual cameras do not flash a black picture-in-picture rectangle at recording start.
- Keep the helper process as the SSOT for screen/window, WASAPI system audio, microphone, webcam, and mux timing.
- Match the requested webcam through Media Foundation friendly names first, then browser device ids/symbolic links, so UI selection remains stable across Chromium and Windows native device namespaces.
- Use the Electron-resolved DirectShow CLSID when the selected virtual camera, for example NVIDIA Broadcast, is registered for DirectShow but absent from Media Foundation enumeration.
- Later: promote the same webcam capture source to a separate editable native
webcamVideoPathif product requirements need post-recording layout edits.
Acceptance:
- Native display/window recordings can include webcam without returning to Electron capture.
npm run test:wgc-webcam:winvalidates the helper path when a webcam is available and skips explicitly when no webcam device exists.- Combined webcam + system audio + microphone produces one MP4 with H.264 video and AAC audio.
5. Native Window Capture
Status: initial implementation in progress. Electron parses the window:<HWND>:... desktop source id through the shared native Windows recording contract and passes windowHandle to the helper. The helper resolves the HWND, validates it with IsWindow, and creates the WGC item with CreateForWindow(HWND). Resize/minimize/move hardening and protected-window diagnostics remain follow-up work.
- Resolve Electron
window:*selections to anHWND. - Use WGC
CreateForWindow(HWND). - Handle window close, minimize, resize, DPI scaling, and monitor moves.
- Return clear errors for unsupported protected windows.
Acceptance:
- Capturing a normal app window works with cursor/audio/mic/webcam.
- Window resize and movement do not corrupt the recording.
6. Runtime Controls
- Add pause/resume commands to the helper.
- Add cancel command that removes partial screen/webcam outputs.
- Keep restart as stop-discard-start from Electron until the helper supports a native restart event.
Acceptance:
- Pause/resume keeps preview duration coherent.
- Cancel leaves no stale media/session/cursor files.
7. Test Pipeline
npm run test:wgc-helper:win: display-only helper smoke test.npm run test:wgc-audio:win: validates AAC track presence and duration.npm run test:wgc-window:win: captures a fixture window by HWND.npm run test:wgc-webcam:win: validates webcam output when a webcam is available, otherwise skips explicitly.- Packaging check: confirms the helper is in
app.asar.unpacked. - Export check: exported MP4s generated from native recordings keep an AAC audio track when the source has audio.
npm run test:wgc-mic:win: validates default-microphone capture writes an AAC track when an input endpoint is available.npm run test:wgc-mixed-audio:win: validates system loopback plus microphone writes one mixed AAC track when endpoint formats are compatible.
Backlog
Native Cursor Click Bounce Is Not Visibly Applied
Status: open. Do not treat Windows native cursor Click Bounce as shipped.
Problem:
- The cursor settings UI exposes
Size,Smoothing,Motion Blur, andClick Bounce. - On Windows native cursor recordings,
Size,Smoothing, andMotion Blurare visibly applied in preview/export. Click Bouncestill has no visible effect in manual packaged-app testing, even after adding click-related sample metadata.
What has already been tried:
- Added
interactionType: "click" | "mouseup" | "move"to native cursor samples. - Added polling-based left-button state through
GetAsyncKeyState. - Added the
GetAsyncKeyStatelow-bit path to catch quick clicks between samples. - Added a PowerShell/C#
WH_MOUSE_LLmouse hook experiment and launched the sampler through a temporary.ps1file to avoid Windows command-line length limits. - Updated
npm run test:cursor-native:winso the diagnostic can observe a synthetic short click and emitclickSampleCount.
Current diagnosis:
- The diagnostic can observe synthetic click events, but this has not translated into a visible
Click Bounceeffect in the real packaged app. - The test currently proves that some click metadata can be recorded, not that the full OpenScreen record -> preview -> export path displays a bounce at the expected time.
- The current native implementation may be animating from metadata that is not present in the real recording session, may be using the wrong timestamp origin, or may be applying a scale change too subtle to notice on the DOM/native cursor path.
Next investigation when resumed:
- Inspect the actual
.cursor.json/session sidecar generated by a packaged-app manual recording and confirm whether real clicks produceinteractionType: "click"at the righttimeMs. - Add a targeted end-to-end fixture that records a known click, loads the generated project, and asserts the preview/export cursor scale changes across adjacent frames.
- Compare the native DOM cursor path against the older
PixiCursorOverlayclick visual state and decide whether native cursor bounce should be a scale-only animation, an additional click ring, or a short explicit keyframe animation independent of sample cadence. - If event capture remains unreliable in the PowerShell sampler, move click events into a small native cursor helper instead of PowerShell/C# script injection.
Ship Criteria
- Windows display capture works with cursor, system audio, microphone, and webcam.
- Windows window capture works with cursor, system audio, microphone, and webcam.
- Preview and export show no cursor position drift.
- Preview and export show no measurable audio/video/webcam drift.
- Windows production builds do not depend on Electron capture fallback.
File: docs/architecture/native-bridge.md
Native Bridge Architecture
Goal
Provide a single, resilient source of truth for platform-native capabilities while keeping Electron transport thin and renderer APIs unified.
Layers
Native adapters
Platform-specific providers implement stable domain interfaces such as cursor telemetry or system asset discovery.Main-process services
Services orchestrate adapters, own runtime state, and expose domain-level operations.Unified IPC transport
Renderer code talks to a singlenative-bridge:invokechannel using versioned contracts.Renderer client
React code should consumesrc/native/client.tsrather than binding directly to ad hoc Electron APIs.
Principles
- Single source of truth: runtime-native state lives in the Electron main process.
- Capability-first: renderer can query support before attempting native behavior.
- Versioned contracts: requests and responses are explicit and evolve predictably.
- Resilience: every response uses a consistent result envelope with stable error codes.
Current rollout
This repository now contains the initial scaffold:
- shared contracts in
src/native/contracts.ts - renderer SDK in
src/native/client.ts - main-process state store in
electron/native-bridge/store.ts - cursor telemetry adapter in
electron/native-bridge/cursor/telemetryCursorAdapter.ts - domain services in
electron/native-bridge/services/* - unified handler registration in
electron/ipc/nativeBridge.ts
The legacy window.electronAPI surface still exists for backward compatibility. New native-facing features should prefer the unified bridge client.
File: electron/native/README.md
Native capture helpers
macOS
macOS native recording will use a ScreenCaptureKit helper with the same process boundary as the Windows WGC helper:
- Electron resolves the selected source, output paths, and user-selected devices.
- The helper receives one structured JSON request.
- The helper owns ScreenCaptureKit/AVFoundation capture, timing, encoding, and muxing.
- Electron persists the resulting media/session manifest and reports helper errors explicitly.
Helper locations:
OPENSCREEN_SCK_CAPTURE_EXE, for local development and diagnostics.electron/native/screencapturekit/build/openscreen-screencapturekit-helper, for locally built Swift output.electron/native/bin/darwin-arm64/openscreen-screencapturekit-helperorelectron/native/bin/darwin-x64/openscreen-screencapturekit-helper, for packaged prebuilt helpers.
The macOS cursor-shape helper is resolved from OPENSCREEN_MAC_CURSOR_HELPER_EXE first, then the matching openscreen-macos-cursor-helper binary in the same local build and packaged electron/native/bin/darwin-${arch} directories.
Build the macOS helper with:
npm run build:native:macOn non-macOS hosts this command exits successfully and does not affect Windows/Linux development. On macOS it builds the Swift package at electron/native/screencapturekit, writes the development binaries to electron/native/screencapturekit/build, and copies redistributable binaries to electron/native/bin/darwin-${arch}.
The current helper implementation supports display/window ScreenCaptureKit video capture, cursor exclusion through SCStreamConfiguration.showsCursor, H.264 encoding, MP4 muxing, and ScreenCaptureKit system audio. It also attempts native ScreenCaptureKit microphone capture when the running macOS version exposes that capability. Webcam recording currently stays as an Electron sidecar and is attached to the same recording session after the native screen capture stops.
Electron exposes is-native-mac-capture-available for capability probing. It resolves the same helper locations listed above and reports missing-helper until a Swift helper binary is present. When available, macOS recording routes screen/window capture through the native helper so editable cursor recordings do not bake the system cursor into the video. Cursor positions are sampled in Electron; when the cursor helper is available and Accessibility is granted, samples are also tagged with link/text cursor hints such as pointer.
See docs/engineering/macos-native-recorder-roadmap.md for the contract, rollout phases, and SSOT rules.
Windows
Windows native recording is resolved from one of these locations:
OPENSCREEN_WGC_CAPTURE_EXE, for local development and diagnostics.electron/native/wgc-capture/build/wgc-capture.exe, for a locally built Ninja helper.electron/native/wgc-capture/build/Release/wgc-capture.exe, for a locally built multi-config helper.electron/native/bin/win32-x64/wgc-capture.exeorelectron/native/bin/win32-arm64/wgc-capture.exe, for packaged prebuilt helpers.
Build the Windows helper with:
npm run build:native:winThe build writes the CMake output to electron/native/wgc-capture/build/wgc-capture.exe and copies the redistributable binary to electron/native/bin/win32-x64/wgc-capture.exe.
The helper contract is process-based: the app starts the process with one JSON argument and sends commands on stdin. stop\n finalizes the recording. During migration the helper prints both newline-delimited JSON events and the legacy text messages Recording started / Recording stopped. Output path: .
Current V2 JSON shape:
{
"schemaVersion": 2,
"recordingId": 123,
"sourceType": "display",
"sourceId": "screen:0:0",
"displayId": 1,
"windowHandle": null,
"outputPath": "C:\\path\\recording-123.mp4",
"videoWidth": 1920,
"videoHeight": 1080,
"fps": 60,
"captureSystemAudio": false,
"captureMic": false,
"microphoneDeviceId": "default",
"microphoneDeviceName": "Microphone (NVIDIA Broadcast)",
"microphoneGain": 1.4,
"webcamEnabled": true,
"webcamDeviceId": "default",
"webcamDeviceName": "Camera (NVIDIA Broadcast)",
"webcamWidth": 1280,
"webcamHeight": 720,
"webcamFps": 30,
"outputs": {
"screenPath": "C:\\path\\recording-123.mp4"
}
}The current helper implementation supports display/window video capture, system audio loopback, selected-microphone capture, Media Foundation webcam capture, and a DirectShow webcam fallback for virtual cameras that are not exposed through Media Foundation. Webcam frames are currently composed into the primary MP4 as a bottom-right picture-in-picture overlay. Browser deviceId values do not always map to Media Foundation symbolic links or WASAPI endpoint IDs, so the renderer passes both browser IDs and user-visible device names. For microphones, the helper tries the requested WASAPI endpoint ID first, then resolves an active capture endpoint by microphoneDeviceName, then falls back to the default endpoint. For webcams, Electron resolves a matching DirectShow filter CLSID for the selected label; the helper uses Media Foundation first, then that exact DirectShow filter when the requested camera is absent from Media Foundation.
Smoke-test the helper with:
npm run test:wgc-helper:win
npm run test:wgc-window:win
npm run test:wgc-audio:win
npm run test:wgc-mic:win
npm run test:wgc-mixed-audio:win
npm run test:wgc-webcam:winTo validate a specific native webcam manually:
$env:OPENSCREEN_WGC_TEST_WEBCAM_DEVICE_NAME = "NVIDIA Broadcast"
npm run test:wgc-webcam:win
Remove-Item Env:OPENSCREEN_WGC_TEST_WEBCAM_DEVICE_NAMETo validate a specific native microphone manually:
$env:OPENSCREEN_WGC_TEST_MICROPHONE_DEVICE_NAME = "Microphone (NVIDIA Broadcast)"
npm run test:wgc-mic:win
Remove-Item Env:OPENSCREEN_WGC_TEST_MICROPHONE_DEVICE_NAME