File: README.md Copy Chapter

Spacedrive

What is Spacedrive? Copy Chapter

Spacedrive is a cross-device data platform. Index files, emails, notes, and external sources. Search everything. Sync via P2P. Keep AI agents safe with built-in screening.

• Content identity — every file gets a BLAKE3 content hash. Same file on two devices produces the same hash. Spacedrive tracks redundancy and deduplication across all your machines.
• Cross-device — see all your files across all your devices in one place. Files on disconnected devices stay in the index and appear as offline.
• P2P sync — devices connect directly via Iroh/QUIC. No servers, no cloud, no single point of failure. Metadata syncs between devices. Files stay where they are.
• Cloud volumes — index S3, Google Drive, Dropbox, OneDrive, Azure, and GCS as first-class volumes alongside local storage.
• Nine views — grid, list, columns, media, size, recents, search, knowledge, and splat. QuickPreview for video, audio, code, documents, 3D, and images.
• Local-first — everything runs on your machine. No data leaves your device unless you choose to sync between your own devices.

Is this a replacement for Finder or Explorer? Copy Section

No. Spacedrive sits above your OS file manager and adds capabilities Finder/Explorer lack:

• Portal across everything — search and browse files across local disks, external drives, NAS, cloud storage, and archived data sources from one interface.
• Operating surface for files — content identity, sidecars, derivative artifacts, rich metadata, sync, and cross-device awareness built into the core model.
• Embeddable and shareable — run it as a desktop app, headless server, hosted file service, or embed the interface and APIs into other products.
• AI-ready by design — indexing and analysis pipelines prepare data ahead of time instead of giving agents raw shell access.
• Safer access model — route AI and automation through structured APIs, permissions, and processing layers instead of direct file operations.

You still use your OS for low-level file interactions. Spacedrive adds the cross-platform, cross-device, cloud-aware, and automation-friendly layer on top.

Data Archival Copy Section

Spacedrive indexes external data sources via script-based adapters: Gmail, Apple Notes, Chrome bookmarks, Obsidian, Slack, GitHub, calendar events, contacts. Each source becomes a searchable repository alongside your files.

Adapters are a folder with an adapter.toml manifest and a sync script in any language. If it reads stdin and prints lines, it works.

Shipped adapters: Gmail, Apple Notes, Chrome Bookmarks, Chrome History, Safari History, Obsidian, OpenCode, Slack, macOS Contacts, macOS Calendar, GitHub.

Spacebot Copy Section

Spacedrive integrates with Spacebot, an open source AI agent runtime. Spacebot runs as an optional separate process. Spacedrive provides the data, permission, and execution layer. Spacebot provides the intelligence.

Each Spacebot instance pairs with one Spacedrive node as its home device. That node authenticates the agent, maintains the device graph, resolves permissions, and forwards operations to peer devices. Every device in your library can reach Spacebot through the paired node over P2P (Iroh/QUIC) without direct network access. One agent runtime serves your entire device fleet.

When Spacebot spawns a worker, that worker can target any device in the library. File reads, shell commands, and operations proxy through Spacedrive to the target device. Talk to the agent from your phone while work executes on a server. Read files from a NAS, run commands on a workstation, report to a laptop — all in one task.

Every operation passes through Spacedrive's permission system: which devices the agent can access, which paths are readable or writable, which operations are allowed, and which require human confirmation. The paired node resolves effective policy before forwarding. One security model, one audit surface across all devices and clouds.

File System Intelligence Copy Section

Spacedrive adds intelligence to your filesystem by combining three layers:

• File intelligence — derivative data like OCR, transcripts, extracted metadata, thumbnails, previews, classifications, and sidecars.
• Directory intelligence — contextual knowledge attached to folders and subtrees ("active projects", "dormant archives", etc).
• Access intelligence — permissions and policy that apply across devices and clouds, routing agents through structured access instead of raw shell commands.

When an agent navigates through Spacedrive, it receives the file listing, subtree context, effective permissions, and summaries. Users can explain how they organize their system. Agents can add attributed notes. Jobs generate summaries from structure and activity. The intelligence stays attached to the filesystem, not buried in temporary session memory.

Safety Screening Copy Section

When enabled, every record passes through a safety pipeline before becoming searchable:

• Prompt Guard 2 — local classifier detects prompt injection in emails, messages, and documents before they enter the index.
• Trust tiers — authored content (your notes) gets balanced screening, external content (email inbox) gets strict screening.
• Quarantine system — flagged records excluded from AI agent queries, reviewable in desktop app.
• Content fencing — search results include trust metadata so agents know what's safe vs untrusted.

No other local data tool screens indexed content before exposing it to AI.

Architecture Copy Chapter

The core is built on four principles:

1. Virtual Distributed Filesystem (VDFS) — files and folders become first-class objects with rich metadata, independent of physical location. Every file gets a universal address (SdPath) that works across devices.

2. Content Identity System — adaptive hashing (BLAKE3 with strategic sampling for large files) creates a unique fingerprint for every piece of content. Enables deduplication, redundancy tracking, and content-based operations.

3. Transactional Actions — every file operation can be previewed before execution. See space savings, conflicts, and estimated time, then approve or cancel. Operations become durable jobs that survive network interruptions and device restarts.

4. Leaderless Sync — peer-to-peer synchronization without central coordinators. Device-specific data uses state replication. Shared metadata uses an HLC-ordered log with deterministic conflict resolution.

The implementation is a single Rust crate with CQRS/DDD architecture. Every operation (file copy, tag create, search query) is a registered action or query with type-safe input/output that auto-generates TypeScript types for the frontend.

spacedrive/
├── core/                  # Rust engine (CQRS/DDD)
├── apps/
│   ├── tauri/             # Desktop app (macOS, Windows, Linux)
│   ├── mobile/            # React Native (iOS, Android)
│   ├── cli/               # CLI and daemon
│   ├── server/            # Headless server
│   └── web/               # Browser client
├── packages/
│   ├── interface/         # Shared React UI
│   ├── ts-client/         # Auto-generated TypeScript client
│   ├── ui/                # Component library
│   └── assets/            # Icons, images, SVGs
├── crates/                # Standalone Rust crates (ffmpeg, crypto, etc.)
├── adapters/              # Script-based data source adapters
└── schemas/               # TOML data type schemas

Getting Started Copy Chapter

Requires Rust 1.81+, Bun 1.3+, just, and Python 3.9+ (for adapters).

git clone https://github.com/spacedriveapp/spacedrive
cd spacedrive

just setup        # bun install + native deps + cargo config
just dev-desktop  # launch the desktop app (auto-starts daemon)
just test         # run all workspace tests

Privacy & Security Copy Chapter

Spacedrive is local-first. Your data stays on your devices.

• End-to-End Encryption — all P2P traffic encrypted via QUIC/TLS
• At-Rest Encryption — libraries can be encrypted on disk (SQLCipher)
• No Telemetry — zero tracking or analytics
• Self-Hostable — run your own relay servers
• Data Sovereignty — you control where your data lives

Optional cloud integration is available for backup and remote access, but it's never required. The cloud service runs unmodified Spacedrive core as a standard P2P device—no special privileges.

Contributing Copy Chapter

• Join Discord to chat with developers and community
• Contributing Guide
• Adapter Guide — write a data source adapter
• SpaceUI — shared design system (clone alongside Spacedrive to work on UI)

License Copy Chapter

FSL-1.1-ALv2 — Functional Source License, converting to Apache 2.0 after two years.

File: apps/server/README.md Copy Chapter

Spacedrive Server

HTTP server for Spacedrive with embedded daemon (RPC only, no web UI).

Overview Copy Chapter

sd-server runs the Spacedrive daemon and exposes RPC endpoints over HTTP. Perfect for:

• NAS deployments (TrueNAS, Unraid, Synology, etc.)
• Headless servers
• Remote access to your Spacedrive libraries
• Docker/container environments
• CLI-only usage with sd-cli

Architecture Copy Chapter

┌─────────────────────────────────────────┐
│         sd-server (HTTP Server)         │
│  ┌───────────────────────────────────┐  │
│  │  Axum HTTP Server (Port 8080)     │  │
│  │  ├─ /health (healthcheck)         │  │
│  │  └─ /rpc (proxy to daemon)        │  │
│  └───────────────────────────────────┘  │
│               ↓                          │
│  ┌───────────────────────────────────┐  │
│  │  Embedded Daemon                  │  │
│  │  (Unix socket: daemon.sock)       │  │
│  │  ├─ Core VDFS                     │  │
│  │  ├─ Indexing                      │  │
│  │  ├─ P2P Networking                │  │
│  │  └─ File Operations               │  │
│  └───────────────────────────────────┘  │
└─────────────────────────────────────────┘

Unlike Tauri (desktop app), the server:

• Embeds the daemon instead of spawning a separate process
• Provides only RPC endpoints (no bundled web UI)
• Proxies RPC requests to daemon via Unix socket
• Provides basic auth for security

Quick Start Copy Chapter

Development (without Docker) Copy Section

1. Build the server:

   bash Copy

   cargo build -p sd-server

2. Run the server:

   bash Copy

   # Development mode (creates temp data dir)
   cargo run -p sd-server
   
   # Production mode (requires DATA_DIR)
   DATA_DIR=/path/to/data cargo run -p sd-server --release

3. Access the RPC endpoint:

   • Health check: http://localhost:8080/health
   • RPC endpoint: http://localhost:8080/rpc
   • Default auth: disabled in dev mode

Docker Deployment (Recommended) Copy Section

Perfect for TrueNAS, Unraid, or any Docker-compatible NAS.

1. Create a .env file:

   bash Copy

   # REQUIRED: Set your credentials
   SD_AUTH=admin:your-secure-password
   
   # Optional: Change port
   PORT=8080
   
   # Optional: Disable auth (NOT RECOMMENDED)
   # SD_AUTH=disabled

2. Start with docker-compose:

   bash Copy

   cd apps/server
   docker-compose up -d

3. Access the server:

   • Navigate to http://your-nas-ip:8080
   • Login with credentials from .env

Configuration Copy Chapter

Environment Variables Copy Section

Authentication Copy Section

IMPORTANT: Always set SD_AUTH in production!

# Single user
SD_AUTH=admin:securepassword123

# Multiple users
SD_AUTH=admin:pass1,user:pass2,readonly:pass3

# Disable (NOT RECOMMENDED - only for trusted networks)
SD_AUTH=disabled

Uses HTTP Basic Authentication. The server will return 401 Unauthorized if credentials don't match.

Data Storage Copy Section

The server stores all data in DATA_DIR:

$DATA_DIR/
├── daemon/
│   └── daemon.sock          # Unix socket for RPC
├── libraries/
│   └── *.sdlibrary/         # Library databases
├── logs/                     # Application logs
└── current_library_id.txt   # Last opened library

Docker volumes: Mounted at /data inside the container.

TrueNAS Setup Copy Chapter

Using TrueNAS SCALE (Docker) Copy Section

1. Navigate to Apps in TrueNAS web UI

2. Click "Launch Docker Image"

3. Configure:

   • Image: Build locally or use pre-built image
   • Port: Map 8080 to host
   • Volume: Mount /mnt/pool/spacedrive to /data
   • Environment:
     • SD_AUTH=admin:yourpassword
     • TZ=America/New_York (your timezone)

4. Add storage pools (optional):

   • Mount your datasets as read-only volumes
   • Example: /mnt/tank/photos → /photos in container

Manual Docker Run Copy Section

docker run -d \
  --name spacedrive \
  -p 8080:8080 \
  -p 7373:7373 \
  -v /mnt/pool/spacedrive:/data \
  -v /mnt/pool/media:/media:ro \
  -e SD_AUTH=admin:password \
  -e TZ=UTC \
  --restart unless-stopped \
  spacedrive/server:latest

Building Copy Chapter

# Build server (RPC only)
cargo build --release -p sd-server

# Run server
./target/release/sd-server --data-dir /path/to/data

You can connect with:

• sd-cli (CLI client)
• Custom HTTP clients via /rpc
• Tauri desktop app configured to connect to this server
• Future web UI (not yet implemented)

Development Workflow Copy Chapter

# Run server in dev mode
cargo run -p sd-server

# Server starts on http://localhost:8080
# Use sd-cli or custom client to interact with RPC endpoint

API Endpoints Copy Chapter

`GET /health` Copy Section

Health check endpoint.

Response: 200 OK with body "OK"

`POST /rpc` Copy Section

JSON-RPC proxy to daemon.

Request:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "query:libraries.list",
  "params": { "include_stats": false }
}

Response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [...]
}

Comparison: Server vs Tauri Copy Chapter

Both use the same Spacedrive core!

Troubleshooting Copy Chapter

Server won't start Copy Section

• Check DATA_DIR exists and is writable
• Verify port 8080 is not in use: lsof -i :8080
• Check logs: RUST_LOG=debug cargo run -p sd-server

Can't connect to daemon Copy Section

• Ensure daemon.sock exists in $DATA_DIR/daemon/
• Check daemon logs in $DATA_DIR/logs/
• Try removing stale socket: rm $DATA_DIR/daemon/daemon.sock

Authentication failing Copy Section

• Verify SD_AUTH format: username:password
• Check browser is sending Basic Auth header
• Test with curl:
  bash Copy

  curl -u admin:password http://localhost:8080/health

Docker build failing Copy Section

• Ensure you're building from repository root:
  bash Copy

  docker build -f apps/server/Dockerfile .

• Check Docker has enough memory (4GB+ recommended)

Contributing Copy Chapter

The server app is part of the Spacedrive v2 monorepo.

Project structure:

apps/server/
├── src/
│   └── main.rs          # Server implementation
├── Cargo.toml           # Dependencies
├── Dockerfile           # Container image
└── docker-compose.yml   # Docker setup

Making changes:

1. Server code: Edit apps/server/src/main.rs
2. Daemon integration: See core/src/infra/daemon/

License Copy Chapter

FSL-1.1-ALv2 - See LICENSE file in repository root.

File: apps/gpui-photo-grid/README.md Copy Chapter

GPUI Photo Grid

High-performance photo grid viewer for Spacedrive built with GPUI.

Features Copy Chapter

• GPU-accelerated rendering with GPUI
• Grid layout with configurable columns and thumbnail size
• Direct connection to Spacedrive daemon
• Loads up to 1000 media files per directory
• Smart thumbnail variant selection

Building Copy Chapter

cargo build --release

Running Copy Chapter

Make sure Spacedrive daemon is running, then:

export SD_LIBRARY_ID="your-library-uuid"
export SD_SOCKET_PATH="$HOME/.spacedrive/daemon.sock"  # optional
export SD_HTTP_URL="http://127.0.0.1:54321"            # optional
export SD_INITIAL_PATH="/"                              # optional

cargo run

Helper Script Copy Chapter

Use the included script to automatically find your library and run:

./run.sh

Environment Variables Copy Chapter

• SD_LIBRARY_ID (required) - Your Spacedrive library UUID
• SD_SOCKET_PATH (optional) - Path to daemon socket (default: ~/.spacedrive/daemon.sock)
• SD_HTTP_URL (optional) - HTTP server URL (default: http://127.0.0.1:54321)
• SD_INITIAL_PATH (optional) - Initial directory path (default: /)

Architecture Copy Chapter

• Uses sd-client crate to communicate with Spacedrive daemon
• Queries files via Unix socket
• Loads thumbnails via HTTP from sidecar server
• GPUI handles image loading, caching, and GPU rendering

Performance Copy Chapter

Current implementation:

• Loads 1000 files at a time
• Renders all visible thumbnails
• Uses GPUI's built-in image caching

Future optimizations:

• Virtual scrolling (only render visible rows)
• Texture streaming with LRU cache
• Infinite scroll with pagination
• Batch image loading

File: core/src/infra/extension/README.md Copy Chapter

WASM Extension System

Status: Basic structure integrated, compiling successfully

This module provides Spacedrive's WebAssembly-based extension system, enabling secure, sandboxed plugins.

What's Implemented Copy Chapter

Core Infrastructure Copy Section

• manager.rs - PluginManager for loading/unloading WASM modules (Wasmer integration)
• host_functions.rs - Skeleton for host_spacedrive_call() and host_spacedrive_log()
• permissions.rs - Capability-based security with rate limiting
• types.rs - Extension manifest format and types

Dependencies Added Copy Section

wasmer = "4.2"
wasmer-middlewares = "4.2"

The Design Copy Chapter

Key Insight: ONE generic host function reuses the entire Wire/Registry infrastructure.

// WASM extensions import:
extern "C" {
    fn spacedrive_call(method, library_id, payload) -> result;
}

// Host function routes to existing registry:
host_spacedrive_call()
  ↓
RpcServer::execute_json_operation()  // EXISTING!
  ↓
LIBRARY_QUERIES/ACTIONS.get()        // EXISTING!
  ↓
Operation::execute()                  // EXISTING!

Result: Zero code duplication. WASM extensions use same operations as CLI/GraphQL/daemon clients.

What's NOT Implemented Yet Copy Chapter

Pending Work Copy Section

1. WASM Memory Interaction (host_functions.rs)

• Read/write strings from WASM linear memory
• Read/write JSON payloads
• UUID handling
• Guest allocator integration

2. Full Wire Bridge (host_functions.rs)

• Call RpcServer::execute_json_operation()
• Permission checking before operation
• Error handling and propagation

3. Extension Operations (core/src/ops/)

• ai.ocr - OCR operation
• ai.classify_text - AI classification
• credentials.store/get - Credential management
• vdfs.write_sidecar - Sidecar file operations

4. Test WASM Module

• Simple "hello world" .wasm file
• Calls spacedrive_call() to test integration
• Validates permission system

5. Extension SDK (separate crate)

• spacedrive-sdk Rust crate
• Type-safe wrapper around spacedrive_call()
• Ergonomic API for extension developers

Next Steps Copy Chapter

Immediate (This Week) Copy Section

1. Implement WASM Memory Helpers

   • Study Wasmer 4.2 API documentation
   • Implement read_string_from_wasm()
   • Implement write_json_to_wasm()
   • Test with simple WASM module

2. Complete host_spacedrive_call()

   • Bridge to execute_json_operation()
   • Add permission checking
   • Error handling

3. Create Test WASM Module

   • Rust project that compiles to WASM
   • Calls spacedrive_call() with test payload
   • Validates round-trip works

Week 2-3 Copy Section

4. Add Extension Operations

   • Implement ai.ocr (Tesseract integration)
   • Implement credentials.store/get
   • Implement vdfs.write_sidecar

5. Build Extension SDK

   • Create spacedrive-sdk crate
   • Type-safe wrappers
   • Documentation

Week 4+ Copy Section

6. Finance Extension
   • Email scanning
   • Receipt processing
   • Full end-to-end test

Architecture Documents Copy Chapter

• WASM_ARCHITECTURE_FINAL.md - Quick reference
• EXTENSION_IPC_DESIGN.md - Detailed design
• EMAIL_INGESTION_EXTENSION_DESIGN.md - Finance extension
• PLATFORM_REVENUE_MODEL.md - Business model

Example Usage (Future) Copy Chapter

// In Spacedrive Core
let mut plugin_manager = PluginManager::new(core.clone(), plugins_dir);
plugin_manager.load_plugin("finance").await?;

// Extension (WASM) calls:
let result = spacedrive_call(
    "query:ai.ocr",
    library_id,
    json!({ "data": pdf_bytes, "options": { "language": "eng" } })
);

Testing Copy Chapter

# Check compilation
cd core && cargo check

# Run tests (once implemented)
cd core && cargo test extension

# Load test plugin (once implemented)
cargo run --bin spacedrive extension load ./plugins/test-plugin

Notes Copy Chapter

• Memory Management: WASM modules must export wasm_alloc(size: i32) -> *mut u8
• Error Handling: Errors returned as JSON { "error": "message" }
• Permissions: Checked on every spacedrive_call()
• Rate Limiting: 1000 requests/minute default

Last Updated: October 2025 - Initial integration

File: core/src/ops/network/sync_setup/README.md Copy Chapter

Library Sync Setup System

Overview Copy Chapter

This module handles library synchronization setup between paired devices. It is intentionally separate from the pairing protocol to maintain clean separation between networking concerns (device authentication) and application concerns (library data organization).

Architecture Copy Chapter

Why Separate from Pairing? Copy Section

1. Separation of Concerns: Pairing is a pure networking operation, while library setup involves application-level business logic
2. User Flexibility: Users may want to pair devices without immediately syncing libraries
3. Progressive Enhancement: Can start simple and evolve independently when sync is fully implemented
4. Clean Testing: Each component tests independently

Components Copy Section

Actions

• LibrarySyncSetupAction (action.rs): CoreAction that sets up library sync between two paired devices
  • Currently implements RegisterOnly mode: registers devices in each other's libraries
  • Future modes: MergeIntoLocal, MergeIntoRemote, CreateShared (awaits full sync implementation)

Queries

• DiscoverRemoteLibrariesQuery (discovery/query.rs): CoreQuery that discovers libraries on a remote paired device
  • Validates device is paired
  • Returns library metadata (name, stats, etc.)
  • Currently returns empty list (requires messaging protocol extension)

Message Types

• LibraryMessage (service/network/protocol/library_messages.rs): Protocol messages for library operations
  • DiscoveryRequest/DiscoveryResponse: Library discovery
  • RegisterDeviceRequest/RegisterDeviceResponse: Device registration

Current Implementation Status Copy Chapter

Phase 1: Basic Structure (Complete) Copy Section

• Directory structure and module organization
• Input/Output types with proper serialization
• CQRS action and query registration
• Device registration in local library database
• Validation of paired devices
• Error handling and logging

Phase 2: Network Implementation (Pending) Copy Section

• LibraryMessage handler in messaging protocol
• Actual library discovery over network
• Remote device registration requests
• Bi-directional library setup

Phase 3: Full Sync Support (Future) Copy Section

• Library merging (MergeIntoLocal, MergeIntoRemote)
• Shared library creation
• Conflict resolution
• Sync job initialization
• Sync leadership management

Usage Flow Copy Chapter

// 1. User pairs two devices (via pairing protocol)
PairGenerateAction::execute() // Device A
PairJoinAction::execute()     // Device B
// Result: Devices are now paired

// 2. User initiates library sync setup
DiscoverRemoteLibrariesQuery {
    device_id: paired_device_id
}
// Result: List of libraries on remote device (currently empty)

// 3. User selects libraries and sync action
LibrarySyncSetupAction {
    local_device_id: device_a_id,
    remote_device_id: device_b_id,
    local_library_id: my_library_id,
    remote_library_id: their_library_id,
    action: LibrarySyncAction::RegisterOnly,
    leader_device_id: device_a_id,
}
// Result: Devices registered in each other's libraries

Database Changes Copy Chapter

The action registers paired devices in library databases:

INSERT INTO device (uuid, name, os, os_version, ...)
VALUES (remote_device_id, 'Remote Device', 'Desktop', '1.0', ...);

This enables:

• Future sync operations
• Device-to-device file operations (Spacedrop)
• Multi-device library awareness

Integration Points Copy Chapter

With Pairing System Copy Section

• Depends on: Device must be in Paired or Connected state
• Location: core/src/service/network/protocol/pairing/

With Library Manager Copy Section

• Uses: LibraryManager::get_library() for validation
• Accesses: Library database for device registration
• Location: core/src/library/manager.rs

With Networking Service Copy Section

• Uses: NetworkingService::device_registry() for device info
• Future: Will use messaging protocol for remote operations
• Location: core/src/service/network/core/

With Sync System (Future) Copy Section

• Will integrate with: Sync jobs, leader election, merge operations
• Design: See docs/core/design/SYNC_DESIGN.md lines 245-311
• Location: TBD (core/src/sync/ when implemented)

API Endpoints Copy Chapter

Query: `query:network.sync_setup.discover` Copy Section

Input:

{
	"device_id": "uuid-of-paired-device"
}

Output:

{
	"device_id": "uuid",
	"device_name": "Device Name",
	"libraries": [
		{
			"id": "uuid",
			"name": "My Library",
			"description": "Optional description",
			"created_at": "2025-01-01T00:00:00Z",
			"statistics": {
				"total_entries": 1000,
				"total_locations": 5,
				"total_size_bytes": 1000000,
				"device_count": 2
			}
		}
	],
	"is_online": true
}

Action: `action:network.sync_setup.input` Copy Section

Input:

{
	"local_device_id": "uuid",
	"remote_device_id": "uuid",
	"local_library_id": "uuid",
	"remote_library_id": "uuid",
	"action": {
		"type": "RegisterOnly"
	},
	"leader_device_id": "uuid"
}

Output:

{
	"success": true,
	"local_library_id": "uuid",
	"remote_library_id": "uuid",
	"devices_registered": true,
	"message": "Devices successfully registered for library access"
}

Testing Recommendations Copy Chapter

1. Unit Tests: Test action validation logic
2. Integration Tests: Test device registration in database
3. E2E Tests: Test full flow from pairing → discovery → setup
4. Error Cases: Test with unpaired devices, invalid libraries, etc.

Future Enhancements Copy Chapter

See SYNC_DESIGN.md for full sync system design including:

• Library merging strategies
• Conflict resolution
• Sync jobs and state machines
• Leader election
• Dependency-aware sync protocol

Notes Copy Chapter

• Remote operations currently log TODO warnings
• Full implementation awaits messaging protocol extension
• Design aligns with provisional sync architecture
• Maintains backward compatibility with existing pairing system

File: core/README.md Copy Chapter

Spacedrive Core

Rust library implementing the Virtual Distributed File System (VDFS) architecture for local-first, AI-native file management.

Architecture Copy Chapter

Structure Copy Section

src/
├── domain/        # Core data models (Entry, Library, Device)
├── ops/           # Operations (actions and queries, CQRS pattern)
├── infra/         # Infrastructure (database, events, wire protocol)
├── service/       # High-level services (network, jobs, sessions)
├── location/      # Location management and indexing
├── library/       # Library lifecycle and operations
├── device/        # Device identity and management
├── volume/        # Volume detection and fingerprinting
├── config/        # Application configuration
├── crypto/        # Cryptographic primitives
└── bin/           # Binaries (cli, daemon)

Core Components Copy Section

Core Manager (Core struct in lib.rs)

• Coordinates all subsystems
• Manages application lifecycle
• Provides unified access to capabilities

Library (library/)

• File-based storage (.sdlibrary directories)
• SQLite database with SeaORM
• Job management and thumbnail generation
• Device registration and sync coordination

Entry-Centric Model (domain/entry.rs)

• Unified representation for files and directories
• Conditional UUIDs (directories immediate, files after content ID)
• On-demand UserMetadata creation
• Relative paths from location roots

Device Management (device/)

• Single device identity per installation
• Persistent across restarts (device.json)
• Sync leadership model per library
• Network address tracking for P2P

Operations Layer (ops/)

• CQRS pattern: Actions (mutations) and Queries (reads)
• Wire protocol with automatic type generation
• Registry system using inventory crate
• Shared across daemon RPC, iOS FFI, and WASM extensions

Infrastructure (infra/)

• api/: Wire protocol dispatcher and RPC server
• event/: Event bus for state changes
• action/: Transactional action system with preview-commit-verify
• query/: Read-optimized query handlers

Networking (service/network/)

• Iroh-based P2P networking
• Device pairing protocol
• File transfer (Spacedrop)
• mDNS local discovery

Jobs (service/jobs/)

• Durable, resumable operations
• MessagePack serialization for state
• Progress reporting and cancellation
• Per-library job managers

Indexing (location/indexer/)

• Five-phase pipeline (discover, classify, extract, thumbnail, cleanup)
• File system watcher integration
• Rules engine with glob patterns
• Resumable with checkpoints

Volume Management (volume/)

• Cross-platform volume detection
• Fingerprinting for identity
• Mount point tracking

Communication Patterns Copy Chapter

Daemon-Client (Desktop/CLI) Copy Section

• Unix socket JSON-RPC 2.0
• Wire method strings (e.g., query:vdfs.list_entries)
• Operations auto-register at compile time

Embedded FFI (iOS/Mobile) Copy Section

• Direct Rust library integration
• Same JSON-RPC protocol over FFI
• Swift client with Specta-generated types

Extensions (WASM) Copy Section

• Sandboxed WASM modules
• Minimal host functions (log, register_job)
• SDK with procedural macros (crates/sdk/)
• Models, jobs, actions, agents, UI manifests

Key Technologies Copy Chapter

• Async runtime: tokio
• Database: SQLite via SeaORM
• Serialization: serde, Specta for type generation
• Networking: Iroh (P2P), mdns-sd (local discovery)
• WASM: wasmer runtime, spacedrive-sdk
• Jobs: inventory for registration, rmp-serde for state
• Crypto: blake3 (content addressing), ed25519 (signing)
• Indexing: notify (fs watcher), globset (rules)

Building Copy Chapter

# Full build
cargo build --release

# With optional features
cargo build --features ffmpeg,ai,heif

# Specific binary
cargo build --bin spacedrive
cargo build --bin daemon

# Run CLI
cargo run --bin spacedrive -- --help

Binaries Copy Chapter

• spacedrive: CLI interface
• daemon: Background daemon process

Development Copy Chapter

• Uses CQRS and DDD patterns
• Operations register via inventory crate macros
• Resumable jobs with MessagePack state serialization
• Type-safe Wire protocol with Specta generation
• Event-driven architecture with EventBus
• No layered architecture (direct Rust patterns)

Testing Copy Chapter

# All tests
cargo test

# Specific module
cargo test --lib location::indexer

# Integration tests
cargo test --test indexer_test

See /docs/core/ for detailed architecture documentation.

File: crates/sdk/README.md Copy Chapter

Spacedrive Extension SDK

Beautiful, type-safe API for building Spacedrive WASM extensions.

Installation Copy Chapter

Add to your extension's Cargo.toml:

[dependencies]
spacedrive-sdk = { path = "../spacedrive-sdk" }

Quick Start Copy Chapter

use spacedrive_sdk::prelude::*;
use spacedrive_sdk::ExtensionContext;

#[no_mangle]
pub extern "C" fn plugin_init() -> i32 {
    spacedrive_sdk::ffi::log_info("Extension started!");
    0 // Success
}

fn process_receipt(ctx: &ExtensionContext, pdf_data: &[u8]) -> Result {
    // Create entry
    let entry = ctx.vdfs().create_entry(CreateEntry {
        name: "Receipt: Starbucks".into(),
        path: "receipts/1.pdf".into(),
        entry_type: "FinancialDocument".into(),
        metadata: None,
    })?;

    // Run OCR
    let ocr_result = ctx.ai().ocr(pdf_data, OcrOptions::default())?;

    // Store result
    ctx.vdfs().write_sidecar(entry.id, "ocr.txt", ocr_result.text.as_bytes())?;

    // Classify with AI
    let receipt = ctx.ai().classify_text(
        &ocr_result.text,
        "Extract vendor, amount, date from this receipt. Return JSON."
    )?;

    ctx.vdfs().write_sidecar(entry.id, "receipt.json",
        serde_json::to_vec(&receipt)?.as_slice()
    )?;

    Ok(entry.id)
}

No Unsafe, No FFI, Just Clean Rust Copy Chapter

Before (raw C bindings):

#[link(wasm_import_module = "spacedrive")]
extern "C" {
    fn spacedrive_call(
        method_ptr: *const u8,
        method_len: usize,
        library_id_ptr: u32,
        payload_ptr: *const u8,
        payload_len: usize
    ) -> u32;
}

// Then manually:
// - Serialize to JSON
// - Get pointer to string
// - Call unsafe function
// - Read result from returned pointer
// - Deserialize JSON
// - Handle errors

After (with SDK):

let entry = ctx.vdfs().create_entry(CreateEntry { ... })?;
let ocr = ctx.ai().ocr(&pdf_data, OcrOptions::default())?;

That's it! Clean, type-safe, ergonomic.

API Reference Copy Chapter

VDFS Operations Copy Section

// Create entries
let entry = ctx.vdfs().create_entry(CreateEntry {
    name: "My File".into(),
    path: "path/to/file".into(),
    entry_type: "Document".into(),
    metadata: None,
})?;

// Write sidecars (store metadata/analysis)
ctx.vdfs().write_sidecar(entry.id, "metadata.json", data)?;

// Read sidecars
let data = ctx.vdfs().read_sidecar(entry.id, "metadata.json")?;

// Update metadata
ctx.vdfs().update_metadata(entry.id, json!({ "category": "Important" }))?;

AI Operations Copy Section

// OCR
let ocr_result = ctx.ai().ocr(&pdf_bytes, OcrOptions {
    language: "eng".into(),
    engine: OcrEngine::Tesseract,
    preprocessing: true,
})?;

// Text classification
let result = ctx.ai().classify_text(
    &text,
    "Extract structured data from this receipt"
)?;

// Embeddings
let embedding = ctx.ai().embed("search query text")?;

Credential Management Copy Section

// Store OAuth token
ctx.credentials().store("gmail_oauth", Credential::oauth2(
    access_token,
    Some(refresh_token),
    3600, // expires_in_seconds
    vec!["https://www.googleapis.com/auth/gmail.readonly".into()]
))?;

// Get credential (auto-refreshes if OAuth)
let cred = ctx.credentials().get("gmail_oauth")?;

// Delete credential
ctx.credentials().delete("old_credential")?;

Job System Copy Section

// Dispatch background job
let job_id = ctx.jobs().dispatch("email_scan", json!({
    "provider": "gmail"
}))?;

// Check status
match ctx.jobs().get_status(job_id)? {
    JobStatus::Running { progress } => {
        ctx.log(&format!("Job {}% complete", progress * 100.0));
    }
    JobStatus::Completed => {
        ctx.log("Job done!");
    }
    _ => {}
}

// Cancel job
ctx.jobs().cancel(job_id)?;

Building Your Extension Copy Chapter

# Build for release (optimized for size)
cargo build --target wasm32-unknown-unknown --release

# Output: target/wasm32-unknown-unknown/release/your_extension.wasm

Required Exports Copy Chapter

Your extension must export these functions:

#[no_mangle]
pub extern "C" fn plugin_init() -> i32 {
    spacedrive_sdk::ffi::log_info("Extension starting!");
    0 // Return 0 for success
}

#[no_mangle]
pub extern "C" fn plugin_cleanup() -> i32 {
    spacedrive_sdk::ffi::log_info("Extension cleanup");
    0 // Return 0 for success
}

The SDK automatically provides wasm_alloc and wasm_free - you don't need to implement them!

manifest.json Copy Chapter

{
  "id": "my-extension",
  "name": "My Extension",
  "version": "0.1.0",
  "description": "What my extension does",
  "author": "Your Name",
  "wasm_file": "my_extension.wasm",
  "permissions": {
    "methods": ["vdfs.", "ai.ocr", "credentials."],
    "libraries": ["*"],
    "rate_limits": {
      "requests_per_minute": 1000
    },
    "max_memory_mb": 512
  }
}

Error Handling Copy Chapter

All operations return Result<T, Error>:

use spacedrive_sdk::prelude::*;

fn my_operation(ctx: &ExtensionContext) -> Result<()> {
    let entry = ctx.vdfs().create_entry(...)?;  // ? operator works!
    ctx.vdfs().write_sidecar(entry.id, "data.json", data)?;
    Ok(())
}

Logging Copy Chapter

ctx.log("Info message");
ctx.log_error("Error message");

// Or directly:
spacedrive_sdk::ffi::log_info("Message");
spacedrive_sdk::ffi::log_debug("Debug");
spacedrive_sdk::ffi::log_warn("Warning");
spacedrive_sdk::ffi::log_error("Error");

Examples Copy Chapter

See extensions/test-extension/ for a complete working example.

**Beautiful API. Zero unsafe code. Just Rust. **

File: crates/sd-client/README.md Copy Chapter

sd-client

Rust client library for connecting to the Spacedrive daemon.

Features Copy Chapter

• Unix socket communication with Spacedrive core
• Type-safe query and action execution
• Media file listing queries
• Thumbnail URL construction
• Smart thumbnail variant selection

Usage Copy Chapter

use sd_client::{SpacedriveClient, SdPath};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create client
    let mut client = SpacedriveClient::new(
        "/path/to/daemon.sock".into(),
        "http://localhost:54321".into(),
    );

    // Set library context
    client.set_library("library-uuid".to_string());

    // Query media files
    let files = client.media_listing(
        SdPath::Physical {
            device_id: "local".to_string(),
            path: "/Users/you/Photos".to_string(),
        },
        Some(1000),
    ).await?;

    // Get thumbnail URLs
    for file in files {
        if let Some(content_id) = file.content_identity {
            if let Some(thumb) = client.select_best_thumbnail(&file.sidecars, 256.0) {
                let url = client.thumbnail_url(
                    &content_id.uuid,
                    &thumb.variant,
                    &thumb.format,
                );
                println!("{}: {}", file.name, url);
            }
        }
    }

    Ok(())
}

Example Copy Chapter

Run the test connection example:

export SD_LIBRARY_ID="your-library-uuid"
export SD_SOCKET_PATH="$HOME/.spacedrive/daemon.sock"  # optional
export SD_HTTP_URL="http://127.0.0.1:54321"            # optional

cargo run --example test_connection

API Copy Chapter

SpacedriveClient Copy Section

• new(socket_path, http_base_url) - Create a new client
• set_library(library_id) - Set the current library context
• execute(wire_method, input) - Execute a query or action
• media_listing(path, limit) - Query media files
• thumbnail_url(content_uuid, variant, format) - Construct thumbnail URL
• select_best_thumbnail(sidecars, target_size) - Pick optimal thumbnail variant

Types Copy Section

• File - File metadata with content identity and sidecars
• ContentIdentity - Content hash and UUID for deduplication
• Sidecar - Generated derivatives (thumbnails, proxies, etc.)
• SdPath - Location-independent file reference
• ImageMediaData - Image-specific metadata (dimensions, date taken)
• VideoMediaData - Video-specific metadata (dimensions, duration)

File: crates/log-analyzer/README.md Copy Chapter

Log Analyzer

A powerful log analysis system for Spacedrive that parses, patterns, and collapses repetitive log entries into queryable insights.

Features Copy Chapter

• Pattern Detection: Automatically identifies log templates and variables
• Collapse Repetitions: Groups consecutive similar logs with statistics
• Sequence Detection: Identifies repeating multi-step patterns (e.g., Commit → Emit × 1000)
• Phase-Based Summary: Aggregates activity by time windows (e.g., "5s intervals showing operation counts")
• Condensed Timeline: Collapses sequences while preserving chronological order
• Queryable Storage: SQLite database for advanced queries with sequences table
• Timeline Generation: Visual timeline of log activity
• Multiple Output Formats: Markdown reports, JSON export, and phase summaries
• Extreme Compression: 99.5% compression achieved on real Spacedrive logs (73,028 lines → 398 sequences)

Quick Start Copy Chapter

Recommended: Phase Summary Copy Section

For most debugging tasks, start with the phase summary:

cargo run --example analyze_sync_log --features cli -- phases test.log --duration 5

This aggregates 85,000+ interleaved log lines into 5-second windows showing operation counts, perfect for understanding sync flow.

As a Library Copy Section

use log_analyzer::LogAnalyzer;

let analyzer = LogAnalyzer::from_file("test.log")?;

// Phase-based summary (most useful)
let summary = analyzer.generate_phase_summary(5)?;  // 5s windows
println!("{}", summary);

// Statistics
println!("Compression: {:.1}%", analyzer.compression_ratio() * 100.0);

As a CLI Tool Copy Section

# Phase summary (recommended - shows "what happened when")
cargo run --example analyze_sync_log --features cli -- phases test.log --duration 5

# Statistics (quick overview)
cargo run --example analyze_sync_log --features cli -- stats test.log

# Condensed timeline (preserves chronological order)
cargo run --example analyze_sync_log --features cli -- condense test.log

# Full analysis with markdown report
cargo run --example analyze_sync_log --features cli -- analyze test.log

# Export to JSON with database
cargo run --example analyze_sync_log --features cli -- analyze test.log \
    --format json \
    --output analysis.json \
    --database analysis.db

How It Works Copy Chapter

1. Parse Copy Section

Extracts structured components from each log line:

2025-11-16T07:19:57.232531Z DEBUG ThreadId(02) sd_core::sync::peer: Recorded ACK peer=1817e146
                 ↓
{
  timestamp: 2025-11-16T07:19:57.232531Z,
  level: DEBUG,
  thread: ThreadId(02),
  module: "sd_core::sync::peer",
  message: "Recorded ACK peer=1817e146"
}

2. Pattern Copy Section

Identifies templates by comparing similar messages:

Log A: "Recorded ACK peer=1817e146"
Log B: "Recorded ACK peer=8ef7a321"
         ↓
Template: "Recorded ACK peer={UUID}"

3. Collapse Copy Section

Groups consecutive instances with statistics:

Template: "Recorded ACK peer={UUID}"
Count: 1992
Duration: 96ms
Variables:
  - UUID: 1992 unique values

4. Detect Sequences Copy Section

Identifies repeating multi-step patterns:

Pattern: [Commit, Emit Event]
Detected: This 2-step sequence repeats 1000× times
Result: 2000 groups → 1 sequence (compression!)

5. Phase-Based Summary (NEW - Most Useful!) Copy Section

Aggregates operations by time windows for high-level flow understanding:

cargo run --example analyze_sync_log --features cli -- phases test.log --duration 5

Output:

## 09:15:28 → 09:15:33 (5s, 12,450 events)

### Key Operations

[4,028×] transaction: Committing device-owned data (entry)
[3,829×] sync_transport: [MockTransport] Delivering message to target
[3,829×] peer: State change applied successfully (entry)
[2,015×] peer: Broadcasting shared change (content_identity)
  [1,936×] peer: Shared change applied successfully
  [  200×] dependency: Added dependency tracking

### By Module

  sd_core::service::sync: 8,450 events
  sd_core::infra::sync: 2,100 events
  sync_realtime_integration_test::helpers: 1,900 events

Benefits:

• Shows "what happened when" without chronological noise
• Highlights warnings/errors per phase
• Aggregates thousands of interleaved async operations
• Perfect for understanding sync flow at a glance

6. Store & Query Copy Section

Save to SQLite for advanced queries:

-- Query sequences
SELECT id, repetitions, description, group_count
FROM sequences
ORDER BY repetitions DESC;

-- Query templates
SELECT template_id, COUNT(*) as count
FROM log_instances
WHERE timestamp BETWEEN '07:19:57' AND '07:20:00'
GROUP BY template_id;

Architecture Copy Chapter

Log File → Parser → Pattern Detector → Collapse Engine → Sequence Detector → Database
                                                                 ↓
                                                       Analysis & Reports

Analysis Pipeline Copy Section

1. Parser: Extract structured log components (73,028 logs)
2. Pattern Detector: Identify templates (102 unique patterns)
3. Collapse Engine: Group consecutive repetitions (64,709 groups)
4. Sequence Detector: Find repeating multi-step patterns (398 sequences)
5. Result: 99.5% compression ratio!

Examples Copy Chapter

See examples/ directory:

• simple.rs - Basic library usage
• analyze_sync_log.rs - Full-featured CLI tool

Testing Copy Chapter

cargo test -p log-analyzer

Performance Copy Chapter

Real-world results from Spacedrive sync test logs:

• Input: 73,028 log lines (40MB+)
• Parsing: ~1 second
• Templates: 102 unique patterns detected
• Groups: 64,709 collapsed groups
• Sequences: 398 multi-step patterns
• Compression: 99.5% (73,028 → 398)

Example sequences detected:

• 1000× "Commit → Emit Event" (2-step pattern)
• 790× "Track Dependency → Add Tracking" (2-step pattern)
• 499× "Commit → Emit → Commit → Emit" (4-step pattern)

Output Modes Copy Chapter

1. Phase Summary (Best for Understanding Flow) Copy Section

Shows aggregated operations per time window - answers "what happened when?"

## 09:15:37 → 09:15:42 (5s, 46,581 events)

[5,374×] Delivering message to target
[3,576×] State change sent successfully
[1,798×] Shared change applied successfully
   [1,798×] Broadcasting shared change
   [1,798×] ACK sent successfully

Use when: Debugging sync flow, understanding system behavior

2. Condensed Timeline (Best for Chronological View) Copy Section

Shows collapsed sequences preserving order - answers "in what sequence?"

[999× SEQUENCE] 09:15:28 → 09:15:38 (2 steps, 10s total)
  Step 1: Committing device-owned data (entry)
  Step 2: Sync event emitted (StateChange)

09:15:38 Location created successfully
[100×] 09:15:38 Indexing jobs still running (200ms)

Use when: Timeline analysis, event ordering matters

3. Full Report (Best for Documentation) Copy Section

Markdown/JSON with complete statistics and templates

Use when: Sharing analysis, generating documentation

Use Cases Copy Chapter

Sync Test Debugging Copy Section

Instead of manually scrolling through 85,000 log lines, get:

Phase Summary:

• See that 4,028 entries committed in 09:15:28-33 window
• See that 3,829 applied successfully (239 missing = stuck in dependencies)
• Spot 24 warnings in 09:15:37-42 window

Condensed Timeline:

• 999× "Commit→Emit" sequence collapsed to 1 line
• Easy anomaly spotting: Missing steps in expected sequences

Performance Analysis Copy Section

• Measure throughput: "20,750 operations/sec"
• Identify bottlenecks: Sequences with long durations
• Compare test runs: Query database for regression analysis

CI Integration Copy Section

• Automated log verification in tests
• Assert expected sequence patterns
• Fail builds on missing or broken sequences

Example Query Copy Section

-- Find the most frequently repeating sequences
SELECT
    s.id,
    s.repetitions,
    s.description,
    s.group_count,
    s.template_sequence
FROM sequences s
ORDER BY s.repetitions DESC
LIMIT 10;

Design Copy Chapter

See /docs/core/design/LOG_ANALYSIS_SYSTEM.md for complete design specification.