## File: crates/obscura/README.md

# obscura

Embeddable Rust API for the [Obscura](https://github.com/h4ckf0r0day/obscura)
headless browser. Drive a real V8 + DOM browser (`Browser`, `Page`, `Element`,
`CookieStore`) directly from Rust, with no separate process or CDP round-trips.

## Install

This crate is not published to crates.io, so depend on it via git. Building it
compiles Obscura from source, including its embedded V8 (`deno_core`), so the
first build is large and slow.

```toml
[dependencies]
obscura = { git = "https://github.com/h4ckf0r0day/obscura", features = ["api"] }
tokio = { version = "1", features = ["rt", "macros"] }
anyhow = "1"
```

## Usage

```rust,no_run
use obscura::Browser;
use std::time::Duration;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let browser = Browser::builder()
        .stealth(true)
        .storage_dir("/tmp/cookies")
        .build()?;

    let mut page = browser.new_page().await?;
    page.goto("https://example.com").await?;

    let el = page.wait_for_selector("a", Duration::from_secs(5)).await?;
    println!("{} -> {:?}", el.text(), el.attribute("href"));

    Ok(())
}
```

See `examples/basic.rs` for a runnable version (`cargo run --example basic`).

---

## File: docs/Adding-a-CDP-method-or-Web-API.md

Two recipes for the two most common extensions: a new CDP method, and a new JS Web API.

## Adding a CDP method

Worked example: `MyDomain.doThing` that takes `{ name }` and returns `{ ok }`.

### 1. Add the handler

Create or edit a file under `crates/obscura-cdp/src/domains/`:

```rust
// crates/obscura-cdp/src/domains/my_domain.rs
use serde_json::{json, Value};
use crate::dispatch::CdpContext;

pub async fn do_thing(
    params: &Value,
    _ctx: &mut CdpContext,
    _session_id: &Option<String>,
) -> Result<Value, String> {
    let name = params.get("name")
        .and_then(|v| v.as_str())
        .ok_or("missing name")?;

    // do the work

    Ok(json!({ "ok": true, "name": name }))
}
```

### 2. Register in the dispatcher

In `crates/obscura-cdp/src/dispatch.rs`, add a match arm:

```rust
"MyDomain.doThing" => domains::my_domain::do_thing(&req.params, ctx, &req.session_id).await,
```

### 3. Test it

`crates/obscura-cdp/tests/cdp_my_domain.rs`:

```rust
use obscura_cdp::dispatch::{dispatch, CdpContext};
use obscura_cdp::types::CdpRequest;
use serde_json::json;

#[tokio::test(flavor = "current_thread")]
async fn my_domain_do_thing_returns_ok() {
    let mut ctx = CdpContext::new();
    let resp = dispatch(&CdpRequest {
        id: 1,
        method: "MyDomain.doThing".into(),
        params: json!({ "name": "test" }),
        session_id: None,
    }, &mut ctx).await;

    assert!(resp.error.is_none());
    assert_eq!(resp.result.unwrap()["ok"], true);
}
```

Run:

```bash
cargo nextest run --release --features render -p obscura-cdp my_domain
```

## Adding a Web API

Worked example: `crypto.subtle.digest`, real implementation backed by a Rust hash op.

### 1. Add the Rust op

In `crates/obscura-js/src/ops.rs`:

```rust
#[op2]
#[buffer]
fn op_subtle_digest(#[string] algorithm: &str, #[buffer] data: &[u8]) -> Vec<u8> {
    use sha1::Digest as _;
    match algorithm.to_ascii_uppercase().as_str() {
        "SHA-1"   => sha1::Sha1::digest(data).to_vec(),
        "SHA-256" => sha2::Sha256::digest(data).to_vec(),
        "SHA-384" => sha2::Sha384::digest(data).to_vec(),
        "SHA-512" => sha2::Sha512::digest(data).to_vec(),
        _         => sha2::Sha256::digest(data).to_vec(),
    }
}
```

### 2. Register the op

In the same file, `build_extension()`:

```rust
ops: std::borrow::Cow::Owned(vec![
    op_dom(),
    op_console_msg(),
    // ...
    op_subtle_digest(),
]),
```

### 3. Add the JS shim

In `crates/obscura-js/js/bootstrap.js`:

```js
globalThis.crypto = globalThis.crypto || {};
globalThis.crypto.subtle = globalThis.crypto.subtle || {};
globalThis.crypto.subtle.digest = function digest(algorithm, data) {
  const algName = typeof algorithm === 'string' ? algorithm : algorithm.name;
  const bytes = data instanceof ArrayBuffer
    ? new Uint8Array(data)
    : new Uint8Array(data.buffer, data.byteOffset, data.byteLength);
  const out = Deno.core.ops.op_subtle_digest(algName, bytes);
  return Promise.resolve(out.buffer);
};
```

### 4. Add a dependency if needed

`crates/obscura-js/Cargo.toml`:

```toml
sha1 = "0.10"
sha2 = "0.10"
```

### 5. Smoke test

```bash
cargo build --release --features render
./target/release/obscura fetch https://example.com --eval "
  crypto.subtle.digest('SHA-256', new TextEncoder().encode('hi'))
    .then(buf => Array.from(new Uint8Array(buf)).map(b => b.toString(16).padStart(2, '0')).join(''))
"
```

## Tips

- Keep the JS shim thin. All side effects go through ops.
- Use `Promise.resolve` to keep async-shaped APIs callable from sync ops.
- Match the spec: Web API names and shapes are checked by Puppeteer / Playwright wrappers.
- DOM mutations go through `op_dom`, not new ops.
- For events that need to fire across handlers, use the existing `_makeListenerBox` helper in `bootstrap.js`.

## Worked examples in the tree

- CDP method with intercept: `crates/obscura-cdp/src/domains/page.rs` `do_navigate`.
- Web API with op + JS shim: `crypto.subtle.digest` (above).
- Web API in pure JS (no op): `DOMParser` in `bootstrap.js`.
- Web API with async event firing: `WebSocket`, `IntersectionObserver` in `bootstrap.js`.

---

## File: docs/Architecture-overview.md

Obscura is a workspace of nine crates.

```
obscura-cli       CLI entry point. fetch, serve, scrape, mcp.
obscura-cdp       Chrome DevTools Protocol server. WebSocket, dispatch, domain handlers.
obscura-browser   Page type, navigation, lifecycle events.
obscura-js        V8 runtime via deno_core. bootstrap.js + Rust ops.
obscura-dom       DOM tree implementation.
obscura-net       HTTP client, stealth client, cookie jar, robots cache, tracker blocklist.
obscura-mcp       Model Context Protocol server.
obscura-render    CSS cascade, retained layout, text shaping, and CPU paint.
obscura           Embeddable Rust library API (Browser, Page, Element, CookieStore).
```

## Request flow

A `Page.navigate` from a CDP client:

```
CDP client (Puppeteer)
        │ WebSocket frame
        ▼
obscura-cdp/server.rs           accept, route by sessionId
        │
        ▼
obscura-cdp/dispatch.rs         method router, acquires v8_lock
        │
        ▼
obscura-cdp/domains/page.rs     Page.navigate handler
        │
        ▼
obscura-browser/page.rs         navigate_with_wait
        │
        ├──► obscura-net/client.rs        HTTP fetch
        │
        ├──► obscura-dom/tree.rs          parse HTML into the tree
        │
        └──► obscura-js/runtime.rs        run inline scripts
                  │
                  └──► bootstrap.js + ops.rs    DOM bindings
```

The dispatcher emits CDP events (`Network.requestWillBeSent`, `Page.frameNavigated`, `Page.lifecycleEvent`) back to the client through the same WebSocket.

## Rendering flow

`obscura-render` consumes the shared DOM and computed style state. Taffy
provides the flex/grid foundation; Obscura adds browser formatting behavior,
text shaping, intrinsic replaced-element sizing, retained geometry, scrolling,
and CPU-backed paint. `obscura-js` exposes renderer-owned geometry to DOM APIs,
`obscura-browser` prepares resources and owns capture, and `obscura-cdp` maps
screenshots, screencast frames, and raster PDF output onto CDP.

Layout is retained between captures and invalidated by relevant DOM, style,
viewport, scroll, animation, font, and resource changes. The same geometry
therefore drives browser APIs and paint instead of maintaining separate
measurement and screenshot models.

Page-owned layout and paint never fetch resources synchronously. Missing
resources load through the page transport, with URL blocking checked both
for navigation/screenshot warmup and for later renderer misses. The DOM-only
`Page::screenshot` fallback (for example after `Page::suspend_js`) also performs
no network requests: external images and fonts absent from its cache remain
unavailable. Resume the page and prepare its resources before capture when
those assets are required. Standalone `obscura-render` callers retain their
existing synchronous-loader behavior.

The relevant paths are `Page::screenshot_with_animation_sample` and
`Page::render_resource_candidates` in `crates/obscura-browser/src/page.rs`,
`ObscuraJsRuntime::take_render_resource_requests` in
`crates/obscura-js/src/runtime.rs`, and `RenderResourceCache` in
`crates/obscura-render/src/paint.rs`.

## Single V8 isolate

All pages in a process share one V8 isolate. The isolate is single-threaded by design.

`obscura_js::v8_lock::global()` is a `tokio::sync::Mutex` that serializes V8 work. A handler that wants to run JS must acquire the lock first:

```rust
let _guard = obscura_js::v8_lock::global().lock().await;
page.evaluate(expr).await
```

The dispatcher routes long-running operations (navigation, eval) through `process_with_interception` in `server.rs`, which spawns the work onto the tokio `LocalSet` and releases the dispatcher to keep handling other CDP messages.

This is why `Target.createTarget` from many concurrent clients works: each `newPage` returns immediately while the actual navigation runs in a spawned task.

## Child frames

`iframe srcdoc` uses a native child realm and document, not a copy in the
parent DOM. Its URL is `about:srcdoc`; relative URLs inherit the creator's
document base, and unsandboxed Web Storage uses the creator's origin.
`srcdoc` takes precedence over `src`, including an empty value. Parser input,
connected `innerHTML`, property/attribute changes and removal use the same
loading path. Replaced or detached documents are retired before stale loads
can run. The iframe's `load` event follows classic author scripts.
Navigation runs as a posted task, after microtasks. Parsing uses srcdoc
standards mode without requiring a doctype and honors script-disabled
`noscript` content.

Sandboxed `srcdoc` enforces opaque document access and script disabling,
including inherited restrictions on nested frames. This does not implement
every HTML sandbox policy. Frame module scripts remain unsupported.
Child-frame CDP evaluation routing and iframe content compositing remain
separate compatibility gaps; loading a document does not establish support
for Playwright frame locators or iframe screenshot fidelity.

## Robustness

One page cannot hang or crash the process. `obscura-js/runtime.rs` provides a V8 termination watchdog (`arm_watchdog`, `run_event_loop_bounded`) that terminates the isolate from a separate thread when synchronous work overruns a budget, because `tokio::time::timeout` cannot preempt synchronous V8. It bounds the post-load settle, the navigation event-loop pumps, and `--eval`. The complete script phase is bounded by `OBSCURA_SCRIPT_DEADLINE_MS`; enhancement modules have a shorter per-module graph-loading/evaluation budget controlled by `OBSCURA_MODULE_BUDGET_MS`, while modules mounting an empty SPA shell receive the full script deadline. `obscura-js/cdp_watchdog.rs` is a single shared watchdog the dispatcher arms around every CDP command, so a runaway page cannot hold the V8 lock and wedge other sessions (tunable via `OBSCURA_CDP_COMMAND_TIMEOUT_MS`). `op_dom` is wrapped in `catch_unwind` so a DOM-op panic degrades to a null result instead of aborting the process through V8's FFI frame, and `obscura-dom/tree.rs` rejects cyclic reparenting that would make tree walks loop forever. Scripted `fetch()`/XHR and module network requests are timeout-bounded (`OBSCURA_FETCH_TIMEOUT_MS`), and the one-shot `fetch` CLI has a process-level hard deadline as a final backstop.

## JS bridge

`obscura-js/js/bootstrap.js` provides the browser globals: `document`, `window`, `navigator`, `location`, observers, fetch, indexedDB, etc.

`obscura-js/src/ops.rs` registers Rust ops that the bootstrap calls into:

```js
Deno.core.ops.op_dom('insert_before', parentNid, refNid, newNid);
```

Adding a Web API usually means:

1. JS shim in `bootstrap.js` that exposes the API surface.
2. Rust op in `ops.rs` that performs the side effect (DOM mutation, fetch, crypto).
3. Register the op in `build_extension()`.

Worked example: [Adding a CDP method or Web API](Adding-a-CDP-method-or-Web-API.md).

## Classic Web Workers

The JavaScript shim executes each classic Worker source once and retains its
message handlers and lexical state. Bare `onmessage` assignments target the
worker scope, and messages posted before the source loads are queued until
initialization finishes. Terminating a worker discards pending messages.

Workers remain emulated within the page runtime, not separate V8 isolates or
OS threads. This is not a complete WorkerGlobalScope implementation.

## CDP session model

Each CDP client connection gets attached to one or more targets.
Session IDs are `"{targetId}-session"`. The dispatcher routes by `sessionId` in the incoming frame to the right `Page`.

Targets are created by `Target.createTarget`. Closing the WebSocket detaches all sessions but leaves the pages running.

## Lifecycle

Lifecycle events are emitted by `obscura-browser/lifecycle.rs` as the page transitions:

```
init → commit → domcontentloaded → load → networkidle2 → networkidle0
```

`waitUntil` on `Page.navigate` blocks until the requested level is reached. The Puppeteer / Playwright `goto` resolves on the matching `Page.lifecycleEvent` client-side.

## Storage

`--storage-dir` persists cookies to `cookies.json` and `localStorage` to `localStorage/<origin>-<digest>.json`: read on process start, written after each navigation that changed something and on graceful shutdown. `sessionStorage` is origin-keyed in memory only, per tab.

## Stealth

`--stealth` swaps the default `reqwest` client for `obscura-net/wreq_client.rs`, which presents a real browser's TLS ClientHello, ALPN, and cipher order (a consistent Chrome fingerprint, not a randomized one) so the TLS layer matches the User-Agent and JS surfaces. It also applies the bundled tracker blocklist before any request leaves the process. Scripted `fetch()`/XHR go through the same stealth client, so subresource requests carry the same fingerprint as the navigation. `--stealth` is a global CLI flag that applies to `fetch`, `serve`, `scrape`, and `mcp`.

## Workspace conventions

- One crate per layer. Cross-crate calls go through the layer above, not sideways.
- All async is `tokio` with a `LocalSet` because V8 is `!Send`.
- All DOM ops go through `op_dom` to keep the JS/Rust boundary narrow.

---

## File: docs/Build-from-source.md

## Requirements

- Rust 1.75+ ([rustup.rs](https://rustup.rs))
- C compiler (gcc or clang)
- ~5 GB free disk space (V8 compiles from source on first build)

Build time depends on the machine and cache state. Cargo reuses unchanged
dependencies, but a release source edit can still require substantial compilation
and linking. For the scoped incremental runtime workflow, see
[Fast edit/test loop](Testing-and-debugging.md#fast-edittest-loop).

## Build

```bash
git clone https://github.com/h4ckf0r0day/obscura.git
cd obscura
cargo build --release -p obscura-cli --bins --features render
```

Binary is at `./target/release/obscura`.

This produces the release binary with geometry, screenshots, screencasting,
and PDF export.

## Rendering and stealth

```bash
cargo build --release -p obscura-cli --bins --features render,stealth
```

This is the complete rendering build with the stealth wreq/BoringSSL transport,
TLS fingerprint randomization, browser-identity protections, and tracker
blocklist. See [Configure stealth and proxies](Configure-stealth-and-proxies.md).

## Without rendering

```bash
cargo build --release -p obscura-cli --bins --no-default-features
cargo build --release -p obscura-cli --bins --no-default-features --features stealth
```

The second command keeps stealth while excluding layout, screenshots,
screencasting, and PDF export.

The stealth feature builds BoringSSL and generates Rust bindings. In addition
to the default requirements, install CMake, Clang, and the libclang/LLVM
development libraries. On Ubuntu/Debian:

```bash
sudo apt-get install build-essential cmake clang libclang-dev llvm-dev
```

On macOS, install the Xcode Command Line Tools and CMake. On Windows, install
the Visual Studio C++ Build Tools, CMake, and LLVM/Clang. Ensure the directory
containing `libclang` is available through `LIBCLANG_PATH` if bindgen cannot
locate it automatically.

On macOS 26 with the standalone Command Line Tools, Apple Clang may not find
libc++ while compiling BoringSSL. Use the active SDK for that build:

```bash
SDK_PATH="$(xcrun --show-sdk-path)"
SDKROOT="$SDK_PATH" CXXFLAGS="-isystem $SDK_PATH/usr/include/c++/v1" \
  cargo build --release -p obscura-cli --bins --features render,stealth
```

## OpenSSL on older systems

If the build fails on the vendored OpenSSL with an AVX-512 assembler error (common on older VPS hosts):

```bash
OPENSSL_NO_VENDOR=1 cargo build --release -p obscura-cli --bins --features render
```

Uses the system OpenSSL instead.

## Run from the build

```bash
./target/release/obscura --version
./target/release/obscura fetch https://example.com --eval "document.title"
```

Install system-wide:

```bash
cargo install --path crates/obscura-cli --features render
```

## Tests

```bash
cargo nextest run --release --features render --no-fail-fast
```

Integration suite:

```bash
python3 tests/test_all.py
```

Use `cargo nextest`, not `cargo test`: runtime tests require process isolation
because the engine owns a single V8 isolate per process.

---

## File: docs/CLI-reference.md

## `obscura`

Most top-level flags apply to every subcommand. `--storage-dir` is the
exception: it is declared on `serve` and `fetch`, and the top-level copy is read
only by `mcp`, which exports it to `OBSCURA_STORAGE_DIR`. Pass it after the
subcommand to use it with `serve` or `fetch`.

```
-v, --verbose                Enable info logging
-p, --port <PORT>            CDP port (default 9222)
    --proxy <URL>            HTTP or SOCKS5 proxy
    --stealth                Consistent browser fingerprint + tracker blocking
    --obey-robots            Respect robots.txt
    --user-agent <UA>        Override the User-Agent
    --storage-dir <DIR>      Persistent cookies and localStorage
    --allow-private-network  Permit loopback / RFC1918 / link-local
    --v8-flags <FLAGS>       Raw V8 flags, applied at startup
-h, --help                   Help
-V, --version                Version
```

## `obscura fetch `

Load a URL and print its content or an evaluated expression.

```
    --dump <FORMAT>          html | text | links | markdown | original | assets | cookies
                             (default html)
    --selector <CSS>         Narrow output to a CSS selector
    --wait <SECONDS>         Fixed post-load delay; omitted uses adaptive settle (5s cap)
    --timeout <SECONDS>      Navigation timeout (default 30)
    --wait-until <LEVEL>     domcontentloaded | load | networkidle2 | networkidle0
                             (default load)
    --user-agent <UA>        Override the User-Agent
    --proxy <URL>            HTTP or SOCKS5 proxy
    --stealth                Consistent browser fingerprint + tracker blocking (global)
-e, --eval <JS>              Evaluate JS, print the result as JSON
-o, --output <FILE>          Write to a file instead of stdout
-s, --screenshot <FILE>      Capture the settled page as PNG (single URL)
-q, --quiet                  Suppress info logging
-v, --verbose                Enable verbose logging
```

`--screenshot` requires a render-enabled build. It uses a 1280×720 viewport by
default and may be combined with `--eval`; the expression runs before capture,
which is useful for scrolling or preparing page state. It is not available in
`--file` batch mode.

When `--wait` is omitted, Obscura drives timers and async work until the page
becomes quiescent, with a five-second ceiling. Supplying `--wait N` instead
requests a fixed `N`-second delay. `--timeout` separately bounds navigation.

`--dump` values:

| Value      | Output                                                    |
| ---------- | --------------------------------------------------------- |
| `html`     | Rendered HTML (default)                                   |
| `text`     | Plain text                                                |
| `markdown` | Markdown conversion                                       |
| `links`    | Every `<a href>`, one URL per line                        |
| `assets`   | Every external resource, one JSON object per line (DOM assets plus `fetch()`/XHR requests) |
| `original` | Raw HTTP response body (binary-safe, bypasses the engine) |
| `cookies`  | All cookies in the jar as a JSON array, including HttpOnly cookies invisible to `document.cookie` |

## `obscura serve`

Run the CDP server. Puppeteer and Playwright connect over WebSocket.

```
-p, --port <PORT>            CDP port (default 9222)
    --host <HOST>            Bind host (default 127.0.0.1)
    --proxy <URL>            HTTP or SOCKS5 proxy
    --user-agent <UA>        Override the User-Agent
    --stealth                Consistent browser fingerprint + tracker blocking (global)
    --workers <N>            Worker processes (default 1)
    --worker-max-connections <N>  Recycle a worker after N CDP connections (--workers > 1)
    --worker-max-rss <MB>    Recycle a worker above this resident size (Linux)
    --worker-max-age <SECS>  Recycle a worker after this long
    --worker-drain-timeout <SECS>  Let a replaced worker finish live connections (default 120)
    --font-dir <DIR>         Recursively load fonts once per worker (repeatable; render build)
    --allow-file-access      Permit CDP clients to navigate to file:// URLs
    --storage-dir <DIR>      Persistent cookies and localStorage
    --allow-private-network  Permit loopback / RFC1918 / link-local
-q, --quiet                  Suppress info logging
-v, --verbose                Enable info logging
```

Default endpoint is `ws://127.0.0.1:9222`.

## `obscura scrape [URLS]...`

Run a JS expression across many URLs in parallel.

```
-e, --eval <JS>              JS to run on each page
    --concurrency <N>        Parallel pages (default 10)
    --format <FORMAT>        Output format (default json)
    --timeout <SECONDS>      Per-URL timeout (default 60)
    --proxy <URL>            HTTP or SOCKS5 proxy
    --stealth                Consistent browser fingerprint + tracker blocking (global)
    --allow-private-network  Permit loopback / RFC1918 / link-local
-q, --quiet                  Suppress info logging
-v, --verbose                Enable verbose logging
```

`--stealth`, `--proxy`, and `--allow-private-network` are global flags: they work before or after any subcommand, so each worker in a `scrape` run inherits stealth too.

Read URLs from stdin with `-`:

```bash
cat urls.txt | obscura scrape - --eval "document.title" --concurrency 20
```

Requires `obscura-worker` next to `obscura` in `PATH`.

## `obscura mcp`

Run obscura as an MCP server.

```
    --http                   HTTP transport instead of stdio
    --font-dir <DIR>         Load additional fonts before rendering (repeatable; render build)
    --host <HOST>            HTTP bind host (default 127.0.0.1)
    --port <PORT>            HTTP port (default 3000)
    --proxy <URL>            HTTP or SOCKS5 proxy
    --user-agent <UA>        Override the User-Agent
    --stealth                Consistent browser fingerprint + tracker blocking (global)
    --allow-private-network  Permit loopback / RFC1918 / link-local
-v, --verbose                Enable info logging
```

`--host` only applies with `--http`. The default `127.0.0.1` keeps the server loopback-only. A non-loopback bind requires `OBSCURA_MCP_TOKEN` (at least 32 bytes); use `OBSCURA_MCP_ALLOWED_ORIGINS` as well when a browser-based client needs access.

Default transport is stdio. See [Use the MCP server](Use-the-MCP-server.md).

Render-enabled builds add `browser_screenshot` and `browser_pdf` to the MCP
tool list. Streaming screencasts are available through CDP rather than MCP.

## Worker navigation responses

`obscura-worker` accepts newline-delimited JSON commands on stdin and emits one
JSON response per command on stdout. For example:

```json
{"cmd":"navigate","url":"https://example.com/"}
```

A completed navigation includes the final document's HTTP status alongside the
existing title and URL:

```json
{"ok":true,"result":{"title":"Example Domain","url":"https://example.com/","status":200}}
```

`status` is the main document's final HTTP response code after redirects, not a
script, image, or child frame's response. HTTP errors such as 404 and 503 still
return `ok: true`: the server responded and the error document can be inspected
with `dump_html`, `dump_text`, or `evaluate`. Navigation without an HTTP response,
such as `about:blank`, a `data:` URL, or a tracker blocked before the request,
returns `status: null`. A transport or
navigation failure retains the existing `{"ok":false,"error":"..."}` envelope
without a `result` object.

The status field is additive. Navigation timing and the other worker commands
are unchanged.

## Worker navigation readiness

`obscura-worker` accepts newline-delimited JSON commands on stdin and writes
one response per command to stdout. Select a readiness mode per navigation:

```json
{"cmd":"navigate","url":"https://example.com","waitUntil":"domcontentloaded"}
```

`waitUntil` accepts `load`, `domcontentloaded`, `networkidle0`, or `networkidle2`.
Omitting it preserves the existing `load` default. The option applies only to
that request. Invalid values return an `ok: false` response without navigating;
the worker remains available for subsequent commands.

These modes use the existing `Page::navigate_with_wait` behavior. Parser scripts
run in every mode. Network-idle modes additionally drive the event loop until
active requests remain at or below zero or two, respectively, for 500 ms, with
the engine's existing five-second idle deadline. `domcontentloaded` uses the
engine's DOM readiness boundary; it does not change script execution or resource
warmup semantics. Successful responses retain their existing `title`, `url`, and `status`
fields.

---

## File: docs/Configure-stealth-and-proxies.md

## Stealth mode

```bash
obscura fetch https://example.com --stealth
obscura serve --stealth
obscura scrape url1 url2 --stealth
obscura mcp --stealth
```

`--stealth` is a global flag, so it works before or after the subcommand and applies to `fetch`, `serve`, `scrape`, and `mcp`. In a `scrape` run each worker inherits it.

What `--stealth` changes:

- Uses the wreq HTTP client with browser-matching TLS fingerprints (ClientHello, ALPN, cipher order).
- Loads a tracker blocklist that drops requests to known analytics and fingerprinting endpoints.
- Bundles webpki roots instead of relying on the system store.

Requires a build that includes the stealth feature. Use a `-stealth` archive
with rendering or a `-no-render-stealth` archive without it. To build the
rendering variant yourself:

```bash
cargo build --release -p obscura-cli --bins --features render,stealth
```

Omit rendering with `cargo build --release -p obscura-cli --bins --no-default-features --features stealth`.

## What stealth handles

- Basic bot detection that checks TLS fingerprint or User-Agent.
- Sites that rely on third-party analytics being reachable.

## What stealth does not handle

- Cloudflare interactive challenges.
- Datadome and Akamai bot manager active challenges.
- CAPTCHAs.
- IP-based rate limiting (use proxies).
- **Passive TCP/IP stack fingerprinting.** Classifiers that inspect the raw SYN
  packet (tcp_options order, window size, window scaling, ip_id) can distinguish
  Linux from macOS even when the TLS and HTTP layers are identical, because those
  fields are set by the OS kernel, not by Obscura. The same binary on Linux
  (`tcp_options: "M1460,S,T,N,W10"`, window 64240) and macOS
  (`tcp_options: "M1460,N,W6,N,N,T,S,E,E"`, window 65535) produces a different
  passive fingerprint. Obscura operates in userspace and cannot override these
  kernel defaults. If TCP/IP passive detection is a concern, run Obscura inside
  a container or VM whose kernel matches the target fingerprint, or route
  connections through a proxy whose TCP stack presents the desired profile.

## Proxies

HTTP proxy:

```bash
obscura fetch https://example.com --proxy http://proxy.example.com:8080
obscura serve --proxy http://proxy.example.com:8080
```

With auth:

```bash
obscura fetch https://example.com --proxy http://user:pass@proxy.example.com:8080
```

SOCKS5:

```bash
obscura fetch https://example.com --proxy socks5://proxy.example.com:1080
```

## Custom User-Agent

```bash
obscura fetch https://example.com --user-agent "Mozilla/5.0 (...) ..."
obscura serve --user-agent "Mozilla/5.0 (...) ..."
```

Default UA matches a recent Chrome on the build platform.

## Browser profile, timezone, and geolocation

The engine presents one of a built-in pool of realistic browser profiles (a mix of Windows and macOS, recent Chrome versions). Each profile keeps `navigator.platform`, `navigator.userAgentData` (platform and platform version), and the UA string internally consistent, so the surfaces a site fingerprints agree with each other. There is no GPU renderer among them: `canvas.getContext('webgl')` returns `null`, so a page cannot read a renderer string at all.

A single stable profile is used by default. One IP cycling through different identities is itself a signal, so rotation is opt-in:

```bash
OBSCURA_PROFILE=2 obscura serve          # pin a specific profile by index
OBSCURA_ROTATE_PROFILE=1 obscura serve   # random profile per browser context
```

Timezone is driven by the process zone so `Date` (`getTimezoneOffset`, `toString`) and `Intl.DateTimeFormat` report the same region. Default is `Europe/Berlin`; set it to match the exit IP:

```bash
OBSCURA_TIMEZONE=America/New_York obscura serve
```

`navigator.geolocation` reports configurable coordinates. Set them as `lat,lon` and keep them consistent with the timezone and proxy region:

```bash
OBSCURA_GEOLOCATION="40.7128,-74.0060" obscura serve
```

Keep these aligned. A rotated or mismatched profile carries no matching TLS or timezone fingerprint, so when you pin a proxy region or TLS fingerprint, leave rotation off and set the timezone and geolocation to the same region. See [Environment variables](Environment-variables.md) for the full list.

## Combine

```bash
obscura serve \
  --stealth \
  --proxy http://user:pass@proxy.example.com:8080 \
  --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ..."
```

---

## File: docs/Connect-Puppeteer-or-Playwright.md

Obscura speaks the Chrome DevTools Protocol over WebSocket. Puppeteer and
Playwright can connect to its CDP endpoint for the supported workflows below.

## Start the server

```bash
obscura serve --port 9222
```

```
obscura listening on ws://127.0.0.1:9222
```

## Puppeteer

```bash
npm install puppeteer-core
```

```js
const puppeteer = require('puppeteer-core');

const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9222',
});

const page = await browser.newPage();
await page.goto('https://example.com');
console.log(await page.title()); // "Example Domain"

await browser.disconnect();
```

Use `puppeteer-core`, not `puppeteer`. The `puppeteer` package bundles a Chrome download.

## Playwright

```bash
npm install playwright
```

```js
const { chromium } = require('playwright');

const browser = await chromium.connectOverCDP('ws://127.0.0.1:9222');
const context = browser.contexts()[0] || await browser.newContext();
const page = await context.newPage();

await page.goto('https://example.com');
console.log(await page.title());

await browser.close();
```

Use `connectOverCDP`, not `connect`. Playwright's `connect` speaks Playwright's own protocol, which obscura does not implement.

## `waitUntil`

Default is `domcontentloaded`. For full subresource load:

```js
await page.goto('https://example.com', { waitUntil: 'load' });
```

| Value              | Returns when                            |
| ------------------ | --------------------------------------- |
| `domcontentloaded` | HTML parsed, scripts ran (default)      |
| `load`             | All subresources finished               |
| `networkidle2`     | ≤2 network connections active for 500ms |
| `networkidle0`     | 0 network connections active for 500ms  |

## Supported

- `page.goto`, `page.reload`, `page.goBack`, `page.goForward`
- `page.evaluate`, `page.evaluateHandle`
- `page.click`, `page.type`, `page.fill`, `page.focus`
- `page.waitForSelector`, `page.waitForFunction`, `page.waitForNavigation`
- `page.cookies`, `page.setCookie`, `context.cookies`
- `page.setRequestInterception`, block / modify
- `page.exposeFunction`
- `page.content`, `page.title`, `page.url`
- `page.screenshot` for viewport, clipped, and full-page capture
- `page.pdf` for raster-backed print output
- raw CDP `Page.startScreencast` with frame acknowledgements (`page.createCDPSession()`
  in Puppeteer; `context.newCDPSession(page)` in Playwright)

DOM-agent frameworks such as browser-use also connect: obscura implements `DOMSnapshot.captureSnapshot` and `Target.targetInfoChanged` for perception, and `DOM.focus` so a focused field receives `Input.dispatchKeyEvent` keystrokes.

## Capture example

```js
await page.setViewport({ width: 1440, height: 1000 });
await page.screenshot({ path: 'viewport.png' });
await page.screenshot({ path: 'full-page.png', fullPage: true });
await page.pdf({ path: 'page.pdf', format: 'A4', printBackground: true });
```

Rendering is included in official binaries and requires `--features render`
for source builds. The client-specific guides cover scrolling, raw CDP
screencasting, and current output limits.

## Raw CDP node queries

`DOM.getAttributes`, `DOM.requestNode`, `DOM.requestChildNodes`, and
`DOM.getNodeForLocation` support the top-level light DOM. Request the document
with `DOM.getDocument` before publishing frontend node IDs. Ancestors and
requested children arrive through session-scoped `DOM.setChildNodes` events;
navigation clears the node cache and emits `DOM.documentUpdated`.

Live mutation events are not implemented yet. Request the document again to
publish newly inserted children of an already-published parent. `requestChildNodes` accepts
`pierce: true` on plain light-DOM subtrees, but rejects requests that would
require frame or shadow-tree traversal. Detached-node publication and
`ignorePointerEventsNone: true` hit testing are not supported. The remaining work is tracked in
[#1291](https://github.com/h4ckf0r0day/obscura/issues/1291).

## Element visibility queries

Render builds implement `element.checkVisibility()` using the rendered flat tree.
Disconnected elements, `display: none` subtrees, `display: contents` elements,
closed-details content, and unassigned light-DOM children have no visible box.
Zero-sized boxes are not automatically rejected. Queries use the element's owning
document, including iframe documents.

The `opacityProperty` / `checkOpacity` and `visibilityProperty` /
`checkVisibilityCSS` options enable the corresponding CSS checks. Without
rendering, only inline styles are checked. CSS `content-visibility: hidden` and
`content-visibility: auto` skipping are not implemented, including the
`contentVisibilityAuto` option.
Listbox option/optgroup boxes are also not rendered, so their visibility queries
currently return `false` even when Chromium returns `true`.

Native table rows and sections can be revealed with explicit `display: table-row`,
`table-row-group`, `table-header-group`, and `table-footer-group` declarations.
CSS-created tables with explicit rows/groups share column tracks and expose row
and group geometry. Full anonymous-table fixup for mixed content is not implemented;
unsupported CSS table structures retain ordinary box layout rather than dropping
their non-table children.

## Current limits

- Pages share one V8 isolate. CPU-bound JavaScript on one page can delay others.
- PDF output is raster-backed; text is not selectable and tagged PDF,
  headers/footers, outlines, and full CSS paged media are not implemented.
- Service workers, native media playback, some Web APIs, and long-tail CSS or
  compositor effects are still incomplete relative to Chromium.

---

## File: docs/Environment-variables.md

## Runtime

### `OBSCURA_ALLOW_PRIVATE_NETWORK`

Allow fetches to loopback (`127.0.0.0/8`), RFC1918 (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), and link-local (`169.254.0.0/16`, including the `169.254.169.254` cloud-metadata endpoint) addresses. The deny-set also covers the unspecified address (`0.0.0.0` / `::`), IPv6 unique-local (`fc00::/7`), and any IPv4-mapped form of the above. Off by default to block SSRF.

The guard validates at DNS-resolution time as well as on literal hosts, so a public hostname that resolves to a forbidden address is rejected at connect time (DNS-rebinding safe), not just hosts written as raw IPs.

Truthy values: `1`, `true`, `yes`, `on`.

```bash
OBSCURA_ALLOW_PRIVATE_NETWORK=1 obscura fetch http://localhost:8080
```

Per-process equivalent: `--allow-private-network` on any subcommand.

### `OBSCURA_NAV_TIMEOUT_MS`

Hard ceiling on a single navigation. Default 30000 (30 seconds). Applies to `Page.navigate` and the CLI `fetch` command.

```bash
OBSCURA_NAV_TIMEOUT_MS=60000 obscura serve
```

### `OBSCURA_NAV_CHAIN_LIMIT`

How many documents a navigation chain may load, the first navigation included. Default 10, which allows the requested document and nine navigations the page itself triggers via `location` assignments or form submissions. Raise the value for an endpoint that chains longer for good reasons, such as an SSO handover across several providers. The low default is what stops a page that resets `location` on every load.

A zero is raised to 1. This loads the requested document. If the page wants to chain further afterwards, the call reports an error, as at any other limit. A value the engine does not read as a number is replaced by the default. This also applies to a negative value and to a value with a trailing space.

The time budget is not tied to this limit. A longer chain usually also needs a higher `OBSCURA_NAV_TIMEOUT_MS`, because its default of 30 seconds applies to the whole chain and not to the individual document.

```bash
OBSCURA_NAV_CHAIN_LIMIT=20 obscura serve
```

### `OBSCURA_SCRIPT_DEADLINE_MS`

Soft deadline for the complete page script-execution phase, including classic scripts and ES modules. Default 30000 (30 seconds). Raise it for a heavy SPA whose initial module is responsible for mounting an otherwise empty document. The engine also uses this value as a hard V8 watchdog budget, with a one-second grace period, so a synchronous script cannot run forever.

```bash
OBSCURA_SCRIPT_DEADLINE_MS=60000 obscura serve
```

### `OBSCURA_MODULE_BUDGET_MS`

Per-module graph-loading and evaluation budget for modules that enhance an already-rendered page. Default 3000 (3 seconds). Raise it when a module such as the Vite HMR client legitimately needs longer to evaluate:

```bash
OBSCURA_MODULE_BUDGET_MS=10000 obscura serve
```

This shorter budget applies when the document body already contains more than 50 descendant nodes, where modules are normally progressive enhancement and should not delay navigation indefinitely. For an unmounted SPA shell, Obscura instead gives each module the full `OBSCURA_SCRIPT_DEADLINE_MS` budget so the app has time to mount. Module network requests remain independently bounded by `OBSCURA_FETCH_TIMEOUT_MS`.

### `OBSCURA_CDP_COMMAND_TIMEOUT_MS`

Per-command deadline for the CDP server. A hung page (a runaway `Runtime.evaluate`, a synchronous DOM op) is terminated after this budget so one bad session cannot hold the shared V8 lock and stall the others. Default 60000 (60 seconds); `0` disables it. Navigation self-bounds via `OBSCURA_NAV_TIMEOUT_MS` well under this.

```bash
OBSCURA_CDP_COMMAND_TIMEOUT_MS=30000 obscura serve
```

### `OBSCURA_CDP_TOKEN`

Bearer token for the CDP discovery and WebSocket endpoints. It is optional on
loopback. A non-loopback bind is refused unless this is set to at least 32
bytes. Pass it as `Authorization: Bearer <token>` in the CDP client's headers.

```bash
OBSCURA_CDP_TOKEN="$(openssl rand -hex 32)" obscura serve --host 0.0.0.0
```

### `OBSCURA_FETCH_TIMEOUT_MS`

Request timeout for scripted `fetch()`, `XMLHttpRequest`, and ES-module loads. Without it a request to a server that accepts the connection but never responds (including a CORS preflight) hangs forever and the XHR is stuck with no completion event. Default 30000 (30 seconds).

```bash
OBSCURA_FETCH_TIMEOUT_MS=15000 obscura serve
```

### `OBSCURA_PROXY`

Default proxy URL used by `obscura-worker` for the parallel `scrape` command when no `--proxy` flag is set.

```bash
OBSCURA_PROXY=http://proxy.example.com:8080 obscura scrape - < urls.txt
```

## Stealth and identity

These tune the browser identity the engine presents so it stays internally consistent. See [Configure stealth and proxies](Configure-stealth-and-proxies.md) for the full picture.

### `OBSCURA_BLOCK_TRACKERS`

Controls the tracker blocklist used by the stealth HTTP transport. It is on by
default so `--stealth` retains its current privacy-first behavior. Set it to
`0`, `false`, `no`, or `off` to keep the stealth TLS/browser fingerprint while
allowing tracker requests.

The setting is read when a stealth client is created. Values are case-insensitive
and surrounding whitespace is ignored; unset, empty, and unrecognized values keep
blocking enabled. Non-stealth transport settings and SSRF protection are unchanged.

```bash
OBSCURA_BLOCK_TRACKERS=0 obscura --stealth fetch https://example.com
```

### `OBSCURA_TIMEZONE`

Pins the process timezone before V8/ICU reads it, so `Date` (`getTimezoneOffset`, `toString`) and `Intl.DateTimeFormat` report one consistent zone. Default `Europe/Berlin`. Set it to match the exit IP's region.

```bash
OBSCURA_TIMEZONE=America/New_York obscura serve
```

### `OBSCURA_GEOLOCATION`

Override the coordinates the `navigator.geolocation` shim reports, as `lat,lon`. Without it the shim reports a fixed default. Keep it consistent with `OBSCURA_TIMEZONE` and the proxy region.

```bash
OBSCURA_GEOLOCATION="40.7128,-74.0060" obscura serve
```

### `OBSCURA_PROFILE`

Pin a specific browser profile from the built-in pool by index (`0`-based). Each profile keeps `navigator.platform`, `userAgentData`, the UA string, and the GPU renderer internally consistent. Without it a single stable profile is used.

```bash
OBSCURA_PROFILE=2 obscura serve
```

### `OBSCURA_ROTATE_PROFILE`

Opt into picking a random profile per browser context instead of the stable default. Leave it off when you pin a TLS fingerprint, proxy region, or timezone, since a rotated profile would no longer match those.

```bash
OBSCURA_ROTATE_PROFILE=1 obscura serve
```

## MCP

### `OBSCURA_MCP_ALLOWED_ORIGINS`

Comma-separated `Origin` allowlist for the HTTP MCP transport (`obscura mcp --http`). Browser requests are refused by default; when set, only listed origins are accepted. Native, non-browser MCP clients (which send no `Origin`) are always allowed.

```bash
OBSCURA_MCP_ALLOWED_ORIGINS="https://app.example.com" obscura mcp --http --host 0.0.0.0
```

### `OBSCURA_MCP_TOKEN`

Bearer token for MCP HTTP requests. It is optional on loopback. A non-loopback
bind is refused unless this is set to at least 32 bytes.

```bash
OBSCURA_MCP_TOKEN="$(openssl rand -hex 32)" obscura mcp --http --host 0.0.0.0
```

## Logging

### `RUST_LOG`

Standard `tracing` filter. Common settings:

```bash
RUST_LOG=obscura=info obscura serve
RUST_LOG=obscura=debug obscura serve
RUST_LOG=obscura_cdp=trace,obscura_browser=debug obscura serve
```

`--verbose` on the CLI is equivalent to `RUST_LOG=obscura=info`.

## Build

### `OPENSSL_NO_VENDOR`

Forces `cargo build` to use the system OpenSSL instead of compiling the vendored copy. Set to `1` on hosts where the vendored OpenSSL fails (older VPS with AVX-512 issues).

```bash
OPENSSL_NO_VENDOR=1 cargo build --release --features render
```

## V8

V8 flags are passed via `--v8-flags`, not environment variables:

```bash
obscura --v8-flags "--max-old-space-size=2048 --expose-gc" serve
```

Defaults are `--max-old-space-size=4096 --max-semi-space-size=4 --optimize-for-size` on 64-bit systems. In the current V8, `--optimize-for-size` overrides the requested semi-space cap to **1 MiB**, not 4 MiB. The old-generation ceiling remains 4 GB.

User flags are appended after the defaults, but V8 applies flag implications afterwards. To change the young-generation cap, also disable size optimization, for example `obscura --v8-flags "--no-optimize-for-size --max-semi-space-size=8" serve`. This can reduce GC CPU at the cost of more memory; measure the complete workload before adopting it. Use `--v8-flags "--print-flag-values"` to inspect the effective settings after V8 initialization.

## HTTP proxy environment

Obscura does not honor `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`. Use `--proxy` or `OBSCURA_PROXY`.

---

## File: docs/Extract-data.md

`--dump` formats the page output without writing JavaScript.

```bash
obscura fetch https://example.com --dump html
obscura fetch https://example.com --dump text
obscura fetch https://example.com --dump markdown
obscura fetch https://example.com --dump links
obscura fetch https://example.com --dump assets
obscura fetch https://example.com --dump original
obscura fetch https://example.com --dump cookies
```

## `html`

Rendered HTML after JavaScript runs. Default.

```bash
obscura fetch https://news.ycombinator.com --dump html > hn.html
```

## `text`

Plain text. No markup.

```bash
obscura fetch https://en.wikipedia.org/wiki/Rust_(programming_language) --dump text
```

## `markdown`

Markdown conversion: headings, lists, links, code blocks, images.

```bash
obscura fetch https://docs.example.com/page --dump markdown > page.md
```

## `links`

Every `<a href>` on the page, one per line.

```bash
obscura fetch https://example.com --dump links
```

## `assets`

Every external resource (stylesheets, scripts, images, fonts, iframes), plus the URLs the page requested through `fetch()`/XHR, one JSON object per line.

```bash
obscura fetch https://example.com --dump assets
```

## `original`

The raw HTML the server sent, before JavaScript ran.

```bash
obscura fetch https://my-spa.example --dump original > before.html
obscura fetch https://my-spa.example --dump html     > after.html
diff before.html after.html
```

## `cookies`

Every cookie in the jar as a JSON array, including HttpOnly cookies that `document.cookie` cannot see. Useful for capturing session tokens set by anti-bot challenges.

```bash
obscura fetch https://example.com --dump cookies
```

## With `--wait-until`

`--dump` runs after the wait condition:

```bash
obscura fetch https://my-spa.example --wait-until load --dump markdown
```

## Pipe and redirect

```bash
obscura fetch https://example.com --dump markdown > example.md
obscura fetch https://example.com --dump text --quiet | wc -w
```

## Reader libraries

After loading a reader library such as [Mozilla Readability](https://github.com/mozilla/readability),
run it on a document clone to leave the page unchanged:

```javascript
const article = new Readability(document.cloneNode(true)).parse();
```

Obscura does not bundle Readability. Supply the library through your automation
client before evaluating this expression. Document clones preserve metadata and
live form-control state. Attribute clones retain their value and namespace.
`DOMParser` HTML documents preserve the document structure and root attributes,
inherit the parser's document URL, and do not execute scripts. XML parsing still
uses the existing limited XML implementation.

---

## File: docs/Installation.md

## Linux x86_64

```bash
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
tar xzf obscura-x86_64-linux.tar.gz
./obscura --version
```

## Linux ARM64

```bash
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-aarch64-linux.tar.gz
tar xzf obscura-aarch64-linux.tar.gz
./obscura --version
```

Linux builds target Ubuntu 22.04 and require glibc 2.35+.

## macOS Apple Silicon

```bash
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-aarch64-macos.tar.gz
tar xzf obscura-aarch64-macos.tar.gz
./obscura --version
```

## macOS Intel

```bash
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-macos.tar.gz
tar xzf obscura-x86_64-macos.tar.gz
./obscura --version
```

## Windows

Download the `.zip` from [Releases](https://github.com/h4ckf0r0day/obscura/releases), extract, run `obscura.exe --version`.

## Arch Linux (AUR)

```bash
yay -S obscura-browser
```

## Docker

```bash
docker run -d --name obscura -p 127.0.0.1:9222:9222 h4ckf0r0day/obscura
```

Image: [h4ckf0r0day/obscura](https://hub.docker.com/r/h4ckf0r0day/obscura). Built on `distroless/cc:nonroot`, with no shell or package manager in the runtime image, running as uid 65532. Note the `-p 127.0.0.1:...` above: it publishes the port to host loopback only. A mounted `--storage-dir` must be writable by uid 65532 — see [Run in production at scale](Run-in-production-at-scale.md#the-container-does-not-run-as-root).

Official archives and the Docker image include the rendering engine. Source
builders must pass `--features render`; see [Build from source](Build-from-source.md).

## From source

See [Build from source](Build-from-source.md).

## What's in the archive

- `obscura`: CLI and CDP server.
- `obscura-worker`: helper for the parallel `scrape` command. Keep both in the same directory.

Archive suffixes identify the feature set: no suffix includes rendering,
`-stealth` includes rendering and stealth, `-no-render` includes neither, and
`-no-render-stealth` includes stealth without rendering.

## Smoke test

```bash
./obscura fetch https://example.com --eval "document.title"
./obscura fetch https://example.com --screenshot smoke.png
```

Expected output: `"Example Domain"`, followed by a nonempty PNG at `smoke.png`.

## Troubleshooting

`cannot execute binary file`: wrong arch. Check `uname -m`.

`GLIBC_2.35 not found`: distro is older than Ubuntu 22.04. Use Docker or build from source.

macOS Gatekeeper warning: `xattr -d com.apple.quarantine ./obscura`.

---

## File: docs/Intercept-and-modify-requests.md

CDP `Fetch.enable` lets you inspect, block, or modify every request the page makes.

## Block by resource type

Puppeteer:

```js
await page.setRequestInterception(true);

page.on('request', req => {
  if (['image', 'media', 'font'].includes(req.resourceType())) {
    req.abort();
  } else {
    req.continue();
  }
});
```

Playwright:

```js
await page.route('**/*', route => {
  if (['image', 'media', 'font'].includes(route.request().resourceType())) {
    route.abort();
  } else {
    route.continue();
  }
});
```

## Block by URL pattern

```js
// Puppeteer
page.on('request', req => {
  const url = req.url();
  if (url.includes('google-analytics.com') || url.includes('doubleclick.net')) {
    req.abort();
  } else {
    req.continue();
  }
});
```

```js
// Playwright
await page.route(/google-analytics\.com|doubleclick\.net/, route => route.abort());
```

## Modify headers

```js
// Puppeteer
page.on('request', req => {
  req.continue({
    headers: { ...req.headers(), 'X-Custom': 'value' },
  });
});
```

```js
// Playwright
await page.route('**/*', route => {
  route.continue({
    headers: { ...route.request().headers(), 'X-Custom': 'value' },
  });
});
```

## Return a fake response

```js
// Puppeteer
page.on('request', req => {
  if (req.url().endsWith('/api/feature-flags')) {
    req.respond({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify({ newDashboard: true }),
    });
  } else {
    req.continue();
  }
});
```

```js
// Playwright
await page.route('**/api/feature-flags', route => {
  route.fulfill({
    status: 200,
    contentType: 'application/json',
    body: JSON.stringify({ newDashboard: true }),
  });
});
```

## Strip analytics in production scrapes

```js
const BLOCK = [
  'google-analytics.com',
  'googletagmanager.com',
  'doubleclick.net',
  'facebook.net',
  'segment.io',
  'mixpanel.com',
  'hotjar.com',
];

page.on('request', req => {
  if (BLOCK.some(host => req.url().includes(host))) {
    req.abort();
  } else {
    req.continue();
  }
});
```

Built-in: `--stealth` ships with a tracker blocklist that handles most of these without per-script setup. See [Configure stealth and proxies](Configure-stealth-and-proxies.md).

## From the Rust library

The patterns above drive interception over CDP from Puppeteer or Playwright. If you embed the engine with the `obscura` crate, the same capability is a native API on `Page`: `on_request` / `on_response` callbacks, an `enable_interception()` channel that can block, mock, or rewrite requests, and `add_preload_script` to run code before the page's own scripts. See [Use as a Rust library](Use-as-a-Rust-library.md#intercept-requests).

---

## File: docs/Markdown-extraction.md

`--dump markdown` converts the rendered page to markdown.

```bash
obscura fetch https://example.com --dump markdown
```

## What gets converted

- Headings (`<h1>` through `<h6>`)
- Paragraphs, line breaks
- Bold, italic, code spans
- Links (with `href`)
- Images (with `src` and `alt`)
- Ordered and unordered lists
- Block quotes
- Code blocks (``, `<code>`)
- Tables

## What gets stripped

- `<script>`, `<style>`, `<noscript>`
- Inline styles
- ARIA attributes
- Tracking pixels and beacons

## Save to file

```bash
obscura fetch https://docs.example.com/page --dump markdown -o page.md
```

## For RAG / LLM context

```bash
obscura fetch https://docs.example.com/page --dump markdown --quiet
```

`--quiet` strips info logging so the output is just markdown.

## Wait for SPA content

For pages that render content client-side:

```bash
obscura fetch https://my-spa.example --wait-until load --dump markdown
```

## Narrow to a region

`--selector` restricts the conversion to a CSS selector:

```bash
obscura fetch https://example.com --selector "main" --dump markdown
obscura fetch https://example.com --selector "article.post" --dump markdown
```

Useful for skipping nav, sidebars, and footers.