{"owner":"h4ckf0r0day","repo":"obscura","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md"],"files":{"AGENTS.md":"# AGENTS.md\n\nGuidance for AI coding agents and contributors working in the Obscura repo.\nThis is the non-obvious stuff you can't infer from the code; read it before\nbuilding, testing, or changing anything.\n\nObscura is a headless browser engine in Rust. It runs real JavaScript through\nV8 (`deno_core`), keeps a real DOM tree, owns its layout and paint pipeline,\nspeaks the Chrome DevTools Protocol, and is a drop-in replacement for headless\nChrome with Puppeteer and Playwright. Rendering and stealth are both first-class\ncapabilities. It targets web scraping and AI-agent automation.\n\n## Build\n\n```bash\nCARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release -p obscura-cli --bins --features render\n\n# Rendering and stealth\nCARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release -p obscura-cli --bins --features render,stealth\n\n# No rendering, with rustls or stealth\nCARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release -p obscura-cli --bins --no-default-features\nCARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release -p obscura-cli --bins --no-default-features --features stealth\n```\n\n- The first build compiles V8 from source: ~5 minutes and a few GB of disk.\n  Incremental builds are seconds.\n- **Iterating on one crate? Scope it:** `cargo build -p obscura-cli`. A bare\n  `cargo build` can re-link the whole workspace; the V8 compile is the cost, so\n  avoid touching it when you don't need to.\n- **Stealth:** `--features render,stealth` retains the complete rendering\n  surface and adds the wreq/BoringSSL transport, fingerprint protections, and\n  tracker blocklist. BoringSSL builds through CMake, so `cmake` must be\n  installed. The rendering build uses rustls and needs neither CMake nor OpenSSL.\n- If the vendored OpenSSL build hits an AVX-512 assembler error on your host,\n  build with `OPENSSL_NO_VENDOR=1`.\n\n## Test\n\nRun tests with **`cargo nextest`, not `cargo test`**:\n\n```bash\ncargo nextest run --release --features render -p <crate>\ncargo nextest run --release --features render --no-fail-fast\n```\n\n`cargo test` runs the whole test binary in one process, but the engine holds a\nsingle V8 isolate per process, so the runtime tests fail under it. `nextest`\nruns each test in its own process, which is the only supported way.\n\nThe authoritative behavioral gate is the **obstacle course** in the companion\nrepo `obscura-benchmark` (33 capability + speed stages, must stay 33/33):\n\n```bash\nOBSCURA_BIN=./target/release/obscura python3 obstacle-course/run.py --runs 1 --warmup 0\n```\n\nIt serves local fixtures, so it is deterministic and offline. WPT conformance\nand the real-world render corpus also live in that repo; report WPT as subtest\npass %, not whole-file pass.\n\n## Before you finish\n\nFor any code change:\n\n1. Run focused release-mode nextest coverage for the crates and repro involved.\n2. Run `cargo nextest run --release --features render --no-fail-fast`.\n3. Run the exact release build shown above.\n4. The obstacle course still reports **33/33**.\n5. For render changes, run deterministic fixtures and broad top/bottom real-site\n   captures using the methodology below.\n6. For stealth changes, re-test with `--stealth` (a non-stealth binary won't\n   exercise the `wreq` path).\n\nDo not bulk-run `cargo fmt`: the tree is not rustfmt-clean, so a blanket format\nproduces a huge unrelated diff. Match the surrounding style in the files you\nedit instead.\n\n## Architecture\n\n- **obscura-cli** — CLI: `fetch` (`--dump assets|html|text|links|markdown|original|cookies`, `--eval <JS>`, `--screenshot <PNG>`), `serve` (CDP server), `scrape`, `mcp`. `--proxy`, `--stealth`, and `--allow-private-network` are global flags: valid before or after the subcommand and applied to `fetch`, `serve`, `scrape`, and `mcp` (a `scrape` run forwards `--stealth` to each worker via `OBSCURA_STEALTH`).\n- **obscura-cdp** — Chrome DevTools Protocol server (WebSocket). Managed page\n  sessions use `\"{targetId}-session\"`; explicit flattened attachments receive\n  distinct session ids so Playwright and Puppeteer can open raw page sessions.\n- **obscura-js** — V8/`deno_core` runtime. `js/bootstrap.js` is the DOM/browser shim; `src/ops.rs` bridges JS to Rust DOM ops; `src/runtime.rs` owns the isolate and the per-page `ObscuraState`.\n- **obscura-dom** — DOM tree (`src/tree.rs`).\n- **obscura-net** — HTTP client (`client.rs`), stealth client (`wreq_client.rs`), cookie jar, robots cache, tracker blocklist.\n- **obscura-browser** — the `Page` type, navigation, JS evaluation.\n- **obscura-render** — selector cascade, computed style, retained layout,\n  scrolling, text shaping, images/SVG/canvas, and CPU-backed paint. The\n  `render` feature powers geometry, screenshots, CDP screencasting, and PDF.\n- **obscura-mcp** — stateful MCP automation tools. Render builds expose\n  `browser_screenshot` and `browser_pdf`; streaming screencasts remain CDP-only.\n- **obscura** — embeddable Rust library API (git dependency; builds V8 locally, not on crates.io). Public request-interception API on `Page`: `add_preload_script`, `enable_interception` (channel of `InterceptedRequest`, resolved with `InterceptResolution::{Continue, Fulfill, Fail}`), and passive `on_request` / `on_response`. `op_fetch_url` invokes these for JS `fetch()`/XHR, so when touching it keep a `Continue` URL rewrite behind `validate_fetch_url` (the SSRF gate, same as redirects).\n\n## Conventions\n\n- **Performance is a hard constraint** (Obscura is ~12x faster and uses ~6x less\n  memory than headless Chrome on framework pages). Keep native Rust fast paths;\n  add a JS fallback only for real spec edge cases. Benchmark old and new\n  revisions interleaved with the same release build, page, network, viewport,\n  settle policy, and capture path. Report distributions and resource use; the\n  noise floor is about plus or minus 10%.\n- **Keep ops panic-safe.** `op_dom` is wrapped in `catch_unwind` so a DOM-op\n  panic returns null instead of aborting the process inside V8's FFI frame. New\n  ops must not unwind into V8.\n- **Commits/PRs/comments:** short and factual, no em dashes, no AI filler.\n\n## Rendering verification\n\nUse deterministic fixtures before real sites. Put generated output in a\ndisposable directory outside the repository:\n\n```bash\nRUN_ROOT=\"$(mktemp -d)\"\nOBSCURA_BIN=./target/release/obscura render-repros/run.sh \"$RUN_ROOT/fixtures\"\nOBSCURA_BIN=./target/release/obscura render-repros/representative-suite/run.sh \"$RUN_ROOT/top\"\nOBSCURA_BIN=./target/release/obscura render-repros/representative-suite/run.sh \"$RUN_ROOT/bottom\" bottom\n```\n\nThe harness accepts `BASELINE_BIN` or `CHROMIUM_BIN` for paired output. A\nlatency-only run may use `SUITE_MODE=latency SETTLE_MS=0`, but zero settle is\nnot valid fidelity evidence.\n\nCompare the same viewport, device scale, identity, network inputs, settle\npolicy, scroll position, animation time, and capture boundary. First confirm\nboth navigations succeeded and both images are nonblank. Then inspect missing\nresources, geometry, text flow, structural edges, clipping, fixed/sticky\nbehavior, and a reduced fixture. Pixel-distance metrics are useful regression\ntripwires, not standalone correctness verdicts. Never add hostname-specific\nlayout, style, or resource behavior.\n\n`render-repros/**` is the tracked public evidence harness. Git-ignored internal\nhandover notes are private working material: do not edit them, link them from\npublic documentation, stage them, or commit them. Do not commit generated\nscreenshots or reports.\n\n## Gotchas\n\n- **DOM mutation arg order:** `insertBefore` / `replaceChild` in `bootstrap.js`\n  pass reference-node vs parent nid in a way that's easy to break. If you touch\n  mutation methods, verify `before()`, `after()`, `replaceWith()`, and\n  `replaceChild()` on connected elements.\n- **Multi-statement `--eval` starting with `const` returns `null`** (V8 gives\n  `const` an empty completion value). Wrap snippets in an IIFE:\n  `(function(){ ...; return result; })()`.\n- **`canAccessOpener` must be in every `TargetInfo` payload**, or strict CDP\n  clients (chromiumoxide) panic.\n- **The DOM reparenting guards in `tree.rs` are load-bearing.** `append_child` /\n  `insert_before` reject cycles (inserting an ancestor of the target is a no-op).\n  A cyclic reparent used to make `descendants()` loop forever and hang the\n  engine on real sites, uninterruptible by tokio or the watchdog. Keep the\n  guards and the `descendants()` length cap.\n- **SSRF:** loopback / RFC1918 / link-local fetches are blocked by default. Use\n  `--allow-private-network` (or `OBSCURA_ALLOW_PRIVATE_NETWORK=1`) for local\n  testing.\n\n## Robustness invariants (do not remove)\n\nOne page must never hang or crash a worker:\n\n- The V8 termination watchdog (`arm_watchdog` / `disarm_watchdog` in\n  `runtime.rs`) terminates the isolate from a separate thread, because\n  `tokio::time::timeout` only cancels at await points and synchronous V8 work\n  runs unbounded.\n- The CLI applies a process-level hard deadline as an absolute backstop.\n- `panic = \"unwind\"` is pinned in the release profile so `catch_unwind` works.\n\n## Stealth\n\nThe stealth features (the `wreq` client, fingerprint and browser-identity\nadjustments) are privacy-first anti-fingerprinting: they present a normal,\nconsistent browser fingerprint (user agent, timezone, navigator properties, and\nsimilar surfaces) so ordinary automation traffic is not singled out. They\ncontain no bot or automation-abuse payload.\n\n## Recommended proxy provider: NodeMaven\n\nFor production scraping you usually want residential or ISP IPs rather than a\ndatacenter address, since many sites block datacenter ranges. We use\n**NodeMaven** and it works well with Obscura: residential/ISP proxies, sticky\nsessions, and per-request country/region targeting. A good default if you don't\nalready have a provider.\n\nRoute Obscura through it with the global `--proxy` flag (use your own\ncredentials):\n\n```bash\n# HTTP\nobscura --proxy http://USER:PASS@gate.nodemaven.com:8080 fetch https://example.com --dump text\n\n# SOCKS5\nobscura --proxy socks5://USER:PASS@gate.nodemaven.com:1080 scrape url1 url2 --concurrency 25\n```\n\nThe targeting options (country, region, session id) live in the proxy username,\nso a sticky session keeps the same exit IP across requests.\n\nSign up: https://go.nodemaven.com/obscuraagent\n\nObscura discount codes: `OBSCURA35` (35% off mobile and residential),\n`OBSCURA40` (40% off ISP / static).\n"}}