{"owner":"six2dez","repo":"reconftw","hasSkills":true,"totalSkillsCount":2,"totalTokensCount":9472,"categories":["claude-rule","project-spec"],"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md","docs/ARCHITECTURE.md"],"skills":{"CLAUDE.md":"\n## Project\n\n**reconFTW**\n\nreconFTW is a comprehensive bash-based reconnaissance automation framework used by bug bounty hunters, penetration testers, and security researchers. It orchestrates 70+ external security tools (Go, Python, Rust) across subdomain enumeration, web probing, OSINT, and vulnerability scanning, producing structured per-target output trees with optional Axiom distributed execution, AI reporting, monitor/incremental mode, and Slack/Telegram/Discord notifications.\n\n**Core Value:** Run one command, get a complete recon picture of a target — passive, active, and vulnerability layers — with resumable checkpoints, structured output, and zero-touch tool orchestration.\n\n### Constraints\n\n- **Tech stack**: Bash 4.3+ — Required for `wait -n`, `mapfile`, associative arrays. macOS users must have Homebrew bash; auto re-exec is best-effort.\n- **External tools**: 70+ runtime dependencies — Most install via `go install @latest` (no version pinning), which is convenient but a known supply-chain risk.\n- **Single process**: All modules sourced into one shell — No subshell isolation between modules; all state shared via globals. Workflow functions must save/restore globals they override (see `passive()` pattern).\n- **Resume semantics**: Checkpoint files are touch-once at `end_func` — Interrupted functions re-run from scratch on next invocation; partial outputs are not detected.\n- **Single-operator**: Designed for one user per target run — No locking, no multi-user state, no concurrent runs against the same target dir.\n- **Output stability**: `Recon//` tree is a public contract — Subdirectory names and filenames are consumed by downstream pipelines, scripts, and parsers; renames are breaking changes.\n- **macOS compatibility**: GNU coreutils + GNU sed + GNU getopt required — System BSD versions are not supported.\n- **CI budget**: Integration-full is weekly cron — Unit + smoke are per-push; adding heavy integration tests must respect this split.\n\n\n\n## Technology Stack\n\n## Languages\n- Bash 4+ - All core framework logic (`reconftw.sh`, `modules/*.sh`, `lib/*.sh`, `install.sh`)\n- Python 3.7+ - Python-backed tools (each runs in isolated `uv venv`): `dorks_hunter`, `CMSeeK`, `EmailHarvester`, `Spoofy`, `SSTImap`, `gato`, `regulator`, `reconftw_ai`, `getjswords.py`\n- Go (latest, min ~1.21) - Primary binary language for ~55 security tools installed via `go install @latest`\n## Runtime\n- Linux (Debian/Ubuntu/RHEL/Arch) or macOS (Apple Silicon / Intel)\n- Docker: Ubuntu 24.04 base image (`Docker/Dockerfile`)\n- ARM64/aarch64, ARMv6l/v7l, and x86_64 all supported for Go binary installs\n- `getopt` (GNU getopt required on macOS via `brew install gnu-getopt`)\n- `nproc` / `sysctl -n hw.ncpu` for CPU core auto-detection\n- `timeout` / `gtimeout` (macOS Homebrew `coreutils`)\n- `gnu-sed` (macOS requires `brew install gnu-sed`)\n- `gnu-coreutils` (macOS requires `brew install coreutils`)\n- Go tools: `go install` (`GOPATH=$HOME/go`, `GOROOT=/usr/local/go`)\n- Python tools: `uv tool install` from GitHub (most) or PyPI (`fray`)\n- Python repo venvs: `uv venv venv && uv pip install -r requirements.txt`\n- Lockfile: None (always installs `@latest`)\n## Frameworks\n- No framework — pure Bash with sourced module files\n- Module loading order: `lib/validation.sh` → `lib/common.sh` → `lib/ui.sh` → `lib/parallel.sh` → `modules/utils.sh` → `modules/core.sh` → `modules/osint.sh` → `modules/subdomains.sh` → `modules/web.sh` → `modules/vulns.sh` → `modules/axiom.sh` → `modules/modes.sh`\n- bats-core (Bash Automated Testing System)\n- GNU make (`Makefile`) — test, lint, format targets\n- shellcheck (error-level) — `make lint`, pre-commit hook\n- shfmt (4-space indent, `-bn`, `-ci`) — `make fmt`, pre-commit hook\n- pre-commit hooks defined in `.pre-commit-config.yaml`\n- semgrep: `.github/workflows/semgrep.yml` (CI only)\n## Key Dependencies\n### Go Tools (installed via `go install @latest`)\n- `subfinder` (projectdiscovery/subfinder) — passive multi-source subdomain discovery\n- `github-subdomains` (gwen001/github-subdomains) — GitHub-based subdomain search\n- `gitlab-subdomains` (gwen001/gitlab-subdomains) — GitLab-based subdomain search\n- `dnstake` (pwnesia/dnstake) — subdomain takeover detection\n- `puredns` (d3mondev/puredns) — mass DNS resolution with wildcard filtering\n- `dnsx` (projectdiscovery/dnsx) — DNS toolkit\n- `massdns` (blechschmidt/massdns) — via repo clone + build\n- `dsieve` (trickest/dsieve) — subdomain filtering\n- `enumerepo` (trickest/enumerepo) — GitHub org repo enumeration\n- `gotator` (Josue87/gotator) — subdomain permutations\n- `analyticsrelationships` (Josue87/analyticsrelationships) — Google Analytics pivoting\n- `roboxtractor` (Josue87/roboxtractor) — robots.txt extractor\n- `crt` (cemulus/crt) — crt.sh search\n- `asnmap` (projectdiscovery/asnmap) — ASN-to-CIDR mapping\n- `mapcidr` (projectdiscovery/mapcidr) — CIDR manipulation\n- `smap` (s0md3v/smap) — passive Shodan-powered port scan\n- `tlsx` (projectdiscovery/tlsx) — TLS certificate harvesting\n- `hakip2host` (hakluke/hakip2host) — reverse IP lookup\n- `cdncheck` (projectdiscovery/cdncheck) — CDN/WAF IP classification\n- `hakoriginfinder` (hakluke/hakoriginfinder) — origin IP discovery behind CDN\n- `inscope` (tomnomnom/hacks/inscope) — scope filtering\n- `csprecon` (edoardottt/csprecon) — CSP-based subdomain discovery\n- `favirecon` (edoardottt/favirecon) — favicon-based tech recon\n- `httpx` (projectdiscovery/httpx) — multi-probe HTTP toolkit\n- `katana` (projectdiscovery/katana) — web crawler\n- `ffuf` (ffuf/ffuf) — web fuzzer\n- `subjs` (lc/subjs) — JavaScript URL extractor\n- `Gxss` (KathanP19/Gxss) — reflected XSS param finder\n- `jsluice` (BishopFox/jsluice) — JS secret/URL extractor\n- `sourcemapper` (denandz/sourcemapper) — JS source map extractor\n- `mantra` (brosck/mantra) — JS/secret scanner\n- `urlfinder` (projectdiscovery/urlfinder) — URL discovery\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — via uv\n- `nmapurls` (sdcampbell/nmapurls) — URL extraction from Nmap XML\n- `naabu` (projectdiscovery/naabu) — fast port scanner\n- `VhostFinder` (wdahlenburg/VhostFinder) — virtual host discovery\n- `shortscan` (bitquark/shortscan) — IIS short filename scanner\n- `nuclei` (projectdiscovery/nuclei) — template-based scanner\n- `dalfox` (hahwul/dalfox) — XSS scanner\n- `crlfuzz` (dwisiswant0/crlfuzz) — CRLF injection scanner\n- `Web-Cache-Vulnerability-Scanner` (Hackmanit) — web cache poisoning\n- `TInjA` (Hackmanit/TInjA) — SSTI scanner\n- `toxicache` (xhzeem/toxicache) — web cache poisoning\n- `second-order` (mhmdiaa/second-order) — broken link/second-order injection\n- `s3scanner` (sa7mon/s3scanner) — S3/GCS/Azure Blob misconfiguration\n- `misconfig-mapper` (intigriti/misconfig-mapper) — third-party misconfiguration\n- `sj` (BishopFox/sj) — Swagger/OpenAPI analysis\n- `grpcurl` (fullstorydev/grpcurl) — gRPC reflection scanner\n- `nerva` (praetorian-inc/nerva) — service fingerprinting\n- `brutus` (praetorian-inc/brutus) — credential spraying\n- `julius` (praetorian-inc/julius) — LLM endpoint probe\n- `titus` (praetorian-inc/titus) — secrets engine\n- `notify` (projectdiscovery/notify) — multi-channel notifications\n- `interactsh-client` (projectdiscovery/interactsh) — OOB callback server\n- `gf` (tomnomnom/gf) — URL pattern grep\n- `anew` (tomnomnom/anew) — append new lines only\n- `unfurl` (tomnomnom/unfurl) — URL parser\n- `qsreplace` (tomnomnom/qsreplace) — querystring replacer\n- `gitdorks_go` (damit5/gitdorks_go) — GitHub dork search\n- `github-endpoints` (gwen001/github-endpoints) — GitHub endpoint discovery\n- `cent` (xm1k3/cent) — nuclei template manager\n- `trufflehog` (trufflesecurity/trufflehog) — secrets scanner (via `go install`)\n- `brutespray` (x90skysn3k/brutespray) — service credential spraying\n### Python Tools (installed via `uv tool install`)\n- `dnsvalidator` (vortexau/dnsvalidator) — DNS resolver validation\n- `interlace` (pry0cc/interlace) — parallel command runner\n- `wafw00f` (EnableSecurity/wafw00f) — WAF fingerprinting\n- `commix` (commixproject/commix) — command injection scanner\n- `waymore` (xnl-h4ck3r/waymore) — passive URL collection\n- `urless` (xnl-h4ck3r/urless) — URL deduplication\n- `ghauri` (r0oth3x49/ghauri) — SQLi scanner (optional)\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — deep link finder\n- `xnldorker` (xnl-h4ck3r/xnldorker) — Google dorker\n- `porch-pirate` (MandConsultingGroup/porch-pirate) — Postman API leaks\n- `p1radup` (iambouali/p1radup) — URL deduplication\n- `subwiz` (hadriansecurity/subwiz) — ML-based subdomain prediction\n- `arjun` (s0md3v/Arjun) — parameter discovery\n- `gqlspection` (doyensec/GQLSpection) — GraphQL deep introspection\n- `postleaksNg` (six2dez/postleaksNG) — Postman public leak search\n- `cewler` (roys/cewler) — web wordlist generator\n- `fray` (dalisecurity/fray) — WAF-aware payload testing (PyPI)\n### Repo-Clone Tools (Python venvs, run via `venv/bin/python3`)\n- `dorks_hunter` (six2dez/dorks_hunter) — Google dork automation\n- `CMSeeK` (Tuhinshubhra/CMSeeK) — CMS fingerprinting\n- `cloud_enum` (initstring/cloud_enum) — AWS/GCP/Azure bucket enumeration\n- `EmailHarvester` (maldevel/EmailHarvester) — email harvesting\n- `SwaggerSpy` (UndeadSec/SwaggerSpy) — Swagger endpoint leak detection\n- `LeakSearch` (JoelGMSec/LeakSearch) — credential leak search\n- `Spoofy` (MattKeeley/Spoofy) — email spoofing check\n- `msftrecon` (Arcanum-Sec/msftrecon) — Microsoft tenant recon\n- `Scopify` (Arcanum-Sec/Scopify) — scope management\n- `regulator` (cramppet/regulator) — regex-based subdomain permutations\n- `SSTImap` (vladko312/SSTImap) — SSTI scanner (alternative engine)\n- `gato` (praetorian-inc/gato) — GitHub Actions audit\n### Repo-Clone Tools (Go build)\n- `ghleaks` (dinosn/ghleaks) — GitHub-wide secret search\n- `nomore403` (devploit/nomore403) — 403 bypass\n- `ffufPostprocessing` (Damian89/ffufPostprocessing) — ffuf result analysis\n- `JSA` (w9w/JSA) — JS analysis\n- `ultimate-nmap-parser` (shifty0g/ultimate-nmap-parser) — Nmap XML parser\n### System-Level Tools (apt/brew/yum)\n- `nmap` — active port scanning\n- `massdns` — DNS resolver (also cloned + built from source)\n- `jq` — JSON processing throughout all modules\n- `exiftool` (perl-Image-ExifTool) — metadata extraction\n- `whois` — domain registration lookup\n- `sqlmap` — SQL injection (system or via repo clone)\n- `testssl.sh` (testssl/testssl.sh) — TLS/SSL misconfiguration testing\n- `medusa` — credential brute-force (system install)\n- `shodan` CLI — installed via `uv tool install shodan`\n### Rust Tools\n- `smugglex` (Cargo) — HTTP request smuggling detection\n- Rustup installed from `https://sh.rustup.rs`\n## Configuration\n- `reconftw.cfg` — sourced after CLI parsing; all feature flags, rate limits, timeouts, wordlist paths, API keys, thread counts\n- `secrets.cfg` (gitignored, auto-sourced) — API keys and tokens separated from main config\n- `secrets.cfg.example` — template showing all supported secret vars\n- Feature flags: `OSINT=true`, `SUBDOMAINS_GENERAL=true`, `VULNS_GENERAL=false`, etc.\n- Rate limits: `HTTPX_RATELIMIT=150`, `NUCLEI_RATELIMIT=150`, `FFUF_RATELIMIT=0`\n- Thread counts: auto-scaled via `AVAILABLE_CORES=$(nproc)` with multipliers per tool\n- Timeouts: per-tool in seconds or minutes (`CMSSCAN_TIMEOUT=3600`, `SUBFINDER_ENUM_TIMEOUT=180`)\n- Wordlist paths: `fuzz_wordlist`, `lfi_wordlist`, `subs_wordlist` etc. under `${WORDLISTS_DIR}`\n- Output: `EXPORT_FORMAT`, `AI_REPORT_TYPE`, `ASSET_STORE`\n- GNU `getopt` long options, parsed in `reconftw.sh` while/case loop\n- All CLI overrides use `CLI_*` pattern and are re-applied after `reconftw.cfg` is sourced\n- Full list from `getopt` call: `domain`, `list`, `recon`, `subdomains`, `passive`, `all`, `web`, `osint`, `zen`, `deep`, `help`, `vps`, `vps-count`, `ai`, `check-tools`, `health-check`, `quick-rescan`, `incremental`, `adaptive-rate`, `dry-run`, `parallel`, `no-parallel`, `monitor`, `monitor-interval`, `monitor-cycles`, `refresh-cache`, `gen-resolvers`, `force`, `export`, `report-only`, `no-report`, `parallel-log`, `quiet`, `verbose`, `no-color`, `log-format`, `show-cache`, `banner`, `no-banner`, `legal`\n- `SHODAN_API_KEY`, `WHOISXML_API`, `PDCP_API_KEY`, `XSS_SERVER`, `COLLAB_SERVER` — preferred over config file\n- `GOROOT`, `GOPATH`, `PATH` — extended by `reconftw.cfg` for Go and Rust binaries\n- `LOGFILE` — per-target log path\n- `config/reconftw_full.cfg` — full-scan preset\n- `config/reconftw_quick.cfg` — quick-scan preset\n- `config/reconftw_stealth.cfg` — low-noise preset\n## Build\n- Default: `go1.23.6` (fetches latest from `https://go.dev/VERSION?m=text`)\n- Installed to `/usr/local/go`; set `install_golang=false` in config to skip\n- Minimum: Python 3.7 (enforced in `install_yum()`)\n- Virtual environments per tool via `uv venv`\n- Root venv at `.venv/` for `getjswords.py` and similar helpers\n## Platform Requirements\n- Bash ≥ 4.3 (for `wait -n` used in `lib/parallel.sh`)\n- Go ≥ 1.21 (tools use SIV module paths like `/v2`, `/v3`)\n- Python ≥ 3.7\n- `uv` package manager\n- Rust / Cargo (for `smugglex`)\n- GNU coreutils, getopt, sed (macOS only via Homebrew)\n- ~5GB free disk space for Go cache, tools, and repos\n- ~1GB RAM minimum (Go compilation)\n- Base image: `ubuntu:24.04`\n- Build arg `INSTALL_AXIOM=true` (default) installs axiom fleet tooling\n- Ports 85-90 exposed (for headless browser tooling)\n- Runs as root (required for raw socket operations by some tools)\n- Health check: `./reconftw.sh --health-check`\n\n\n\n## Conventions\n\n## Shell Settings\n## Source Guard Pattern\n- `lib/common.sh`: `[[ -n \"$_COMMON_SH_LOADED\" ]] && return 0`\n- `lib/parallel.sh`: `[[ -n \"$_PARALLEL_SH_LOADED\" ]] && return 0`\n- `lib/ui.sh`: `[[ -n \"${_UI_SH_LOADED:-}\" ]] && return 0`\n## Function Naming\n- Public module functions: `snake_case` prefixed with module context (`sub_passive`, `sub_crt`, `geo_info`)\n- Private helpers: `_snake_case` prefix (`_print_status`, `_print_error`, `_print_module_start`, `_parallel_emit_job_output`)\n- UI layer: `ui_` prefix (`ui_init`, `ui_header`, `ui_summary`, `ui_batch_end`)\n- Lifecycle wrappers: `start_func` / `end_func` (call these at top/bottom of every recon function)\n- Validation functions: `validate_*` / `sanitize_*` (defined in `lib/validation.sh` and `modules/utils.sh`)\n## Variable Naming\n- Config flags: `SUBPASSIVE`, `SUBCRT`, `PARALLEL_MODE`, `OUTPUT_VERBOSITY`\n- Runtime state: `LOGFILE`, `SCRIPTPATH`, `DIFF`, `DRY_RUN`, `AXIOM`\n- Error codes: `E_SUCCESS=0`, `E_INVALID_DOMAIN=20`, `E_INVALID_IP=21` (readonly, defined in `lib/validation.sh`)\n## CLI Flag Pattern\n## Output / UI Conventions\n- `OUTPUT_VERBOSITY=0` (quiet): only errors/FAIL printed\n- `OUTPUT_VERBOSITY=1` (normal, default): OK/WARN/FAIL/SKIP status lines\n- `OUTPUT_VERBOSITY=2` (verbose): all of the above + INFO messages + start_func messages\n## Function Lifecycle (start_func / end_func)\n- `start_func name desc` — logs to LOGFILE, sets per-function start timestamp, emits INFO at verbosity >= 2\n- `end_func message name [status]` — touches checkpoint file, calculates elapsed time, calls `_print_status`\n- `skip_notification reason` — emits SKIP/CACHE badge; reasons: `\"disabled\"`, `\"mode\"`, `\"processed\"`, `\"processed-visible\"`, `\"noinput\"`\n## File Checkpointing (Resumability)\n- `end_func` creates the checkpoint: `touch \"$called_fn_dir/.${fn}\"`\n- DIFF mode (`DIFF=true`) bypasses checkpoint — forces re-execution\n- Helper `should_run()` in `lib/common.sh` provides a cleaner gate: `if should_run \"FLAG_VAR\"; then`\n## Error Handling\n- `E_SUCCESS=0`, `E_GENERAL=1`, `E_MISSING_DEP=2`, `E_INVALID_INPUT=3`\n## Validation Functions\n| Function | Purpose |\n|----------|---------|\n| `validate_domain()` | RFC domain check + injection character rejection |\n| `validate_ipv4()` | Octet range validation |\n| `validate_integer()` | Numeric range check |\n| `validate_boolean()` | Accepts `true`/`false` only (not `1`/`0`/`yes`/`no`) |\n| `validate_file_readable()` | Exists + readable + is-a-file |\n| `sanitize_interlace_input()` | Removes shell metacharacters from input files (canonical in `lib/validation.sh`) |\n| `sanitize_domain()` | Strips URL components, lowercases, rejects injection (in `modules/utils.sh`) |\n| `is_in_scope_host()` | Anchored hostname scope check (prevents substring false positives) |\n| `filter_in_scope_urls()` | Python3-based URL scope check (scheme, userinfo, host) |\n## Path and CWD Conventions\n## Parallel Execution\n## Import / Sourcing Order\n## Logging\n- All tool output redirected to `$LOGFILE`: `command ... 2>>\"$LOGFILE\" >/dev/null`\n- Structured JSON logging via `log_json level func message [key=val]` (optional, `STRUCTURED_LOGGING=true`)\n- `redact_secrets()` scrubs `REDACT_VARS` and `REGISTERED_SECRETS` from log lines\n- `register_secret \"$value\"` must be called before logging any secret value\n## Comments\n\n\n\n## Architecture\n\n## System Overview\n```text\n```\n## Component Responsibilities\n| Component | Responsibility | File |\n|-----------|----------------|------|\n| Entry point & CLI parser | getopt argument parsing, config sourcing, mode dispatch | `reconftw.sh` |\n| Mode orchestration | `start`/`end`, workflow functions (`recon`, `passive`, `all`, `vulns`, `osint`, `subs_menu`, `webs_menu`, `zen_menu`, `monitor_mode`) | `modules/modes.sh` |\n| Function lifecycle | `start_func`/`end_func`, checkpointing, logging, notifications, reporting, plugins, health check | `modules/core.sh` |\n| Subdomain enumeration | All `sub_*` functions, `subtakeover`, `zonetransfer`, `s3buckets`, `geo_info` | `modules/subdomains.sh` |\n| Web analysis | `webprobe_full`, `screenshot`, `nuclei_check`, `fuzz`, `jschecks`, `urlchecks`, `waf_checks`, and 20+ others | `modules/web.sh` |\n| Vulnerability scanning | `xss`, `ssrf_checks`, `sqli`, `crlf_checks`, `lfi`, `ssti`, `smuggling`, `fuzzparams`, `nuclei_dast`, and others | `modules/vulns.sh` |\n| OSINT collection | `domain_info`, `ip_info`, `emails`, `google_dorks`, `github_leaks`, `github_actions_audit`, `cloud_enum_scan`, etc. | `modules/osint.sh` |\n| Shared utilities | `run_command`, `sed_i`, `deleteOutScoped`, `validate_config`, `cache_*`, `checkpoint_*`, `circuit_breaker_*`, rate-limit adaption | `modules/utils.sh` |\n| Axiom/distributed mode | `axiom_launch`, `axiom_shutdown`, `axiom_selected`, `resolvers_update`, `ipcidr_target` | `modules/axiom.sh` |\n| Parallel execution | `parallel_funcs`, `_throttle_jobs`, job heartbeat, progress live display, log mode output | `lib/parallel.sh` |\n| Input validation/sanitization | `sanitize_domain`, `sanitize_ip`, `validate_domain`, `validate_integer`, `_sanitize_list_entry` | `lib/validation.sh` |\n| Shared file/counter utilities | `ensure_dirs`, `ensure_webs_all`, `safe_backup`, `count_lines`, incident tracking | `lib/common.sh` |\n| UI presentation layer | `_print_status`, `_print_msg`, `_print_section`, `_print_rule`, `ui_header`, `ui_summary`, TTY detection, color management, JSONL output | `lib/ui.sh` |\n| Configuration | All runtime settings (~350 variables); sourced after CLI parse | `reconftw.cfg` |\n## Pattern Overview\n- Single process: `reconftw.sh` sources all libraries and modules at startup; every function lives in the same shell environment\n- No subshell isolation between modules — all state is shared via global variables\n- File-based checkpointing: `called_fn_dir/.funcname` sentinel files prevent re-running completed functions across invocations\n- CLI-over-config: `reconftw.cfg` provides defaults; CLI flags set `CLI_*` variables that are re-applied after config sourcing to guarantee they cannot be overwritten\n- All external tool invocations go through `run_command()` which handles dry-run mode, adaptive rate limiting, axiom dispatch, and debug logging\n## Layers\n- Purpose: Bootstrap, macOS re-exec, module loading, getopt CLI parsing, config sourcing, CLI override re-application, mode dispatch\n- Location: `reconftw.sh`\n- Contains: `normalize_vps_count_args()`, the main `while/case` getopt loop, the config `source` sequence, CLI override if-blocks, the final `case $opt_mode` dispatch\n- Depends on: All libraries (sourced first), all modules (sourced second)\n- Used by: End user / CI\n- Purpose: Reusable utilities with no side effects; loadable independently for tests\n- Location: `lib/validation.sh`, `lib/common.sh`, `lib/ui.sh`, `lib/parallel.sh`\n- Contains: Input sanitization, file helpers, UI/color/progress, parallel job management\n- Depends on: Nothing (source-guarded with `_*_LOADED` pattern)\n- Used by: All modules and reconftw.sh\n- Purpose: Implement all scanning, analysis, and orchestration functions\n- Location: `modules/`\n- Contains: All recon, vuln, OSINT, web, subdomain functions\n- Depends on: Libraries (always loaded first), `reconftw.cfg` variables, external tools on PATH\n- Used by: modes.sh orchestrates all others; reconftw.sh dispatches to modes.sh\n- Purpose: Default runtime values for ~350 flags/paths/limits; can be overridden by `secrets.cfg` and custom config\n- Location: `reconftw.cfg`, optionally `secrets.cfg`, optionally `$CUSTOM_CONFIG`\n- Contains: Module enable/disable flags, tool flags, API key env-var references, paths, parallelism settings, verbosity, Axiom settings\n- Depends on: Nothing\n- Used by: Sourced by `reconftw.sh` between CLI parse and CLI override re-application\n- Purpose: Store per-target findings in a stable directory hierarchy\n- Location: `Recon//` (created at `start()` time by `modules/modes.sh`)\n- Contains: Standard subdirectories listed below\n- Depends on: `start()` in modes.sh creates the directory tree\n## Data Flow\n### Primary Recon Request Path (`-r` / `--recon`)\n### Function Execution Path (every leaf module function)\n### Parallel Execution Path\n### Axiom Distributed Scan Flow\n- Global bash variables throughout (no encapsulation). Config vars, target vars (`domain`, `dir`, `called_fn_dir`, `LOGFILE`), and result counters are all globals\n- `passive()` saves/restores module-enable globals before overriding them (`modules/modes.sh:549-611`)\n## Key Abstractions\n- Purpose: Lifecycle wrapper around every leaf scanning function\n- Examples: Used in every function in `modules/subdomains.sh`, `modules/web.sh`, `modules/vulns.sh`, `modules/osint.sh`\n- Pattern: `start_func \"${FUNCNAME[0]}\" \"description\"` at top; `end_func \"output path\" \"${FUNCNAME[0]}\"` at bottom; creates checkpoint file on end\n- Purpose: Prevent re-running completed functions across multiple invocations of the same target\n- Location: `Recon//.called_fn/.funcname`\n- Pattern: Each function tests `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]`; `end_func` writes the sentinel via `touch \"$called_fn_dir/.${fn}\"`\n- Purpose: Universal external-tool gate for dry-run preview, axiom dispatch, adaptive rate limiting, and debug logging\n- Location: `modules/utils.sh:468`\n- Pattern: All tool calls inside module functions use `run_command ` rather than direct invocation\n- Purpose: Allow modules to be sourced multiple times (test re-sourcing, `--source-only`) without re-executing\n- Pattern: `[[ -n \"$_FOO_LOADED\" ]] && return 0` at top of each lib file (`lib/common.sh:6`, `lib/parallel.sh:6`, `lib/ui.sh:5`, `lib/validation.sh` — validation uses error-code guards instead)\n- Purpose: Run independent module functions concurrently up to `PARALLEL_MAX_JOBS`\n- Pattern: `parallel_funcs N func_a func_b func_c` — each function spawned as a background subshell; used in `recon()`, `osint()`, `vulns()` for independent groups\n- Purpose: Transparent axiom/local fallback wrapper — if axiom fails during a module, retries locally\n- Location: `modules/modes.sh:656`\n- Pattern: All module calls inside `subs_menu`, `webs_menu`, `recon`, `passive` use this wrapper\n## Entry Points\n- Location: `reconftw.sh`\n- Triggers: Direct execution (`./reconftw.sh -d example.com -r`)\n- Responsibilities: Bootstrap, all module loading, CLI parse, config source, mode dispatch\n- Location: `reconftw.sh:123-125`\n- Triggers: `./reconftw.sh --source-only` (used by bats test `setup()` blocks)\n- Responsibilities: Sources all modules without executing any recon\n- Location: `modules/modes.sh:13`\n- Triggers: Called at the top of most workflow functions (`recon`, `subs_menu`, `passive`, `osint`, `zen_menu`)\n- Responsibilities: Create output directory tree, init LOGFILE, init cache/incremental/DNS/plugins, set global `dir` and `called_fn_dir`\n- Location: `modules/modes.sh:286`\n- Triggers: Called at the bottom of most workflow functions\n- Responsibilities: AI report, cleanup, Faraday, screenshot diffs, plugin events, hotlist, `export_reports()`, timing summary\n## Output Directory Structure\n```\n```\n## Verbosity and Output Controls\n- `0` (quiet): Only errors and final summary printed to terminal; banner suppressed\n- `1` (normal, default): Errors + warnings printed; `notification()` info/good suppressed\n- `2` (verbose): All `notification()` calls, PID info, full parallel output, `start_func` messages, `print_timing_summary`\n- `summary`: One badge line per completed parallel job\n- `tail`: Last `PARALLEL_TAIL_LINES` (default 20, doubled on failure) from each job's log\n- `full`: Complete captured stdout from each job\n- `jsonl-strict`: Forces `OUTPUT_VERBOSITY=0`, emits only machine-readable JSONL\n## Architectural Constraints\n- **Threading:** Single-threaded bash with optional background subshells via `parallel_funcs`; `wait -n` (bash 4.3+) used for job throttling in `_throttle_jobs`\n- **Global state:** All config vars, `domain`, `dir`, `called_fn_dir`, `LOGFILE`, `start`, `runtime`, `DIFF`, `AXIOM`, and hundreds of module-enable flags are module-level globals; any sourced function can read or mutate them\n- **Circular imports:** None by design — reconftw.sh sources libs first, then modules in explicit dependency order (`utils.sh` → `core.sh` → `osint.sh` → `subdomains.sh` → `web.sh` → `vulns.sh` → `axiom.sh` → `modes.sh`)\n- **macOS bash version:** reconftw.sh re-execs itself under Homebrew bash ≥ 4 on macOS (system bash is 3.2); `lib/parallel.sh` requires bash 4.3+ for `wait -n`\n- **Working directory:** `start()` calls `cd \"$dir\"` (the per-target output dir) before any module function runs; all relative paths inside modules resolve against the target dir. `reconftw.sh` captures `startdir=${PWD}` before this\n- **No subshell isolation per module:** Modules are sourced functions, not subprocess commands. A `return` inside a module returns from the function; an `exit` would kill the whole shell\n## Anti-Patterns\n### Direct external tool calls without `run_command`\n### Writing checkpoint files manually\n### Skipping the `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]` guard\n### Overriding config globals without save/restore in workflow functions\n## Error Handling\n- ERR trap in `start()` logs function name, line number, and command to `$LOGFILE` and calls `explain_err()` (`modules/modes.sh:140`)\n- Non-zero exit from `parallel_funcs` increments `RECON_OSINT_PARALLEL_FAILURES` and sets `RECON_PARTIAL_RUN=true`\n- `run_module_with_axiom_failover` catches axiom mid-run failures and retries locally\n- Circuit-breaker helpers (`circuit_breaker_is_open`, `circuit_breaker_record_failure`) in `modules/utils.sh:1190` for persistent tool failures\n## Cross-Cutting Concerns\n\n\n\n## Project Skills\n\nNo project skills found. Add skills to any of: `.claude/skills/`, `.agents/skills/`, `.cursor/skills/`, `.github/skills/`, or `.codex/skills/` with a `SKILL.md` index file.\n\n\n\n## GSD Workflow Enforcement\n\nBefore using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.\n\nUse these entry points:\n- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks\n- `/gsd-debug` for investigation and bug fixing\n- `/gsd-execute-phase` for planned phase work\n\nDo not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.\n\n\n\n\n\n## Developer Profile\n\n> Profile not yet configured. Run `/gsd-profile-user` to generate your developer profile.\n> This section is managed by `generate-claude-profile` -- do not edit manually.\n\n","docs/ARCHITECTURE.md":"# reconFTW Architecture\n\n## Overview\n\nreconFTW is a modular reconnaissance framework. The codebase is organized as follows:\n\n```\nreconftw/\n├── reconftw.sh # Main entry point, argument parsing, orchestration\n├── reconftw.cfg # Main configuration file (all settings)\n├── modules/ # Phase-specific modules (what to do)\n│ ├── utils.sh # Utility functions (cleanup, sanitization, caching)\n│ ├── core.sh # Framework core (logging, tools check, lifecycle)\n│ ├── osint.sh # OSINT gathering functions\n│ ├── subdomains.sh # Subdomain enumeration\n│ ├── web.sh # Web analysis and crawling\n│ ├── vulns.sh # Vulnerability scanning\n│ ├── axiom.sh # Ax (formerly Axiom) distributed scanning\n│ └── modes.sh # Scan mode orchestration\n├── lib/ # Pure utility libraries (how to do it)\n│ ├── config.sh # [FUTURE] Modular config loader\n│ └── validation.sh # Input validation functions\n└── tests/ # Test suite\n ├── unit/ # Unit tests (bats)\n ├── security/ # Security tests\n └── mocks/ # Tool mocks for testing\n```\n\n## Module Load Order\n\nModules are sourced in this specific order (defined in `reconftw.sh`):\n\n1. `modules/utils.sh` - Base utilities (must be first)\n2. `modules/core.sh` - Framework infrastructure\n3. `modules/osint.sh` - OSINT functions\n4. `modules/subdomains.sh` - Subdomain enumeration\n5. `modules/web.sh` - Web analysis\n6. `modules/vulns.sh` - Vulnerability scanning\n7. `modules/axiom.sh` - Distributed scanning\n8. `modules/modes.sh` - Mode orchestration (must be last)\n\n## Data Flow\n\n```\nTarget Input → Sanitization → Phase Execution → Output Files\n ↓ ↓ ↓ ↓\n --domain sanitize_domain start_func Recon/$domain/\n --list sanitize_ip [tool calls] subdomains/\n --cidr end_func webs/vulns/\n```\n\n## Key Functions\n\n### Lifecycle Functions (core.sh)\n- `start_func(name, description)` - Begin a function, check if already done\n- `end_func(message, name)` - Complete a function, mark as done\n\n### Utility Functions (utils.sh)\n- `sanitize_domain(domain)` - Sanitize domain input (lowercase, safe chars)\n- `sanitize_ip(ip)` - Sanitize IP/CIDR input\n- `should_run_deep(count, limit)` - Check if DEEP mode should run\n- `deleteOutScoped(scopefile, targetfile)` - Remove out-of-scope entries\n- `run_command(cmd, args...)` - Execute with logging (respects DRY_RUN)\n\n### DNS Resolution Functions (utils.sh)\n\nAll DNS resolution goes through two global wrappers that auto-select between puredns and dnsx based on the network environment:\n\n- `_get_local_ip()` - Get the primary local IP address (cross-platform). Uses `route -n get default` on macOS and `ip -4 route get 1.1.1.1` on Linux to find the correct interface regardless of naming.\n- `_ip_is_public_ipv4(ip)` - Conservative check for publicly routable IPv4. Treats RFC1918, CGNAT (100.64.0.0/10), loopback, link-local, multicast/reserved as non-public.\n- `_is_behind_nat()` - Detect if running behind NAT (or unknown network) by checking the primary local IP. Defaults to NAT-safe behavior if detection fails.\n- `init_dns_resolver()` - Evaluates resolver selection once per run and caches it in `DNS_RESOLVER_SELECTED` when `DNS_RESOLVER=auto`.\n- `_select_dns_resolver()` - Select DNS resolver based on `DNS_RESOLVER` config. `puredns`/`dnsx` force the resolver; `auto` uses the cached selection from `init_dns_resolver()` (with a safe fallback if unset).\n- `_resolve_domains(input_file, output_file)` - Resolve a list of domains. Uses puredns (massdns + wildcard filter) on VPS or dnsx (Go net resolver + basic wildcard threshold) behind NAT.\n- `_bruteforce_domains(wordlist, target_domain, output_file)` - Bruteforce subdomains. Equivalent to `puredns bruteforce wordlist domain` or `dnsx -d domain -w wordlist`.\n\n**Why two resolvers:** puredns uses massdns which sends raw UDP packets to thousands of resolvers simultaneously. This floods the NAT table on consumer routers and can kill connectivity. dnsx uses Go's standard net resolver through the OS networking stack, which is NAT-friendly but lacks puredns's advanced wildcard detection.\n\n### Permutation Functions (subdomains.sh)\n\n- `_select_permutations_wordlist(source_file)` - Choose wordlist based on `PERMUTATIONS_WORDLIST_MODE` and subdomain count. In `auto` mode: full list (849 words) for <=100 subs or DEEP mode, short list (162 words) for >100 subs.\n- `_run_permutation_engine(source_file)` - Run gotator with the auto-selected wordlist.\n- `_generate_permutation_candidates(source_file, output_file)` - Generate permutation candidates with byte-size limit.\n\n### Validation Functions (lib/validation.sh)\n- `validate_domain(domain)` - Validate domain format\n- `validate_ipv4(ip)` - Validate IPv4 address\n- `validate_boolean(value)` - Validate boolean string\n\n## Global Variables\n\n### Required (set before module load)\n- `$SCRIPTPATH` - Path to reconftw installation\n- `$domain` - Target domain\n- `$dir` - Output directory for current scan\n\n### Configuration (from reconftw.cfg)\n- `$DEEP` - Enable deep/thorough scanning\n- `$DEEP_LIMIT` - Threshold for auto-deep mode\n- `$AXIOM` - Enable distributed scanning\n- `$*_THREADS` - Thread counts per tool\n- `$DNS_RESOLVER` - DNS resolver selection (`auto`|`puredns`|`dnsx`)\n- `$DNS_RESOLVER_SELECTED` - Cached resolver selection for this run (set by `init_dns_resolver()` when `DNS_RESOLVER=auto`)\n- `$PERMUTATIONS_WORDLIST_MODE` - Wordlist selection (`auto`|`full`|`short`)\n- `$PERMUTATIONS_SHORT_THRESHOLD` - Subdomain count threshold for short wordlist (default: 100)\n\n## Error Codes\n\nDefined in `lib/validation.sh`:\n- `E_SUCCESS=0` - Success\n- `E_GENERAL=1` - General error\n- `E_MISSING_DEP=2` - Missing dependency\n- `E_INVALID_INPUT=3` - Invalid user input\n- `E_CONFIG=8` - Configuration error\n\n## Testing\n\n```bash\nmake test # Unit tests only\nmake test-security # Security tests only\nmake test-all # All tests\nmake lint # Shellcheck errors\nmake lint-fix # Shellcheck warnings with context\n```\n"},"files":{"CLAUDE.md":"\n## Project\n\n**reconFTW**\n\nreconFTW is a comprehensive bash-based reconnaissance automation framework used by bug bounty hunters, penetration testers, and security researchers. It orchestrates 70+ external security tools (Go, Python, Rust) across subdomain enumeration, web probing, OSINT, and vulnerability scanning, producing structured per-target output trees with optional Axiom distributed execution, AI reporting, monitor/incremental mode, and Slack/Telegram/Discord notifications.\n\n**Core Value:** Run one command, get a complete recon picture of a target — passive, active, and vulnerability layers — with resumable checkpoints, structured output, and zero-touch tool orchestration.\n\n### Constraints\n\n- **Tech stack**: Bash 4.3+ — Required for `wait -n`, `mapfile`, associative arrays. macOS users must have Homebrew bash; auto re-exec is best-effort.\n- **External tools**: 70+ runtime dependencies — Most install via `go install @latest` (no version pinning), which is convenient but a known supply-chain risk.\n- **Single process**: All modules sourced into one shell — No subshell isolation between modules; all state shared via globals. Workflow functions must save/restore globals they override (see `passive()` pattern).\n- **Resume semantics**: Checkpoint files are touch-once at `end_func` — Interrupted functions re-run from scratch on next invocation; partial outputs are not detected.\n- **Single-operator**: Designed for one user per target run — No locking, no multi-user state, no concurrent runs against the same target dir.\n- **Output stability**: `Recon//` tree is a public contract — Subdirectory names and filenames are consumed by downstream pipelines, scripts, and parsers; renames are breaking changes.\n- **macOS compatibility**: GNU coreutils + GNU sed + GNU getopt required — System BSD versions are not supported.\n- **CI budget**: Integration-full is weekly cron — Unit + smoke are per-push; adding heavy integration tests must respect this split.\n\n\n\n## Technology Stack\n\n## Languages\n- Bash 4+ - All core framework logic (`reconftw.sh`, `modules/*.sh`, `lib/*.sh`, `install.sh`)\n- Python 3.7+ - Python-backed tools (each runs in isolated `uv venv`): `dorks_hunter`, `CMSeeK`, `EmailHarvester`, `Spoofy`, `SSTImap`, `gato`, `regulator`, `reconftw_ai`, `getjswords.py`\n- Go (latest, min ~1.21) - Primary binary language for ~55 security tools installed via `go install @latest`\n## Runtime\n- Linux (Debian/Ubuntu/RHEL/Arch) or macOS (Apple Silicon / Intel)\n- Docker: Ubuntu 24.04 base image (`Docker/Dockerfile`)\n- ARM64/aarch64, ARMv6l/v7l, and x86_64 all supported for Go binary installs\n- `getopt` (GNU getopt required on macOS via `brew install gnu-getopt`)\n- `nproc` / `sysctl -n hw.ncpu` for CPU core auto-detection\n- `timeout` / `gtimeout` (macOS Homebrew `coreutils`)\n- `gnu-sed` (macOS requires `brew install gnu-sed`)\n- `gnu-coreutils` (macOS requires `brew install coreutils`)\n- Go tools: `go install` (`GOPATH=$HOME/go`, `GOROOT=/usr/local/go`)\n- Python tools: `uv tool install` from GitHub (most) or PyPI (`fray`)\n- Python repo venvs: `uv venv venv && uv pip install -r requirements.txt`\n- Lockfile: None (always installs `@latest`)\n## Frameworks\n- No framework — pure Bash with sourced module files\n- Module loading order: `lib/validation.sh` → `lib/common.sh` → `lib/ui.sh` → `lib/parallel.sh` → `modules/utils.sh` → `modules/core.sh` → `modules/osint.sh` → `modules/subdomains.sh` → `modules/web.sh` → `modules/vulns.sh` → `modules/axiom.sh` → `modules/modes.sh`\n- bats-core (Bash Automated Testing System)\n- GNU make (`Makefile`) — test, lint, format targets\n- shellcheck (error-level) — `make lint`, pre-commit hook\n- shfmt (4-space indent, `-bn`, `-ci`) — `make fmt`, pre-commit hook\n- pre-commit hooks defined in `.pre-commit-config.yaml`\n- semgrep: `.github/workflows/semgrep.yml` (CI only)\n## Key Dependencies\n### Go Tools (installed via `go install @latest`)\n- `subfinder` (projectdiscovery/subfinder) — passive multi-source subdomain discovery\n- `github-subdomains` (gwen001/github-subdomains) — GitHub-based subdomain search\n- `gitlab-subdomains` (gwen001/gitlab-subdomains) — GitLab-based subdomain search\n- `dnstake` (pwnesia/dnstake) — subdomain takeover detection\n- `puredns` (d3mondev/puredns) — mass DNS resolution with wildcard filtering\n- `dnsx` (projectdiscovery/dnsx) — DNS toolkit\n- `massdns` (blechschmidt/massdns) — via repo clone + build\n- `dsieve` (trickest/dsieve) — subdomain filtering\n- `enumerepo` (trickest/enumerepo) — GitHub org repo enumeration\n- `gotator` (Josue87/gotator) — subdomain permutations\n- `analyticsrelationships` (Josue87/analyticsrelationships) — Google Analytics pivoting\n- `roboxtractor` (Josue87/roboxtractor) — robots.txt extractor\n- `crt` (cemulus/crt) — crt.sh search\n- `asnmap` (projectdiscovery/asnmap) — ASN-to-CIDR mapping\n- `mapcidr` (projectdiscovery/mapcidr) — CIDR manipulation\n- `smap` (s0md3v/smap) — passive Shodan-powered port scan\n- `tlsx` (projectdiscovery/tlsx) — TLS certificate harvesting\n- `hakip2host` (hakluke/hakip2host) — reverse IP lookup\n- `cdncheck` (projectdiscovery/cdncheck) — CDN/WAF IP classification\n- `hakoriginfinder` (hakluke/hakoriginfinder) — origin IP discovery behind CDN\n- `inscope` (tomnomnom/hacks/inscope) — scope filtering\n- `csprecon` (edoardottt/csprecon) — CSP-based subdomain discovery\n- `favirecon` (edoardottt/favirecon) — favicon-based tech recon\n- `httpx` (projectdiscovery/httpx) — multi-probe HTTP toolkit\n- `katana` (projectdiscovery/katana) — web crawler\n- `ffuf` (ffuf/ffuf) — web fuzzer\n- `subjs` (lc/subjs) — JavaScript URL extractor\n- `Gxss` (KathanP19/Gxss) — reflected XSS param finder\n- `jsluice` (BishopFox/jsluice) — JS secret/URL extractor\n- `sourcemapper` (denandz/sourcemapper) — JS source map extractor\n- `mantra` (brosck/mantra) — JS/secret scanner\n- `urlfinder` (projectdiscovery/urlfinder) — URL discovery\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — via uv\n- `nmapurls` (sdcampbell/nmapurls) — URL extraction from Nmap XML\n- `naabu` (projectdiscovery/naabu) — fast port scanner\n- `VhostFinder` (wdahlenburg/VhostFinder) — virtual host discovery\n- `shortscan` (bitquark/shortscan) — IIS short filename scanner\n- `nuclei` (projectdiscovery/nuclei) — template-based scanner\n- `dalfox` (hahwul/dalfox) — XSS scanner\n- `crlfuzz` (dwisiswant0/crlfuzz) — CRLF injection scanner\n- `Web-Cache-Vulnerability-Scanner` (Hackmanit) — web cache poisoning\n- `TInjA` (Hackmanit/TInjA) — SSTI scanner\n- `toxicache` (xhzeem/toxicache) — web cache poisoning\n- `second-order` (mhmdiaa/second-order) — broken link/second-order injection\n- `s3scanner` (sa7mon/s3scanner) — S3/GCS/Azure Blob misconfiguration\n- `misconfig-mapper` (intigriti/misconfig-mapper) — third-party misconfiguration\n- `sj` (BishopFox/sj) — Swagger/OpenAPI analysis\n- `grpcurl` (fullstorydev/grpcurl) — gRPC reflection scanner\n- `nerva` (praetorian-inc/nerva) — service fingerprinting\n- `brutus` (praetorian-inc/brutus) — credential spraying\n- `julius` (praetorian-inc/julius) — LLM endpoint probe\n- `titus` (praetorian-inc/titus) — secrets engine\n- `notify` (projectdiscovery/notify) — multi-channel notifications\n- `interactsh-client` (projectdiscovery/interactsh) — OOB callback server\n- `gf` (tomnomnom/gf) — URL pattern grep\n- `anew` (tomnomnom/anew) — append new lines only\n- `unfurl` (tomnomnom/unfurl) — URL parser\n- `qsreplace` (tomnomnom/qsreplace) — querystring replacer\n- `gitdorks_go` (damit5/gitdorks_go) — GitHub dork search\n- `github-endpoints` (gwen001/github-endpoints) — GitHub endpoint discovery\n- `cent` (xm1k3/cent) — nuclei template manager\n- `trufflehog` (trufflesecurity/trufflehog) — secrets scanner (via `go install`)\n- `brutespray` (x90skysn3k/brutespray) — service credential spraying\n### Python Tools (installed via `uv tool install`)\n- `dnsvalidator` (vortexau/dnsvalidator) — DNS resolver validation\n- `interlace` (pry0cc/interlace) — parallel command runner\n- `wafw00f` (EnableSecurity/wafw00f) — WAF fingerprinting\n- `commix` (commixproject/commix) — command injection scanner\n- `waymore` (xnl-h4ck3r/waymore) — passive URL collection\n- `urless` (xnl-h4ck3r/urless) — URL deduplication\n- `ghauri` (r0oth3x49/ghauri) — SQLi scanner (optional)\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — deep link finder\n- `xnldorker` (xnl-h4ck3r/xnldorker) — Google dorker\n- `porch-pirate` (MandConsultingGroup/porch-pirate) — Postman API leaks\n- `p1radup` (iambouali/p1radup) — URL deduplication\n- `subwiz` (hadriansecurity/subwiz) — ML-based subdomain prediction\n- `arjun` (s0md3v/Arjun) — parameter discovery\n- `gqlspection` (doyensec/GQLSpection) — GraphQL deep introspection\n- `postleaksNg` (six2dez/postleaksNG) — Postman public leak search\n- `cewler` (roys/cewler) — web wordlist generator\n- `fray` (dalisecurity/fray) — WAF-aware payload testing (PyPI)\n### Repo-Clone Tools (Python venvs, run via `venv/bin/python3`)\n- `dorks_hunter` (six2dez/dorks_hunter) — Google dork automation\n- `CMSeeK` (Tuhinshubhra/CMSeeK) — CMS fingerprinting\n- `cloud_enum` (initstring/cloud_enum) — AWS/GCP/Azure bucket enumeration\n- `EmailHarvester` (maldevel/EmailHarvester) — email harvesting\n- `SwaggerSpy` (UndeadSec/SwaggerSpy) — Swagger endpoint leak detection\n- `LeakSearch` (JoelGMSec/LeakSearch) — credential leak search\n- `Spoofy` (MattKeeley/Spoofy) — email spoofing check\n- `msftrecon` (Arcanum-Sec/msftrecon) — Microsoft tenant recon\n- `Scopify` (Arcanum-Sec/Scopify) — scope management\n- `regulator` (cramppet/regulator) — regex-based subdomain permutations\n- `SSTImap` (vladko312/SSTImap) — SSTI scanner (alternative engine)\n- `gato` (praetorian-inc/gato) — GitHub Actions audit\n### Repo-Clone Tools (Go build)\n- `ghleaks` (dinosn/ghleaks) — GitHub-wide secret search\n- `nomore403` (devploit/nomore403) — 403 bypass\n- `ffufPostprocessing` (Damian89/ffufPostprocessing) — ffuf result analysis\n- `JSA` (w9w/JSA) — JS analysis\n- `ultimate-nmap-parser` (shifty0g/ultimate-nmap-parser) — Nmap XML parser\n### System-Level Tools (apt/brew/yum)\n- `nmap` — active port scanning\n- `massdns` — DNS resolver (also cloned + built from source)\n- `jq` — JSON processing throughout all modules\n- `exiftool` (perl-Image-ExifTool) — metadata extraction\n- `whois` — domain registration lookup\n- `sqlmap` — SQL injection (system or via repo clone)\n- `testssl.sh` (testssl/testssl.sh) — TLS/SSL misconfiguration testing\n- `medusa` — credential brute-force (system install)\n- `shodan` CLI — installed via `uv tool install shodan`\n### Rust Tools\n- `smugglex` (Cargo) — HTTP request smuggling detection\n- Rustup installed from `https://sh.rustup.rs`\n## Configuration\n- `reconftw.cfg` — sourced after CLI parsing; all feature flags, rate limits, timeouts, wordlist paths, API keys, thread counts\n- `secrets.cfg` (gitignored, auto-sourced) — API keys and tokens separated from main config\n- `secrets.cfg.example` — template showing all supported secret vars\n- Feature flags: `OSINT=true`, `SUBDOMAINS_GENERAL=true`, `VULNS_GENERAL=false`, etc.\n- Rate limits: `HTTPX_RATELIMIT=150`, `NUCLEI_RATELIMIT=150`, `FFUF_RATELIMIT=0`\n- Thread counts: auto-scaled via `AVAILABLE_CORES=$(nproc)` with multipliers per tool\n- Timeouts: per-tool in seconds or minutes (`CMSSCAN_TIMEOUT=3600`, `SUBFINDER_ENUM_TIMEOUT=180`)\n- Wordlist paths: `fuzz_wordlist`, `lfi_wordlist`, `subs_wordlist` etc. under `${WORDLISTS_DIR}`\n- Output: `EXPORT_FORMAT`, `AI_REPORT_TYPE`, `ASSET_STORE`\n- GNU `getopt` long options, parsed in `reconftw.sh` while/case loop\n- All CLI overrides use `CLI_*` pattern and are re-applied after `reconftw.cfg` is sourced\n- Full list from `getopt` call: `domain`, `list`, `recon`, `subdomains`, `passive`, `all`, `web`, `osint`, `zen`, `deep`, `help`, `vps`, `vps-count`, `ai`, `check-tools`, `health-check`, `quick-rescan`, `incremental`, `adaptive-rate`, `dry-run`, `parallel`, `no-parallel`, `monitor`, `monitor-interval`, `monitor-cycles`, `refresh-cache`, `gen-resolvers`, `force`, `export`, `report-only`, `no-report`, `parallel-log`, `quiet`, `verbose`, `no-color`, `log-format`, `show-cache`, `banner`, `no-banner`, `legal`\n- `SHODAN_API_KEY`, `WHOISXML_API`, `PDCP_API_KEY`, `XSS_SERVER`, `COLLAB_SERVER` — preferred over config file\n- `GOROOT`, `GOPATH`, `PATH` — extended by `reconftw.cfg` for Go and Rust binaries\n- `LOGFILE` — per-target log path\n- `config/reconftw_full.cfg` — full-scan preset\n- `config/reconftw_quick.cfg` — quick-scan preset\n- `config/reconftw_stealth.cfg` — low-noise preset\n## Build\n- Default: `go1.23.6` (fetches latest from `https://go.dev/VERSION?m=text`)\n- Installed to `/usr/local/go`; set `install_golang=false` in config to skip\n- Minimum: Python 3.7 (enforced in `install_yum()`)\n- Virtual environments per tool via `uv venv`\n- Root venv at `.venv/` for `getjswords.py` and similar helpers\n## Platform Requirements\n- Bash ≥ 4.3 (for `wait -n` used in `lib/parallel.sh`)\n- Go ≥ 1.21 (tools use SIV module paths like `/v2`, `/v3`)\n- Python ≥ 3.7\n- `uv` package manager\n- Rust / Cargo (for `smugglex`)\n- GNU coreutils, getopt, sed (macOS only via Homebrew)\n- ~5GB free disk space for Go cache, tools, and repos\n- ~1GB RAM minimum (Go compilation)\n- Base image: `ubuntu:24.04`\n- Build arg `INSTALL_AXIOM=true` (default) installs axiom fleet tooling\n- Ports 85-90 exposed (for headless browser tooling)\n- Runs as root (required for raw socket operations by some tools)\n- Health check: `./reconftw.sh --health-check`\n\n\n\n## Conventions\n\n## Shell Settings\n## Source Guard Pattern\n- `lib/common.sh`: `[[ -n \"$_COMMON_SH_LOADED\" ]] && return 0`\n- `lib/parallel.sh`: `[[ -n \"$_PARALLEL_SH_LOADED\" ]] && return 0`\n- `lib/ui.sh`: `[[ -n \"${_UI_SH_LOADED:-}\" ]] && return 0`\n## Function Naming\n- Public module functions: `snake_case` prefixed with module context (`sub_passive`, `sub_crt`, `geo_info`)\n- Private helpers: `_snake_case` prefix (`_print_status`, `_print_error`, `_print_module_start`, `_parallel_emit_job_output`)\n- UI layer: `ui_` prefix (`ui_init`, `ui_header`, `ui_summary`, `ui_batch_end`)\n- Lifecycle wrappers: `start_func` / `end_func` (call these at top/bottom of every recon function)\n- Validation functions: `validate_*` / `sanitize_*` (defined in `lib/validation.sh` and `modules/utils.sh`)\n## Variable Naming\n- Config flags: `SUBPASSIVE`, `SUBCRT`, `PARALLEL_MODE`, `OUTPUT_VERBOSITY`\n- Runtime state: `LOGFILE`, `SCRIPTPATH`, `DIFF`, `DRY_RUN`, `AXIOM`\n- Error codes: `E_SUCCESS=0`, `E_INVALID_DOMAIN=20`, `E_INVALID_IP=21` (readonly, defined in `lib/validation.sh`)\n## CLI Flag Pattern\n## Output / UI Conventions\n- `OUTPUT_VERBOSITY=0` (quiet): only errors/FAIL printed\n- `OUTPUT_VERBOSITY=1` (normal, default): OK/WARN/FAIL/SKIP status lines\n- `OUTPUT_VERBOSITY=2` (verbose): all of the above + INFO messages + start_func messages\n## Function Lifecycle (start_func / end_func)\n- `start_func name desc` — logs to LOGFILE, sets per-function start timestamp, emits INFO at verbosity >= 2\n- `end_func message name [status]` — touches checkpoint file, calculates elapsed time, calls `_print_status`\n- `skip_notification reason` — emits SKIP/CACHE badge; reasons: `\"disabled\"`, `\"mode\"`, `\"processed\"`, `\"processed-visible\"`, `\"noinput\"`\n## File Checkpointing (Resumability)\n- `end_func` creates the checkpoint: `touch \"$called_fn_dir/.${fn}\"`\n- DIFF mode (`DIFF=true`) bypasses checkpoint — forces re-execution\n- Helper `should_run()` in `lib/common.sh` provides a cleaner gate: `if should_run \"FLAG_VAR\"; then`\n## Error Handling\n- `E_SUCCESS=0`, `E_GENERAL=1`, `E_MISSING_DEP=2`, `E_INVALID_INPUT=3`\n## Validation Functions\n| Function | Purpose |\n|----------|---------|\n| `validate_domain()` | RFC domain check + injection character rejection |\n| `validate_ipv4()` | Octet range validation |\n| `validate_integer()` | Numeric range check |\n| `validate_boolean()` | Accepts `true`/`false` only (not `1`/`0`/`yes`/`no`) |\n| `validate_file_readable()` | Exists + readable + is-a-file |\n| `sanitize_interlace_input()` | Removes shell metacharacters from input files (canonical in `lib/validation.sh`) |\n| `sanitize_domain()` | Strips URL components, lowercases, rejects injection (in `modules/utils.sh`) |\n| `is_in_scope_host()` | Anchored hostname scope check (prevents substring false positives) |\n| `filter_in_scope_urls()` | Python3-based URL scope check (scheme, userinfo, host) |\n## Path and CWD Conventions\n## Parallel Execution\n## Import / Sourcing Order\n## Logging\n- All tool output redirected to `$LOGFILE`: `command ... 2>>\"$LOGFILE\" >/dev/null`\n- Structured JSON logging via `log_json level func message [key=val]` (optional, `STRUCTURED_LOGGING=true`)\n- `redact_secrets()` scrubs `REDACT_VARS` and `REGISTERED_SECRETS` from log lines\n- `register_secret \"$value\"` must be called before logging any secret value\n## Comments\n\n\n\n## Architecture\n\n## System Overview\n```text\n```\n## Component Responsibilities\n| Component | Responsibility | File |\n|-----------|----------------|------|\n| Entry point & CLI parser | getopt argument parsing, config sourcing, mode dispatch | `reconftw.sh` |\n| Mode orchestration | `start`/`end`, workflow functions (`recon`, `passive`, `all`, `vulns`, `osint`, `subs_menu`, `webs_menu`, `zen_menu`, `monitor_mode`) | `modules/modes.sh` |\n| Function lifecycle | `start_func`/`end_func`, checkpointing, logging, notifications, reporting, plugins, health check | `modules/core.sh` |\n| Subdomain enumeration | All `sub_*` functions, `subtakeover`, `zonetransfer`, `s3buckets`, `geo_info` | `modules/subdomains.sh` |\n| Web analysis | `webprobe_full`, `screenshot`, `nuclei_check`, `fuzz`, `jschecks`, `urlchecks`, `waf_checks`, and 20+ others | `modules/web.sh` |\n| Vulnerability scanning | `xss`, `ssrf_checks`, `sqli`, `crlf_checks`, `lfi`, `ssti`, `smuggling`, `fuzzparams`, `nuclei_dast`, and others | `modules/vulns.sh` |\n| OSINT collection | `domain_info`, `ip_info`, `emails`, `google_dorks`, `github_leaks`, `github_actions_audit`, `cloud_enum_scan`, etc. | `modules/osint.sh` |\n| Shared utilities | `run_command`, `sed_i`, `deleteOutScoped`, `validate_config`, `cache_*`, `checkpoint_*`, `circuit_breaker_*`, rate-limit adaption | `modules/utils.sh` |\n| Axiom/distributed mode | `axiom_launch`, `axiom_shutdown`, `axiom_selected`, `resolvers_update`, `ipcidr_target` | `modules/axiom.sh` |\n| Parallel execution | `parallel_funcs`, `_throttle_jobs`, job heartbeat, progress live display, log mode output | `lib/parallel.sh` |\n| Input validation/sanitization | `sanitize_domain`, `sanitize_ip`, `validate_domain`, `validate_integer`, `_sanitize_list_entry` | `lib/validation.sh` |\n| Shared file/counter utilities | `ensure_dirs`, `ensure_webs_all`, `safe_backup`, `count_lines`, incident tracking | `lib/common.sh` |\n| UI presentation layer | `_print_status`, `_print_msg`, `_print_section`, `_print_rule`, `ui_header`, `ui_summary`, TTY detection, color management, JSONL output | `lib/ui.sh` |\n| Configuration | All runtime settings (~350 variables); sourced after CLI parse | `reconftw.cfg` |\n## Pattern Overview\n- Single process: `reconftw.sh` sources all libraries and modules at startup; every function lives in the same shell environment\n- No subshell isolation between modules — all state is shared via global variables\n- File-based checkpointing: `called_fn_dir/.funcname` sentinel files prevent re-running completed functions across invocations\n- CLI-over-config: `reconftw.cfg` provides defaults; CLI flags set `CLI_*` variables that are re-applied after config sourcing to guarantee they cannot be overwritten\n- All external tool invocations go through `run_command()` which handles dry-run mode, adaptive rate limiting, axiom dispatch, and debug logging\n## Layers\n- Purpose: Bootstrap, macOS re-exec, module loading, getopt CLI parsing, config sourcing, CLI override re-application, mode dispatch\n- Location: `reconftw.sh`\n- Contains: `normalize_vps_count_args()`, the main `while/case` getopt loop, the config `source` sequence, CLI override if-blocks, the final `case $opt_mode` dispatch\n- Depends on: All libraries (sourced first), all modules (sourced second)\n- Used by: End user / CI\n- Purpose: Reusable utilities with no side effects; loadable independently for tests\n- Location: `lib/validation.sh`, `lib/common.sh`, `lib/ui.sh`, `lib/parallel.sh`\n- Contains: Input sanitization, file helpers, UI/color/progress, parallel job management\n- Depends on: Nothing (source-guarded with `_*_LOADED` pattern)\n- Used by: All modules and reconftw.sh\n- Purpose: Implement all scanning, analysis, and orchestration functions\n- Location: `modules/`\n- Contains: All recon, vuln, OSINT, web, subdomain functions\n- Depends on: Libraries (always loaded first), `reconftw.cfg` variables, external tools on PATH\n- Used by: modes.sh orchestrates all others; reconftw.sh dispatches to modes.sh\n- Purpose: Default runtime values for ~350 flags/paths/limits; can be overridden by `secrets.cfg` and custom config\n- Location: `reconftw.cfg`, optionally `secrets.cfg`, optionally `$CUSTOM_CONFIG`\n- Contains: Module enable/disable flags, tool flags, API key env-var references, paths, parallelism settings, verbosity, Axiom settings\n- Depends on: Nothing\n- Used by: Sourced by `reconftw.sh` between CLI parse and CLI override re-application\n- Purpose: Store per-target findings in a stable directory hierarchy\n- Location: `Recon//` (created at `start()` time by `modules/modes.sh`)\n- Contains: Standard subdirectories listed below\n- Depends on: `start()` in modes.sh creates the directory tree\n## Data Flow\n### Primary Recon Request Path (`-r` / `--recon`)\n### Function Execution Path (every leaf module function)\n### Parallel Execution Path\n### Axiom Distributed Scan Flow\n- Global bash variables throughout (no encapsulation). Config vars, target vars (`domain`, `dir`, `called_fn_dir`, `LOGFILE`), and result counters are all globals\n- `passive()` saves/restores module-enable globals before overriding them (`modules/modes.sh:549-611`)\n## Key Abstractions\n- Purpose: Lifecycle wrapper around every leaf scanning function\n- Examples: Used in every function in `modules/subdomains.sh`, `modules/web.sh`, `modules/vulns.sh`, `modules/osint.sh`\n- Pattern: `start_func \"${FUNCNAME[0]}\" \"description\"` at top; `end_func \"output path\" \"${FUNCNAME[0]}\"` at bottom; creates checkpoint file on end\n- Purpose: Prevent re-running completed functions across multiple invocations of the same target\n- Location: `Recon//.called_fn/.funcname`\n- Pattern: Each function tests `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]`; `end_func` writes the sentinel via `touch \"$called_fn_dir/.${fn}\"`\n- Purpose: Universal external-tool gate for dry-run preview, axiom dispatch, adaptive rate limiting, and debug logging\n- Location: `modules/utils.sh:468`\n- Pattern: All tool calls inside module functions use `run_command ` rather than direct invocation\n- Purpose: Allow modules to be sourced multiple times (test re-sourcing, `--source-only`) without re-executing\n- Pattern: `[[ -n \"$_FOO_LOADED\" ]] && return 0` at top of each lib file (`lib/common.sh:6`, `lib/parallel.sh:6`, `lib/ui.sh:5`, `lib/validation.sh` — validation uses error-code guards instead)\n- Purpose: Run independent module functions concurrently up to `PARALLEL_MAX_JOBS`\n- Pattern: `parallel_funcs N func_a func_b func_c` — each function spawned as a background subshell; used in `recon()`, `osint()`, `vulns()` for independent groups\n- Purpose: Transparent axiom/local fallback wrapper — if axiom fails during a module, retries locally\n- Location: `modules/modes.sh:656`\n- Pattern: All module calls inside `subs_menu`, `webs_menu`, `recon`, `passive` use this wrapper\n## Entry Points\n- Location: `reconftw.sh`\n- Triggers: Direct execution (`./reconftw.sh -d example.com -r`)\n- Responsibilities: Bootstrap, all module loading, CLI parse, config source, mode dispatch\n- Location: `reconftw.sh:123-125`\n- Triggers: `./reconftw.sh --source-only` (used by bats test `setup()` blocks)\n- Responsibilities: Sources all modules without executing any recon\n- Location: `modules/modes.sh:13`\n- Triggers: Called at the top of most workflow functions (`recon`, `subs_menu`, `passive`, `osint`, `zen_menu`)\n- Responsibilities: Create output directory tree, init LOGFILE, init cache/incremental/DNS/plugins, set global `dir` and `called_fn_dir`\n- Location: `modules/modes.sh:286`\n- Triggers: Called at the bottom of most workflow functions\n- Responsibilities: AI report, cleanup, Faraday, screenshot diffs, plugin events, hotlist, `export_reports()`, timing summary\n## Output Directory Structure\n```\n```\n## Verbosity and Output Controls\n- `0` (quiet): Only errors and final summary printed to terminal; banner suppressed\n- `1` (normal, default): Errors + warnings printed; `notification()` info/good suppressed\n- `2` (verbose): All `notification()` calls, PID info, full parallel output, `start_func` messages, `print_timing_summary`\n- `summary`: One badge line per completed parallel job\n- `tail`: Last `PARALLEL_TAIL_LINES` (default 20, doubled on failure) from each job's log\n- `full`: Complete captured stdout from each job\n- `jsonl-strict`: Forces `OUTPUT_VERBOSITY=0`, emits only machine-readable JSONL\n## Architectural Constraints\n- **Threading:** Single-threaded bash with optional background subshells via `parallel_funcs`; `wait -n` (bash 4.3+) used for job throttling in `_throttle_jobs`\n- **Global state:** All config vars, `domain`, `dir`, `called_fn_dir`, `LOGFILE`, `start`, `runtime`, `DIFF`, `AXIOM`, and hundreds of module-enable flags are module-level globals; any sourced function can read or mutate them\n- **Circular imports:** None by design — reconftw.sh sources libs first, then modules in explicit dependency order (`utils.sh` → `core.sh` → `osint.sh` → `subdomains.sh` → `web.sh` → `vulns.sh` → `axiom.sh` → `modes.sh`)\n- **macOS bash version:** reconftw.sh re-execs itself under Homebrew bash ≥ 4 on macOS (system bash is 3.2); `lib/parallel.sh` requires bash 4.3+ for `wait -n`\n- **Working directory:** `start()` calls `cd \"$dir\"` (the per-target output dir) before any module function runs; all relative paths inside modules resolve against the target dir. `reconftw.sh` captures `startdir=${PWD}` before this\n- **No subshell isolation per module:** Modules are sourced functions, not subprocess commands. A `return` inside a module returns from the function; an `exit` would kill the whole shell\n## Anti-Patterns\n### Direct external tool calls without `run_command`\n### Writing checkpoint files manually\n### Skipping the `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]` guard\n### Overriding config globals without save/restore in workflow functions\n## Error Handling\n- ERR trap in `start()` logs function name, line number, and command to `$LOGFILE` and calls `explain_err()` (`modules/modes.sh:140`)\n- Non-zero exit from `parallel_funcs` increments `RECON_OSINT_PARALLEL_FAILURES` and sets `RECON_PARTIAL_RUN=true`\n- `run_module_with_axiom_failover` catches axiom mid-run failures and retries locally\n- Circuit-breaker helpers (`circuit_breaker_is_open`, `circuit_breaker_record_failure`) in `modules/utils.sh:1190` for persistent tool failures\n## Cross-Cutting Concerns\n\n\n\n## Project Skills\n\nNo project skills found. Add skills to any of: `.claude/skills/`, `.agents/skills/`, `.cursor/skills/`, `.github/skills/`, or `.codex/skills/` with a `SKILL.md` index file.\n\n\n\n## GSD Workflow Enforcement\n\nBefore using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.\n\nUse these entry points:\n- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks\n- `/gsd-debug` for investigation and bug fixing\n- `/gsd-execute-phase` for planned phase work\n\nDo not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.\n\n\n\n\n\n## Developer Profile\n\n> Profile not yet configured. Run `/gsd-profile-user` to generate your developer profile.\n> This section is managed by `generate-claude-profile` -- do not edit manually.\n\n","docs/ARCHITECTURE.md":"# reconFTW Architecture\n\n## Overview\n\nreconFTW is a modular reconnaissance framework. The codebase is organized as follows:\n\n```\nreconftw/\n├── reconftw.sh # Main entry point, argument parsing, orchestration\n├── reconftw.cfg # Main configuration file (all settings)\n├── modules/ # Phase-specific modules (what to do)\n│ ├── utils.sh # Utility functions (cleanup, sanitization, caching)\n│ ├── core.sh # Framework core (logging, tools check, lifecycle)\n│ ├── osint.sh # OSINT gathering functions\n│ ├── subdomains.sh # Subdomain enumeration\n│ ├── web.sh # Web analysis and crawling\n│ ├── vulns.sh # Vulnerability scanning\n│ ├── axiom.sh # Ax (formerly Axiom) distributed scanning\n│ └── modes.sh # Scan mode orchestration\n├── lib/ # Pure utility libraries (how to do it)\n│ ├── config.sh # [FUTURE] Modular config loader\n│ └── validation.sh # Input validation functions\n└── tests/ # Test suite\n ├── unit/ # Unit tests (bats)\n ├── security/ # Security tests\n └── mocks/ # Tool mocks for testing\n```\n\n## Module Load Order\n\nModules are sourced in this specific order (defined in `reconftw.sh`):\n\n1. `modules/utils.sh` - Base utilities (must be first)\n2. `modules/core.sh` - Framework infrastructure\n3. `modules/osint.sh` - OSINT functions\n4. `modules/subdomains.sh` - Subdomain enumeration\n5. `modules/web.sh` - Web analysis\n6. `modules/vulns.sh` - Vulnerability scanning\n7. `modules/axiom.sh` - Distributed scanning\n8. `modules/modes.sh` - Mode orchestration (must be last)\n\n## Data Flow\n\n```\nTarget Input → Sanitization → Phase Execution → Output Files\n ↓ ↓ ↓ ↓\n --domain sanitize_domain start_func Recon/$domain/\n --list sanitize_ip [tool calls] subdomains/\n --cidr end_func webs/vulns/\n```\n\n## Key Functions\n\n### Lifecycle Functions (core.sh)\n- `start_func(name, description)` - Begin a function, check if already done\n- `end_func(message, name)` - Complete a function, mark as done\n\n### Utility Functions (utils.sh)\n- `sanitize_domain(domain)` - Sanitize domain input (lowercase, safe chars)\n- `sanitize_ip(ip)` - Sanitize IP/CIDR input\n- `should_run_deep(count, limit)` - Check if DEEP mode should run\n- `deleteOutScoped(scopefile, targetfile)` - Remove out-of-scope entries\n- `run_command(cmd, args...)` - Execute with logging (respects DRY_RUN)\n\n### DNS Resolution Functions (utils.sh)\n\nAll DNS resolution goes through two global wrappers that auto-select between puredns and dnsx based on the network environment:\n\n- `_get_local_ip()` - Get the primary local IP address (cross-platform). Uses `route -n get default` on macOS and `ip -4 route get 1.1.1.1` on Linux to find the correct interface regardless of naming.\n- `_ip_is_public_ipv4(ip)` - Conservative check for publicly routable IPv4. Treats RFC1918, CGNAT (100.64.0.0/10), loopback, link-local, multicast/reserved as non-public.\n- `_is_behind_nat()` - Detect if running behind NAT (or unknown network) by checking the primary local IP. Defaults to NAT-safe behavior if detection fails.\n- `init_dns_resolver()` - Evaluates resolver selection once per run and caches it in `DNS_RESOLVER_SELECTED` when `DNS_RESOLVER=auto`.\n- `_select_dns_resolver()` - Select DNS resolver based on `DNS_RESOLVER` config. `puredns`/`dnsx` force the resolver; `auto` uses the cached selection from `init_dns_resolver()` (with a safe fallback if unset).\n- `_resolve_domains(input_file, output_file)` - Resolve a list of domains. Uses puredns (massdns + wildcard filter) on VPS or dnsx (Go net resolver + basic wildcard threshold) behind NAT.\n- `_bruteforce_domains(wordlist, target_domain, output_file)` - Bruteforce subdomains. Equivalent to `puredns bruteforce wordlist domain` or `dnsx -d domain -w wordlist`.\n\n**Why two resolvers:** puredns uses massdns which sends raw UDP packets to thousands of resolvers simultaneously. This floods the NAT table on consumer routers and can kill connectivity. dnsx uses Go's standard net resolver through the OS networking stack, which is NAT-friendly but lacks puredns's advanced wildcard detection.\n\n### Permutation Functions (subdomains.sh)\n\n- `_select_permutations_wordlist(source_file)` - Choose wordlist based on `PERMUTATIONS_WORDLIST_MODE` and subdomain count. In `auto` mode: full list (849 words) for <=100 subs or DEEP mode, short list (162 words) for >100 subs.\n- `_run_permutation_engine(source_file)` - Run gotator with the auto-selected wordlist.\n- `_generate_permutation_candidates(source_file, output_file)` - Generate permutation candidates with byte-size limit.\n\n### Validation Functions (lib/validation.sh)\n- `validate_domain(domain)` - Validate domain format\n- `validate_ipv4(ip)` - Validate IPv4 address\n- `validate_boolean(value)` - Validate boolean string\n\n## Global Variables\n\n### Required (set before module load)\n- `$SCRIPTPATH` - Path to reconftw installation\n- `$domain` - Target domain\n- `$dir` - Output directory for current scan\n\n### Configuration (from reconftw.cfg)\n- `$DEEP` - Enable deep/thorough scanning\n- `$DEEP_LIMIT` - Threshold for auto-deep mode\n- `$AXIOM` - Enable distributed scanning\n- `$*_THREADS` - Thread counts per tool\n- `$DNS_RESOLVER` - DNS resolver selection (`auto`|`puredns`|`dnsx`)\n- `$DNS_RESOLVER_SELECTED` - Cached resolver selection for this run (set by `init_dns_resolver()` when `DNS_RESOLVER=auto`)\n- `$PERMUTATIONS_WORDLIST_MODE` - Wordlist selection (`auto`|`full`|`short`)\n- `$PERMUTATIONS_SHORT_THRESHOLD` - Subdomain count threshold for short wordlist (default: 100)\n\n## Error Codes\n\nDefined in `lib/validation.sh`:\n- `E_SUCCESS=0` - Success\n- `E_GENERAL=1` - General error\n- `E_MISSING_DEP=2` - Missing dependency\n- `E_INVALID_INPUT=3` - Invalid user input\n- `E_CONFIG=8` - Configuration error\n\n## Testing\n\n```bash\nmake test # Unit tests only\nmake test-security # Security tests only\nmake test-all # All tests\nmake lint # Shellcheck errors\nmake lint-fix # Shellcheck warnings with context\n```\n"},"items":[{"name":"ARCHITECTURE.md","path":"docs/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/six2dez/reconftw/HEAD/docs/ARCHITECTURE.md","title":"docs - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# reconFTW Architecture\n\n## Overview\n\nreconFTW is a modular reconnaissance framework. The codebase is organized as follows:\n\n```\nreconftw/\n├── reconftw.sh # Main entry point, argument parsing, orchestration\n├── reconftw.cfg # Main configuration file (all settings)\n├── modules/ # Phase-specific modules (what to do)\n│ ├── utils.sh # Utility functions (cleanup, sanitization, caching)\n│ ├── core.sh # Framework core (logging, tools check, lifecycle)\n│ ├── osint.sh # OSINT gathering functions\n│ ├── subdomains.sh # Subdomain enumeration\n│ ├── web.sh # Web analysis and crawling\n│ ├── vulns.sh # Vulnerability scanning\n│ ├── axiom.sh # Ax (formerly Axiom) distributed scanning\n│ └── modes.sh # Scan mode orchestration\n├── lib/ # Pure utility libraries (how to do it)\n│ ├── config.sh # [FUTURE] Modular config loader\n│ └── validation.sh # Input validation functions\n└── tests/ # Test suite\n ├── unit/ # Unit tests (bats)\n ├── security/ # Security tests\n └── mocks/ # Tool mocks for testing\n```\n\n## Module Load Order\n\nModules are sourced in this specific order (defined in `reconftw.sh`):\n\n1. `modules/utils.sh` - Base utilities (must be first)\n2. `modules/core.sh` - Framework infrastructure\n3. `modules/osint.sh` - OSINT functions\n4. `modules/subdomains.sh` - Subdomain enumeration\n5. `modules/web.sh` - Web analysis\n6. `modules/vulns.sh` - Vulnerability scanning\n7. `modules/axiom.sh` - Distributed scanning\n8. `modules/modes.sh` - Mode orchestration (must be last)\n\n## Data Flow\n\n```\nTarget Input → Sanitization → Phase Execution → Output Files\n ↓ ↓ ↓ ↓\n --domain sanitize_domain start_func Recon/$domain/\n --list sanitize_ip [tool calls] subdomains/\n --cidr end_func webs/vulns/\n```\n\n## Key Functions\n\n### Lifecycle Functions (core.sh)\n- `start_func(name, description)` - Begin a function, check if already done\n- `end_func(message, name)` - Complete a function, mark as done\n\n### Utility Functions (utils.sh)\n- `sanitize_domain(domain)` - Sanitize domain input (lowercase, safe chars)\n- `sanitize_ip(ip)` - Sanitize IP/CIDR input\n- `should_run_deep(count, limit)` - Check if DEEP mode should run\n- `deleteOutScoped(scopefile, targetfile)` - Remove out-of-scope entries\n- `run_command(cmd, args...)` - Execute with logging (respects DRY_RUN)\n\n### DNS Resolution Functions (utils.sh)\n\nAll DNS resolution goes through two global wrappers that auto-select between puredns and dnsx based on the network environment:\n\n- `_get_local_ip()` - Get the primary local IP address (cross-platform). Uses `route -n get default` on macOS and `ip -4 route get 1.1.1.1` on Linux to find the correct interface regardless of naming.\n- `_ip_is_public_ipv4(ip)` - Conservative check for publicly routable IPv4. Treats RFC1918, CGNAT (100.64.0.0/10), loopback, link-local, multicast/reserved as non-public.\n- `_is_behind_nat()` - Detect if running behind NAT (or unknown network) by checking the primary local IP. Defaults to NAT-safe behavior if detection fails.\n- `init_dns_resolver()` - Evaluates resolver selection once per run and caches it in `DNS_RESOLVER_SELECTED` when `DNS_RESOLVER=auto`.\n- `_select_dns_resolver()` - Select DNS resolver based on `DNS_RESOLVER` config. `puredns`/`dnsx` force the resolver; `auto` uses the cached selection from `init_dns_resolver()` (with a safe fallback if unset).\n- `_resolve_domains(input_file, output_file)` - Resolve a list of domains. Uses puredns (massdns + wildcard filter) on VPS or dnsx (Go net resolver + basic wildcard threshold) behind NAT.\n- `_bruteforce_domains(wordlist, target_domain, output_file)` - Bruteforce subdomains. Equivalent to `puredns bruteforce wordlist domain` or `dnsx -d domain -w wordlist`.\n\n**Why two resolvers:** puredns uses massdns which sends raw UDP packets to thousands of resolvers simultaneously. This floods the NAT table on consumer routers and can kill connectivity. dnsx uses Go's standard net resolver through the OS networking stack, which is NAT-friendly but lacks puredns's advanced wildcard detection.\n\n### Permutation Functions (subdomains.sh)\n\n- `_select_permutations_wordlist(source_file)` - Choose wordlist based on `PERMUTATIONS_WORDLIST_MODE` and subdomain count. In `auto` mode: full list (849 words) for <=100 subs or DEEP mode, short list (162 words) for >100 subs.\n- `_run_permutation_engine(source_file)` - Run gotator with the auto-selected wordlist.\n- `_generate_permutation_candidates(source_file, output_file)` - Generate permutation candidates with byte-size limit.\n\n### Validation Functions (lib/validation.sh)\n- `validate_domain(domain)` - Validate domain format\n- `validate_ipv4(ip)` - Validate IPv4 address\n- `validate_boolean(value)` - Validate boolean string\n\n## Global Variables\n\n### Required (set before module load)\n- `$SCRIPTPATH` - Path to reconftw installation\n- `$domain` - Target domain\n- `$dir` - Output directory for current scan\n\n### Configuration (from reconftw.cfg)\n- `$DEEP` - Enable deep/thorough scanning\n- `$DEEP_LIMIT` - Threshold for auto-deep mode\n- `$AXIOM` - Enable distributed scanning\n- `$*_THREADS` - Thread counts per tool\n- `$DNS_RESOLVER` - DNS resolver selection (`auto`|`puredns`|`dnsx`)\n- `$DNS_RESOLVER_SELECTED` - Cached resolver selection for this run (set by `init_dns_resolver()` when `DNS_RESOLVER=auto`)\n- `$PERMUTATIONS_WORDLIST_MODE` - Wordlist selection (`auto`|`full`|`short`)\n- `$PERMUTATIONS_SHORT_THRESHOLD` - Subdomain count threshold for short wordlist (default: 100)\n\n## Error Codes\n\nDefined in `lib/validation.sh`:\n- `E_SUCCESS=0` - Success\n- `E_GENERAL=1` - General error\n- `E_MISSING_DEP=2` - Missing dependency\n- `E_INVALID_INPUT=3` - Invalid user input\n- `E_CONFIG=8` - Configuration error\n\n## Testing\n\n```bash\nmake test # Unit tests only\nmake test-security # Security tests only\nmake test-all # All tests\nmake lint # Shellcheck errors\nmake lint-fix # Shellcheck warnings with context\n```\n","isInternal":false,"tokens":1537,"sizeBytes":6299},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/six2dez/reconftw/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"\n## Project\n\n**reconFTW**\n\nreconFTW is a comprehensive bash-based reconnaissance automation framework used by bug bounty hunters, penetration testers, and security researchers. It orchestrates 70+ external security tools (Go, Python, Rust) across subdomain enumeration, web probing, OSINT, and vulnerability scanning, producing structured per-target output trees with optional Axiom distributed execution, AI reporting, monitor/incremental mode, and Slack/Telegram/Discord notifications.\n\n**Core Value:** Run one command, get a complete recon picture of a target — passive, active, and vulnerability layers — with resumable checkpoints, structured output, and zero-touch tool orchestration.\n\n### Constraints\n\n- **Tech stack**: Bash 4.3+ — Required for `wait -n`, `mapfile`, associative arrays. macOS users must have Homebrew bash; auto re-exec is best-effort.\n- **External tools**: 70+ runtime dependencies — Most install via `go install @latest` (no version pinning), which is convenient but a known supply-chain risk.\n- **Single process**: All modules sourced into one shell — No subshell isolation between modules; all state shared via globals. Workflow functions must save/restore globals they override (see `passive()` pattern).\n- **Resume semantics**: Checkpoint files are touch-once at `end_func` — Interrupted functions re-run from scratch on next invocation; partial outputs are not detected.\n- **Single-operator**: Designed for one user per target run — No locking, no multi-user state, no concurrent runs against the same target dir.\n- **Output stability**: `Recon//` tree is a public contract — Subdirectory names and filenames are consumed by downstream pipelines, scripts, and parsers; renames are breaking changes.\n- **macOS compatibility**: GNU coreutils + GNU sed + GNU getopt required — System BSD versions are not supported.\n- **CI budget**: Integration-full is weekly cron — Unit + smoke are per-push; adding heavy integration tests must respect this split.\n\n\n\n## Technology Stack\n\n## Languages\n- Bash 4+ - All core framework logic (`reconftw.sh`, `modules/*.sh`, `lib/*.sh`, `install.sh`)\n- Python 3.7+ - Python-backed tools (each runs in isolated `uv venv`): `dorks_hunter`, `CMSeeK`, `EmailHarvester`, `Spoofy`, `SSTImap`, `gato`, `regulator`, `reconftw_ai`, `getjswords.py`\n- Go (latest, min ~1.21) - Primary binary language for ~55 security tools installed via `go install @latest`\n## Runtime\n- Linux (Debian/Ubuntu/RHEL/Arch) or macOS (Apple Silicon / Intel)\n- Docker: Ubuntu 24.04 base image (`Docker/Dockerfile`)\n- ARM64/aarch64, ARMv6l/v7l, and x86_64 all supported for Go binary installs\n- `getopt` (GNU getopt required on macOS via `brew install gnu-getopt`)\n- `nproc` / `sysctl -n hw.ncpu` for CPU core auto-detection\n- `timeout` / `gtimeout` (macOS Homebrew `coreutils`)\n- `gnu-sed` (macOS requires `brew install gnu-sed`)\n- `gnu-coreutils` (macOS requires `brew install coreutils`)\n- Go tools: `go install` (`GOPATH=$HOME/go`, `GOROOT=/usr/local/go`)\n- Python tools: `uv tool install` from GitHub (most) or PyPI (`fray`)\n- Python repo venvs: `uv venv venv && uv pip install -r requirements.txt`\n- Lockfile: None (always installs `@latest`)\n## Frameworks\n- No framework — pure Bash with sourced module files\n- Module loading order: `lib/validation.sh` → `lib/common.sh` → `lib/ui.sh` → `lib/parallel.sh` → `modules/utils.sh` → `modules/core.sh` → `modules/osint.sh` → `modules/subdomains.sh` → `modules/web.sh` → `modules/vulns.sh` → `modules/axiom.sh` → `modules/modes.sh`\n- bats-core (Bash Automated Testing System)\n- GNU make (`Makefile`) — test, lint, format targets\n- shellcheck (error-level) — `make lint`, pre-commit hook\n- shfmt (4-space indent, `-bn`, `-ci`) — `make fmt`, pre-commit hook\n- pre-commit hooks defined in `.pre-commit-config.yaml`\n- semgrep: `.github/workflows/semgrep.yml` (CI only)\n## Key Dependencies\n### Go Tools (installed via `go install @latest`)\n- `subfinder` (projectdiscovery/subfinder) — passive multi-source subdomain discovery\n- `github-subdomains` (gwen001/github-subdomains) — GitHub-based subdomain search\n- `gitlab-subdomains` (gwen001/gitlab-subdomains) — GitLab-based subdomain search\n- `dnstake` (pwnesia/dnstake) — subdomain takeover detection\n- `puredns` (d3mondev/puredns) — mass DNS resolution with wildcard filtering\n- `dnsx` (projectdiscovery/dnsx) — DNS toolkit\n- `massdns` (blechschmidt/massdns) — via repo clone + build\n- `dsieve` (trickest/dsieve) — subdomain filtering\n- `enumerepo` (trickest/enumerepo) — GitHub org repo enumeration\n- `gotator` (Josue87/gotator) — subdomain permutations\n- `analyticsrelationships` (Josue87/analyticsrelationships) — Google Analytics pivoting\n- `roboxtractor` (Josue87/roboxtractor) — robots.txt extractor\n- `crt` (cemulus/crt) — crt.sh search\n- `asnmap` (projectdiscovery/asnmap) — ASN-to-CIDR mapping\n- `mapcidr` (projectdiscovery/mapcidr) — CIDR manipulation\n- `smap` (s0md3v/smap) — passive Shodan-powered port scan\n- `tlsx` (projectdiscovery/tlsx) — TLS certificate harvesting\n- `hakip2host` (hakluke/hakip2host) — reverse IP lookup\n- `cdncheck` (projectdiscovery/cdncheck) — CDN/WAF IP classification\n- `hakoriginfinder` (hakluke/hakoriginfinder) — origin IP discovery behind CDN\n- `inscope` (tomnomnom/hacks/inscope) — scope filtering\n- `csprecon` (edoardottt/csprecon) — CSP-based subdomain discovery\n- `favirecon` (edoardottt/favirecon) — favicon-based tech recon\n- `httpx` (projectdiscovery/httpx) — multi-probe HTTP toolkit\n- `katana` (projectdiscovery/katana) — web crawler\n- `ffuf` (ffuf/ffuf) — web fuzzer\n- `subjs` (lc/subjs) — JavaScript URL extractor\n- `Gxss` (KathanP19/Gxss) — reflected XSS param finder\n- `jsluice` (BishopFox/jsluice) — JS secret/URL extractor\n- `sourcemapper` (denandz/sourcemapper) — JS source map extractor\n- `mantra` (brosck/mantra) — JS/secret scanner\n- `urlfinder` (projectdiscovery/urlfinder) — URL discovery\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — via uv\n- `nmapurls` (sdcampbell/nmapurls) — URL extraction from Nmap XML\n- `naabu` (projectdiscovery/naabu) — fast port scanner\n- `VhostFinder` (wdahlenburg/VhostFinder) — virtual host discovery\n- `shortscan` (bitquark/shortscan) — IIS short filename scanner\n- `nuclei` (projectdiscovery/nuclei) — template-based scanner\n- `dalfox` (hahwul/dalfox) — XSS scanner\n- `crlfuzz` (dwisiswant0/crlfuzz) — CRLF injection scanner\n- `Web-Cache-Vulnerability-Scanner` (Hackmanit) — web cache poisoning\n- `TInjA` (Hackmanit/TInjA) — SSTI scanner\n- `toxicache` (xhzeem/toxicache) — web cache poisoning\n- `second-order` (mhmdiaa/second-order) — broken link/second-order injection\n- `s3scanner` (sa7mon/s3scanner) — S3/GCS/Azure Blob misconfiguration\n- `misconfig-mapper` (intigriti/misconfig-mapper) — third-party misconfiguration\n- `sj` (BishopFox/sj) — Swagger/OpenAPI analysis\n- `grpcurl` (fullstorydev/grpcurl) — gRPC reflection scanner\n- `nerva` (praetorian-inc/nerva) — service fingerprinting\n- `brutus` (praetorian-inc/brutus) — credential spraying\n- `julius` (praetorian-inc/julius) — LLM endpoint probe\n- `titus` (praetorian-inc/titus) — secrets engine\n- `notify` (projectdiscovery/notify) — multi-channel notifications\n- `interactsh-client` (projectdiscovery/interactsh) — OOB callback server\n- `gf` (tomnomnom/gf) — URL pattern grep\n- `anew` (tomnomnom/anew) — append new lines only\n- `unfurl` (tomnomnom/unfurl) — URL parser\n- `qsreplace` (tomnomnom/qsreplace) — querystring replacer\n- `gitdorks_go` (damit5/gitdorks_go) — GitHub dork search\n- `github-endpoints` (gwen001/github-endpoints) — GitHub endpoint discovery\n- `cent` (xm1k3/cent) — nuclei template manager\n- `trufflehog` (trufflesecurity/trufflehog) — secrets scanner (via `go install`)\n- `brutespray` (x90skysn3k/brutespray) — service credential spraying\n### Python Tools (installed via `uv tool install`)\n- `dnsvalidator` (vortexau/dnsvalidator) — DNS resolver validation\n- `interlace` (pry0cc/interlace) — parallel command runner\n- `wafw00f` (EnableSecurity/wafw00f) — WAF fingerprinting\n- `commix` (commixproject/commix) — command injection scanner\n- `waymore` (xnl-h4ck3r/waymore) — passive URL collection\n- `urless` (xnl-h4ck3r/urless) — URL deduplication\n- `ghauri` (r0oth3x49/ghauri) — SQLi scanner (optional)\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — deep link finder\n- `xnldorker` (xnl-h4ck3r/xnldorker) — Google dorker\n- `porch-pirate` (MandConsultingGroup/porch-pirate) — Postman API leaks\n- `p1radup` (iambouali/p1radup) — URL deduplication\n- `subwiz` (hadriansecurity/subwiz) — ML-based subdomain prediction\n- `arjun` (s0md3v/Arjun) — parameter discovery\n- `gqlspection` (doyensec/GQLSpection) — GraphQL deep introspection\n- `postleaksNg` (six2dez/postleaksNG) — Postman public leak search\n- `cewler` (roys/cewler) — web wordlist generator\n- `fray` (dalisecurity/fray) — WAF-aware payload testing (PyPI)\n### Repo-Clone Tools (Python venvs, run via `venv/bin/python3`)\n- `dorks_hunter` (six2dez/dorks_hunter) — Google dork automation\n- `CMSeeK` (Tuhinshubhra/CMSeeK) — CMS fingerprinting\n- `cloud_enum` (initstring/cloud_enum) — AWS/GCP/Azure bucket enumeration\n- `EmailHarvester` (maldevel/EmailHarvester) — email harvesting\n- `SwaggerSpy` (UndeadSec/SwaggerSpy) — Swagger endpoint leak detection\n- `LeakSearch` (JoelGMSec/LeakSearch) — credential leak search\n- `Spoofy` (MattKeeley/Spoofy) — email spoofing check\n- `msftrecon` (Arcanum-Sec/msftrecon) — Microsoft tenant recon\n- `Scopify` (Arcanum-Sec/Scopify) — scope management\n- `regulator` (cramppet/regulator) — regex-based subdomain permutations\n- `SSTImap` (vladko312/SSTImap) — SSTI scanner (alternative engine)\n- `gato` (praetorian-inc/gato) — GitHub Actions audit\n### Repo-Clone Tools (Go build)\n- `ghleaks` (dinosn/ghleaks) — GitHub-wide secret search\n- `nomore403` (devploit/nomore403) — 403 bypass\n- `ffufPostprocessing` (Damian89/ffufPostprocessing) — ffuf result analysis\n- `JSA` (w9w/JSA) — JS analysis\n- `ultimate-nmap-parser` (shifty0g/ultimate-nmap-parser) — Nmap XML parser\n### System-Level Tools (apt/brew/yum)\n- `nmap` — active port scanning\n- `massdns` — DNS resolver (also cloned + built from source)\n- `jq` — JSON processing throughout all modules\n- `exiftool` (perl-Image-ExifTool) — metadata extraction\n- `whois` — domain registration lookup\n- `sqlmap` — SQL injection (system or via repo clone)\n- `testssl.sh` (testssl/testssl.sh) — TLS/SSL misconfiguration testing\n- `medusa` — credential brute-force (system install)\n- `shodan` CLI — installed via `uv tool install shodan`\n### Rust Tools\n- `smugglex` (Cargo) — HTTP request smuggling detection\n- Rustup installed from `https://sh.rustup.rs`\n## Configuration\n- `reconftw.cfg` — sourced after CLI parsing; all feature flags, rate limits, timeouts, wordlist paths, API keys, thread counts\n- `secrets.cfg` (gitignored, auto-sourced) — API keys and tokens separated from main config\n- `secrets.cfg.example` — template showing all supported secret vars\n- Feature flags: `OSINT=true`, `SUBDOMAINS_GENERAL=true`, `VULNS_GENERAL=false`, etc.\n- Rate limits: `HTTPX_RATELIMIT=150`, `NUCLEI_RATELIMIT=150`, `FFUF_RATELIMIT=0`\n- Thread counts: auto-scaled via `AVAILABLE_CORES=$(nproc)` with multipliers per tool\n- Timeouts: per-tool in seconds or minutes (`CMSSCAN_TIMEOUT=3600`, `SUBFINDER_ENUM_TIMEOUT=180`)\n- Wordlist paths: `fuzz_wordlist`, `lfi_wordlist`, `subs_wordlist` etc. under `${WORDLISTS_DIR}`\n- Output: `EXPORT_FORMAT`, `AI_REPORT_TYPE`, `ASSET_STORE`\n- GNU `getopt` long options, parsed in `reconftw.sh` while/case loop\n- All CLI overrides use `CLI_*` pattern and are re-applied after `reconftw.cfg` is sourced\n- Full list from `getopt` call: `domain`, `list`, `recon`, `subdomains`, `passive`, `all`, `web`, `osint`, `zen`, `deep`, `help`, `vps`, `vps-count`, `ai`, `check-tools`, `health-check`, `quick-rescan`, `incremental`, `adaptive-rate`, `dry-run`, `parallel`, `no-parallel`, `monitor`, `monitor-interval`, `monitor-cycles`, `refresh-cache`, `gen-resolvers`, `force`, `export`, `report-only`, `no-report`, `parallel-log`, `quiet`, `verbose`, `no-color`, `log-format`, `show-cache`, `banner`, `no-banner`, `legal`\n- `SHODAN_API_KEY`, `WHOISXML_API`, `PDCP_API_KEY`, `XSS_SERVER`, `COLLAB_SERVER` — preferred over config file\n- `GOROOT`, `GOPATH`, `PATH` — extended by `reconftw.cfg` for Go and Rust binaries\n- `LOGFILE` — per-target log path\n- `config/reconftw_full.cfg` — full-scan preset\n- `config/reconftw_quick.cfg` — quick-scan preset\n- `config/reconftw_stealth.cfg` — low-noise preset\n## Build\n- Default: `go1.23.6` (fetches latest from `https://go.dev/VERSION?m=text`)\n- Installed to `/usr/local/go`; set `install_golang=false` in config to skip\n- Minimum: Python 3.7 (enforced in `install_yum()`)\n- Virtual environments per tool via `uv venv`\n- Root venv at `.venv/` for `getjswords.py` and similar helpers\n## Platform Requirements\n- Bash ≥ 4.3 (for `wait -n` used in `lib/parallel.sh`)\n- Go ≥ 1.21 (tools use SIV module paths like `/v2`, `/v3`)\n- Python ≥ 3.7\n- `uv` package manager\n- Rust / Cargo (for `smugglex`)\n- GNU coreutils, getopt, sed (macOS only via Homebrew)\n- ~5GB free disk space for Go cache, tools, and repos\n- ~1GB RAM minimum (Go compilation)\n- Base image: `ubuntu:24.04`\n- Build arg `INSTALL_AXIOM=true` (default) installs axiom fleet tooling\n- Ports 85-90 exposed (for headless browser tooling)\n- Runs as root (required for raw socket operations by some tools)\n- Health check: `./reconftw.sh --health-check`\n\n\n\n## Conventions\n\n## Shell Settings\n## Source Guard Pattern\n- `lib/common.sh`: `[[ -n \"$_COMMON_SH_LOADED\" ]] && return 0`\n- `lib/parallel.sh`: `[[ -n \"$_PARALLEL_SH_LOADED\" ]] && return 0`\n- `lib/ui.sh`: `[[ -n \"${_UI_SH_LOADED:-}\" ]] && return 0`\n## Function Naming\n- Public module functions: `snake_case` prefixed with module context (`sub_passive`, `sub_crt`, `geo_info`)\n- Private helpers: `_snake_case` prefix (`_print_status`, `_print_error`, `_print_module_start`, `_parallel_emit_job_output`)\n- UI layer: `ui_` prefix (`ui_init`, `ui_header`, `ui_summary`, `ui_batch_end`)\n- Lifecycle wrappers: `start_func` / `end_func` (call these at top/bottom of every recon function)\n- Validation functions: `validate_*` / `sanitize_*` (defined in `lib/validation.sh` and `modules/utils.sh`)\n## Variable Naming\n- Config flags: `SUBPASSIVE`, `SUBCRT`, `PARALLEL_MODE`, `OUTPUT_VERBOSITY`\n- Runtime state: `LOGFILE`, `SCRIPTPATH`, `DIFF`, `DRY_RUN`, `AXIOM`\n- Error codes: `E_SUCCESS=0`, `E_INVALID_DOMAIN=20`, `E_INVALID_IP=21` (readonly, defined in `lib/validation.sh`)\n## CLI Flag Pattern\n## Output / UI Conventions\n- `OUTPUT_VERBOSITY=0` (quiet): only errors/FAIL printed\n- `OUTPUT_VERBOSITY=1` (normal, default): OK/WARN/FAIL/SKIP status lines\n- `OUTPUT_VERBOSITY=2` (verbose): all of the above + INFO messages + start_func messages\n## Function Lifecycle (start_func / end_func)\n- `start_func name desc` — logs to LOGFILE, sets per-function start timestamp, emits INFO at verbosity >= 2\n- `end_func message name [status]` — touches checkpoint file, calculates elapsed time, calls `_print_status`\n- `skip_notification reason` — emits SKIP/CACHE badge; reasons: `\"disabled\"`, `\"mode\"`, `\"processed\"`, `\"processed-visible\"`, `\"noinput\"`\n## File Checkpointing (Resumability)\n- `end_func` creates the checkpoint: `touch \"$called_fn_dir/.${fn}\"`\n- DIFF mode (`DIFF=true`) bypasses checkpoint — forces re-execution\n- Helper `should_run()` in `lib/common.sh` provides a cleaner gate: `if should_run \"FLAG_VAR\"; then`\n## Error Handling\n- `E_SUCCESS=0`, `E_GENERAL=1`, `E_MISSING_DEP=2`, `E_INVALID_INPUT=3`\n## Validation Functions\n| Function | Purpose |\n|----------|---------|\n| `validate_domain()` | RFC domain check + injection character rejection |\n| `validate_ipv4()` | Octet range validation |\n| `validate_integer()` | Numeric range check |\n| `validate_boolean()` | Accepts `true`/`false` only (not `1`/`0`/`yes`/`no`) |\n| `validate_file_readable()` | Exists + readable + is-a-file |\n| `sanitize_interlace_input()` | Removes shell metacharacters from input files (canonical in `lib/validation.sh`) |\n| `sanitize_domain()` | Strips URL components, lowercases, rejects injection (in `modules/utils.sh`) |\n| `is_in_scope_host()` | Anchored hostname scope check (prevents substring false positives) |\n| `filter_in_scope_urls()` | Python3-based URL scope check (scheme, userinfo, host) |\n## Path and CWD Conventions\n## Parallel Execution\n## Import / Sourcing Order\n## Logging\n- All tool output redirected to `$LOGFILE`: `command ... 2>>\"$LOGFILE\" >/dev/null`\n- Structured JSON logging via `log_json level func message [key=val]` (optional, `STRUCTURED_LOGGING=true`)\n- `redact_secrets()` scrubs `REDACT_VARS` and `REGISTERED_SECRETS` from log lines\n- `register_secret \"$value\"` must be called before logging any secret value\n## Comments\n\n\n\n## Architecture\n\n## System Overview\n```text\n```\n## Component Responsibilities\n| Component | Responsibility | File |\n|-----------|----------------|------|\n| Entry point & CLI parser | getopt argument parsing, config sourcing, mode dispatch | `reconftw.sh` |\n| Mode orchestration | `start`/`end`, workflow functions (`recon`, `passive`, `all`, `vulns`, `osint`, `subs_menu`, `webs_menu`, `zen_menu`, `monitor_mode`) | `modules/modes.sh` |\n| Function lifecycle | `start_func`/`end_func`, checkpointing, logging, notifications, reporting, plugins, health check | `modules/core.sh` |\n| Subdomain enumeration | All `sub_*` functions, `subtakeover`, `zonetransfer`, `s3buckets`, `geo_info` | `modules/subdomains.sh` |\n| Web analysis | `webprobe_full`, `screenshot`, `nuclei_check`, `fuzz`, `jschecks`, `urlchecks`, `waf_checks`, and 20+ others | `modules/web.sh` |\n| Vulnerability scanning | `xss`, `ssrf_checks`, `sqli`, `crlf_checks`, `lfi`, `ssti`, `smuggling`, `fuzzparams`, `nuclei_dast`, and others | `modules/vulns.sh` |\n| OSINT collection | `domain_info`, `ip_info`, `emails`, `google_dorks`, `github_leaks`, `github_actions_audit`, `cloud_enum_scan`, etc. | `modules/osint.sh` |\n| Shared utilities | `run_command`, `sed_i`, `deleteOutScoped`, `validate_config`, `cache_*`, `checkpoint_*`, `circuit_breaker_*`, rate-limit adaption | `modules/utils.sh` |\n| Axiom/distributed mode | `axiom_launch`, `axiom_shutdown`, `axiom_selected`, `resolvers_update`, `ipcidr_target` | `modules/axiom.sh` |\n| Parallel execution | `parallel_funcs`, `_throttle_jobs`, job heartbeat, progress live display, log mode output | `lib/parallel.sh` |\n| Input validation/sanitization | `sanitize_domain`, `sanitize_ip`, `validate_domain`, `validate_integer`, `_sanitize_list_entry` | `lib/validation.sh` |\n| Shared file/counter utilities | `ensure_dirs`, `ensure_webs_all`, `safe_backup`, `count_lines`, incident tracking | `lib/common.sh` |\n| UI presentation layer | `_print_status`, `_print_msg`, `_print_section`, `_print_rule`, `ui_header`, `ui_summary`, TTY detection, color management, JSONL output | `lib/ui.sh` |\n| Configuration | All runtime settings (~350 variables); sourced after CLI parse | `reconftw.cfg` |\n## Pattern Overview\n- Single process: `reconftw.sh` sources all libraries and modules at startup; every function lives in the same shell environment\n- No subshell isolation between modules — all state is shared via global variables\n- File-based checkpointing: `called_fn_dir/.funcname` sentinel files prevent re-running completed functions across invocations\n- CLI-over-config: `reconftw.cfg` provides defaults; CLI flags set `CLI_*` variables that are re-applied after config sourcing to guarantee they cannot be overwritten\n- All external tool invocations go through `run_command()` which handles dry-run mode, adaptive rate limiting, axiom dispatch, and debug logging\n## Layers\n- Purpose: Bootstrap, macOS re-exec, module loading, getopt CLI parsing, config sourcing, CLI override re-application, mode dispatch\n- Location: `reconftw.sh`\n- Contains: `normalize_vps_count_args()`, the main `while/case` getopt loop, the config `source` sequence, CLI override if-blocks, the final `case $opt_mode` dispatch\n- Depends on: All libraries (sourced first), all modules (sourced second)\n- Used by: End user / CI\n- Purpose: Reusable utilities with no side effects; loadable independently for tests\n- Location: `lib/validation.sh`, `lib/common.sh`, `lib/ui.sh`, `lib/parallel.sh`\n- Contains: Input sanitization, file helpers, UI/color/progress, parallel job management\n- Depends on: Nothing (source-guarded with `_*_LOADED` pattern)\n- Used by: All modules and reconftw.sh\n- Purpose: Implement all scanning, analysis, and orchestration functions\n- Location: `modules/`\n- Contains: All recon, vuln, OSINT, web, subdomain functions\n- Depends on: Libraries (always loaded first), `reconftw.cfg` variables, external tools on PATH\n- Used by: modes.sh orchestrates all others; reconftw.sh dispatches to modes.sh\n- Purpose: Default runtime values for ~350 flags/paths/limits; can be overridden by `secrets.cfg` and custom config\n- Location: `reconftw.cfg`, optionally `secrets.cfg`, optionally `$CUSTOM_CONFIG`\n- Contains: Module enable/disable flags, tool flags, API key env-var references, paths, parallelism settings, verbosity, Axiom settings\n- Depends on: Nothing\n- Used by: Sourced by `reconftw.sh` between CLI parse and CLI override re-application\n- Purpose: Store per-target findings in a stable directory hierarchy\n- Location: `Recon//` (created at `start()` time by `modules/modes.sh`)\n- Contains: Standard subdirectories listed below\n- Depends on: `start()` in modes.sh creates the directory tree\n## Data Flow\n### Primary Recon Request Path (`-r` / `--recon`)\n### Function Execution Path (every leaf module function)\n### Parallel Execution Path\n### Axiom Distributed Scan Flow\n- Global bash variables throughout (no encapsulation). Config vars, target vars (`domain`, `dir`, `called_fn_dir`, `LOGFILE`), and result counters are all globals\n- `passive()` saves/restores module-enable globals before overriding them (`modules/modes.sh:549-611`)\n## Key Abstractions\n- Purpose: Lifecycle wrapper around every leaf scanning function\n- Examples: Used in every function in `modules/subdomains.sh`, `modules/web.sh`, `modules/vulns.sh`, `modules/osint.sh`\n- Pattern: `start_func \"${FUNCNAME[0]}\" \"description\"` at top; `end_func \"output path\" \"${FUNCNAME[0]}\"` at bottom; creates checkpoint file on end\n- Purpose: Prevent re-running completed functions across multiple invocations of the same target\n- Location: `Recon//.called_fn/.funcname`\n- Pattern: Each function tests `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]`; `end_func` writes the sentinel via `touch \"$called_fn_dir/.${fn}\"`\n- Purpose: Universal external-tool gate for dry-run preview, axiom dispatch, adaptive rate limiting, and debug logging\n- Location: `modules/utils.sh:468`\n- Pattern: All tool calls inside module functions use `run_command ` rather than direct invocation\n- Purpose: Allow modules to be sourced multiple times (test re-sourcing, `--source-only`) without re-executing\n- Pattern: `[[ -n \"$_FOO_LOADED\" ]] && return 0` at top of each lib file (`lib/common.sh:6`, `lib/parallel.sh:6`, `lib/ui.sh:5`, `lib/validation.sh` — validation uses error-code guards instead)\n- Purpose: Run independent module functions concurrently up to `PARALLEL_MAX_JOBS`\n- Pattern: `parallel_funcs N func_a func_b func_c` — each function spawned as a background subshell; used in `recon()`, `osint()`, `vulns()` for independent groups\n- Purpose: Transparent axiom/local fallback wrapper — if axiom fails during a module, retries locally\n- Location: `modules/modes.sh:656`\n- Pattern: All module calls inside `subs_menu`, `webs_menu`, `recon`, `passive` use this wrapper\n## Entry Points\n- Location: `reconftw.sh`\n- Triggers: Direct execution (`./reconftw.sh -d example.com -r`)\n- Responsibilities: Bootstrap, all module loading, CLI parse, config source, mode dispatch\n- Location: `reconftw.sh:123-125`\n- Triggers: `./reconftw.sh --source-only` (used by bats test `setup()` blocks)\n- Responsibilities: Sources all modules without executing any recon\n- Location: `modules/modes.sh:13`\n- Triggers: Called at the top of most workflow functions (`recon`, `subs_menu`, `passive`, `osint`, `zen_menu`)\n- Responsibilities: Create output directory tree, init LOGFILE, init cache/incremental/DNS/plugins, set global `dir` and `called_fn_dir`\n- Location: `modules/modes.sh:286`\n- Triggers: Called at the bottom of most workflow functions\n- Responsibilities: AI report, cleanup, Faraday, screenshot diffs, plugin events, hotlist, `export_reports()`, timing summary\n## Output Directory Structure\n```\n```\n## Verbosity and Output Controls\n- `0` (quiet): Only errors and final summary printed to terminal; banner suppressed\n- `1` (normal, default): Errors + warnings printed; `notification()` info/good suppressed\n- `2` (verbose): All `notification()` calls, PID info, full parallel output, `start_func` messages, `print_timing_summary`\n- `summary`: One badge line per completed parallel job\n- `tail`: Last `PARALLEL_TAIL_LINES` (default 20, doubled on failure) from each job's log\n- `full`: Complete captured stdout from each job\n- `jsonl-strict`: Forces `OUTPUT_VERBOSITY=0`, emits only machine-readable JSONL\n## Architectural Constraints\n- **Threading:** Single-threaded bash with optional background subshells via `parallel_funcs`; `wait -n` (bash 4.3+) used for job throttling in `_throttle_jobs`\n- **Global state:** All config vars, `domain`, `dir`, `called_fn_dir`, `LOGFILE`, `start`, `runtime`, `DIFF`, `AXIOM`, and hundreds of module-enable flags are module-level globals; any sourced function can read or mutate them\n- **Circular imports:** None by design — reconftw.sh sources libs first, then modules in explicit dependency order (`utils.sh` → `core.sh` → `osint.sh` → `subdomains.sh` → `web.sh` → `vulns.sh` → `axiom.sh` → `modes.sh`)\n- **macOS bash version:** reconftw.sh re-execs itself under Homebrew bash ≥ 4 on macOS (system bash is 3.2); `lib/parallel.sh` requires bash 4.3+ for `wait -n`\n- **Working directory:** `start()` calls `cd \"$dir\"` (the per-target output dir) before any module function runs; all relative paths inside modules resolve against the target dir. `reconftw.sh` captures `startdir=${PWD}` before this\n- **No subshell isolation per module:** Modules are sourced functions, not subprocess commands. A `return` inside a module returns from the function; an `exit` would kill the whole shell\n## Anti-Patterns\n### Direct external tool calls without `run_command`\n### Writing checkpoint files manually\n### Skipping the `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]` guard\n### Overriding config globals without save/restore in workflow functions\n## Error Handling\n- ERR trap in `start()` logs function name, line number, and command to `$LOGFILE` and calls `explain_err()` (`modules/modes.sh:140`)\n- Non-zero exit from `parallel_funcs` increments `RECON_OSINT_PARALLEL_FAILURES` and sets `RECON_PARTIAL_RUN=true`\n- `run_module_with_axiom_failover` catches axiom mid-run failures and retries locally\n- Circuit-breaker helpers (`circuit_breaker_is_open`, `circuit_breaker_record_failure`) in `modules/utils.sh:1190` for persistent tool failures\n## Cross-Cutting Concerns\n\n\n\n## Project Skills\n\nNo project skills found. Add skills to any of: `.claude/skills/`, `.agents/skills/`, `.cursor/skills/`, `.github/skills/`, or `.codex/skills/` with a `SKILL.md` index file.\n\n\n\n## GSD Workflow Enforcement\n\nBefore using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.\n\nUse these entry points:\n- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks\n- `/gsd-debug` for investigation and bug fixing\n- `/gsd-execute-phase` for planned phase work\n\nDo not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.\n\n\n\n\n\n## Developer Profile\n\n> Profile not yet configured. Run `/gsd-profile-user` to generate your developer profile.\n> This section is managed by `generate-claude-profile` -- do not edit manually.\n\n","isInternal":false,"tokens":7935,"sizeBytes":28775}],"systemPromptSnippet":"\n\n# reconFTW Architecture\n\n## Overview\n\nreconFTW is a modular reconnaissance framework. The codebase is organized as follows:\n\n```\nreconftw/\n├── reconftw.sh # Main entry point, argument parsing, orchestration\n├── reconftw.cfg # Main configuration file (all settings)\n├── modules/ # Phase-specific modules (what to do)\n│ ├── utils.sh # Utility functions (cleanup, sanitization, caching)\n│ ├── core.sh # Framework core (logging, tools check, lifecycle)\n│ ├── osint.sh # OSINT gathering functions\n│ ├── subdomains.sh # Subdomain enumeration\n│ ├── web.sh # Web analysis and crawling\n│ ├── vulns.sh # Vulnerability scanning\n│ ├── axiom.sh # Ax (formerly Axiom) distributed scanning\n│ └── modes.sh # Scan mode orchestration\n├── lib/ # Pure utility libraries (how to do it)\n│ ├── config.sh # [FUTURE] Modular config loader\n│ └── validation.sh # Input validation functions\n└── tests/ # Test suite\n ├── unit/ # Unit tests (bats)\n ├── security/ # Security tests\n └── mocks/ # Tool mocks for testing\n```\n\n## Module Load Order\n\nModules are sourced in this specific order (defined in `reconftw.sh`):\n\n1. `modules/utils.sh` - Base utilities (must be first)\n2. `modules/core.sh` - Framework infrastructure\n3. `modules/osint.sh` - OSINT functions\n4. `modules/subdomains.sh` - Subdomain enumeration\n5. `modules/web.sh` - Web analysis\n6. `modules/vulns.sh` - Vulnerability scanning\n7. `modules/axiom.sh` - Distributed scanning\n8. `modules/modes.sh` - Mode orchestration (must be last)\n\n## Data Flow\n\n```\nTarget Input → Sanitization → Phase Execution → Output Files\n ↓ ↓ ↓ ↓\n --domain sanitize_domain start_func Recon/$domain/\n --list sanitize_ip [tool calls] subdomains/\n --cidr end_func webs/vulns/\n```\n\n## Key Functions\n\n### Lifecycle Functions (core.sh)\n- `start_func(name, description)` - Begin a function, check if already done\n- `end_func(message, name)` - Complete a function, mark as done\n\n### Utility Functions (utils.sh)\n- `sanitize_domain(domain)` - Sanitize domain input (lowercase, safe chars)\n- `sanitize_ip(ip)` - Sanitize IP/CIDR input\n- `should_run_deep(count, limit)` - Check if DEEP mode should run\n- `deleteOutScoped(scopefile, targetfile)` - Remove out-of-scope entries\n- `run_command(cmd, args...)` - Execute with logging (respects DRY_RUN)\n\n### DNS Resolution Functions (utils.sh)\n\nAll DNS resolution goes through two global wrappers that auto-select between puredns and dnsx based on the network environment:\n\n- `_get_local_ip()` - Get the primary local IP address (cross-platform). Uses `route -n get default` on macOS and `ip -4 route get 1.1.1.1` on Linux to find the correct interface regardless of naming.\n- `_ip_is_public_ipv4(ip)` - Conservative check for publicly routable IPv4. Treats RFC1918, CGNAT (100.64.0.0/10), loopback, link-local, multicast/reserved as non-public.\n- `_is_behind_nat()` - Detect if running behind NAT (or unknown network) by checking the primary local IP. Defaults to NAT-safe behavior if detection fails.\n- `init_dns_resolver()` - Evaluates resolver selection once per run and caches it in `DNS_RESOLVER_SELECTED` when `DNS_RESOLVER=auto`.\n- `_select_dns_resolver()` - Select DNS resolver based on `DNS_RESOLVER` config. `puredns`/`dnsx` force the resolver; `auto` uses the cached selection from `init_dns_resolver()` (with a safe fallback if unset).\n- `_resolve_domains(input_file, output_file)` - Resolve a list of domains. Uses puredns (massdns + wildcard filter) on VPS or dnsx (Go net resolver + basic wildcard threshold) behind NAT.\n- `_bruteforce_domains(wordlist, target_domain, output_file)` - Bruteforce subdomains. Equivalent to `puredns bruteforce wordlist domain` or `dnsx -d domain -w wordlist`.\n\n**Why two resolvers:** puredns uses massdns which sends raw UDP packets to thousands of resolvers simultaneously. This floods the NAT table on consumer routers and can kill connectivity. dnsx uses Go's standard net resolver through the OS networking stack, which is NAT-friendly but lacks puredns's advanced wildcard detection.\n\n### Permutation Functions (subdomains.sh)\n\n- `_select_permutations_wordlist(source_file)` - Choose wordlist based on `PERMUTATIONS_WORDLIST_MODE` and subdomain count. In `auto` mode: full list (849 words) for <=100 subs or DEEP mode, short list (162 words) for >100 subs.\n- `_run_permutation_engine(source_file)` - Run gotator with the auto-selected wordlist.\n- `_generate_permutation_candidates(source_file, output_file)` - Generate permutation candidates with byte-size limit.\n\n### Validation Functions (lib/validation.sh)\n- `validate_domain(domain)` - Validate domain format\n- `validate_ipv4(ip)` - Validate IPv4 address\n- `validate_boolean(value)` - Validate boolean string\n\n## Global Variables\n\n### Required (set before module load)\n- `$SCRIPTPATH` - Path to reconftw installation\n- `$domain` - Target domain\n- `$dir` - Output directory for current scan\n\n### Configuration (from reconftw.cfg)\n- `$DEEP` - Enable deep/thorough scanning\n- `$DEEP_LIMIT` - Threshold for auto-deep mode\n- `$AXIOM` - Enable distributed scanning\n- `$*_THREADS` - Thread counts per tool\n- `$DNS_RESOLVER` - DNS resolver selection (`auto`|`puredns`|`dnsx`)\n- `$DNS_RESOLVER_SELECTED` - Cached resolver selection for this run (set by `init_dns_resolver()` when `DNS_RESOLVER=auto`)\n- `$PERMUTATIONS_WORDLIST_MODE` - Wordlist selection (`auto`|`full`|`short`)\n- `$PERMUTATIONS_SHORT_THRESHOLD` - Subdomain count threshold for short wordlist (default: 100)\n\n## Error Codes\n\nDefined in `lib/validation.sh`:\n- `E_SUCCESS=0` - Success\n- `E_GENERAL=1` - General error\n- `E_MISSING_DEP=2` - Missing dependency\n- `E_INVALID_INPUT=3` - Invalid user input\n- `E_CONFIG=8` - Configuration error\n\n## Testing\n\n```bash\nmake test # Unit tests only\nmake test-security # Security tests only\nmake test-all # All tests\nmake lint # Shellcheck errors\nmake lint-fix # Shellcheck warnings with context\n```\n\n\n\n## Project\n\n**reconFTW**\n\nreconFTW is a comprehensive bash-based reconnaissance automation framework used by bug bounty hunters, penetration testers, and security researchers. It orchestrates 70+ external security tools (Go, Python, Rust) across subdomain enumeration, web probing, OSINT, and vulnerability scanning, producing structured per-target output trees with optional Axiom distributed execution, AI reporting, monitor/incremental mode, and Slack/Telegram/Discord notifications.\n\n**Core Value:** Run one command, get a complete recon picture of a target — passive, active, and vulnerability layers — with resumable checkpoints, structured output, and zero-touch tool orchestration.\n\n### Constraints\n\n- **Tech stack**: Bash 4.3+ — Required for `wait -n`, `mapfile`, associative arrays. macOS users must have Homebrew bash; auto re-exec is best-effort.\n- **External tools**: 70+ runtime dependencies — Most install via `go install @latest` (no version pinning), which is convenient but a known supply-chain risk.\n- **Single process**: All modules sourced into one shell — No subshell isolation between modules; all state shared via globals. Workflow functions must save/restore globals they override (see `passive()` pattern).\n- **Resume semantics**: Checkpoint files are touch-once at `end_func` — Interrupted functions re-run from scratch on next invocation; partial outputs are not detected.\n- **Single-operator**: Designed for one user per target run — No locking, no multi-user state, no concurrent runs against the same target dir.\n- **Output stability**: `Recon//` tree is a public contract — Subdirectory names and filenames are consumed by downstream pipelines, scripts, and parsers; renames are breaking changes.\n- **macOS compatibility**: GNU coreutils + GNU sed + GNU getopt required — System BSD versions are not supported.\n- **CI budget**: Integration-full is weekly cron — Unit + smoke are per-push; adding heavy integration tests must respect this split.\n\n\n\n## Technology Stack\n\n## Languages\n- Bash 4+ - All core framework logic (`reconftw.sh`, `modules/*.sh`, `lib/*.sh`, `install.sh`)\n- Python 3.7+ - Python-backed tools (each runs in isolated `uv venv`): `dorks_hunter`, `CMSeeK`, `EmailHarvester`, `Spoofy`, `SSTImap`, `gato`, `regulator`, `reconftw_ai`, `getjswords.py`\n- Go (latest, min ~1.21) - Primary binary language for ~55 security tools installed via `go install @latest`\n## Runtime\n- Linux (Debian/Ubuntu/RHEL/Arch) or macOS (Apple Silicon / Intel)\n- Docker: Ubuntu 24.04 base image (`Docker/Dockerfile`)\n- ARM64/aarch64, ARMv6l/v7l, and x86_64 all supported for Go binary installs\n- `getopt` (GNU getopt required on macOS via `brew install gnu-getopt`)\n- `nproc` / `sysctl -n hw.ncpu` for CPU core auto-detection\n- `timeout` / `gtimeout` (macOS Homebrew `coreutils`)\n- `gnu-sed` (macOS requires `brew install gnu-sed`)\n- `gnu-coreutils` (macOS requires `brew install coreutils`)\n- Go tools: `go install` (`GOPATH=$HOME/go`, `GOROOT=/usr/local/go`)\n- Python tools: `uv tool install` from GitHub (most) or PyPI (`fray`)\n- Python repo venvs: `uv venv venv && uv pip install -r requirements.txt`\n- Lockfile: None (always installs `@latest`)\n## Frameworks\n- No framework — pure Bash with sourced module files\n- Module loading order: `lib/validation.sh` → `lib/common.sh` → `lib/ui.sh` → `lib/parallel.sh` → `modules/utils.sh` → `modules/core.sh` → `modules/osint.sh` → `modules/subdomains.sh` → `modules/web.sh` → `modules/vulns.sh` → `modules/axiom.sh` → `modules/modes.sh`\n- bats-core (Bash Automated Testing System)\n- GNU make (`Makefile`) — test, lint, format targets\n- shellcheck (error-level) — `make lint`, pre-commit hook\n- shfmt (4-space indent, `-bn`, `-ci`) — `make fmt`, pre-commit hook\n- pre-commit hooks defined in `.pre-commit-config.yaml`\n- semgrep: `.github/workflows/semgrep.yml` (CI only)\n## Key Dependencies\n### Go Tools (installed via `go install @latest`)\n- `subfinder` (projectdiscovery/subfinder) — passive multi-source subdomain discovery\n- `github-subdomains` (gwen001/github-subdomains) — GitHub-based subdomain search\n- `gitlab-subdomains` (gwen001/gitlab-subdomains) — GitLab-based subdomain search\n- `dnstake` (pwnesia/dnstake) — subdomain takeover detection\n- `puredns` (d3mondev/puredns) — mass DNS resolution with wildcard filtering\n- `dnsx` (projectdiscovery/dnsx) — DNS toolkit\n- `massdns` (blechschmidt/massdns) — via repo clone + build\n- `dsieve` (trickest/dsieve) — subdomain filtering\n- `enumerepo` (trickest/enumerepo) — GitHub org repo enumeration\n- `gotator` (Josue87/gotator) — subdomain permutations\n- `analyticsrelationships` (Josue87/analyticsrelationships) — Google Analytics pivoting\n- `roboxtractor` (Josue87/roboxtractor) — robots.txt extractor\n- `crt` (cemulus/crt) — crt.sh search\n- `asnmap` (projectdiscovery/asnmap) — ASN-to-CIDR mapping\n- `mapcidr` (projectdiscovery/mapcidr) — CIDR manipulation\n- `smap` (s0md3v/smap) — passive Shodan-powered port scan\n- `tlsx` (projectdiscovery/tlsx) — TLS certificate harvesting\n- `hakip2host` (hakluke/hakip2host) — reverse IP lookup\n- `cdncheck` (projectdiscovery/cdncheck) — CDN/WAF IP classification\n- `hakoriginfinder` (hakluke/hakoriginfinder) — origin IP discovery behind CDN\n- `inscope` (tomnomnom/hacks/inscope) — scope filtering\n- `csprecon` (edoardottt/csprecon) — CSP-based subdomain discovery\n- `favirecon` (edoardottt/favirecon) — favicon-based tech recon\n- `httpx` (projectdiscovery/httpx) — multi-probe HTTP toolkit\n- `katana` (projectdiscovery/katana) — web crawler\n- `ffuf` (ffuf/ffuf) — web fuzzer\n- `subjs` (lc/subjs) — JavaScript URL extractor\n- `Gxss` (KathanP19/Gxss) — reflected XSS param finder\n- `jsluice` (BishopFox/jsluice) — JS secret/URL extractor\n- `sourcemapper` (denandz/sourcemapper) — JS source map extractor\n- `mantra` (brosck/mantra) — JS/secret scanner\n- `urlfinder` (projectdiscovery/urlfinder) — URL discovery\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — via uv\n- `nmapurls` (sdcampbell/nmapurls) — URL extraction from Nmap XML\n- `naabu` (projectdiscovery/naabu) — fast port scanner\n- `VhostFinder` (wdahlenburg/VhostFinder) — virtual host discovery\n- `shortscan` (bitquark/shortscan) — IIS short filename scanner\n- `nuclei` (projectdiscovery/nuclei) — template-based scanner\n- `dalfox` (hahwul/dalfox) — XSS scanner\n- `crlfuzz` (dwisiswant0/crlfuzz) — CRLF injection scanner\n- `Web-Cache-Vulnerability-Scanner` (Hackmanit) — web cache poisoning\n- `TInjA` (Hackmanit/TInjA) — SSTI scanner\n- `toxicache` (xhzeem/toxicache) — web cache poisoning\n- `second-order` (mhmdiaa/second-order) — broken link/second-order injection\n- `s3scanner` (sa7mon/s3scanner) — S3/GCS/Azure Blob misconfiguration\n- `misconfig-mapper` (intigriti/misconfig-mapper) — third-party misconfiguration\n- `sj` (BishopFox/sj) — Swagger/OpenAPI analysis\n- `grpcurl` (fullstorydev/grpcurl) — gRPC reflection scanner\n- `nerva` (praetorian-inc/nerva) — service fingerprinting\n- `brutus` (praetorian-inc/brutus) — credential spraying\n- `julius` (praetorian-inc/julius) — LLM endpoint probe\n- `titus` (praetorian-inc/titus) — secrets engine\n- `notify` (projectdiscovery/notify) — multi-channel notifications\n- `interactsh-client` (projectdiscovery/interactsh) — OOB callback server\n- `gf` (tomnomnom/gf) — URL pattern grep\n- `anew` (tomnomnom/anew) — append new lines only\n- `unfurl` (tomnomnom/unfurl) — URL parser\n- `qsreplace` (tomnomnom/qsreplace) — querystring replacer\n- `gitdorks_go` (damit5/gitdorks_go) — GitHub dork search\n- `github-endpoints` (gwen001/github-endpoints) — GitHub endpoint discovery\n- `cent` (xm1k3/cent) — nuclei template manager\n- `trufflehog` (trufflesecurity/trufflehog) — secrets scanner (via `go install`)\n- `brutespray` (x90skysn3k/brutespray) — service credential spraying\n### Python Tools (installed via `uv tool install`)\n- `dnsvalidator` (vortexau/dnsvalidator) — DNS resolver validation\n- `interlace` (pry0cc/interlace) — parallel command runner\n- `wafw00f` (EnableSecurity/wafw00f) — WAF fingerprinting\n- `commix` (commixproject/commix) — command injection scanner\n- `waymore` (xnl-h4ck3r/waymore) — passive URL collection\n- `urless` (xnl-h4ck3r/urless) — URL deduplication\n- `ghauri` (r0oth3x49/ghauri) — SQLi scanner (optional)\n- `xnLinkFinder` (xnl-h4ck3r/xnLinkFinder) — deep link finder\n- `xnldorker` (xnl-h4ck3r/xnldorker) — Google dorker\n- `porch-pirate` (MandConsultingGroup/porch-pirate) — Postman API leaks\n- `p1radup` (iambouali/p1radup) — URL deduplication\n- `subwiz` (hadriansecurity/subwiz) — ML-based subdomain prediction\n- `arjun` (s0md3v/Arjun) — parameter discovery\n- `gqlspection` (doyensec/GQLSpection) — GraphQL deep introspection\n- `postleaksNg` (six2dez/postleaksNG) — Postman public leak search\n- `cewler` (roys/cewler) — web wordlist generator\n- `fray` (dalisecurity/fray) — WAF-aware payload testing (PyPI)\n### Repo-Clone Tools (Python venvs, run via `venv/bin/python3`)\n- `dorks_hunter` (six2dez/dorks_hunter) — Google dork automation\n- `CMSeeK` (Tuhinshubhra/CMSeeK) — CMS fingerprinting\n- `cloud_enum` (initstring/cloud_enum) — AWS/GCP/Azure bucket enumeration\n- `EmailHarvester` (maldevel/EmailHarvester) — email harvesting\n- `SwaggerSpy` (UndeadSec/SwaggerSpy) — Swagger endpoint leak detection\n- `LeakSearch` (JoelGMSec/LeakSearch) — credential leak search\n- `Spoofy` (MattKeeley/Spoofy) — email spoofing check\n- `msftrecon` (Arcanum-Sec/msftrecon) — Microsoft tenant recon\n- `Scopify` (Arcanum-Sec/Scopify) — scope management\n- `regulator` (cramppet/regulator) — regex-based subdomain permutations\n- `SSTImap` (vladko312/SSTImap) — SSTI scanner (alternative engine)\n- `gato` (praetorian-inc/gato) — GitHub Actions audit\n### Repo-Clone Tools (Go build)\n- `ghleaks` (dinosn/ghleaks) — GitHub-wide secret search\n- `nomore403` (devploit/nomore403) — 403 bypass\n- `ffufPostprocessing` (Damian89/ffufPostprocessing) — ffuf result analysis\n- `JSA` (w9w/JSA) — JS analysis\n- `ultimate-nmap-parser` (shifty0g/ultimate-nmap-parser) — Nmap XML parser\n### System-Level Tools (apt/brew/yum)\n- `nmap` — active port scanning\n- `massdns` — DNS resolver (also cloned + built from source)\n- `jq` — JSON processing throughout all modules\n- `exiftool` (perl-Image-ExifTool) — metadata extraction\n- `whois` — domain registration lookup\n- `sqlmap` — SQL injection (system or via repo clone)\n- `testssl.sh` (testssl/testssl.sh) — TLS/SSL misconfiguration testing\n- `medusa` — credential brute-force (system install)\n- `shodan` CLI — installed via `uv tool install shodan`\n### Rust Tools\n- `smugglex` (Cargo) — HTTP request smuggling detection\n- Rustup installed from `https://sh.rustup.rs`\n## Configuration\n- `reconftw.cfg` — sourced after CLI parsing; all feature flags, rate limits, timeouts, wordlist paths, API keys, thread counts\n- `secrets.cfg` (gitignored, auto-sourced) — API keys and tokens separated from main config\n- `secrets.cfg.example` — template showing all supported secret vars\n- Feature flags: `OSINT=true`, `SUBDOMAINS_GENERAL=true`, `VULNS_GENERAL=false`, etc.\n- Rate limits: `HTTPX_RATELIMIT=150`, `NUCLEI_RATELIMIT=150`, `FFUF_RATELIMIT=0`\n- Thread counts: auto-scaled via `AVAILABLE_CORES=$(nproc)` with multipliers per tool\n- Timeouts: per-tool in seconds or minutes (`CMSSCAN_TIMEOUT=3600`, `SUBFINDER_ENUM_TIMEOUT=180`)\n- Wordlist paths: `fuzz_wordlist`, `lfi_wordlist`, `subs_wordlist` etc. under `${WORDLISTS_DIR}`\n- Output: `EXPORT_FORMAT`, `AI_REPORT_TYPE`, `ASSET_STORE`\n- GNU `getopt` long options, parsed in `reconftw.sh` while/case loop\n- All CLI overrides use `CLI_*` pattern and are re-applied after `reconftw.cfg` is sourced\n- Full list from `getopt` call: `domain`, `list`, `recon`, `subdomains`, `passive`, `all`, `web`, `osint`, `zen`, `deep`, `help`, `vps`, `vps-count`, `ai`, `check-tools`, `health-check`, `quick-rescan`, `incremental`, `adaptive-rate`, `dry-run`, `parallel`, `no-parallel`, `monitor`, `monitor-interval`, `monitor-cycles`, `refresh-cache`, `gen-resolvers`, `force`, `export`, `report-only`, `no-report`, `parallel-log`, `quiet`, `verbose`, `no-color`, `log-format`, `show-cache`, `banner`, `no-banner`, `legal`\n- `SHODAN_API_KEY`, `WHOISXML_API`, `PDCP_API_KEY`, `XSS_SERVER`, `COLLAB_SERVER` — preferred over config file\n- `GOROOT`, `GOPATH`, `PATH` — extended by `reconftw.cfg` for Go and Rust binaries\n- `LOGFILE` — per-target log path\n- `config/reconftw_full.cfg` — full-scan preset\n- `config/reconftw_quick.cfg` — quick-scan preset\n- `config/reconftw_stealth.cfg` — low-noise preset\n## Build\n- Default: `go1.23.6` (fetches latest from `https://go.dev/VERSION?m=text`)\n- Installed to `/usr/local/go`; set `install_golang=false` in config to skip\n- Minimum: Python 3.7 (enforced in `install_yum()`)\n- Virtual environments per tool via `uv venv`\n- Root venv at `.venv/` for `getjswords.py` and similar helpers\n## Platform Requirements\n- Bash ≥ 4.3 (for `wait -n` used in `lib/parallel.sh`)\n- Go ≥ 1.21 (tools use SIV module paths like `/v2`, `/v3`)\n- Python ≥ 3.7\n- `uv` package manager\n- Rust / Cargo (for `smugglex`)\n- GNU coreutils, getopt, sed (macOS only via Homebrew)\n- ~5GB free disk space for Go cache, tools, and repos\n- ~1GB RAM minimum (Go compilation)\n- Base image: `ubuntu:24.04`\n- Build arg `INSTALL_AXIOM=true` (default) installs axiom fleet tooling\n- Ports 85-90 exposed (for headless browser tooling)\n- Runs as root (required for raw socket operations by some tools)\n- Health check: `./reconftw.sh --health-check`\n\n\n\n## Conventions\n\n## Shell Settings\n## Source Guard Pattern\n- `lib/common.sh`: `[[ -n \"$_COMMON_SH_LOADED\" ]] && return 0`\n- `lib/parallel.sh`: `[[ -n \"$_PARALLEL_SH_LOADED\" ]] && return 0`\n- `lib/ui.sh`: `[[ -n \"${_UI_SH_LOADED:-}\" ]] && return 0`\n## Function Naming\n- Public module functions: `snake_case` prefixed with module context (`sub_passive`, `sub_crt`, `geo_info`)\n- Private helpers: `_snake_case` prefix (`_print_status`, `_print_error`, `_print_module_start`, `_parallel_emit_job_output`)\n- UI layer: `ui_` prefix (`ui_init`, `ui_header`, `ui_summary`, `ui_batch_end`)\n- Lifecycle wrappers: `start_func` / `end_func` (call these at top/bottom of every recon function)\n- Validation functions: `validate_*` / `sanitize_*` (defined in `lib/validation.sh` and `modules/utils.sh`)\n## Variable Naming\n- Config flags: `SUBPASSIVE`, `SUBCRT`, `PARALLEL_MODE`, `OUTPUT_VERBOSITY`\n- Runtime state: `LOGFILE`, `SCRIPTPATH`, `DIFF`, `DRY_RUN`, `AXIOM`\n- Error codes: `E_SUCCESS=0`, `E_INVALID_DOMAIN=20`, `E_INVALID_IP=21` (readonly, defined in `lib/validation.sh`)\n## CLI Flag Pattern\n## Output / UI Conventions\n- `OUTPUT_VERBOSITY=0` (quiet): only errors/FAIL printed\n- `OUTPUT_VERBOSITY=1` (normal, default): OK/WARN/FAIL/SKIP status lines\n- `OUTPUT_VERBOSITY=2` (verbose): all of the above + INFO messages + start_func messages\n## Function Lifecycle (start_func / end_func)\n- `start_func name desc` — logs to LOGFILE, sets per-function start timestamp, emits INFO at verbosity >= 2\n- `end_func message name [status]` — touches checkpoint file, calculates elapsed time, calls `_print_status`\n- `skip_notification reason` — emits SKIP/CACHE badge; reasons: `\"disabled\"`, `\"mode\"`, `\"processed\"`, `\"processed-visible\"`, `\"noinput\"`\n## File Checkpointing (Resumability)\n- `end_func` creates the checkpoint: `touch \"$called_fn_dir/.${fn}\"`\n- DIFF mode (`DIFF=true`) bypasses checkpoint — forces re-execution\n- Helper `should_run()` in `lib/common.sh` provides a cleaner gate: `if should_run \"FLAG_VAR\"; then`\n## Error Handling\n- `E_SUCCESS=0`, `E_GENERAL=1`, `E_MISSING_DEP=2`, `E_INVALID_INPUT=3`\n## Validation Functions\n| Function | Purpose |\n|----------|---------|\n| `validate_domain()` | RFC domain check + injection character rejection |\n| `validate_ipv4()` | Octet range validation |\n| `validate_integer()` | Numeric range check |\n| `validate_boolean()` | Accepts `true`/`false` only (not `1`/`0`/`yes`/`no`) |\n| `validate_file_readable()` | Exists + readable + is-a-file |\n| `sanitize_interlace_input()` | Removes shell metacharacters from input files (canonical in `lib/validation.sh`) |\n| `sanitize_domain()` | Strips URL components, lowercases, rejects injection (in `modules/utils.sh`) |\n| `is_in_scope_host()` | Anchored hostname scope check (prevents substring false positives) |\n| `filter_in_scope_urls()` | Python3-based URL scope check (scheme, userinfo, host) |\n## Path and CWD Conventions\n## Parallel Execution\n## Import / Sourcing Order\n## Logging\n- All tool output redirected to `$LOGFILE`: `command ... 2>>\"$LOGFILE\" >/dev/null`\n- Structured JSON logging via `log_json level func message [key=val]` (optional, `STRUCTURED_LOGGING=true`)\n- `redact_secrets()` scrubs `REDACT_VARS` and `REGISTERED_SECRETS` from log lines\n- `register_secret \"$value\"` must be called before logging any secret value\n## Comments\n\n\n\n## Architecture\n\n## System Overview\n```text\n```\n## Component Responsibilities\n| Component | Responsibility | File |\n|-----------|----------------|------|\n| Entry point & CLI parser | getopt argument parsing, config sourcing, mode dispatch | `reconftw.sh` |\n| Mode orchestration | `start`/`end`, workflow functions (`recon`, `passive`, `all`, `vulns`, `osint`, `subs_menu`, `webs_menu`, `zen_menu`, `monitor_mode`) | `modules/modes.sh` |\n| Function lifecycle | `start_func`/`end_func`, checkpointing, logging, notifications, reporting, plugins, health check | `modules/core.sh` |\n| Subdomain enumeration | All `sub_*` functions, `subtakeover`, `zonetransfer`, `s3buckets`, `geo_info` | `modules/subdomains.sh` |\n| Web analysis | `webprobe_full`, `screenshot`, `nuclei_check`, `fuzz`, `jschecks`, `urlchecks`, `waf_checks`, and 20+ others | `modules/web.sh` |\n| Vulnerability scanning | `xss`, `ssrf_checks`, `sqli`, `crlf_checks`, `lfi`, `ssti`, `smuggling`, `fuzzparams`, `nuclei_dast`, and others | `modules/vulns.sh` |\n| OSINT collection | `domain_info`, `ip_info`, `emails`, `google_dorks`, `github_leaks`, `github_actions_audit`, `cloud_enum_scan`, etc. | `modules/osint.sh` |\n| Shared utilities | `run_command`, `sed_i`, `deleteOutScoped`, `validate_config`, `cache_*`, `checkpoint_*`, `circuit_breaker_*`, rate-limit adaption | `modules/utils.sh` |\n| Axiom/distributed mode | `axiom_launch`, `axiom_shutdown`, `axiom_selected`, `resolvers_update`, `ipcidr_target` | `modules/axiom.sh` |\n| Parallel execution | `parallel_funcs`, `_throttle_jobs`, job heartbeat, progress live display, log mode output | `lib/parallel.sh` |\n| Input validation/sanitization | `sanitize_domain`, `sanitize_ip`, `validate_domain`, `validate_integer`, `_sanitize_list_entry` | `lib/validation.sh` |\n| Shared file/counter utilities | `ensure_dirs`, `ensure_webs_all`, `safe_backup`, `count_lines`, incident tracking | `lib/common.sh` |\n| UI presentation layer | `_print_status`, `_print_msg`, `_print_section`, `_print_rule`, `ui_header`, `ui_summary`, TTY detection, color management, JSONL output | `lib/ui.sh` |\n| Configuration | All runtime settings (~350 variables); sourced after CLI parse | `reconftw.cfg` |\n## Pattern Overview\n- Single process: `reconftw.sh` sources all libraries and modules at startup; every function lives in the same shell environment\n- No subshell isolation between modules — all state is shared via global variables\n- File-based checkpointing: `called_fn_dir/.funcname` sentinel files prevent re-running completed functions across invocations\n- CLI-over-config: `reconftw.cfg` provides defaults; CLI flags set `CLI_*` variables that are re-applied after config sourcing to guarantee they cannot be overwritten\n- All external tool invocations go through `run_command()` which handles dry-run mode, adaptive rate limiting, axiom dispatch, and debug logging\n## Layers\n- Purpose: Bootstrap, macOS re-exec, module loading, getopt CLI parsing, config sourcing, CLI override re-application, mode dispatch\n- Location: `reconftw.sh`\n- Contains: `normalize_vps_count_args()`, the main `while/case` getopt loop, the config `source` sequence, CLI override if-blocks, the final `case $opt_mode` dispatch\n- Depends on: All libraries (sourced first), all modules (sourced second)\n- Used by: End user / CI\n- Purpose: Reusable utilities with no side effects; loadable independently for tests\n- Location: `lib/validation.sh`, `lib/common.sh`, `lib/ui.sh`, `lib/parallel.sh`\n- Contains: Input sanitization, file helpers, UI/color/progress, parallel job management\n- Depends on: Nothing (source-guarded with `_*_LOADED` pattern)\n- Used by: All modules and reconftw.sh\n- Purpose: Implement all scanning, analysis, and orchestration functions\n- Location: `modules/`\n- Contains: All recon, vuln, OSINT, web, subdomain functions\n- Depends on: Libraries (always loaded first), `reconftw.cfg` variables, external tools on PATH\n- Used by: modes.sh orchestrates all others; reconftw.sh dispatches to modes.sh\n- Purpose: Default runtime values for ~350 flags/paths/limits; can be overridden by `secrets.cfg` and custom config\n- Location: `reconftw.cfg`, optionally `secrets.cfg`, optionally `$CUSTOM_CONFIG`\n- Contains: Module enable/disable flags, tool flags, API key env-var references, paths, parallelism settings, verbosity, Axiom settings\n- Depends on: Nothing\n- Used by: Sourced by `reconftw.sh` between CLI parse and CLI override re-application\n- Purpose: Store per-target findings in a stable directory hierarchy\n- Location: `Recon//` (created at `start()` time by `modules/modes.sh`)\n- Contains: Standard subdirectories listed below\n- Depends on: `start()` in modes.sh creates the directory tree\n## Data Flow\n### Primary Recon Request Path (`-r` / `--recon`)\n### Function Execution Path (every leaf module function)\n### Parallel Execution Path\n### Axiom Distributed Scan Flow\n- Global bash variables throughout (no encapsulation). Config vars, target vars (`domain`, `dir`, `called_fn_dir`, `LOGFILE`), and result counters are all globals\n- `passive()` saves/restores module-enable globals before overriding them (`modules/modes.sh:549-611`)\n## Key Abstractions\n- Purpose: Lifecycle wrapper around every leaf scanning function\n- Examples: Used in every function in `modules/subdomains.sh`, `modules/web.sh`, `modules/vulns.sh`, `modules/osint.sh`\n- Pattern: `start_func \"${FUNCNAME[0]}\" \"description\"` at top; `end_func \"output path\" \"${FUNCNAME[0]}\"` at bottom; creates checkpoint file on end\n- Purpose: Prevent re-running completed functions across multiple invocations of the same target\n- Location: `Recon//.called_fn/.funcname`\n- Pattern: Each function tests `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]`; `end_func` writes the sentinel via `touch \"$called_fn_dir/.${fn}\"`\n- Purpose: Universal external-tool gate for dry-run preview, axiom dispatch, adaptive rate limiting, and debug logging\n- Location: `modules/utils.sh:468`\n- Pattern: All tool calls inside module functions use `run_command ` rather than direct invocation\n- Purpose: Allow modules to be sourced multiple times (test re-sourcing, `--source-only`) without re-executing\n- Pattern: `[[ -n \"$_FOO_LOADED\" ]] && return 0` at top of each lib file (`lib/common.sh:6`, `lib/parallel.sh:6`, `lib/ui.sh:5`, `lib/validation.sh` — validation uses error-code guards instead)\n- Purpose: Run independent module functions concurrently up to `PARALLEL_MAX_JOBS`\n- Pattern: `parallel_funcs N func_a func_b func_c` — each function spawned as a background subshell; used in `recon()`, `osint()`, `vulns()` for independent groups\n- Purpose: Transparent axiom/local fallback wrapper — if axiom fails during a module, retries locally\n- Location: `modules/modes.sh:656`\n- Pattern: All module calls inside `subs_menu`, `webs_menu`, `recon`, `passive` use this wrapper\n## Entry Points\n- Location: `reconftw.sh`\n- Triggers: Direct execution (`./reconftw.sh -d example.com -r`)\n- Responsibilities: Bootstrap, all module loading, CLI parse, config source, mode dispatch\n- Location: `reconftw.sh:123-125`\n- Triggers: `./reconftw.sh --source-only` (used by bats test `setup()` blocks)\n- Responsibilities: Sources all modules without executing any recon\n- Location: `modules/modes.sh:13`\n- Triggers: Called at the top of most workflow functions (`recon`, `subs_menu`, `passive`, `osint`, `zen_menu`)\n- Responsibilities: Create output directory tree, init LOGFILE, init cache/incremental/DNS/plugins, set global `dir` and `called_fn_dir`\n- Location: `modules/modes.sh:286`\n- Triggers: Called at the bottom of most workflow functions\n- Responsibilities: AI report, cleanup, Faraday, screenshot diffs, plugin events, hotlist, `export_reports()`, timing summary\n## Output Directory Structure\n```\n```\n## Verbosity and Output Controls\n- `0` (quiet): Only errors and final summary printed to terminal; banner suppressed\n- `1` (normal, default): Errors + warnings printed; `notification()` info/good suppressed\n- `2` (verbose): All `notification()` calls, PID info, full parallel output, `start_func` messages, `print_timing_summary`\n- `summary`: One badge line per completed parallel job\n- `tail`: Last `PARALLEL_TAIL_LINES` (default 20, doubled on failure) from each job's log\n- `full`: Complete captured stdout from each job\n- `jsonl-strict`: Forces `OUTPUT_VERBOSITY=0`, emits only machine-readable JSONL\n## Architectural Constraints\n- **Threading:** Single-threaded bash with optional background subshells via `parallel_funcs`; `wait -n` (bash 4.3+) used for job throttling in `_throttle_jobs`\n- **Global state:** All config vars, `domain`, `dir`, `called_fn_dir`, `LOGFILE`, `start`, `runtime`, `DIFF`, `AXIOM`, and hundreds of module-enable flags are module-level globals; any sourced function can read or mutate them\n- **Circular imports:** None by design — reconftw.sh sources libs first, then modules in explicit dependency order (`utils.sh` → `core.sh` → `osint.sh` → `subdomains.sh` → `web.sh` → `vulns.sh` → `axiom.sh` → `modes.sh`)\n- **macOS bash version:** reconftw.sh re-execs itself under Homebrew bash ≥ 4 on macOS (system bash is 3.2); `lib/parallel.sh` requires bash 4.3+ for `wait -n`\n- **Working directory:** `start()` calls `cd \"$dir\"` (the per-target output dir) before any module function runs; all relative paths inside modules resolve against the target dir. `reconftw.sh` captures `startdir=${PWD}` before this\n- **No subshell isolation per module:** Modules are sourced functions, not subprocess commands. A `return` inside a module returns from the function; an `exit` would kill the whole shell\n## Anti-Patterns\n### Direct external tool calls without `run_command`\n### Writing checkpoint files manually\n### Skipping the `[[ ! -f \"$called_fn_dir/.${FUNCNAME[0]}\" ]] || [[ $DIFF == true ]]` guard\n### Overriding config globals without save/restore in workflow functions\n## Error Handling\n- ERR trap in `start()` logs function name, line number, and command to `$LOGFILE` and calls `explain_err()` (`modules/modes.sh:140`)\n- Non-zero exit from `parallel_funcs` increments `RECON_OSINT_PARALLEL_FAILURES` and sets `RECON_PARTIAL_RUN=true`\n- `run_module_with_axiom_failover` catches axiom mid-run failures and retries locally\n- Circuit-breaker helpers (`circuit_breaker_is_open`, `circuit_breaker_record_failure`) in `modules/utils.sh:1190` for persistent tool failures\n## Cross-Cutting Concerns\n\n\n\n## Project Skills\n\nNo project skills found. Add skills to any of: `.claude/skills/`, `.agents/skills/`, `.cursor/skills/`, `.github/skills/`, or `.codex/skills/` with a `SKILL.md` index file.\n\n\n\n## GSD Workflow Enforcement\n\nBefore using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.\n\nUse these entry points:\n- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks\n- `/gsd-debug` for investigation and bug fixing\n- `/gsd-execute-phase` for planned phase work\n\nDo not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.\n\n\n\n\n\n## Developer Profile\n\n> Profile not yet configured. Run `/gsd-profile-user` to generate your developer profile.\n> This section is managed by `generate-claude-profile` -- do not edit manually.\n\n"}