{"owner":"sqlmapproject","repo":"sqlmap","hasSkills":true,"totalSkillsCount":1,"totalTokensCount":3852,"categories":["project-spec"],"hasMcp":false,"mcpConfig":null,"found":["doc/ARCHITECTURE.md"],"skills":{"doc/ARCHITECTURE.md":"# sqlmap architecture\n\nA contributor-oriented map of how sqlmap is put together: the major components,\nhow a run flows through them, and where to start looking for a given concern.\n\n> This is a map, not a spec. It describes the durable structure and data flow; for\n> exact signatures, option names, and enumerable lists (tampers, DBMSes, options),\n> the source is authoritative. **When this document disagrees with the code, the code wins.**\n\nsqlmap runs on both Python 2.7 and 3.x; sources are kept pure-ASCII unless a literal\nnon-ASCII byte is unavoidable. Compatibility shims live in `lib/core/compat.py` and\n`thirdparty/six`.\n\n---\n\n## 1. Entry points\n\n| Entry | File | Purpose |\n|-------|------|---------|\n| CLI | `sqlmap.py` -> `main()` | the scanner. Applies runtime patches, parses options, runs a scan. |\n| REST API | `sqlmapapi.py` | `-s` server / `-c` client wrappers around `lib/utils/api.py`. |\n| Library | `import sqlmap` -> `lib/utils/library.py` | `scan()` / `scanFromRequest()` for programmatic callers; like the API, it drives the engine as a subprocess. |\n\n`main()` (sqlmap.py) does, in order: `dirtyPatches()` (monkey-patches stdlib for\nquirks/security - see below), `setPaths()`, `init()` (option parsing + environment\nsetup), then dispatches to `start()` for a normal scan, or to the self-tests\n(`--smoke` / `--vuln-test` / `--api-test`) in `lib/core/testing.py`.\n\n---\n\n## 2. Global state: `conf` and `kb`\n\nAlmost everything hangs off two process-global singletons defined in `lib/core/data.py`,\nboth `AttribDict` (attribute-accessible dicts; missing keys read back as `None`):\n\n- **`conf`** - the resolved user configuration (options + derived settings). What the\n  user asked for.\n- **`kb`** (\"knowledge base\") - mutable runtime state discovered during a run\n  (identified DBMS, injection points, page templates, caches, locks, counters).\n\nThe configuration pipeline (`lib/core/`):\n\n- `parse/cmdline.py` - argparse definition of every CLI option.\n- `core/optiondict.py` - option name -> type map (used for config-file/API coercion).\n- `core/defaults.py` - default values.\n- `core/option.py` - the heavy lifter: `_setConfAttributes()`, `_setKnowledgeBaseAttributes()`,\n  `_setHTTPHandlers()` (installs the global urllib opener incl. keep-alive), DBMS/encoding\n  setup, etc. Merges CLI + config file + defaults into `conf`/`kb`.\n- `core/settings.py` - constants, version, regexes, thresholds. **New constants go here.**\n\nIdentifiers in the codebase are camelCase.\n\n---\n\n## 3. Top-level layout\n\n| Path | Responsibility |\n|------|----------------|\n| `lib/core/` | conf/kb model, common helpers, settings, enums, dump, session, agent, option parsing |\n| `lib/controller/` | the scan orchestrator (`controller.py`), detection checks (`checks.py`), enumeration dispatch (`action.py`), DBMS handler selection (`handler.py`) |\n| `lib/request/` | HTTP layer: `connect.py` (sending), `comparison.py` (the true/false oracle), `inject.py` (value extraction), protocol handlers, response processing |\n| `lib/techniques/` | the exploitation engines: `blind/{inference,multibit}.py`, `error/use.py`, `union/{test,use}.py`, `dns/`, plus one directory per non-SQL family (`ssti/`, `nosql/`, `xpath/`, ...) |\n| `lib/parse/` | parsing of inputs: CLI, config, HTTP request/log files, HTML, sitemap, and the XML payload/boundary loader (`payloads.py`) |\n| `lib/utils/` | feature modules: `api.py` (REST), `hashdb.py` (session), `crawler.py`, `hash.py` (cracking), `har.py`, `brute.py`, `search.py`, ... |\n| `lib/takeover/` | OS-level takeover: shells, file access, UDF, registry, Metasploit, `xp_cmdshell` |\n| `plugins/generic/` | DBMS-agnostic enumeration/fingerprint/filesystem/takeover base classes |\n| `plugins/dbms/<dbms>/` | per-DBMS subclasses + dialect (one dir per supported DBMS) |\n| `tamper/` | payload-mutation scripts (WAF bypass), one `tamper()` per file |\n| `data/xml/` | the data-driven engine: `boundaries.xml`, `payloads/*.xml`, `queries.xml`, `errors.xml` |\n| `data/` (other) | wordlists/common tables/columns (`txt/`), UDFs (`udf/`), stored procs (`procs/`), shells (`shell/`) |\n| `tests/` | stdlib-unittest suite (offline); see section 11 |\n| `thirdparty/` | vendored dependencies (six, bottle, chardet, ...) - no pip at runtime |\n| `extra/` | auxiliary tools (e.g. `vulnserver` used by `--vuln-test`) |\n\n---\n\n## 4. The scan lifecycle (`lib/controller/controller.py: start()`)\n\nFor each target:\n\n1. **Target setup** - `initTargetEnv()` / `setupTargetEnv()` (`lib/core/target.py`):\n   resolve URL/params, open the per-target output dir and session file\n   (`conf.hashDBFile`), and **resume** anything already known (DBMS, injection points,\n   cached values) from the session.\n2. **Connection & profiling** (`lib/controller/checks.py`): `checkConnection()`,\n   `checkWaf()` (fills `kb.identifiedWafs`), `checkStability()` /\n   dynamic-content detection (establishes `kb.pageTemplate`, `kb.matchRatio`).\n3. **Heuristics** - `heuristicCheckSqlInjection()` (cheap error-based hint).\n4. **Detection** - `checkSqlInjection(place, parameter, value)` per parameter, driven by\n   the data engine (section 5). Confirmed points are appended to `kb.injections`.\n5. **Fingerprint & handler** - `lib/controller/handler.py: setHandler()` identifies the\n   back-end DBMS and assigns `conf.dbmsHandler`, the object through which all\n   enumeration is dispatched (section 7).\n6. **Action** - `action()` (`lib/controller/action.py`) routes the requested operation\n   (`--banner`, `--dbs`, `--tables`, `--dump`, `--sql-query`, `--os-shell`, ...) to\n   `conf.dbmsHandler` methods, and feeds results to `conf.dumper`.\n\nIf nothing is injectable, the dead-end advisory (level/risk, technique, `--text-only`,\n`--tamper` - definitive when `kb.identifiedWafs` is set) is raised as\n`SqlmapNotVulnerableException`.\n\n---\n\n## 5. The data-driven detection engine\n\nDetection behavior lives in **data, not code** - `data/xml/`, loaded by\n`lib/parse/payloads.py` (`loadBoundaries()`, `loadPayloads()`):\n\n- **`boundaries.xml`** - injection *boundaries*: prefix/suffix pairs and the\n  clause/where/parameter-type context they apply to (e.g. quote vs. numeric contexts).\n- **`payloads/*.xml`** - the *tests*, one file per technique\n  (`boolean_blind`, `error_based`, `inline_query`, `stacked_queries`, `time_blind`,\n  `union_query`), each with the request template and the comparison/grep logic that\n  decides success.\n\n`getSortedInjectionTests()` (`lib/core/common.py`) orders the candidate tests by the\nidentified/likely DBMS, `--level`, and `--risk`. The **agent** (`lib/core/agent.py`)\nforges the actual payload string - applying boundary prefix/suffix, the `[RANDNUM]`/\n`[DELIMITER]`-style markers, comments, and tamper scripts. Requests go out via\n`lib/request/connect.py`; the **oracle** `lib/request/comparison.py` decides true/false\nby comparing the response against `kb.pageTemplate` (difflib ratio vs. `kb.matchRatio`,\nplus titles/errors/HTTP-code signals).\n\n---\n\n## 6. Exploitation techniques\n\nOnce a parameter is injectable, value extraction is dispatched by\n`lib/request/inject.py: getValue()` to the matching engine in `lib/techniques/`:\n\n| Technique | Engine | Mechanism |\n|-----------|--------|-----------|\n| boolean-based blind | `blind/inference.py: bisection()` | binary-search each character via true/false oracle |\n| multi-bit blind (`--multi-bit`) | `blind/multibit.py` | opt-in; reads several bits per request out of the rows a listing page renders |\n| time-based blind / stacked | `blind/inference.py` (time compare) | same bisection, oracle is a measured delay |\n| error-based | `error/use.py: errorUse()` | parse the value straight out of a provoked DB error |\n| UNION query | `union/{test,use}.py` | column-count detection then `UNION SELECT` extraction |\n| inline query | (inline, via inject) | value embedded in the original query position |\n| DNS exfiltration | `dns/` | `--dns-domain` out-of-band channel |\n\n`bisection()` is the hot loop; it caches the `--charset` table in\n`kb.cache.charsetAsciiTbl` and respects the `kb.disableShiftTable` runaway-guard latch\n(intentional). Multi-threaded extraction is coordinated via `kb.locks` and\n`getCurrentThreadData()` (`lib/core/threads.py`).\n\n**Non-SQL engines.** `--nosql`, `--xpath`, `--ldap`, `--ssti`, `--graphql`, `--hql`, `--sparql`,\n`--odata`, `--xslt`, `--xxe` and `--jwt` do not go through the pipeline above at all. Each one is a\nself-contained scanner in `lib/techniques/<family>/inject.py`, dispatched from `controller.py` and\nregistered once in `NONSQL_TECHNIQUES` (`lib/core/settings.py`) - that tuple is what the rest of the\ncode tests against, so a new family is declared there and nowhere else. The response comparison,\nreflection stripping and blind-bit classification they all need live in `lib/utils/nonsql.py`\ninstead of being copied per family.\n\n---\n\n## 7. DBMS abstraction\n\nEnumeration is DBMS-agnostic at the top and specialized underneath:\n\n- **`plugins/generic/`** - base classes for each concern: `fingerprint.py`,\n  `enumeration.py`, `databases.py`, `entries.py`, `users.py`, `filesystem.py`,\n  `takeover.py`, `syntax.py`, `misc.py`, `search.py`, `custom.py`, `connector.py`\n  (direct DB connection for `-d`).\n- **`plugins/dbms/<dbms>/`** - one directory per supported DBMS, subclassing the generic\n  pieces and supplying dialect specifics.\n- **`data/xml/queries.xml`** - per-DBMS SQL query templates (banner, current user, table\n  enumeration, casting, etc.) keyed by DBMS. The generic code asks for a query by name;\n  the dialect comes from XML. It also carries the per-DBMS `<gadgets>` - side-effecting scalar\n  expressions (e.g. PostgreSQL `dblink_exec`) that `getGadget()` (`lib/request/inject.py`) probes\n  once and caches in `kb.gadget`, so `--sql-query` / `--file-write` / `--os-cmd` still work from an\n  injection point that has no stacked queries.\n\n`conf.dbmsHandler` (set in `handler.py`) is the live object that `action()` calls into.\n\n---\n\n## 8. Output and session\n\n- **Output** - `conf.dumper` is a `Dump` instance (`lib/core/dump.py`): console tables\n  plus per-table file export in CSV / HTML / SQLITE / JSONL (`--dump-format`). Logging\n  is via `logger` (`lib/core/log.py`).\n- **Session / resume** - each target gets a SQLite session file\n  (`<output>/<host>/session.sqlite`). `hashDBWrite()` / `hashDBRetrieve()`\n  (`lib/core/common.py`, backed by `lib/utils/hashdb.py`) cache injection points,\n  fingerprint, and extracted values so a re-run *resumes* instead of re-testing\n  (`--flush-session` discards it; `--fresh-queries` ignores cached query results). A\n  stale-session nudge fires on resume when the file is older than `HASHDB_STALE_DAYS`.\n\n---\n\n## 9. Request layer and tampering\n\n`lib/request/connect.py` (`Connect.getPage`) is the single HTTP chokepoint. Around it:\nprotocol handlers (`httpshandler`, `redirecthandler`, `chunkedhandler`, `rangehandler`,\npersistent connections via `lib/request/keepalive.py`), response processing (`basic.py`), and the\ncomparison oracle (`comparison.py`). Alternative transports hang off the same chokepoint: the\nstandard-library HTTP/2 client (`http2.py`, `--http2`, which also carries the `--timeless` oracle in\n`timeless.py`) and WebSocket targets (`websocket.py`).\n\n**Tamper scripts** (`tamper/`) mutate the payload just before sending to evade WAF/IPS.\nEach file exposes a `tamper(payload, **kwargs)` and a `__priority__`; `--tamper=a,b,c`\nchains them in priority order. They are payload-string transforms only (no engine\ncoupling), which is why they compose freely. When a WAF/IPS is identified, `lib/utils/wafbypass.py`\nranks the plausible candidates and `_autoWafBypass()` (`controller.py`) trials them one at a time -\na candidate is adopted only if re-running the detection through it brings the injection back.\n\n---\n\n## 10. REST API and JSON report\n\n`lib/utils/api.py` runs a Bottle server (`sqlmapapi.py -s`) that drives sqlmap scans as\nsubprocesses and exposes them over HTTP. Key pieces: `DataStore`/`Task` (task registry),\nan IPC SQLite `Database` (the subprocess writes results/logs/errors back through\n`StdDbOut`), and the route handlers (`/task/*`, `/option/*`, `/scan/*`, `/version`, ...).\nThe contract is documented in `sqlmapapi.yaml` (OpenAPI) and `REST-API.md`.\n\n`--report-json` reuses the *same* assembly code (`_assembleData` / `_sanitizeScanData`)\nthat the `/scan/<id>/data` endpoint uses, so the CLI report and the API result can't\ndrift; `RESTAPI_VERSION` is the API contract version (major exposed as integer).\n\n---\n\n## 11. Tests and self-tests\n\nTwo complementary layers:\n\n- **Offline unit/regression suite** (`tests/`) - stdlib `unittest` only (no pytest/pip),\n  green on py2 + py3. `_testutils.py` bootstraps global state and provides the\n  property/fuzz harness (`Rng` - a cross-version-identical PRNG - and `for_all`). Run:\n  `python -B -m unittest discover -s tests -p \"test_*.py\"` (`-B` matters: a cached `.pyc`\n  makes a `getFileType(__file__)` doctest see `binary`).\n- **In-tree self-tests** (`lib/core/testing.py`, hidden switches): `--smoke-test`\n  (doctests + regex sanity over the whole tree), `--vuln-test` (end-to-end scans against\n  the bundled `extra/vulnserver`), `--api-test` (live REST round-trip). The CI workflow\n  (`.github/workflows/tests.yml`) runs all of these.\n\n---\n\n## 12. \"Where do I start for ...?\"\n\n| I want to change... | Start in |\n|---------------------|----------|\n| a CLI option | `lib/parse/cmdline.py` (+ `optiondict.py`, `defaults.py`) |\n| a constant/threshold | `lib/core/settings.py` |\n| how injection is *detected* | `data/xml/boundaries.xml` + `data/xml/payloads/*.xml`, then `lib/controller/checks.py` |\n| how a value is *extracted* | `lib/request/inject.py` + the relevant `lib/techniques/` engine |\n| a non-SQL technique (SSTI, NoSQL, XPath, ...) | `lib/techniques/<family>/inject.py` (+ `NONSQL_TECHNIQUES` in `settings.py`) |\n| the true/false decision | `lib/request/comparison.py` |\n| a per-DBMS query/dialect | `data/xml/queries.xml` + `plugins/dbms/<dbms>/` |\n| enumeration behavior | `plugins/generic/*.py` |\n| dump/output format | `lib/core/dump.py` |\n| a WAF-bypass transform | add a file under `tamper/` |\n| the REST API surface | `lib/utils/api.py` (+ keep `sqlmapapi.yaml` in sync) |\n| session/resume behavior | `lib/utils/hashdb.py` + `hashDB*` in `lib/core/common.py` |\n| a stdlib monkey-patch / security shim | `lib/core/patch.py` |\n"},"files":{"doc/ARCHITECTURE.md":"# sqlmap architecture\n\nA contributor-oriented map of how sqlmap is put together: the major components,\nhow a run flows through them, and where to start looking for a given concern.\n\n> This is a map, not a spec. It describes the durable structure and data flow; for\n> exact signatures, option names, and enumerable lists (tampers, DBMSes, options),\n> the source is authoritative. **When this document disagrees with the code, the code wins.**\n\nsqlmap runs on both Python 2.7 and 3.x; sources are kept pure-ASCII unless a literal\nnon-ASCII byte is unavoidable. Compatibility shims live in `lib/core/compat.py` and\n`thirdparty/six`.\n\n---\n\n## 1. Entry points\n\n| Entry | File | Purpose |\n|-------|------|---------|\n| CLI | `sqlmap.py` -> `main()` | the scanner. Applies runtime patches, parses options, runs a scan. |\n| REST API | `sqlmapapi.py` | `-s` server / `-c` client wrappers around `lib/utils/api.py`. |\n| Library | `import sqlmap` -> `lib/utils/library.py` | `scan()` / `scanFromRequest()` for programmatic callers; like the API, it drives the engine as a subprocess. |\n\n`main()` (sqlmap.py) does, in order: `dirtyPatches()` (monkey-patches stdlib for\nquirks/security - see below), `setPaths()`, `init()` (option parsing + environment\nsetup), then dispatches to `start()` for a normal scan, or to the self-tests\n(`--smoke` / `--vuln-test` / `--api-test`) in `lib/core/testing.py`.\n\n---\n\n## 2. Global state: `conf` and `kb`\n\nAlmost everything hangs off two process-global singletons defined in `lib/core/data.py`,\nboth `AttribDict` (attribute-accessible dicts; missing keys read back as `None`):\n\n- **`conf`** - the resolved user configuration (options + derived settings). What the\n  user asked for.\n- **`kb`** (\"knowledge base\") - mutable runtime state discovered during a run\n  (identified DBMS, injection points, page templates, caches, locks, counters).\n\nThe configuration pipeline (`lib/core/`):\n\n- `parse/cmdline.py` - argparse definition of every CLI option.\n- `core/optiondict.py` - option name -> type map (used for config-file/API coercion).\n- `core/defaults.py` - default values.\n- `core/option.py` - the heavy lifter: `_setConfAttributes()`, `_setKnowledgeBaseAttributes()`,\n  `_setHTTPHandlers()` (installs the global urllib opener incl. keep-alive), DBMS/encoding\n  setup, etc. Merges CLI + config file + defaults into `conf`/`kb`.\n- `core/settings.py` - constants, version, regexes, thresholds. **New constants go here.**\n\nIdentifiers in the codebase are camelCase.\n\n---\n\n## 3. Top-level layout\n\n| Path | Responsibility |\n|------|----------------|\n| `lib/core/` | conf/kb model, common helpers, settings, enums, dump, session, agent, option parsing |\n| `lib/controller/` | the scan orchestrator (`controller.py`), detection checks (`checks.py`), enumeration dispatch (`action.py`), DBMS handler selection (`handler.py`) |\n| `lib/request/` | HTTP layer: `connect.py` (sending), `comparison.py` (the true/false oracle), `inject.py` (value extraction), protocol handlers, response processing |\n| `lib/techniques/` | the exploitation engines: `blind/{inference,multibit}.py`, `error/use.py`, `union/{test,use}.py`, `dns/`, plus one directory per non-SQL family (`ssti/`, `nosql/`, `xpath/`, ...) |\n| `lib/parse/` | parsing of inputs: CLI, config, HTTP request/log files, HTML, sitemap, and the XML payload/boundary loader (`payloads.py`) |\n| `lib/utils/` | feature modules: `api.py` (REST), `hashdb.py` (session), `crawler.py`, `hash.py` (cracking), `har.py`, `brute.py`, `search.py`, ... |\n| `lib/takeover/` | OS-level takeover: shells, file access, UDF, registry, Metasploit, `xp_cmdshell` |\n| `plugins/generic/` | DBMS-agnostic enumeration/fingerprint/filesystem/takeover base classes |\n| `plugins/dbms/<dbms>/` | per-DBMS subclasses + dialect (one dir per supported DBMS) |\n| `tamper/` | payload-mutation scripts (WAF bypass), one `tamper()` per file |\n| `data/xml/` | the data-driven engine: `boundaries.xml`, `payloads/*.xml`, `queries.xml`, `errors.xml` |\n| `data/` (other) | wordlists/common tables/columns (`txt/`), UDFs (`udf/`), stored procs (`procs/`), shells (`shell/`) |\n| `tests/` | stdlib-unittest suite (offline); see section 11 |\n| `thirdparty/` | vendored dependencies (six, bottle, chardet, ...) - no pip at runtime |\n| `extra/` | auxiliary tools (e.g. `vulnserver` used by `--vuln-test`) |\n\n---\n\n## 4. The scan lifecycle (`lib/controller/controller.py: start()`)\n\nFor each target:\n\n1. **Target setup** - `initTargetEnv()` / `setupTargetEnv()` (`lib/core/target.py`):\n   resolve URL/params, open the per-target output dir and session file\n   (`conf.hashDBFile`), and **resume** anything already known (DBMS, injection points,\n   cached values) from the session.\n2. **Connection & profiling** (`lib/controller/checks.py`): `checkConnection()`,\n   `checkWaf()` (fills `kb.identifiedWafs`), `checkStability()` /\n   dynamic-content detection (establishes `kb.pageTemplate`, `kb.matchRatio`).\n3. **Heuristics** - `heuristicCheckSqlInjection()` (cheap error-based hint).\n4. **Detection** - `checkSqlInjection(place, parameter, value)` per parameter, driven by\n   the data engine (section 5). Confirmed points are appended to `kb.injections`.\n5. **Fingerprint & handler** - `lib/controller/handler.py: setHandler()` identifies the\n   back-end DBMS and assigns `conf.dbmsHandler`, the object through which all\n   enumeration is dispatched (section 7).\n6. **Action** - `action()` (`lib/controller/action.py`) routes the requested operation\n   (`--banner`, `--dbs`, `--tables`, `--dump`, `--sql-query`, `--os-shell`, ...) to\n   `conf.dbmsHandler` methods, and feeds results to `conf.dumper`.\n\nIf nothing is injectable, the dead-end advisory (level/risk, technique, `--text-only`,\n`--tamper` - definitive when `kb.identifiedWafs` is set) is raised as\n`SqlmapNotVulnerableException`.\n\n---\n\n## 5. The data-driven detection engine\n\nDetection behavior lives in **data, not code** - `data/xml/`, loaded by\n`lib/parse/payloads.py` (`loadBoundaries()`, `loadPayloads()`):\n\n- **`boundaries.xml`** - injection *boundaries*: prefix/suffix pairs and the\n  clause/where/parameter-type context they apply to (e.g. quote vs. numeric contexts).\n- **`payloads/*.xml`** - the *tests*, one file per technique\n  (`boolean_blind`, `error_based`, `inline_query`, `stacked_queries`, `time_blind`,\n  `union_query`), each with the request template and the comparison/grep logic that\n  decides success.\n\n`getSortedInjectionTests()` (`lib/core/common.py`) orders the candidate tests by the\nidentified/likely DBMS, `--level`, and `--risk`. The **agent** (`lib/core/agent.py`)\nforges the actual payload string - applying boundary prefix/suffix, the `[RANDNUM]`/\n`[DELIMITER]`-style markers, comments, and tamper scripts. Requests go out via\n`lib/request/connect.py`; the **oracle** `lib/request/comparison.py` decides true/false\nby comparing the response against `kb.pageTemplate` (difflib ratio vs. `kb.matchRatio`,\nplus titles/errors/HTTP-code signals).\n\n---\n\n## 6. Exploitation techniques\n\nOnce a parameter is injectable, value extraction is dispatched by\n`lib/request/inject.py: getValue()` to the matching engine in `lib/techniques/`:\n\n| Technique | Engine | Mechanism |\n|-----------|--------|-----------|\n| boolean-based blind | `blind/inference.py: bisection()` | binary-search each character via true/false oracle |\n| multi-bit blind (`--multi-bit`) | `blind/multibit.py` | opt-in; reads several bits per request out of the rows a listing page renders |\n| time-based blind / stacked | `blind/inference.py` (time compare) | same bisection, oracle is a measured delay |\n| error-based | `error/use.py: errorUse()` | parse the value straight out of a provoked DB error |\n| UNION query | `union/{test,use}.py` | column-count detection then `UNION SELECT` extraction |\n| inline query | (inline, via inject) | value embedded in the original query position |\n| DNS exfiltration | `dns/` | `--dns-domain` out-of-band channel |\n\n`bisection()` is the hot loop; it caches the `--charset` table in\n`kb.cache.charsetAsciiTbl` and respects the `kb.disableShiftTable` runaway-guard latch\n(intentional). Multi-threaded extraction is coordinated via `kb.locks` and\n`getCurrentThreadData()` (`lib/core/threads.py`).\n\n**Non-SQL engines.** `--nosql`, `--xpath`, `--ldap`, `--ssti`, `--graphql`, `--hql`, `--sparql`,\n`--odata`, `--xslt`, `--xxe` and `--jwt` do not go through the pipeline above at all. Each one is a\nself-contained scanner in `lib/techniques/<family>/inject.py`, dispatched from `controller.py` and\nregistered once in `NONSQL_TECHNIQUES` (`lib/core/settings.py`) - that tuple is what the rest of the\ncode tests against, so a new family is declared there and nowhere else. The response comparison,\nreflection stripping and blind-bit classification they all need live in `lib/utils/nonsql.py`\ninstead of being copied per family.\n\n---\n\n## 7. DBMS abstraction\n\nEnumeration is DBMS-agnostic at the top and specialized underneath:\n\n- **`plugins/generic/`** - base classes for each concern: `fingerprint.py`,\n  `enumeration.py`, `databases.py`, `entries.py`, `users.py`, `filesystem.py`,\n  `takeover.py`, `syntax.py`, `misc.py`, `search.py`, `custom.py`, `connector.py`\n  (direct DB connection for `-d`).\n- **`plugins/dbms/<dbms>/`** - one directory per supported DBMS, subclassing the generic\n  pieces and supplying dialect specifics.\n- **`data/xml/queries.xml`** - per-DBMS SQL query templates (banner, current user, table\n  enumeration, casting, etc.) keyed by DBMS. The generic code asks for a query by name;\n  the dialect comes from XML. It also carries the per-DBMS `<gadgets>` - side-effecting scalar\n  expressions (e.g. PostgreSQL `dblink_exec`) that `getGadget()` (`lib/request/inject.py`) probes\n  once and caches in `kb.gadget`, so `--sql-query` / `--file-write` / `--os-cmd` still work from an\n  injection point that has no stacked queries.\n\n`conf.dbmsHandler` (set in `handler.py`) is the live object that `action()` calls into.\n\n---\n\n## 8. Output and session\n\n- **Output** - `conf.dumper` is a `Dump` instance (`lib/core/dump.py`): console tables\n  plus per-table file export in CSV / HTML / SQLITE / JSONL (`--dump-format`). Logging\n  is via `logger` (`lib/core/log.py`).\n- **Session / resume** - each target gets a SQLite session file\n  (`<output>/<host>/session.sqlite`). `hashDBWrite()` / `hashDBRetrieve()`\n  (`lib/core/common.py`, backed by `lib/utils/hashdb.py`) cache injection points,\n  fingerprint, and extracted values so a re-run *resumes* instead of re-testing\n  (`--flush-session` discards it; `--fresh-queries` ignores cached query results). A\n  stale-session nudge fires on resume when the file is older than `HASHDB_STALE_DAYS`.\n\n---\n\n## 9. Request layer and tampering\n\n`lib/request/connect.py` (`Connect.getPage`) is the single HTTP chokepoint. Around it:\nprotocol handlers (`httpshandler`, `redirecthandler`, `chunkedhandler`, `rangehandler`,\npersistent connections via `lib/request/keepalive.py`), response processing (`basic.py`), and the\ncomparison oracle (`comparison.py`). Alternative transports hang off the same chokepoint: the\nstandard-library HTTP/2 client (`http2.py`, `--http2`, which also carries the `--timeless` oracle in\n`timeless.py`) and WebSocket targets (`websocket.py`).\n\n**Tamper scripts** (`tamper/`) mutate the payload just before sending to evade WAF/IPS.\nEach file exposes a `tamper(payload, **kwargs)` and a `__priority__`; `--tamper=a,b,c`\nchains them in priority order. They are payload-string transforms only (no engine\ncoupling), which is why they compose freely. When a WAF/IPS is identified, `lib/utils/wafbypass.py`\nranks the plausible candidates and `_autoWafBypass()` (`controller.py`) trials them one at a time -\na candidate is adopted only if re-running the detection through it brings the injection back.\n\n---\n\n## 10. REST API and JSON report\n\n`lib/utils/api.py` runs a Bottle server (`sqlmapapi.py -s`) that drives sqlmap scans as\nsubprocesses and exposes them over HTTP. Key pieces: `DataStore`/`Task` (task registry),\nan IPC SQLite `Database` (the subprocess writes results/logs/errors back through\n`StdDbOut`), and the route handlers (`/task/*`, `/option/*`, `/scan/*`, `/version`, ...).\nThe contract is documented in `sqlmapapi.yaml` (OpenAPI) and `REST-API.md`.\n\n`--report-json` reuses the *same* assembly code (`_assembleData` / `_sanitizeScanData`)\nthat the `/scan/<id>/data` endpoint uses, so the CLI report and the API result can't\ndrift; `RESTAPI_VERSION` is the API contract version (major exposed as integer).\n\n---\n\n## 11. Tests and self-tests\n\nTwo complementary layers:\n\n- **Offline unit/regression suite** (`tests/`) - stdlib `unittest` only (no pytest/pip),\n  green on py2 + py3. `_testutils.py` bootstraps global state and provides the\n  property/fuzz harness (`Rng` - a cross-version-identical PRNG - and `for_all`). Run:\n  `python -B -m unittest discover -s tests -p \"test_*.py\"` (`-B` matters: a cached `.pyc`\n  makes a `getFileType(__file__)` doctest see `binary`).\n- **In-tree self-tests** (`lib/core/testing.py`, hidden switches): `--smoke-test`\n  (doctests + regex sanity over the whole tree), `--vuln-test` (end-to-end scans against\n  the bundled `extra/vulnserver`), `--api-test` (live REST round-trip). The CI workflow\n  (`.github/workflows/tests.yml`) runs all of these.\n\n---\n\n## 12. \"Where do I start for ...?\"\n\n| I want to change... | Start in |\n|---------------------|----------|\n| a CLI option | `lib/parse/cmdline.py` (+ `optiondict.py`, `defaults.py`) |\n| a constant/threshold | `lib/core/settings.py` |\n| how injection is *detected* | `data/xml/boundaries.xml` + `data/xml/payloads/*.xml`, then `lib/controller/checks.py` |\n| how a value is *extracted* | `lib/request/inject.py` + the relevant `lib/techniques/` engine |\n| a non-SQL technique (SSTI, NoSQL, XPath, ...) | `lib/techniques/<family>/inject.py` (+ `NONSQL_TECHNIQUES` in `settings.py`) |\n| the true/false decision | `lib/request/comparison.py` |\n| a per-DBMS query/dialect | `data/xml/queries.xml` + `plugins/dbms/<dbms>/` |\n| enumeration behavior | `plugins/generic/*.py` |\n| dump/output format | `lib/core/dump.py` |\n| a WAF-bypass transform | add a file under `tamper/` |\n| the REST API surface | `lib/utils/api.py` (+ keep `sqlmapapi.yaml` in sync) |\n| session/resume behavior | `lib/utils/hashdb.py` + `hashDB*` in `lib/core/common.py` |\n| a stdlib monkey-patch / security shim | `lib/core/patch.py` |\n"},"items":[{"name":"ARCHITECTURE.md","path":"doc/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/sqlmapproject/sqlmap/HEAD/doc/ARCHITECTURE.md","title":"doc - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# sqlmap architecture\n\nA contributor-oriented map of how sqlmap is put together: the major components,\nhow a run flows through them, and where to start looking for a given concern.\n\n> This is a map, not a spec. It describes the durable structure and data flow; for\n> exact signatures, option names, and enumerable lists (tampers, DBMSes, options),\n> the source is authoritative. **When this document disagrees with the code, the code wins.**\n\nsqlmap runs on both Python 2.7 and 3.x; sources are kept pure-ASCII unless a literal\nnon-ASCII byte is unavoidable. Compatibility shims live in `lib/core/compat.py` and\n`thirdparty/six`.\n\n---\n\n## 1. Entry points\n\n| Entry | File | Purpose |\n|-------|------|---------|\n| CLI | `sqlmap.py` -> `main()` | the scanner. Applies runtime patches, parses options, runs a scan. |\n| REST API | `sqlmapapi.py` | `-s` server / `-c` client wrappers around `lib/utils/api.py`. |\n| Library | `import sqlmap` -> `lib/utils/library.py` | `scan()` / `scanFromRequest()` for programmatic callers; like the API, it drives the engine as a subprocess. |\n\n`main()` (sqlmap.py) does, in order: `dirtyPatches()` (monkey-patches stdlib for\nquirks/security - see below), `setPaths()`, `init()` (option parsing + environment\nsetup), then dispatches to `start()` for a normal scan, or to the self-tests\n(`--smoke` / `--vuln-test` / `--api-test`) in `lib/core/testing.py`.\n\n---\n\n## 2. Global state: `conf` and `kb`\n\nAlmost everything hangs off two process-global singletons defined in `lib/core/data.py`,\nboth `AttribDict` (attribute-accessible dicts; missing keys read back as `None`):\n\n- **`conf`** - the resolved user configuration (options + derived settings). What the\n  user asked for.\n- **`kb`** (\"knowledge base\") - mutable runtime state discovered during a run\n  (identified DBMS, injection points, page templates, caches, locks, counters).\n\nThe configuration pipeline (`lib/core/`):\n\n- `parse/cmdline.py` - argparse definition of every CLI option.\n- `core/optiondict.py` - option name -> type map (used for config-file/API coercion).\n- `core/defaults.py` - default values.\n- `core/option.py` - the heavy lifter: `_setConfAttributes()`, `_setKnowledgeBaseAttributes()`,\n  `_setHTTPHandlers()` (installs the global urllib opener incl. keep-alive), DBMS/encoding\n  setup, etc. Merges CLI + config file + defaults into `conf`/`kb`.\n- `core/settings.py` - constants, version, regexes, thresholds. **New constants go here.**\n\nIdentifiers in the codebase are camelCase.\n\n---\n\n## 3. Top-level layout\n\n| Path | Responsibility |\n|------|----------------|\n| `lib/core/` | conf/kb model, common helpers, settings, enums, dump, session, agent, option parsing |\n| `lib/controller/` | the scan orchestrator (`controller.py`), detection checks (`checks.py`), enumeration dispatch (`action.py`), DBMS handler selection (`handler.py`) |\n| `lib/request/` | HTTP layer: `connect.py` (sending), `comparison.py` (the true/false oracle), `inject.py` (value extraction), protocol handlers, response processing |\n| `lib/techniques/` | the exploitation engines: `blind/{inference,multibit}.py`, `error/use.py`, `union/{test,use}.py`, `dns/`, plus one directory per non-SQL family (`ssti/`, `nosql/`, `xpath/`, ...) |\n| `lib/parse/` | parsing of inputs: CLI, config, HTTP request/log files, HTML, sitemap, and the XML payload/boundary loader (`payloads.py`) |\n| `lib/utils/` | feature modules: `api.py` (REST), `hashdb.py` (session), `crawler.py`, `hash.py` (cracking), `har.py`, `brute.py`, `search.py`, ... |\n| `lib/takeover/` | OS-level takeover: shells, file access, UDF, registry, Metasploit, `xp_cmdshell` |\n| `plugins/generic/` | DBMS-agnostic enumeration/fingerprint/filesystem/takeover base classes |\n| `plugins/dbms/<dbms>/` | per-DBMS subclasses + dialect (one dir per supported DBMS) |\n| `tamper/` | payload-mutation scripts (WAF bypass), one `tamper()` per file |\n| `data/xml/` | the data-driven engine: `boundaries.xml`, `payloads/*.xml`, `queries.xml`, `errors.xml` |\n| `data/` (other) | wordlists/common tables/columns (`txt/`), UDFs (`udf/`), stored procs (`procs/`), shells (`shell/`) |\n| `tests/` | stdlib-unittest suite (offline); see section 11 |\n| `thirdparty/` | vendored dependencies (six, bottle, chardet, ...) - no pip at runtime |\n| `extra/` | auxiliary tools (e.g. `vulnserver` used by `--vuln-test`) |\n\n---\n\n## 4. The scan lifecycle (`lib/controller/controller.py: start()`)\n\nFor each target:\n\n1. **Target setup** - `initTargetEnv()` / `setupTargetEnv()` (`lib/core/target.py`):\n   resolve URL/params, open the per-target output dir and session file\n   (`conf.hashDBFile`), and **resume** anything already known (DBMS, injection points,\n   cached values) from the session.\n2. **Connection & profiling** (`lib/controller/checks.py`): `checkConnection()`,\n   `checkWaf()` (fills `kb.identifiedWafs`), `checkStability()` /\n   dynamic-content detection (establishes `kb.pageTemplate`, `kb.matchRatio`).\n3. **Heuristics** - `heuristicCheckSqlInjection()` (cheap error-based hint).\n4. **Detection** - `checkSqlInjection(place, parameter, value)` per parameter, driven by\n   the data engine (section 5). Confirmed points are appended to `kb.injections`.\n5. **Fingerprint & handler** - `lib/controller/handler.py: setHandler()` identifies the\n   back-end DBMS and assigns `conf.dbmsHandler`, the object through which all\n   enumeration is dispatched (section 7).\n6. **Action** - `action()` (`lib/controller/action.py`) routes the requested operation\n   (`--banner`, `--dbs`, `--tables`, `--dump`, `--sql-query`, `--os-shell`, ...) to\n   `conf.dbmsHandler` methods, and feeds results to `conf.dumper`.\n\nIf nothing is injectable, the dead-end advisory (level/risk, technique, `--text-only`,\n`--tamper` - definitive when `kb.identifiedWafs` is set) is raised as\n`SqlmapNotVulnerableException`.\n\n---\n\n## 5. The data-driven detection engine\n\nDetection behavior lives in **data, not code** - `data/xml/`, loaded by\n`lib/parse/payloads.py` (`loadBoundaries()`, `loadPayloads()`):\n\n- **`boundaries.xml`** - injection *boundaries*: prefix/suffix pairs and the\n  clause/where/parameter-type context they apply to (e.g. quote vs. numeric contexts).\n- **`payloads/*.xml`** - the *tests*, one file per technique\n  (`boolean_blind`, `error_based`, `inline_query`, `stacked_queries`, `time_blind`,\n  `union_query`), each with the request template and the comparison/grep logic that\n  decides success.\n\n`getSortedInjectionTests()` (`lib/core/common.py`) orders the candidate tests by the\nidentified/likely DBMS, `--level`, and `--risk`. The **agent** (`lib/core/agent.py`)\nforges the actual payload string - applying boundary prefix/suffix, the `[RANDNUM]`/\n`[DELIMITER]`-style markers, comments, and tamper scripts. Requests go out via\n`lib/request/connect.py`; the **oracle** `lib/request/comparison.py` decides true/false\nby comparing the response against `kb.pageTemplate` (difflib ratio vs. `kb.matchRatio`,\nplus titles/errors/HTTP-code signals).\n\n---\n\n## 6. Exploitation techniques\n\nOnce a parameter is injectable, value extraction is dispatched by\n`lib/request/inject.py: getValue()` to the matching engine in `lib/techniques/`:\n\n| Technique | Engine | Mechanism |\n|-----------|--------|-----------|\n| boolean-based blind | `blind/inference.py: bisection()` | binary-search each character via true/false oracle |\n| multi-bit blind (`--multi-bit`) | `blind/multibit.py` | opt-in; reads several bits per request out of the rows a listing page renders |\n| time-based blind / stacked | `blind/inference.py` (time compare) | same bisection, oracle is a measured delay |\n| error-based | `error/use.py: errorUse()` | parse the value straight out of a provoked DB error |\n| UNION query | `union/{test,use}.py` | column-count detection then `UNION SELECT` extraction |\n| inline query | (inline, via inject) | value embedded in the original query position |\n| DNS exfiltration | `dns/` | `--dns-domain` out-of-band channel |\n\n`bisection()` is the hot loop; it caches the `--charset` table in\n`kb.cache.charsetAsciiTbl` and respects the `kb.disableShiftTable` runaway-guard latch\n(intentional). Multi-threaded extraction is coordinated via `kb.locks` and\n`getCurrentThreadData()` (`lib/core/threads.py`).\n\n**Non-SQL engines.** `--nosql`, `--xpath`, `--ldap`, `--ssti`, `--graphql`, `--hql`, `--sparql`,\n`--odata`, `--xslt`, `--xxe` and `--jwt` do not go through the pipeline above at all. Each one is a\nself-contained scanner in `lib/techniques/<family>/inject.py`, dispatched from `controller.py` and\nregistered once in `NONSQL_TECHNIQUES` (`lib/core/settings.py`) - that tuple is what the rest of the\ncode tests against, so a new family is declared there and nowhere else. The response comparison,\nreflection stripping and blind-bit classification they all need live in `lib/utils/nonsql.py`\ninstead of being copied per family.\n\n---\n\n## 7. DBMS abstraction\n\nEnumeration is DBMS-agnostic at the top and specialized underneath:\n\n- **`plugins/generic/`** - base classes for each concern: `fingerprint.py`,\n  `enumeration.py`, `databases.py`, `entries.py`, `users.py`, `filesystem.py`,\n  `takeover.py`, `syntax.py`, `misc.py`, `search.py`, `custom.py`, `connector.py`\n  (direct DB connection for `-d`).\n- **`plugins/dbms/<dbms>/`** - one directory per supported DBMS, subclassing the generic\n  pieces and supplying dialect specifics.\n- **`data/xml/queries.xml`** - per-DBMS SQL query templates (banner, current user, table\n  enumeration, casting, etc.) keyed by DBMS. The generic code asks for a query by name;\n  the dialect comes from XML. It also carries the per-DBMS `<gadgets>` - side-effecting scalar\n  expressions (e.g. PostgreSQL `dblink_exec`) that `getGadget()` (`lib/request/inject.py`) probes\n  once and caches in `kb.gadget`, so `--sql-query` / `--file-write` / `--os-cmd` still work from an\n  injection point that has no stacked queries.\n\n`conf.dbmsHandler` (set in `handler.py`) is the live object that `action()` calls into.\n\n---\n\n## 8. Output and session\n\n- **Output** - `conf.dumper` is a `Dump` instance (`lib/core/dump.py`): console tables\n  plus per-table file export in CSV / HTML / SQLITE / JSONL (`--dump-format`). Logging\n  is via `logger` (`lib/core/log.py`).\n- **Session / resume** - each target gets a SQLite session file\n  (`<output>/<host>/session.sqlite`). `hashDBWrite()` / `hashDBRetrieve()`\n  (`lib/core/common.py`, backed by `lib/utils/hashdb.py`) cache injection points,\n  fingerprint, and extracted values so a re-run *resumes* instead of re-testing\n  (`--flush-session` discards it; `--fresh-queries` ignores cached query results). A\n  stale-session nudge fires on resume when the file is older than `HASHDB_STALE_DAYS`.\n\n---\n\n## 9. Request layer and tampering\n\n`lib/request/connect.py` (`Connect.getPage`) is the single HTTP chokepoint. Around it:\nprotocol handlers (`httpshandler`, `redirecthandler`, `chunkedhandler`, `rangehandler`,\npersistent connections via `lib/request/keepalive.py`), response processing (`basic.py`), and the\ncomparison oracle (`comparison.py`). Alternative transports hang off the same chokepoint: the\nstandard-library HTTP/2 client (`http2.py`, `--http2`, which also carries the `--timeless` oracle in\n`timeless.py`) and WebSocket targets (`websocket.py`).\n\n**Tamper scripts** (`tamper/`) mutate the payload just before sending to evade WAF/IPS.\nEach file exposes a `tamper(payload, **kwargs)` and a `__priority__`; `--tamper=a,b,c`\nchains them in priority order. They are payload-string transforms only (no engine\ncoupling), which is why they compose freely. When a WAF/IPS is identified, `lib/utils/wafbypass.py`\nranks the plausible candidates and `_autoWafBypass()` (`controller.py`) trials them one at a time -\na candidate is adopted only if re-running the detection through it brings the injection back.\n\n---\n\n## 10. REST API and JSON report\n\n`lib/utils/api.py` runs a Bottle server (`sqlmapapi.py -s`) that drives sqlmap scans as\nsubprocesses and exposes them over HTTP. Key pieces: `DataStore`/`Task` (task registry),\nan IPC SQLite `Database` (the subprocess writes results/logs/errors back through\n`StdDbOut`), and the route handlers (`/task/*`, `/option/*`, `/scan/*`, `/version`, ...).\nThe contract is documented in `sqlmapapi.yaml` (OpenAPI) and `REST-API.md`.\n\n`--report-json` reuses the *same* assembly code (`_assembleData` / `_sanitizeScanData`)\nthat the `/scan/<id>/data` endpoint uses, so the CLI report and the API result can't\ndrift; `RESTAPI_VERSION` is the API contract version (major exposed as integer).\n\n---\n\n## 11. Tests and self-tests\n\nTwo complementary layers:\n\n- **Offline unit/regression suite** (`tests/`) - stdlib `unittest` only (no pytest/pip),\n  green on py2 + py3. `_testutils.py` bootstraps global state and provides the\n  property/fuzz harness (`Rng` - a cross-version-identical PRNG - and `for_all`). Run:\n  `python -B -m unittest discover -s tests -p \"test_*.py\"` (`-B` matters: a cached `.pyc`\n  makes a `getFileType(__file__)` doctest see `binary`).\n- **In-tree self-tests** (`lib/core/testing.py`, hidden switches): `--smoke-test`\n  (doctests + regex sanity over the whole tree), `--vuln-test` (end-to-end scans against\n  the bundled `extra/vulnserver`), `--api-test` (live REST round-trip). The CI workflow\n  (`.github/workflows/tests.yml`) runs all of these.\n\n---\n\n## 12. \"Where do I start for ...?\"\n\n| I want to change... | Start in |\n|---------------------|----------|\n| a CLI option | `lib/parse/cmdline.py` (+ `optiondict.py`, `defaults.py`) |\n| a constant/threshold | `lib/core/settings.py` |\n| how injection is *detected* | `data/xml/boundaries.xml` + `data/xml/payloads/*.xml`, then `lib/controller/checks.py` |\n| how a value is *extracted* | `lib/request/inject.py` + the relevant `lib/techniques/` engine |\n| a non-SQL technique (SSTI, NoSQL, XPath, ...) | `lib/techniques/<family>/inject.py` (+ `NONSQL_TECHNIQUES` in `settings.py`) |\n| the true/false decision | `lib/request/comparison.py` |\n| a per-DBMS query/dialect | `data/xml/queries.xml` + `plugins/dbms/<dbms>/` |\n| enumeration behavior | `plugins/generic/*.py` |\n| dump/output format | `lib/core/dump.py` |\n| a WAF-bypass transform | add a file under `tamper/` |\n| the REST API surface | `lib/utils/api.py` (+ keep `sqlmapapi.yaml` in sync) |\n| session/resume behavior | `lib/utils/hashdb.py` + `hashDB*` in `lib/core/common.py` |\n| a stdlib monkey-patch / security shim | `lib/core/patch.py` |\n","isInternal":false,"tokens":3852,"sizeBytes":14367}],"systemPromptSnippet":"<yakaai_skills repo=\"sqlmapproject/sqlmap\">\n<!-- File: doc/ARCHITECTURE.md (Tokens: ~3852 | Category: project-spec) -->\n# sqlmap architecture\n\nA contributor-oriented map of how sqlmap is put together: the major components,\nhow a run flows through them, and where to start looking for a given concern.\n\n> This is a map, not a spec. It describes the durable structure and data flow; for\n> exact signatures, option names, and enumerable lists (tampers, DBMSes, options),\n> the source is authoritative. **When this document disagrees with the code, the code wins.**\n\nsqlmap runs on both Python 2.7 and 3.x; sources are kept pure-ASCII unless a literal\nnon-ASCII byte is unavoidable. Compatibility shims live in `lib/core/compat.py` and\n`thirdparty/six`.\n\n---\n\n## 1. Entry points\n\n| Entry | File | Purpose |\n|-------|------|---------|\n| CLI | `sqlmap.py` -> `main()` | the scanner. Applies runtime patches, parses options, runs a scan. |\n| REST API | `sqlmapapi.py` | `-s` server / `-c` client wrappers around `lib/utils/api.py`. |\n| Library | `import sqlmap` -> `lib/utils/library.py` | `scan()` / `scanFromRequest()` for programmatic callers; like the API, it drives the engine as a subprocess. |\n\n`main()` (sqlmap.py) does, in order: `dirtyPatches()` (monkey-patches stdlib for\nquirks/security - see below), `setPaths()`, `init()` (option parsing + environment\nsetup), then dispatches to `start()` for a normal scan, or to the self-tests\n(`--smoke` / `--vuln-test` / `--api-test`) in `lib/core/testing.py`.\n\n---\n\n## 2. Global state: `conf` and `kb`\n\nAlmost everything hangs off two process-global singletons defined in `lib/core/data.py`,\nboth `AttribDict` (attribute-accessible dicts; missing keys read back as `None`):\n\n- **`conf`** - the resolved user configuration (options + derived settings). What the\n  user asked for.\n- **`kb`** (\"knowledge base\") - mutable runtime state discovered during a run\n  (identified DBMS, injection points, page templates, caches, locks, counters).\n\nThe configuration pipeline (`lib/core/`):\n\n- `parse/cmdline.py` - argparse definition of every CLI option.\n- `core/optiondict.py` - option name -> type map (used for config-file/API coercion).\n- `core/defaults.py` - default values.\n- `core/option.py` - the heavy lifter: `_setConfAttributes()`, `_setKnowledgeBaseAttributes()`,\n  `_setHTTPHandlers()` (installs the global urllib opener incl. keep-alive), DBMS/encoding\n  setup, etc. Merges CLI + config file + defaults into `conf`/`kb`.\n- `core/settings.py` - constants, version, regexes, thresholds. **New constants go here.**\n\nIdentifiers in the codebase are camelCase.\n\n---\n\n## 3. Top-level layout\n\n| Path | Responsibility |\n|------|----------------|\n| `lib/core/` | conf/kb model, common helpers, settings, enums, dump, session, agent, option parsing |\n| `lib/controller/` | the scan orchestrator (`controller.py`), detection checks (`checks.py`), enumeration dispatch (`action.py`), DBMS handler selection (`handler.py`) |\n| `lib/request/` | HTTP layer: `connect.py` (sending), `comparison.py` (the true/false oracle), `inject.py` (value extraction), protocol handlers, response processing |\n| `lib/techniques/` | the exploitation engines: `blind/{inference,multibit}.py`, `error/use.py`, `union/{test,use}.py`, `dns/`, plus one directory per non-SQL family (`ssti/`, `nosql/`, `xpath/`, ...) |\n| `lib/parse/` | parsing of inputs: CLI, config, HTTP request/log files, HTML, sitemap, and the XML payload/boundary loader (`payloads.py`) |\n| `lib/utils/` | feature modules: `api.py` (REST), `hashdb.py` (session), `crawler.py`, `hash.py` (cracking), `har.py`, `brute.py`, `search.py`, ... |\n| `lib/takeover/` | OS-level takeover: shells, file access, UDF, registry, Metasploit, `xp_cmdshell` |\n| `plugins/generic/` | DBMS-agnostic enumeration/fingerprint/filesystem/takeover base classes |\n| `plugins/dbms/<dbms>/` | per-DBMS subclasses + dialect (one dir per supported DBMS) |\n| `tamper/` | payload-mutation scripts (WAF bypass), one `tamper()` per file |\n| `data/xml/` | the data-driven engine: `boundaries.xml`, `payloads/*.xml`, `queries.xml`, `errors.xml` |\n| `data/` (other) | wordlists/common tables/columns (`txt/`), UDFs (`udf/`), stored procs (`procs/`), shells (`shell/`) |\n| `tests/` | stdlib-unittest suite (offline); see section 11 |\n| `thirdparty/` | vendored dependencies (six, bottle, chardet, ...) - no pip at runtime |\n| `extra/` | auxiliary tools (e.g. `vulnserver` used by `--vuln-test`) |\n\n---\n\n## 4. The scan lifecycle (`lib/controller/controller.py: start()`)\n\nFor each target:\n\n1. **Target setup** - `initTargetEnv()` / `setupTargetEnv()` (`lib/core/target.py`):\n   resolve URL/params, open the per-target output dir and session file\n   (`conf.hashDBFile`), and **resume** anything already known (DBMS, injection points,\n   cached values) from the session.\n2. **Connection & profiling** (`lib/controller/checks.py`): `checkConnection()`,\n   `checkWaf()` (fills `kb.identifiedWafs`), `checkStability()` /\n   dynamic-content detection (establishes `kb.pageTemplate`, `kb.matchRatio`).\n3. **Heuristics** - `heuristicCheckSqlInjection()` (cheap error-based hint).\n4. **Detection** - `checkSqlInjection(place, parameter, value)` per parameter, driven by\n   the data engine (section 5). Confirmed points are appended to `kb.injections`.\n5. **Fingerprint & handler** - `lib/controller/handler.py: setHandler()` identifies the\n   back-end DBMS and assigns `conf.dbmsHandler`, the object through which all\n   enumeration is dispatched (section 7).\n6. **Action** - `action()` (`lib/controller/action.py`) routes the requested operation\n   (`--banner`, `--dbs`, `--tables`, `--dump`, `--sql-query`, `--os-shell`, ...) to\n   `conf.dbmsHandler` methods, and feeds results to `conf.dumper`.\n\nIf nothing is injectable, the dead-end advisory (level/risk, technique, `--text-only`,\n`--tamper` - definitive when `kb.identifiedWafs` is set) is raised as\n`SqlmapNotVulnerableException`.\n\n---\n\n## 5. The data-driven detection engine\n\nDetection behavior lives in **data, not code** - `data/xml/`, loaded by\n`lib/parse/payloads.py` (`loadBoundaries()`, `loadPayloads()`):\n\n- **`boundaries.xml`** - injection *boundaries*: prefix/suffix pairs and the\n  clause/where/parameter-type context they apply to (e.g. quote vs. numeric contexts).\n- **`payloads/*.xml`** - the *tests*, one file per technique\n  (`boolean_blind`, `error_based`, `inline_query`, `stacked_queries`, `time_blind`,\n  `union_query`), each with the request template and the comparison/grep logic that\n  decides success.\n\n`getSortedInjectionTests()` (`lib/core/common.py`) orders the candidate tests by the\nidentified/likely DBMS, `--level`, and `--risk`. The **agent** (`lib/core/agent.py`)\nforges the actual payload string - applying boundary prefix/suffix, the `[RANDNUM]`/\n`[DELIMITER]`-style markers, comments, and tamper scripts. Requests go out via\n`lib/request/connect.py`; the **oracle** `lib/request/comparison.py` decides true/false\nby comparing the response against `kb.pageTemplate` (difflib ratio vs. `kb.matchRatio`,\nplus titles/errors/HTTP-code signals).\n\n---\n\n## 6. Exploitation techniques\n\nOnce a parameter is injectable, value extraction is dispatched by\n`lib/request/inject.py: getValue()` to the matching engine in `lib/techniques/`:\n\n| Technique | Engine | Mechanism |\n|-----------|--------|-----------|\n| boolean-based blind | `blind/inference.py: bisection()` | binary-search each character via true/false oracle |\n| multi-bit blind (`--multi-bit`) | `blind/multibit.py` | opt-in; reads several bits per request out of the rows a listing page renders |\n| time-based blind / stacked | `blind/inference.py` (time compare) | same bisection, oracle is a measured delay |\n| error-based | `error/use.py: errorUse()` | parse the value straight out of a provoked DB error |\n| UNION query | `union/{test,use}.py` | column-count detection then `UNION SELECT` extraction |\n| inline query | (inline, via inject) | value embedded in the original query position |\n| DNS exfiltration | `dns/` | `--dns-domain` out-of-band channel |\n\n`bisection()` is the hot loop; it caches the `--charset` table in\n`kb.cache.charsetAsciiTbl` and respects the `kb.disableShiftTable` runaway-guard latch\n(intentional). Multi-threaded extraction is coordinated via `kb.locks` and\n`getCurrentThreadData()` (`lib/core/threads.py`).\n\n**Non-SQL engines.** `--nosql`, `--xpath`, `--ldap`, `--ssti`, `--graphql`, `--hql`, `--sparql`,\n`--odata`, `--xslt`, `--xxe` and `--jwt` do not go through the pipeline above at all. Each one is a\nself-contained scanner in `lib/techniques/<family>/inject.py`, dispatched from `controller.py` and\nregistered once in `NONSQL_TECHNIQUES` (`lib/core/settings.py`) - that tuple is what the rest of the\ncode tests against, so a new family is declared there and nowhere else. The response comparison,\nreflection stripping and blind-bit classification they all need live in `lib/utils/nonsql.py`\ninstead of being copied per family.\n\n---\n\n## 7. DBMS abstraction\n\nEnumeration is DBMS-agnostic at the top and specialized underneath:\n\n- **`plugins/generic/`** - base classes for each concern: `fingerprint.py`,\n  `enumeration.py`, `databases.py`, `entries.py`, `users.py`, `filesystem.py`,\n  `takeover.py`, `syntax.py`, `misc.py`, `search.py`, `custom.py`, `connector.py`\n  (direct DB connection for `-d`).\n- **`plugins/dbms/<dbms>/`** - one directory per supported DBMS, subclassing the generic\n  pieces and supplying dialect specifics.\n- **`data/xml/queries.xml`** - per-DBMS SQL query templates (banner, current user, table\n  enumeration, casting, etc.) keyed by DBMS. The generic code asks for a query by name;\n  the dialect comes from XML. It also carries the per-DBMS `<gadgets>` - side-effecting scalar\n  expressions (e.g. PostgreSQL `dblink_exec`) that `getGadget()` (`lib/request/inject.py`) probes\n  once and caches in `kb.gadget`, so `--sql-query` / `--file-write` / `--os-cmd` still work from an\n  injection point that has no stacked queries.\n\n`conf.dbmsHandler` (set in `handler.py`) is the live object that `action()` calls into.\n\n---\n\n## 8. Output and session\n\n- **Output** - `conf.dumper` is a `Dump` instance (`lib/core/dump.py`): console tables\n  plus per-table file export in CSV / HTML / SQLITE / JSONL (`--dump-format`). Logging\n  is via `logger` (`lib/core/log.py`).\n- **Session / resume** - each target gets a SQLite session file\n  (`<output>/<host>/session.sqlite`). `hashDBWrite()` / `hashDBRetrieve()`\n  (`lib/core/common.py`, backed by `lib/utils/hashdb.py`) cache injection points,\n  fingerprint, and extracted values so a re-run *resumes* instead of re-testing\n  (`--flush-session` discards it; `--fresh-queries` ignores cached query results). A\n  stale-session nudge fires on resume when the file is older than `HASHDB_STALE_DAYS`.\n\n---\n\n## 9. Request layer and tampering\n\n`lib/request/connect.py` (`Connect.getPage`) is the single HTTP chokepoint. Around it:\nprotocol handlers (`httpshandler`, `redirecthandler`, `chunkedhandler`, `rangehandler`,\npersistent connections via `lib/request/keepalive.py`), response processing (`basic.py`), and the\ncomparison oracle (`comparison.py`). Alternative transports hang off the same chokepoint: the\nstandard-library HTTP/2 client (`http2.py`, `--http2`, which also carries the `--timeless` oracle in\n`timeless.py`) and WebSocket targets (`websocket.py`).\n\n**Tamper scripts** (`tamper/`) mutate the payload just before sending to evade WAF/IPS.\nEach file exposes a `tamper(payload, **kwargs)` and a `__priority__`; `--tamper=a,b,c`\nchains them in priority order. They are payload-string transforms only (no engine\ncoupling), which is why they compose freely. When a WAF/IPS is identified, `lib/utils/wafbypass.py`\nranks the plausible candidates and `_autoWafBypass()` (`controller.py`) trials them one at a time -\na candidate is adopted only if re-running the detection through it brings the injection back.\n\n---\n\n## 10. REST API and JSON report\n\n`lib/utils/api.py` runs a Bottle server (`sqlmapapi.py -s`) that drives sqlmap scans as\nsubprocesses and exposes them over HTTP. Key pieces: `DataStore`/`Task` (task registry),\nan IPC SQLite `Database` (the subprocess writes results/logs/errors back through\n`StdDbOut`), and the route handlers (`/task/*`, `/option/*`, `/scan/*`, `/version`, ...).\nThe contract is documented in `sqlmapapi.yaml` (OpenAPI) and `REST-API.md`.\n\n`--report-json` reuses the *same* assembly code (`_assembleData` / `_sanitizeScanData`)\nthat the `/scan/<id>/data` endpoint uses, so the CLI report and the API result can't\ndrift; `RESTAPI_VERSION` is the API contract version (major exposed as integer).\n\n---\n\n## 11. Tests and self-tests\n\nTwo complementary layers:\n\n- **Offline unit/regression suite** (`tests/`) - stdlib `unittest` only (no pytest/pip),\n  green on py2 + py3. `_testutils.py` bootstraps global state and provides the\n  property/fuzz harness (`Rng` - a cross-version-identical PRNG - and `for_all`). Run:\n  `python -B -m unittest discover -s tests -p \"test_*.py\"` (`-B` matters: a cached `.pyc`\n  makes a `getFileType(__file__)` doctest see `binary`).\n- **In-tree self-tests** (`lib/core/testing.py`, hidden switches): `--smoke-test`\n  (doctests + regex sanity over the whole tree), `--vuln-test` (end-to-end scans against\n  the bundled `extra/vulnserver`), `--api-test` (live REST round-trip). The CI workflow\n  (`.github/workflows/tests.yml`) runs all of these.\n\n---\n\n## 12. \"Where do I start for ...?\"\n\n| I want to change... | Start in |\n|---------------------|----------|\n| a CLI option | `lib/parse/cmdline.py` (+ `optiondict.py`, `defaults.py`) |\n| a constant/threshold | `lib/core/settings.py` |\n| how injection is *detected* | `data/xml/boundaries.xml` + `data/xml/payloads/*.xml`, then `lib/controller/checks.py` |\n| how a value is *extracted* | `lib/request/inject.py` + the relevant `lib/techniques/` engine |\n| a non-SQL technique (SSTI, NoSQL, XPath, ...) | `lib/techniques/<family>/inject.py` (+ `NONSQL_TECHNIQUES` in `settings.py`) |\n| the true/false decision | `lib/request/comparison.py` |\n| a per-DBMS query/dialect | `data/xml/queries.xml` + `plugins/dbms/<dbms>/` |\n| enumeration behavior | `plugins/generic/*.py` |\n| dump/output format | `lib/core/dump.py` |\n| a WAF-bypass transform | add a file under `tamper/` |\n| the REST API surface | `lib/utils/api.py` (+ keep `sqlmapapi.yaml` in sync) |\n| session/resume behavior | `lib/utils/hashdb.py` + `hashDB*` in `lib/core/common.py` |\n| a stdlib monkey-patch / security shim | `lib/core/patch.py` |\n</yakaai_skills>"}