{"owner":"projectdiscovery","repo":"nuclei","hasSkills":true,"totalSkillsCount":6,"totalTokensCount":7790,"categories":["plugin-manifest","claude-rule","project-spec"],"hasMcp":false,"mcpConfig":null,"found":["pkg/js/devtools/README.md","pkg/js/devtools/scrapefuncs/README.md","pkg/js/devtools/tsgen/README.md","pkg/js/devtools/bindgen/README.md","CLAUDE.md","DESIGN.md"],"skills":{"pkg/js/devtools/README.md":"## devtools\n\ndevtools contains tools and scripts to automate boring tasks related to javascript layer/ packages.\n\n### bindgen\n\n[bindgen](./bindgen/README.md) is a tool that automatically generated bindings for native go packages with 'goja'\n\n\n### scrapefuncs\n\n[scrapefuncs](./scrapefuncs/README.md) is a tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n\n### Generating API Reference (aka static site using javascript files using jsdoc)\n\n```console\njsdoc -R [Homepage.md] -r -d api_reference -t [optional: jsdoc theme to use] generated/js\n```\n\ngenerated static site will be available at `api_reference/` directory and can be verified using simplehttpserver\n\n```console\nsimplehttpserver\n```\n\nand then open `http://localhost:8000/` in browser\n\n\n### Notes\n\nwe currently use [clean-jsdoc-theme](https://www.npmjs.com/package/clean-jsdoc-theme) demo at [sample-jsproto-docs/](https://projectdiscovery.github.io/js-proto-docs/)","pkg/js/devtools/scrapefuncs/README.md":"## scrapefuncs\n\nscrapefuncs is go/ast based tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n### Usage\n\n```console\nUsage of ./scrapefuncs:\n  -dir string\n    \tdirectory to process (default \"pkg/js/global\")\n  -key string\n    \topenai api key\n  -keyfile string\n    \topenai api key file\n  -out string\n    \toutput js file with declarations of all global functions\n```\n\n\n### Example\n\n```console\n$ ./scrapefuncs -keyfile ~/.openai.key                                   \n[+] Scraped 7 functions\n\nName: Rand\nSignatures: \"Rand(n int) []byte\"\nDescription: Rand returns a random byte slice of length n\n\nName: RandInt\nSignatures: \"RandInt() int\"\nDescription: RandInt returns a random int\n\nName: log\nSignatures: \"log(msg string)\"\nSignatures: \"log(msg map[string]interface{})\"\nDescription: log prints given input to stdout with [JS] prefix for debugging purposes \n\nName: getNetworkPort\nSignatures: \"getNetworkPort(port string, defaultPort string) string\"\nDescription: getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n\nName: isPortOpen\nSignatures: \"isPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isPortOpen checks if given TCP port is open on host. timeout is optional and defaults to 5 seconds\n\nName: isUDPPortOpen\nSignatures: \"isUDPPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isUDPPortOpen checks if the given UDP port is open on the host. Timeout is optional and defaults to 5 seconds.\n\nName: ToBytes\nSignatures: \"ToBytes(...interface{}) []byte\"\nDescription: ToBytes converts given input to byte slice\n\nName: ToString\nSignatures: \"ToString(...interface{}) string\"\nDescription: ToString converts given input to string\n\n\n[+] Generating jsdoc for all functions\n\n/**\n * Rand returns a random byte slice of length n\n * Rand(n int) []byte\n * @function\n * @param {number} n - The length of the byte slice.\n */\nfunction Rand(n) {\n    // implemented in go\n};\n\n/**\n * RandInt returns a random int\n * RandInt() int\n * @function\n */\nfunction RandInt() {\n    // implemented in go\n};\n\n/**\n * log prints given input to stdout with [JS] prefix for debugging purposes\n * log(msg string)\n * log(msg map[string]interface{})\n * @function\n * @param {string|Object} msg - The message to print.\n */\nfunction log(msg) {\n    // implemented in go\n};\n\n/**\n * getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n * getNetworkPort(port string, defaultPort string) string\n * @function\n * @param {string} port - The port to check.\n * @param {string} defaultPort - The default port to return if the port is colliding.\n */\nfunction getNetworkPort(port, defaultPort) {\n    // implemented in go\n};\n\n/**\n * isPortOpen checks if given port is open on host. timeout is optional and defaults to 5 seconds\n * isPortOpen(host string, port string, [timeout int]) bool\n * @function\n * @param {string} host - The host to check.\n * @param {string} port - The port to check.\n * @param {number} [timeout=5] - The timeout in seconds.\n */\nfunction isPortOpen(host, port, timeout = 5) {\n    // implemented in go\n};\n\n/**\n * ToBytes converts given input to byte slice\n * ToBytes(...interface{}) []byte\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToBytes(...args) {\n    // implemented in go\n};\n\n/**\n * ToString converts given input to string\n * ToString(...interface{}) string\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToString(...args) {\n    // implemented in go\n};\n```","pkg/js/devtools/tsgen/README.md":"# tsgen\n\ntsgen is devtool to generate dummy typescript code for goja node modules written in golang. These generated typescript code can be compiled to generate .d.ts files to provide intellisense to editors like vscode and documentation for the node modules.","pkg/js/devtools/bindgen/README.md":"## bindgen (aka bindings generator)\n\nbindgen is a tool that automatically generated bindings for native go packages with 'goja'\n\nNative Go packages are available [here](../../libs/)\n\nGenerated Output is available [here](../../generated/)\n\nbindgen generates 3 different types of outputs \n\n- `go` => this directory contains corresponding goja bindings (actual bindings code) ex: [kerberos.go](../../generated/go/libkerberos/kerberos.go)\n- `js` => this is more of a javascript **representation** of all exposed functions and types etc. in javascript ex: [kerberos.js](../../generated/js/libkerberos/kerberos.js) and does not server any functional purpose other than reference\n- `markdown` => autogenerated markdown documentation for each library / package ex: [kerberos.md](../../generated/markdown/libkerberos/kerberos.md)\n\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nNuclei is a modern, high-performance vulnerability scanner built in Go that leverages YAML-based templates for customizable vulnerability detection. It supports multiple protocols (HTTP, DNS, TCP, SSL, WebSocket, WHOIS, JavaScript, Code) and is designed for zero false positives through real-world condition simulation.\n\n## Development Commands\n\n### Building and Testing\n- `make build` - Build the main nuclei binary to ./bin/nuclei\n- `make test` - Run unit tests with race detection\n- `make integration` - Run the native integration suite via `go test -tags=integration ./internal/tests/integration`\n- `make functional` - CI-only functional suite entry point\n- `make vet` - Run go vet for code analysis\n- `make tidy` - Clean up go modules\n\n### Validation and Linting\n- `make template-validate` - Validate nuclei templates using the built binary\n- `go fmt ./...` - Format Go code\n- `go vet ./...` - Static analysis\n\n### Development Tools\n- `make devtools-all` - Build all development tools (bindgen, tsgen, scrapefuncs)\n- `make jsupdate-all` - Update JavaScript bindings and TypeScript definitions\n- `make docs` - Generate documentation\n- `make memogen` - Generate memoization code for JavaScript libraries\n\n### Testing Specific Components\n- Run single test: `go test -v ./pkg/path/to/package -run TestName`\n- Integration tests are in `internal/tests/integration/` and run via `go test -tags=integration ./internal/tests/integration`\n\n## Architecture Overview\n\n### Core Components\n- **cmd/nuclei** - Main CLI entry point with flag parsing and configuration\n- **internal/runner** - Core runner that orchestrates the entire scanning process\n- **pkg/core** - Execution engine with work pools and template clustering\n- **pkg/templates** - Template parsing, compilation, and management\n- **pkg/protocols** - Protocol implementations (HTTP, DNS, Network, etc.)\n- **pkg/operators** - Matching and extraction logic (matchers/extractors)\n- **pkg/catalog** - Template discovery and loading from disk/remote sources\n\n### Protocol Architecture\nEach protocol (HTTP, DNS, Network, etc.) implements:\n- Request interface with Compile(), ExecuteWithResults(), Match(), Extract() methods\n- Operators embedding for matching/extraction functionality\n- Protocol-specific request building and execution logic\n\n### Template System\n- Templates are YAML files defining vulnerability detection logic\n- Compiled into executable requests with operators (matchers/extractors)\n- Support for workflows (multistep template execution)\n- Template clustering optimizes identical requests across multiple templates\n\n### Key Execution Flow\n1. Template loading and compilation via pkg/catalog/loader\n2. Input provider setup for targets\n3. Engine creation with work pools for concurrency\n4. Template execution with result collection via operators\n5. Output writing and reporting integration\n\n### JavaScript Integration\n- Custom JavaScript runtime for code protocol templates\n- Auto-generated bindings in pkg/js/generated/\n- Library implementations in pkg/js/libs/\n- Development tools for binding generation in pkg/js/devtools/\n\n## Template Development\n- Templates located in separate nuclei-templates repository\n- YAML format with info, requests, and operators sections  \n- Support for multiple protocol types in single template\n- Built-in DSL functions for dynamic content generation\n- Template validation available via `make template-validate`\n\n## Key Directories\n- **lib/** - SDK for embedding nuclei as a library\n- **examples/** - Usage examples for different scenarios\n- **internal/tests/integration/** - Native integration suite and owned testdata for harness coverage\n- **internal/tests/functional/** - CI-only native functional comparison suite and testcase assets\n- **pkg/fuzz/** - Fuzzing engine and DAST capabilities\n- **pkg/input/** - Input processing for various formats (Burp, OpenAPI, etc.)\n- **pkg/reporting/** - Result export and issue tracking integrations\nSee SECURITY_CONTEXT.md for this repo's known vulnerabilities and recurring weak spots. Check it before writing or reviewing security-sensitive code.\n","DESIGN.md":"# Nuclei Architecture Document\n\nA brief overview of Nuclei Engine architecture. This document will be kept updated as the engine progresses.\n\n## pkg/templates\n\n### Template\n\nTemplate is the basic unit of input to the engine which describes the requests to be made, matching to be done, data to extract, etc.\n\nThe template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers. \n\nAny attributes etc. required for the template, engine or requests to function are also set here.\n\nWorkflows are also compiled, their templates are loaded and compiled as well. Any validations etc. on the paths provided are also done here.\n\n`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.\n\n### Preprocessors\n\nPreprocessors are also applied here which can do things at template level. They get data of the template which they can alter at will on runtime. This is used in the engine to do random string generation.\n\nCustom processor can be used if they satisfy the following interface.\n\n```go\ntype Preprocessor interface {\n\tProcess(data []byte) []byte\n}\n```\n\n## pkg/model\n\nModel package implements Information structure for Nuclei Templates. `Info` contains all major metadata information for the template. `Classification` structure can also be used to provide additional context to vulnerability data.\n\nIt also specifies a `WorkflowLoader` interface that is used during workflow loading in template compilation stage.\n\n```go\ntype WorkflowLoader interface {\n\tGetTemplatePathsByTags(tags []string) []string\n\tGetTemplatePaths(templatesList []string, noValidate bool) []string\n}\n```\n\n## pkg/protocols\n\nProtocols package implements all the request protocols supported by Nuclei. This includes http, dns, network, headless and file requests as of now. \n\n### Request\n\nIt exposes a `Request` interface that is implemented by all the request protocols supported.\n\n```go\n// Request is an interface implemented any protocol based request generator.\ntype Request interface {\n\tCompile(options *ExecuterOptions) error\n\tRequests() int\n\tGetID() string\n\tMatch(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\tExtract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\tExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback OutputEventCallback) error\n\tMakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent\n\tMakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent\n\tGetCompiledOperators() []*operators.Operators\n}\n```\n\nMany of these methods are similar across protocols while some are very protocol specific. \n\nA brief overview of the methods is provided below -\n\n- **Compile** - Compiles the request with provided options.\n- **Requests** - Returns total requests made.\n- **GetID** - Returns any ID for the request\n- **Match** - Used to perform matching for patterns using matchers\n- **Extract** - Used to perform extraction for patterns using extractors\n- **ExecuteWithResults** - Request execution function for input.\n- **MakeResultEventItem** - Creates a single result event for the intermediate `InternalWrappedEvent` output structure.\n- **MakeResultEvent** - Returns a slice of results based on an `InternalWrappedEvent` internal output event.\n- **GetCompiledOperators** - Returns the compiled operators for the request.\n\n`MakeDefaultResultEvent` function can be used as a default for `MakeResultEvent` function when no protocol-specific features need to be implemented for result generation. \n\nFor reference protocol requests implementations, one can look at the below packages  - \n\n1. [pkg/protocols/http](./pkg/protocols/http)\n2. [pkg/protocols/dns](./pkg/protocols/dns)\n3. [pkg/protocols/network](./pkg/protocols/network)\n\n### Executer\n\nAll these different requests interfaces are converted to an Executer which is also an interface defined in `pkg/protocols` which is used during final execution of the template.\n\n```go\n// Executer is an interface implemented any protocol based request executer.\ntype Executer interface {\n\tCompile() error\n\tRequests() int\n\tExecute(input string) (bool, error)\n\tExecuteWithResults(input string, callback OutputEventCallback) error\n}\n```\n\nThe `ExecuteWithResults` function accepts a callback, which gets provided with results during execution in form of `*output.InternalWrappedEvent` structure.\n\nThe default executer is provided in `pkg/protocols/common/executer` . It takes a list of Requests and relevant `ExecuterOptions` and implements the Executer interface required for template execution. The executer during Template compilation process is created from this package and used as-is.\n\nA different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates`  We have a single HTTP  request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.\n\nFor Workflow execution, a separate RunWorkflow function is used which executes the workflow independently of the template execution.\n\nWith this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.\n\n## internal/runner\n\n### Template loading\n\nThe first process after all CLI specific initialisation is the loading of template/workflow paths that the user wants to run. This is done by the packages described below.\n\n#### pkg/catalog\n\nThis package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template and current user directory.\n\nThe syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.\n\n\n\nNext step is the initialisation of the reporting modules which is handled in `pkg/reporting`.  \n\n#### pkg/reporting\n\nReporting module contains exporters and trackers as well as a module for deduplication and a module for result formatting. \n\nExporters and Trackers are interfaces defined in pkg/reporting.\n\n```go\n// Tracker is an interface implemented by an issue tracker\ntype Tracker interface {\n\tCreateIssue(event *output.ResultEvent) error\n}\n\n// Exporter is an interface implemented by an issue exporter\ntype Exporter interface {\n\tClose() error\n\tExport(event *output.ResultEvent) error\n}\n```\n\nExporters include `Elasticsearch`, `markdown`, `sarif` . Trackers include `GitHub`, `GitLab` and `Jira`.\n\nEach exporter and trackers implement their own configuration in YAML format and are very modular in nature, so adding new ones is easy.\n\n\n\nAfter reading all the inputs from various sources and initialisation other miscellaneous options, the next bit is the output writing which is done using `pkg/output` module.\n\n#### pkg/output\n\nOutput package implements the output writing functionality for Nuclei.\n\nOutput Writer implements the Writer interface which is called each time a result is found for nuclei.\n\n```go\n// Writer is an interface which writes output to somewhere for nuclei events.\ntype Writer interface {\n\tClose()\n\tColorizer() aurora.Aurora\n\tWrite(*ResultEvent) error\n\tRequest(templateID, url, requestType string, err error)\n}\n```\n\nResultEvent structure is passed to the Nuclei Output Writer which contains the entire detail of a found result. Various intermediary types like `InternalWrappedEvent` and `InternalEvent` are used throughout nuclei protocols and matchers to describe results in various stages of execution.\n\n\n\n Interactsh is also initialised if it is not explicitly disabled. \n\n#### pkg/protocols/common/interactsh\n\nInteractsh module is used to provide automatic Out-of-Band vulnerability identification in Nuclei. \n\nIt uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.\n\nPolling for interactions and server registration only starts when a template uses the interactsh module and is executed by nuclei. After that no registration is required for the entire run.\n\n\n\n### RunEnumeration\n\nNext we arrive in the `RunEnumeration` function of the runner.\n\n`HostErrorsCache` is initialised which is used throughout the run of Nuclei enumeration to keep track of errors per host and skip further requests if the errors are greater than the provided threshold. The functionality for the error tracking cache is defined in [hosterrorscache.go](https://github.com/projectdiscovery/nuclei/blob/main/pkg/protocols/common/hosterrorscache/hosterrorscache.go) and is pretty simplistic in nature.\n\nNext the `WorkflowLoader` is initialised which used to load workflows. It exists in `pkg/parsers/workflow_loader.go`\n\nThe loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc. to return compiled `Templates` and `Workflows`. \n\n#### pkg/catalog/loader\n\nFirst the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module.  Next the path filter module is used to remove the excluded template/workflows paths.\n\n`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc. filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.\n\n`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.\n\nClustering module comes in next whose job is to cluster identical HTTP GET requests together (as a lot of the templates perform the same get requests many times, it's a good way to save many requests on large scans with lots of templates). \n\n### pkg/operators\n\nOperators package implements all the matching and extracting logic of Nuclei. \n\n```go\n// Operators contain the operators that can be applied on protocols\ntype Operators struct {\n\tMatchers []*matchers.Matcher\n\tExtractors []*extractors.Extractor\n\tMatchersCondition string\n}\n```\n\nA protocol only needs to embed the `operators.Operators` type shown above, and it can utilise all the matching/extracting functionality of nuclei.\n\n```go\n// MatchFunc performs matching operation for a matcher on model and returns true or false.\ntype MatchFunc func(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\n// ExtractFunc performs extracting operation for an extractor on model and returns true or false.\ntype ExtractFunc func(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\n// Execute executes the operators on data and returns a result structure\nfunc (operators *Operators) Execute(data map[string]interface{}, match MatchFunc, extract ExtractFunc, isDebug bool) (*Result, bool) \n```\n\nThe core of this process is the Execute function which takes an input dictionary as well as a Match and Extract function and return a `Result` structure which is used later during nuclei execution to check for results.\n\n```go\n// Result is a result structure created from operators running on data.\ntype Result struct {\n\tMatched bool\n\tExtracted bool\n\tMatches map[string][]string\n\tExtracts map[string][]string\n\tOutputExtracts []string\n\tDynamicValues map[string]interface{}\n\tPayloadValues map[string]interface{}\n}\n```\n\nThe internal logics for matching and extracting for things like words, regexes, jq, paths, etc. is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.\n\n\n### Template Execution\n\n`pkg/core` provides the engine mechanism which runs the templates/workflows on inputs. It exposes an `Execute` function which does the task of execution while also doing template clustering. The clustering can also be disabled optionally by the user.\n \nAn example of using the core engine is provided below.\n\n```go\nengine := core.New(r.options)\nengine.SetExecuterOptions(executerOpts)\nresults := engine.ExecuteWithOpts(finalTemplates, r.hmapInputProvider, true)\n```\n\n### Adding a New Protocol\n\nProtocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc. are implemented in form of protocol requests.\n\nA protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.\n\nThe code for the websocket protocol is contained in `pkg/protocols/others/websocket`. \n\nBelow a high level skeleton of the websocket implementation is provided with all the important parts present.\n\n```go\npackage websocket\n\n// Request is a request for the Websocket protocol\ntype Request struct {\n\t// Operators for the current request go here.\n\toperators.Operators `yaml:\",inline,omitempty\"`\n\tCompiledOperators   *operators.Operators `yaml:\"-\"`\n\n\t// description: |\n\t//   Address contains address for the request\n\tAddress string `yaml:\"address,omitempty\" jsonschema:\"title=address for the websocket request,description=Address contains address for the request\"`\n\n    // declarations here\n}\n\n// Compile compiles the request generators preparing any requests possible.\nfunc (r *Request) Compile(options *protocols.ExecuterOptions) error {\n\tr.options = options\n\n    // request compilation here as well as client creation\n \n\tif len(r.Matchers) > 0 || len(r.Extractors) > 0 {\n\t\tcompiled := &r.Operators\n\t\tif err := compiled.Compile(); err != nil {\n\t\t\treturn errors.Wrap(err, \"could not compile operators\")\n\t\t}\n\t\tr.CompiledOperators = compiled\n\t}\n\treturn nil\n}\n\n// Requests returns the total number of requests the rule will perform\nfunc (r *Request) Requests() int {\n\tif r.generator != nil {\n\t\treturn r.generator.NewIterator().Total()\n\t}\n\treturn 1\n}\n\n// GetID returns the ID for the request if any.\nfunc (r *Request) GetID() string {\n\treturn \"\"\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) ExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n    // payloads init here\n\tif err := r.executeRequestWithPayloads(input, hostname, value, previous, callback); err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) executeRequestWithPayloads(input, hostname string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n\theader := http.Header{}\n\n    // make the actual request here after setting all options\n\n\tevent := eventcreator.CreateEventWithAdditionalOptions(r, data, r.options.Options.Debug || r.options.Options.DebugResponse, func(internalWrappedEvent *output.InternalWrappedEvent) {\n\t\tinternalWrappedEvent.OperatorsResult.PayloadValues = payloadValues\n\t})\n\tif r.options.Options.Debug || r.options.Options.DebugResponse {\n\t\tresponseOutput := responseBuilder.String()\n\t\tgologger.Debug().Msgf(\"[%s] Dumped Websocket response for %s\", r.options.TemplateID, input)\n\t\tgologger.Print().Msgf(\"%s\", responsehighlighter.Highlight(event.OperatorsResult, responseOutput, r.options.Options.NoColor))\n\t}\n\n\tcallback(event)\n\treturn nil\n}\n\nfunc (r *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent {\n\tdata := &output.ResultEvent{\n\t\tTemplateID:       types.ToString(r.options.TemplateID),\n\t\tTemplatePath:     types.ToString(r.options.TemplatePath),\n\t\t// ... setting more values for result event\n\t}\n\treturn data\n}\n\n// Match performs matching operation for a matcher on model and returns:\n// true and a list of matched snippets if the matcher type is supports it\n// otherwise false and an empty string slice\nfunc (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string) {\n\treturn protocols.MakeDefaultMatchFunc(data, matcher)\n}\n\n// Extract performs extracting operation for an extractor on model and returns true or false.\nfunc (r *Request) Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{} {\n\treturn protocols.MakeDefaultExtractFunc(data, matcher)\n}\n\n// MakeResultEvent creates a result event from internal wrapped event\nfunc (r *Request) MakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent {\n\treturn protocols.MakeDefaultResultEvent(r, wrapped)\n}\n\n// GetCompiledOperators returns a list of the compiled operators\nfunc (r *Request) GetCompiledOperators() []*operators.Operators {\n\treturn []*operators.Operators{r.CompiledOperators}\n}\n\n// Type returns the type of the protocol request\nfunc (r *Request) Type() templateTypes.ProtocolType {\n\treturn templateTypes.WebsocketProtocol\n}\n```\n\nAlmost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, `GetCompiledOperators`, etc. which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.\n\n`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.\n\nStep by step description of how to add a new protocol to Nuclei - \n\n1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with fewer options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `pkg/templates/types/types.go`.\n\n2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.\n\n```go\n\nimport (\n\t...\n\t\"github.com/projectdiscovery/nuclei/v3/pkg/protocols/others/websocket\"\n)\n\n// Template is a YAML input file which defines all the requests and\n// other metadata for a template.\ntype Template struct {\n\t...\n\t// description: |\n\t//   Websocket contains the Websocket request to make in the template.\n\tRequestsWebsocket []*websocket.Request `yaml:\"websocket,omitempty\" json:\"websocket,omitempty\" jsonschema:\"title=websocket requests to make,description=Websocket requests to make for the template\"`\n\t...\n}\n```\n\nAlso add the protocol case to the `Type` function as well as the `TemplateTypes` array in the same `templates.go` file.\n\n```go\n// TemplateTypes is a list of accepted template types\nvar TemplateTypes = []string{\n\t...\n\t\"websocket\",\n}\n\n// Type returns the type of the template\nfunc (t *Template) Type() templateTypes.ProtocolType {\n\t...\n\tcase len(t.RequestsWebsocket) > 0:\n\t\treturn templateTypes.WebsocketProtocol\n\tdefault:\n\t\treturn \"\"\n\t}\n}\n```\n\n3. Add the protocol request to the `Requests` function and `compileProtocolRequests` function in the `compile.go` file in same directory.\n\n```go\n\n// Requests return the total request count for the template\nfunc (template *Template) Requests() int {\n\treturn len(template.RequestsDNS) +\n\t\t...\n\t\tlen(template.RequestsSSL) +\n\t\tlen(template.RequestsWebsocket)\n}\n\n\n// compileProtocolRequests compiles all the protocol requests for the template\nfunc (template *Template) compileProtocolRequests(options protocols.ExecuterOptions) error {\n\t...\n\n\tcase len(template.RequestsWebsocket) > 0:\n\t\trequests = template.convertRequestToProtocolsRequest(template.RequestsWebsocket)\n\t}\n\ttemplate.Executer = executer.NewExecuter(requests, &options)\n\treturn nil\n}\n```\n\nThat's it, you've added a new protocol to Nuclei. The next good step would be to add native integration coverage under `internal/tests/integration` and run it with `go test -tags=integration ./internal/tests/integration`.\n\n\n## Profiling and Tracing\n\nTo analyze Nuclei's performance and resource usage, you can generate CPU & memory profiles and trace files using the `-profile-mem` flag:\n\n```bash\nnuclei -t nuclei-templates/ -u https://example.com -profile-mem=nuclei-$(git describe --tags)\n```\n\nThis command creates three files:\n\n* `nuclei.cpu`: CPU profile\n* `nuclei.mem`: Memory (heap) profile\n* `nuclei.trace`: Execution trace\n\n### Analyzing the CPU/Memory Profiles\n\n* View the profile in the terminal:\n\n```bash\ngo tool pprof nuclei.{cpu,mem}\n```\n\n* Display overall CPU time for processing $$N$$ targets:\n\n```\ngo tool pprof -top nuclei.cpu | grep \"Total samples\"\n```\n\n* Display top memory consumers:\n\n```bash\ngo tool pprof -top nuclei.mem | grep \"$(go list -m)\" | head -10\n```\n\n* Visualize the profile in a web browser:\n\n```bash\ngo tool pprof -http=:$(shuf -i 1000-99999 -n 1) nuclei.{cpu,mem}\n```\n\n### Analyzing the Trace File\n\nTo examine the execution trace:\n\n```bash\ngo tool trace nuclei.trace\n```\n\nThese tools help identify performance bottlenecks and memory leaks, allowing for targeted optimizations of Nuclei's codebase.\n\n## Project Structure\n\n- [pkg/reporting](./pkg/reporting) - Reporting modules for nuclei.\n- [pkg/reporting/exporters/sarif](./pkg/reporting/exporters/sarif) - Sarif Result Exporter\n- [pkg/reporting/exporters/markdown](./pkg/reporting/exporters/markdown) - Markdown Result Exporter\n- [pkg/reporting/exporters/es](./pkg/reporting/exporters/es) - Elasticsearch Result Exporter\n- [pkg/reporting/dedupe](./pkg/reporting/dedupe) - Dedupe module for Results\n- [pkg/reporting/trackers/gitlab](./pkg/reporting/trackers/gitlab) - GitLab Issue Tracker Exporter\n- [pkg/reporting/trackers/jira](./pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter\n- [pkg/reporting/trackers/github](./pkg/reporting/trackers/github) - GitHub Issue Tracker Exporter\n- [pkg/reporting/format](./pkg/reporting/format) - Result Formatting Functions\n- [pkg/parsers](./pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.\n- [pkg/types](./pkg/types) - Contains CLI options as well as misc helper functions.\n- [pkg/progress](./pkg/progress) - Progress tracking\n- [pkg/operators](./pkg/operators) - Operators for Nuclei\n- [pkg/operators/common/dsl](./pkg/operators/common/dsl) - DSL functions for Nuclei YAML Syntax\n- [pkg/operators/matchers](./pkg/operators/matchers) - Matchers implementation\n- [pkg/operators/extractors](./pkg/operators/extractors) - Extractors implementation\n- [pkg/catalog](./pkg/catalog) - Template loading from disk helpers\n- [pkg/catalog/config](./pkg/catalog/config) - Internal configuration management\n- [pkg/catalog/loader](./pkg/catalog/loader) - Implements loading and validation of templates and workflows.\n- [pkg/catalog/loader/filter](./pkg/catalog/loader/filter) - Filter filters templates based on tags and paths\n- [pkg/output](./pkg/output) - Output module for nuclei\n- [pkg/workflows](./pkg/workflows) - Workflow execution logic + declarations\n- [pkg/utils](./pkg/utils) - Utility functions\n- [pkg/model](./pkg/model) - Template Info + misc\n- [pkg/templates](./pkg/templates) - Templates core starting point\n- [pkg/templates/cache](./pkg/templates/cache) - Templates cache\n- [pkg/protocols](./pkg/protocols) - Protocol Specification\n- [pkg/protocols/file](./pkg/protocols/file) - File protocol\n- [pkg/protocols/network](./pkg/protocols/network) - Network protocol\n- [pkg/protocols/common/expressions](./pkg/protocols/common/expressions) - Expression evaluation + Templating Support\n- [pkg/protocols/common/interactsh](./pkg/protocols/common/interactsh) - Interactsh integration\n- [pkg/protocols/common/generators](./pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc.)\n- [pkg/protocols/common/executer](./pkg/protocols/common/executer) - Default Template Executer\n- [pkg/protocols/common/replacer](./pkg/protocols/common/replacer) - Template replacement helpers\n- [pkg/protocols/common/helpers/eventcreator](./pkg/protocols/common/helpers/eventcreator) - Result event creator\n- [pkg/protocols/common/helpers/responsehighlighter](./pkg/protocols/common/helpers/responsehighlighter) - Debug response highlighter\n- [pkg/protocols/common/helpers/deserialization](./pkg/protocols/common/helpers/deserialization) - Deserialization helper functions\n- [pkg/protocols/common/hosterrorscache](./pkg/protocols/common/hosterrorscache) - Host errors cache for tracking erroring hosts\n- [pkg/protocols/offlinehttp](./pkg/protocols/offlinehttp) - Offline http protocol\n- [pkg/protocols/http](./pkg/protocols/http) - HTTP protocol\n- [pkg/protocols/http/race](./pkg/protocols/http/race) - HTTP Race Module\n- [pkg/protocols/http/raw](./pkg/protocols/http/raw) - HTTP Raw Request Support\n- [pkg/protocols/headless](./pkg/protocols/headless) - Headless Module\n- [pkg/protocols/headless/engine](./pkg/protocols/headless/engine) - Internal Headless implementation\n- [pkg/protocols/dns](./pkg/protocols/dns) - DNS protocol\n- [pkg/projectfile](./pkg/projectfile) - Project File Implementation\n\n### Notes\n\n1. The matching as well as interim output functionality is a bit complex, we should simplify it a bit as well.\n"},"files":{"pkg/js/devtools/README.md":"## devtools\n\ndevtools contains tools and scripts to automate boring tasks related to javascript layer/ packages.\n\n### bindgen\n\n[bindgen](./bindgen/README.md) is a tool that automatically generated bindings for native go packages with 'goja'\n\n\n### scrapefuncs\n\n[scrapefuncs](./scrapefuncs/README.md) is a tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n\n### Generating API Reference (aka static site using javascript files using jsdoc)\n\n```console\njsdoc -R [Homepage.md] -r -d api_reference -t [optional: jsdoc theme to use] generated/js\n```\n\ngenerated static site will be available at `api_reference/` directory and can be verified using simplehttpserver\n\n```console\nsimplehttpserver\n```\n\nand then open `http://localhost:8000/` in browser\n\n\n### Notes\n\nwe currently use [clean-jsdoc-theme](https://www.npmjs.com/package/clean-jsdoc-theme) demo at [sample-jsproto-docs/](https://projectdiscovery.github.io/js-proto-docs/)","pkg/js/devtools/scrapefuncs/README.md":"## scrapefuncs\n\nscrapefuncs is go/ast based tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n### Usage\n\n```console\nUsage of ./scrapefuncs:\n  -dir string\n    \tdirectory to process (default \"pkg/js/global\")\n  -key string\n    \topenai api key\n  -keyfile string\n    \topenai api key file\n  -out string\n    \toutput js file with declarations of all global functions\n```\n\n\n### Example\n\n```console\n$ ./scrapefuncs -keyfile ~/.openai.key                                   \n[+] Scraped 7 functions\n\nName: Rand\nSignatures: \"Rand(n int) []byte\"\nDescription: Rand returns a random byte slice of length n\n\nName: RandInt\nSignatures: \"RandInt() int\"\nDescription: RandInt returns a random int\n\nName: log\nSignatures: \"log(msg string)\"\nSignatures: \"log(msg map[string]interface{})\"\nDescription: log prints given input to stdout with [JS] prefix for debugging purposes \n\nName: getNetworkPort\nSignatures: \"getNetworkPort(port string, defaultPort string) string\"\nDescription: getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n\nName: isPortOpen\nSignatures: \"isPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isPortOpen checks if given TCP port is open on host. timeout is optional and defaults to 5 seconds\n\nName: isUDPPortOpen\nSignatures: \"isUDPPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isUDPPortOpen checks if the given UDP port is open on the host. Timeout is optional and defaults to 5 seconds.\n\nName: ToBytes\nSignatures: \"ToBytes(...interface{}) []byte\"\nDescription: ToBytes converts given input to byte slice\n\nName: ToString\nSignatures: \"ToString(...interface{}) string\"\nDescription: ToString converts given input to string\n\n\n[+] Generating jsdoc for all functions\n\n/**\n * Rand returns a random byte slice of length n\n * Rand(n int) []byte\n * @function\n * @param {number} n - The length of the byte slice.\n */\nfunction Rand(n) {\n    // implemented in go\n};\n\n/**\n * RandInt returns a random int\n * RandInt() int\n * @function\n */\nfunction RandInt() {\n    // implemented in go\n};\n\n/**\n * log prints given input to stdout with [JS] prefix for debugging purposes\n * log(msg string)\n * log(msg map[string]interface{})\n * @function\n * @param {string|Object} msg - The message to print.\n */\nfunction log(msg) {\n    // implemented in go\n};\n\n/**\n * getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n * getNetworkPort(port string, defaultPort string) string\n * @function\n * @param {string} port - The port to check.\n * @param {string} defaultPort - The default port to return if the port is colliding.\n */\nfunction getNetworkPort(port, defaultPort) {\n    // implemented in go\n};\n\n/**\n * isPortOpen checks if given port is open on host. timeout is optional and defaults to 5 seconds\n * isPortOpen(host string, port string, [timeout int]) bool\n * @function\n * @param {string} host - The host to check.\n * @param {string} port - The port to check.\n * @param {number} [timeout=5] - The timeout in seconds.\n */\nfunction isPortOpen(host, port, timeout = 5) {\n    // implemented in go\n};\n\n/**\n * ToBytes converts given input to byte slice\n * ToBytes(...interface{}) []byte\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToBytes(...args) {\n    // implemented in go\n};\n\n/**\n * ToString converts given input to string\n * ToString(...interface{}) string\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToString(...args) {\n    // implemented in go\n};\n```","pkg/js/devtools/tsgen/README.md":"# tsgen\n\ntsgen is devtool to generate dummy typescript code for goja node modules written in golang. These generated typescript code can be compiled to generate .d.ts files to provide intellisense to editors like vscode and documentation for the node modules.","pkg/js/devtools/bindgen/README.md":"## bindgen (aka bindings generator)\n\nbindgen is a tool that automatically generated bindings for native go packages with 'goja'\n\nNative Go packages are available [here](../../libs/)\n\nGenerated Output is available [here](../../generated/)\n\nbindgen generates 3 different types of outputs \n\n- `go` => this directory contains corresponding goja bindings (actual bindings code) ex: [kerberos.go](../../generated/go/libkerberos/kerberos.go)\n- `js` => this is more of a javascript **representation** of all exposed functions and types etc. in javascript ex: [kerberos.js](../../generated/js/libkerberos/kerberos.js) and does not server any functional purpose other than reference\n- `markdown` => autogenerated markdown documentation for each library / package ex: [kerberos.md](../../generated/markdown/libkerberos/kerberos.md)\n\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nNuclei is a modern, high-performance vulnerability scanner built in Go that leverages YAML-based templates for customizable vulnerability detection. It supports multiple protocols (HTTP, DNS, TCP, SSL, WebSocket, WHOIS, JavaScript, Code) and is designed for zero false positives through real-world condition simulation.\n\n## Development Commands\n\n### Building and Testing\n- `make build` - Build the main nuclei binary to ./bin/nuclei\n- `make test` - Run unit tests with race detection\n- `make integration` - Run the native integration suite via `go test -tags=integration ./internal/tests/integration`\n- `make functional` - CI-only functional suite entry point\n- `make vet` - Run go vet for code analysis\n- `make tidy` - Clean up go modules\n\n### Validation and Linting\n- `make template-validate` - Validate nuclei templates using the built binary\n- `go fmt ./...` - Format Go code\n- `go vet ./...` - Static analysis\n\n### Development Tools\n- `make devtools-all` - Build all development tools (bindgen, tsgen, scrapefuncs)\n- `make jsupdate-all` - Update JavaScript bindings and TypeScript definitions\n- `make docs` - Generate documentation\n- `make memogen` - Generate memoization code for JavaScript libraries\n\n### Testing Specific Components\n- Run single test: `go test -v ./pkg/path/to/package -run TestName`\n- Integration tests are in `internal/tests/integration/` and run via `go test -tags=integration ./internal/tests/integration`\n\n## Architecture Overview\n\n### Core Components\n- **cmd/nuclei** - Main CLI entry point with flag parsing and configuration\n- **internal/runner** - Core runner that orchestrates the entire scanning process\n- **pkg/core** - Execution engine with work pools and template clustering\n- **pkg/templates** - Template parsing, compilation, and management\n- **pkg/protocols** - Protocol implementations (HTTP, DNS, Network, etc.)\n- **pkg/operators** - Matching and extraction logic (matchers/extractors)\n- **pkg/catalog** - Template discovery and loading from disk/remote sources\n\n### Protocol Architecture\nEach protocol (HTTP, DNS, Network, etc.) implements:\n- Request interface with Compile(), ExecuteWithResults(), Match(), Extract() methods\n- Operators embedding for matching/extraction functionality\n- Protocol-specific request building and execution logic\n\n### Template System\n- Templates are YAML files defining vulnerability detection logic\n- Compiled into executable requests with operators (matchers/extractors)\n- Support for workflows (multistep template execution)\n- Template clustering optimizes identical requests across multiple templates\n\n### Key Execution Flow\n1. Template loading and compilation via pkg/catalog/loader\n2. Input provider setup for targets\n3. Engine creation with work pools for concurrency\n4. Template execution with result collection via operators\n5. Output writing and reporting integration\n\n### JavaScript Integration\n- Custom JavaScript runtime for code protocol templates\n- Auto-generated bindings in pkg/js/generated/\n- Library implementations in pkg/js/libs/\n- Development tools for binding generation in pkg/js/devtools/\n\n## Template Development\n- Templates located in separate nuclei-templates repository\n- YAML format with info, requests, and operators sections  \n- Support for multiple protocol types in single template\n- Built-in DSL functions for dynamic content generation\n- Template validation available via `make template-validate`\n\n## Key Directories\n- **lib/** - SDK for embedding nuclei as a library\n- **examples/** - Usage examples for different scenarios\n- **internal/tests/integration/** - Native integration suite and owned testdata for harness coverage\n- **internal/tests/functional/** - CI-only native functional comparison suite and testcase assets\n- **pkg/fuzz/** - Fuzzing engine and DAST capabilities\n- **pkg/input/** - Input processing for various formats (Burp, OpenAPI, etc.)\n- **pkg/reporting/** - Result export and issue tracking integrations\nSee SECURITY_CONTEXT.md for this repo's known vulnerabilities and recurring weak spots. Check it before writing or reviewing security-sensitive code.\n","DESIGN.md":"# Nuclei Architecture Document\n\nA brief overview of Nuclei Engine architecture. This document will be kept updated as the engine progresses.\n\n## pkg/templates\n\n### Template\n\nTemplate is the basic unit of input to the engine which describes the requests to be made, matching to be done, data to extract, etc.\n\nThe template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers. \n\nAny attributes etc. required for the template, engine or requests to function are also set here.\n\nWorkflows are also compiled, their templates are loaded and compiled as well. Any validations etc. on the paths provided are also done here.\n\n`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.\n\n### Preprocessors\n\nPreprocessors are also applied here which can do things at template level. They get data of the template which they can alter at will on runtime. This is used in the engine to do random string generation.\n\nCustom processor can be used if they satisfy the following interface.\n\n```go\ntype Preprocessor interface {\n\tProcess(data []byte) []byte\n}\n```\n\n## pkg/model\n\nModel package implements Information structure for Nuclei Templates. `Info` contains all major metadata information for the template. `Classification` structure can also be used to provide additional context to vulnerability data.\n\nIt also specifies a `WorkflowLoader` interface that is used during workflow loading in template compilation stage.\n\n```go\ntype WorkflowLoader interface {\n\tGetTemplatePathsByTags(tags []string) []string\n\tGetTemplatePaths(templatesList []string, noValidate bool) []string\n}\n```\n\n## pkg/protocols\n\nProtocols package implements all the request protocols supported by Nuclei. This includes http, dns, network, headless and file requests as of now. \n\n### Request\n\nIt exposes a `Request` interface that is implemented by all the request protocols supported.\n\n```go\n// Request is an interface implemented any protocol based request generator.\ntype Request interface {\n\tCompile(options *ExecuterOptions) error\n\tRequests() int\n\tGetID() string\n\tMatch(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\tExtract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\tExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback OutputEventCallback) error\n\tMakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent\n\tMakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent\n\tGetCompiledOperators() []*operators.Operators\n}\n```\n\nMany of these methods are similar across protocols while some are very protocol specific. \n\nA brief overview of the methods is provided below -\n\n- **Compile** - Compiles the request with provided options.\n- **Requests** - Returns total requests made.\n- **GetID** - Returns any ID for the request\n- **Match** - Used to perform matching for patterns using matchers\n- **Extract** - Used to perform extraction for patterns using extractors\n- **ExecuteWithResults** - Request execution function for input.\n- **MakeResultEventItem** - Creates a single result event for the intermediate `InternalWrappedEvent` output structure.\n- **MakeResultEvent** - Returns a slice of results based on an `InternalWrappedEvent` internal output event.\n- **GetCompiledOperators** - Returns the compiled operators for the request.\n\n`MakeDefaultResultEvent` function can be used as a default for `MakeResultEvent` function when no protocol-specific features need to be implemented for result generation. \n\nFor reference protocol requests implementations, one can look at the below packages  - \n\n1. [pkg/protocols/http](./pkg/protocols/http)\n2. [pkg/protocols/dns](./pkg/protocols/dns)\n3. [pkg/protocols/network](./pkg/protocols/network)\n\n### Executer\n\nAll these different requests interfaces are converted to an Executer which is also an interface defined in `pkg/protocols` which is used during final execution of the template.\n\n```go\n// Executer is an interface implemented any protocol based request executer.\ntype Executer interface {\n\tCompile() error\n\tRequests() int\n\tExecute(input string) (bool, error)\n\tExecuteWithResults(input string, callback OutputEventCallback) error\n}\n```\n\nThe `ExecuteWithResults` function accepts a callback, which gets provided with results during execution in form of `*output.InternalWrappedEvent` structure.\n\nThe default executer is provided in `pkg/protocols/common/executer` . It takes a list of Requests and relevant `ExecuterOptions` and implements the Executer interface required for template execution. The executer during Template compilation process is created from this package and used as-is.\n\nA different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates`  We have a single HTTP  request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.\n\nFor Workflow execution, a separate RunWorkflow function is used which executes the workflow independently of the template execution.\n\nWith this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.\n\n## internal/runner\n\n### Template loading\n\nThe first process after all CLI specific initialisation is the loading of template/workflow paths that the user wants to run. This is done by the packages described below.\n\n#### pkg/catalog\n\nThis package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template and current user directory.\n\nThe syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.\n\n\n\nNext step is the initialisation of the reporting modules which is handled in `pkg/reporting`.  \n\n#### pkg/reporting\n\nReporting module contains exporters and trackers as well as a module for deduplication and a module for result formatting. \n\nExporters and Trackers are interfaces defined in pkg/reporting.\n\n```go\n// Tracker is an interface implemented by an issue tracker\ntype Tracker interface {\n\tCreateIssue(event *output.ResultEvent) error\n}\n\n// Exporter is an interface implemented by an issue exporter\ntype Exporter interface {\n\tClose() error\n\tExport(event *output.ResultEvent) error\n}\n```\n\nExporters include `Elasticsearch`, `markdown`, `sarif` . Trackers include `GitHub`, `GitLab` and `Jira`.\n\nEach exporter and trackers implement their own configuration in YAML format and are very modular in nature, so adding new ones is easy.\n\n\n\nAfter reading all the inputs from various sources and initialisation other miscellaneous options, the next bit is the output writing which is done using `pkg/output` module.\n\n#### pkg/output\n\nOutput package implements the output writing functionality for Nuclei.\n\nOutput Writer implements the Writer interface which is called each time a result is found for nuclei.\n\n```go\n// Writer is an interface which writes output to somewhere for nuclei events.\ntype Writer interface {\n\tClose()\n\tColorizer() aurora.Aurora\n\tWrite(*ResultEvent) error\n\tRequest(templateID, url, requestType string, err error)\n}\n```\n\nResultEvent structure is passed to the Nuclei Output Writer which contains the entire detail of a found result. Various intermediary types like `InternalWrappedEvent` and `InternalEvent` are used throughout nuclei protocols and matchers to describe results in various stages of execution.\n\n\n\n Interactsh is also initialised if it is not explicitly disabled. \n\n#### pkg/protocols/common/interactsh\n\nInteractsh module is used to provide automatic Out-of-Band vulnerability identification in Nuclei. \n\nIt uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.\n\nPolling for interactions and server registration only starts when a template uses the interactsh module and is executed by nuclei. After that no registration is required for the entire run.\n\n\n\n### RunEnumeration\n\nNext we arrive in the `RunEnumeration` function of the runner.\n\n`HostErrorsCache` is initialised which is used throughout the run of Nuclei enumeration to keep track of errors per host and skip further requests if the errors are greater than the provided threshold. The functionality for the error tracking cache is defined in [hosterrorscache.go](https://github.com/projectdiscovery/nuclei/blob/main/pkg/protocols/common/hosterrorscache/hosterrorscache.go) and is pretty simplistic in nature.\n\nNext the `WorkflowLoader` is initialised which used to load workflows. It exists in `pkg/parsers/workflow_loader.go`\n\nThe loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc. to return compiled `Templates` and `Workflows`. \n\n#### pkg/catalog/loader\n\nFirst the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module.  Next the path filter module is used to remove the excluded template/workflows paths.\n\n`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc. filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.\n\n`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.\n\nClustering module comes in next whose job is to cluster identical HTTP GET requests together (as a lot of the templates perform the same get requests many times, it's a good way to save many requests on large scans with lots of templates). \n\n### pkg/operators\n\nOperators package implements all the matching and extracting logic of Nuclei. \n\n```go\n// Operators contain the operators that can be applied on protocols\ntype Operators struct {\n\tMatchers []*matchers.Matcher\n\tExtractors []*extractors.Extractor\n\tMatchersCondition string\n}\n```\n\nA protocol only needs to embed the `operators.Operators` type shown above, and it can utilise all the matching/extracting functionality of nuclei.\n\n```go\n// MatchFunc performs matching operation for a matcher on model and returns true or false.\ntype MatchFunc func(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\n// ExtractFunc performs extracting operation for an extractor on model and returns true or false.\ntype ExtractFunc func(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\n// Execute executes the operators on data and returns a result structure\nfunc (operators *Operators) Execute(data map[string]interface{}, match MatchFunc, extract ExtractFunc, isDebug bool) (*Result, bool) \n```\n\nThe core of this process is the Execute function which takes an input dictionary as well as a Match and Extract function and return a `Result` structure which is used later during nuclei execution to check for results.\n\n```go\n// Result is a result structure created from operators running on data.\ntype Result struct {\n\tMatched bool\n\tExtracted bool\n\tMatches map[string][]string\n\tExtracts map[string][]string\n\tOutputExtracts []string\n\tDynamicValues map[string]interface{}\n\tPayloadValues map[string]interface{}\n}\n```\n\nThe internal logics for matching and extracting for things like words, regexes, jq, paths, etc. is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.\n\n\n### Template Execution\n\n`pkg/core` provides the engine mechanism which runs the templates/workflows on inputs. It exposes an `Execute` function which does the task of execution while also doing template clustering. The clustering can also be disabled optionally by the user.\n \nAn example of using the core engine is provided below.\n\n```go\nengine := core.New(r.options)\nengine.SetExecuterOptions(executerOpts)\nresults := engine.ExecuteWithOpts(finalTemplates, r.hmapInputProvider, true)\n```\n\n### Adding a New Protocol\n\nProtocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc. are implemented in form of protocol requests.\n\nA protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.\n\nThe code for the websocket protocol is contained in `pkg/protocols/others/websocket`. \n\nBelow a high level skeleton of the websocket implementation is provided with all the important parts present.\n\n```go\npackage websocket\n\n// Request is a request for the Websocket protocol\ntype Request struct {\n\t// Operators for the current request go here.\n\toperators.Operators `yaml:\",inline,omitempty\"`\n\tCompiledOperators   *operators.Operators `yaml:\"-\"`\n\n\t// description: |\n\t//   Address contains address for the request\n\tAddress string `yaml:\"address,omitempty\" jsonschema:\"title=address for the websocket request,description=Address contains address for the request\"`\n\n    // declarations here\n}\n\n// Compile compiles the request generators preparing any requests possible.\nfunc (r *Request) Compile(options *protocols.ExecuterOptions) error {\n\tr.options = options\n\n    // request compilation here as well as client creation\n \n\tif len(r.Matchers) > 0 || len(r.Extractors) > 0 {\n\t\tcompiled := &r.Operators\n\t\tif err := compiled.Compile(); err != nil {\n\t\t\treturn errors.Wrap(err, \"could not compile operators\")\n\t\t}\n\t\tr.CompiledOperators = compiled\n\t}\n\treturn nil\n}\n\n// Requests returns the total number of requests the rule will perform\nfunc (r *Request) Requests() int {\n\tif r.generator != nil {\n\t\treturn r.generator.NewIterator().Total()\n\t}\n\treturn 1\n}\n\n// GetID returns the ID for the request if any.\nfunc (r *Request) GetID() string {\n\treturn \"\"\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) ExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n    // payloads init here\n\tif err := r.executeRequestWithPayloads(input, hostname, value, previous, callback); err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) executeRequestWithPayloads(input, hostname string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n\theader := http.Header{}\n\n    // make the actual request here after setting all options\n\n\tevent := eventcreator.CreateEventWithAdditionalOptions(r, data, r.options.Options.Debug || r.options.Options.DebugResponse, func(internalWrappedEvent *output.InternalWrappedEvent) {\n\t\tinternalWrappedEvent.OperatorsResult.PayloadValues = payloadValues\n\t})\n\tif r.options.Options.Debug || r.options.Options.DebugResponse {\n\t\tresponseOutput := responseBuilder.String()\n\t\tgologger.Debug().Msgf(\"[%s] Dumped Websocket response for %s\", r.options.TemplateID, input)\n\t\tgologger.Print().Msgf(\"%s\", responsehighlighter.Highlight(event.OperatorsResult, responseOutput, r.options.Options.NoColor))\n\t}\n\n\tcallback(event)\n\treturn nil\n}\n\nfunc (r *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent {\n\tdata := &output.ResultEvent{\n\t\tTemplateID:       types.ToString(r.options.TemplateID),\n\t\tTemplatePath:     types.ToString(r.options.TemplatePath),\n\t\t// ... setting more values for result event\n\t}\n\treturn data\n}\n\n// Match performs matching operation for a matcher on model and returns:\n// true and a list of matched snippets if the matcher type is supports it\n// otherwise false and an empty string slice\nfunc (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string) {\n\treturn protocols.MakeDefaultMatchFunc(data, matcher)\n}\n\n// Extract performs extracting operation for an extractor on model and returns true or false.\nfunc (r *Request) Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{} {\n\treturn protocols.MakeDefaultExtractFunc(data, matcher)\n}\n\n// MakeResultEvent creates a result event from internal wrapped event\nfunc (r *Request) MakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent {\n\treturn protocols.MakeDefaultResultEvent(r, wrapped)\n}\n\n// GetCompiledOperators returns a list of the compiled operators\nfunc (r *Request) GetCompiledOperators() []*operators.Operators {\n\treturn []*operators.Operators{r.CompiledOperators}\n}\n\n// Type returns the type of the protocol request\nfunc (r *Request) Type() templateTypes.ProtocolType {\n\treturn templateTypes.WebsocketProtocol\n}\n```\n\nAlmost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, `GetCompiledOperators`, etc. which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.\n\n`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.\n\nStep by step description of how to add a new protocol to Nuclei - \n\n1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with fewer options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `pkg/templates/types/types.go`.\n\n2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.\n\n```go\n\nimport (\n\t...\n\t\"github.com/projectdiscovery/nuclei/v3/pkg/protocols/others/websocket\"\n)\n\n// Template is a YAML input file which defines all the requests and\n// other metadata for a template.\ntype Template struct {\n\t...\n\t// description: |\n\t//   Websocket contains the Websocket request to make in the template.\n\tRequestsWebsocket []*websocket.Request `yaml:\"websocket,omitempty\" json:\"websocket,omitempty\" jsonschema:\"title=websocket requests to make,description=Websocket requests to make for the template\"`\n\t...\n}\n```\n\nAlso add the protocol case to the `Type` function as well as the `TemplateTypes` array in the same `templates.go` file.\n\n```go\n// TemplateTypes is a list of accepted template types\nvar TemplateTypes = []string{\n\t...\n\t\"websocket\",\n}\n\n// Type returns the type of the template\nfunc (t *Template) Type() templateTypes.ProtocolType {\n\t...\n\tcase len(t.RequestsWebsocket) > 0:\n\t\treturn templateTypes.WebsocketProtocol\n\tdefault:\n\t\treturn \"\"\n\t}\n}\n```\n\n3. Add the protocol request to the `Requests` function and `compileProtocolRequests` function in the `compile.go` file in same directory.\n\n```go\n\n// Requests return the total request count for the template\nfunc (template *Template) Requests() int {\n\treturn len(template.RequestsDNS) +\n\t\t...\n\t\tlen(template.RequestsSSL) +\n\t\tlen(template.RequestsWebsocket)\n}\n\n\n// compileProtocolRequests compiles all the protocol requests for the template\nfunc (template *Template) compileProtocolRequests(options protocols.ExecuterOptions) error {\n\t...\n\n\tcase len(template.RequestsWebsocket) > 0:\n\t\trequests = template.convertRequestToProtocolsRequest(template.RequestsWebsocket)\n\t}\n\ttemplate.Executer = executer.NewExecuter(requests, &options)\n\treturn nil\n}\n```\n\nThat's it, you've added a new protocol to Nuclei. The next good step would be to add native integration coverage under `internal/tests/integration` and run it with `go test -tags=integration ./internal/tests/integration`.\n\n\n## Profiling and Tracing\n\nTo analyze Nuclei's performance and resource usage, you can generate CPU & memory profiles and trace files using the `-profile-mem` flag:\n\n```bash\nnuclei -t nuclei-templates/ -u https://example.com -profile-mem=nuclei-$(git describe --tags)\n```\n\nThis command creates three files:\n\n* `nuclei.cpu`: CPU profile\n* `nuclei.mem`: Memory (heap) profile\n* `nuclei.trace`: Execution trace\n\n### Analyzing the CPU/Memory Profiles\n\n* View the profile in the terminal:\n\n```bash\ngo tool pprof nuclei.{cpu,mem}\n```\n\n* Display overall CPU time for processing $$N$$ targets:\n\n```\ngo tool pprof -top nuclei.cpu | grep \"Total samples\"\n```\n\n* Display top memory consumers:\n\n```bash\ngo tool pprof -top nuclei.mem | grep \"$(go list -m)\" | head -10\n```\n\n* Visualize the profile in a web browser:\n\n```bash\ngo tool pprof -http=:$(shuf -i 1000-99999 -n 1) nuclei.{cpu,mem}\n```\n\n### Analyzing the Trace File\n\nTo examine the execution trace:\n\n```bash\ngo tool trace nuclei.trace\n```\n\nThese tools help identify performance bottlenecks and memory leaks, allowing for targeted optimizations of Nuclei's codebase.\n\n## Project Structure\n\n- [pkg/reporting](./pkg/reporting) - Reporting modules for nuclei.\n- [pkg/reporting/exporters/sarif](./pkg/reporting/exporters/sarif) - Sarif Result Exporter\n- [pkg/reporting/exporters/markdown](./pkg/reporting/exporters/markdown) - Markdown Result Exporter\n- [pkg/reporting/exporters/es](./pkg/reporting/exporters/es) - Elasticsearch Result Exporter\n- [pkg/reporting/dedupe](./pkg/reporting/dedupe) - Dedupe module for Results\n- [pkg/reporting/trackers/gitlab](./pkg/reporting/trackers/gitlab) - GitLab Issue Tracker Exporter\n- [pkg/reporting/trackers/jira](./pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter\n- [pkg/reporting/trackers/github](./pkg/reporting/trackers/github) - GitHub Issue Tracker Exporter\n- [pkg/reporting/format](./pkg/reporting/format) - Result Formatting Functions\n- [pkg/parsers](./pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.\n- [pkg/types](./pkg/types) - Contains CLI options as well as misc helper functions.\n- [pkg/progress](./pkg/progress) - Progress tracking\n- [pkg/operators](./pkg/operators) - Operators for Nuclei\n- [pkg/operators/common/dsl](./pkg/operators/common/dsl) - DSL functions for Nuclei YAML Syntax\n- [pkg/operators/matchers](./pkg/operators/matchers) - Matchers implementation\n- [pkg/operators/extractors](./pkg/operators/extractors) - Extractors implementation\n- [pkg/catalog](./pkg/catalog) - Template loading from disk helpers\n- [pkg/catalog/config](./pkg/catalog/config) - Internal configuration management\n- [pkg/catalog/loader](./pkg/catalog/loader) - Implements loading and validation of templates and workflows.\n- [pkg/catalog/loader/filter](./pkg/catalog/loader/filter) - Filter filters templates based on tags and paths\n- [pkg/output](./pkg/output) - Output module for nuclei\n- [pkg/workflows](./pkg/workflows) - Workflow execution logic + declarations\n- [pkg/utils](./pkg/utils) - Utility functions\n- [pkg/model](./pkg/model) - Template Info + misc\n- [pkg/templates](./pkg/templates) - Templates core starting point\n- [pkg/templates/cache](./pkg/templates/cache) - Templates cache\n- [pkg/protocols](./pkg/protocols) - Protocol Specification\n- [pkg/protocols/file](./pkg/protocols/file) - File protocol\n- [pkg/protocols/network](./pkg/protocols/network) - Network protocol\n- [pkg/protocols/common/expressions](./pkg/protocols/common/expressions) - Expression evaluation + Templating Support\n- [pkg/protocols/common/interactsh](./pkg/protocols/common/interactsh) - Interactsh integration\n- [pkg/protocols/common/generators](./pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc.)\n- [pkg/protocols/common/executer](./pkg/protocols/common/executer) - Default Template Executer\n- [pkg/protocols/common/replacer](./pkg/protocols/common/replacer) - Template replacement helpers\n- [pkg/protocols/common/helpers/eventcreator](./pkg/protocols/common/helpers/eventcreator) - Result event creator\n- [pkg/protocols/common/helpers/responsehighlighter](./pkg/protocols/common/helpers/responsehighlighter) - Debug response highlighter\n- [pkg/protocols/common/helpers/deserialization](./pkg/protocols/common/helpers/deserialization) - Deserialization helper functions\n- [pkg/protocols/common/hosterrorscache](./pkg/protocols/common/hosterrorscache) - Host errors cache for tracking erroring hosts\n- [pkg/protocols/offlinehttp](./pkg/protocols/offlinehttp) - Offline http protocol\n- [pkg/protocols/http](./pkg/protocols/http) - HTTP protocol\n- [pkg/protocols/http/race](./pkg/protocols/http/race) - HTTP Race Module\n- [pkg/protocols/http/raw](./pkg/protocols/http/raw) - HTTP Raw Request Support\n- [pkg/protocols/headless](./pkg/protocols/headless) - Headless Module\n- [pkg/protocols/headless/engine](./pkg/protocols/headless/engine) - Internal Headless implementation\n- [pkg/protocols/dns](./pkg/protocols/dns) - DNS protocol\n- [pkg/projectfile](./pkg/projectfile) - Project File Implementation\n\n### Notes\n\n1. The matching as well as interim output functionality is a bit complex, we should simplify it a bit as well.\n"},"items":[{"name":"DESIGN.md","path":"DESIGN.md","rawUrl":"https://raw.githubusercontent.com/projectdiscovery/nuclei/HEAD/DESIGN.md","title":"UI Design System & Style Tokens","category":"project-spec","format":"markdown","content":"# Nuclei Architecture Document\n\nA brief overview of Nuclei Engine architecture. This document will be kept updated as the engine progresses.\n\n## pkg/templates\n\n### Template\n\nTemplate is the basic unit of input to the engine which describes the requests to be made, matching to be done, data to extract, etc.\n\nThe template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers. \n\nAny attributes etc. required for the template, engine or requests to function are also set here.\n\nWorkflows are also compiled, their templates are loaded and compiled as well. Any validations etc. on the paths provided are also done here.\n\n`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.\n\n### Preprocessors\n\nPreprocessors are also applied here which can do things at template level. They get data of the template which they can alter at will on runtime. This is used in the engine to do random string generation.\n\nCustom processor can be used if they satisfy the following interface.\n\n```go\ntype Preprocessor interface {\n\tProcess(data []byte) []byte\n}\n```\n\n## pkg/model\n\nModel package implements Information structure for Nuclei Templates. `Info` contains all major metadata information for the template. `Classification` structure can also be used to provide additional context to vulnerability data.\n\nIt also specifies a `WorkflowLoader` interface that is used during workflow loading in template compilation stage.\n\n```go\ntype WorkflowLoader interface {\n\tGetTemplatePathsByTags(tags []string) []string\n\tGetTemplatePaths(templatesList []string, noValidate bool) []string\n}\n```\n\n## pkg/protocols\n\nProtocols package implements all the request protocols supported by Nuclei. This includes http, dns, network, headless and file requests as of now. \n\n### Request\n\nIt exposes a `Request` interface that is implemented by all the request protocols supported.\n\n```go\n// Request is an interface implemented any protocol based request generator.\ntype Request interface {\n\tCompile(options *ExecuterOptions) error\n\tRequests() int\n\tGetID() string\n\tMatch(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\tExtract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\tExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback OutputEventCallback) error\n\tMakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent\n\tMakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent\n\tGetCompiledOperators() []*operators.Operators\n}\n```\n\nMany of these methods are similar across protocols while some are very protocol specific. \n\nA brief overview of the methods is provided below -\n\n- **Compile** - Compiles the request with provided options.\n- **Requests** - Returns total requests made.\n- **GetID** - Returns any ID for the request\n- **Match** - Used to perform matching for patterns using matchers\n- **Extract** - Used to perform extraction for patterns using extractors\n- **ExecuteWithResults** - Request execution function for input.\n- **MakeResultEventItem** - Creates a single result event for the intermediate `InternalWrappedEvent` output structure.\n- **MakeResultEvent** - Returns a slice of results based on an `InternalWrappedEvent` internal output event.\n- **GetCompiledOperators** - Returns the compiled operators for the request.\n\n`MakeDefaultResultEvent` function can be used as a default for `MakeResultEvent` function when no protocol-specific features need to be implemented for result generation. \n\nFor reference protocol requests implementations, one can look at the below packages  - \n\n1. [pkg/protocols/http](./pkg/protocols/http)\n2. [pkg/protocols/dns](./pkg/protocols/dns)\n3. [pkg/protocols/network](./pkg/protocols/network)\n\n### Executer\n\nAll these different requests interfaces are converted to an Executer which is also an interface defined in `pkg/protocols` which is used during final execution of the template.\n\n```go\n// Executer is an interface implemented any protocol based request executer.\ntype Executer interface {\n\tCompile() error\n\tRequests() int\n\tExecute(input string) (bool, error)\n\tExecuteWithResults(input string, callback OutputEventCallback) error\n}\n```\n\nThe `ExecuteWithResults` function accepts a callback, which gets provided with results during execution in form of `*output.InternalWrappedEvent` structure.\n\nThe default executer is provided in `pkg/protocols/common/executer` . It takes a list of Requests and relevant `ExecuterOptions` and implements the Executer interface required for template execution. The executer during Template compilation process is created from this package and used as-is.\n\nA different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates`  We have a single HTTP  request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.\n\nFor Workflow execution, a separate RunWorkflow function is used which executes the workflow independently of the template execution.\n\nWith this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.\n\n## internal/runner\n\n### Template loading\n\nThe first process after all CLI specific initialisation is the loading of template/workflow paths that the user wants to run. This is done by the packages described below.\n\n#### pkg/catalog\n\nThis package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template and current user directory.\n\nThe syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.\n\n\n\nNext step is the initialisation of the reporting modules which is handled in `pkg/reporting`.  \n\n#### pkg/reporting\n\nReporting module contains exporters and trackers as well as a module for deduplication and a module for result formatting. \n\nExporters and Trackers are interfaces defined in pkg/reporting.\n\n```go\n// Tracker is an interface implemented by an issue tracker\ntype Tracker interface {\n\tCreateIssue(event *output.ResultEvent) error\n}\n\n// Exporter is an interface implemented by an issue exporter\ntype Exporter interface {\n\tClose() error\n\tExport(event *output.ResultEvent) error\n}\n```\n\nExporters include `Elasticsearch`, `markdown`, `sarif` . Trackers include `GitHub`, `GitLab` and `Jira`.\n\nEach exporter and trackers implement their own configuration in YAML format and are very modular in nature, so adding new ones is easy.\n\n\n\nAfter reading all the inputs from various sources and initialisation other miscellaneous options, the next bit is the output writing which is done using `pkg/output` module.\n\n#### pkg/output\n\nOutput package implements the output writing functionality for Nuclei.\n\nOutput Writer implements the Writer interface which is called each time a result is found for nuclei.\n\n```go\n// Writer is an interface which writes output to somewhere for nuclei events.\ntype Writer interface {\n\tClose()\n\tColorizer() aurora.Aurora\n\tWrite(*ResultEvent) error\n\tRequest(templateID, url, requestType string, err error)\n}\n```\n\nResultEvent structure is passed to the Nuclei Output Writer which contains the entire detail of a found result. Various intermediary types like `InternalWrappedEvent` and `InternalEvent` are used throughout nuclei protocols and matchers to describe results in various stages of execution.\n\n\n\n Interactsh is also initialised if it is not explicitly disabled. \n\n#### pkg/protocols/common/interactsh\n\nInteractsh module is used to provide automatic Out-of-Band vulnerability identification in Nuclei. \n\nIt uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.\n\nPolling for interactions and server registration only starts when a template uses the interactsh module and is executed by nuclei. After that no registration is required for the entire run.\n\n\n\n### RunEnumeration\n\nNext we arrive in the `RunEnumeration` function of the runner.\n\n`HostErrorsCache` is initialised which is used throughout the run of Nuclei enumeration to keep track of errors per host and skip further requests if the errors are greater than the provided threshold. The functionality for the error tracking cache is defined in [hosterrorscache.go](https://github.com/projectdiscovery/nuclei/blob/main/pkg/protocols/common/hosterrorscache/hosterrorscache.go) and is pretty simplistic in nature.\n\nNext the `WorkflowLoader` is initialised which used to load workflows. It exists in `pkg/parsers/workflow_loader.go`\n\nThe loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc. to return compiled `Templates` and `Workflows`. \n\n#### pkg/catalog/loader\n\nFirst the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module.  Next the path filter module is used to remove the excluded template/workflows paths.\n\n`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc. filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.\n\n`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.\n\nClustering module comes in next whose job is to cluster identical HTTP GET requests together (as a lot of the templates perform the same get requests many times, it's a good way to save many requests on large scans with lots of templates). \n\n### pkg/operators\n\nOperators package implements all the matching and extracting logic of Nuclei. \n\n```go\n// Operators contain the operators that can be applied on protocols\ntype Operators struct {\n\tMatchers []*matchers.Matcher\n\tExtractors []*extractors.Extractor\n\tMatchersCondition string\n}\n```\n\nA protocol only needs to embed the `operators.Operators` type shown above, and it can utilise all the matching/extracting functionality of nuclei.\n\n```go\n// MatchFunc performs matching operation for a matcher on model and returns true or false.\ntype MatchFunc func(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\n// ExtractFunc performs extracting operation for an extractor on model and returns true or false.\ntype ExtractFunc func(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\n// Execute executes the operators on data and returns a result structure\nfunc (operators *Operators) Execute(data map[string]interface{}, match MatchFunc, extract ExtractFunc, isDebug bool) (*Result, bool) \n```\n\nThe core of this process is the Execute function which takes an input dictionary as well as a Match and Extract function and return a `Result` structure which is used later during nuclei execution to check for results.\n\n```go\n// Result is a result structure created from operators running on data.\ntype Result struct {\n\tMatched bool\n\tExtracted bool\n\tMatches map[string][]string\n\tExtracts map[string][]string\n\tOutputExtracts []string\n\tDynamicValues map[string]interface{}\n\tPayloadValues map[string]interface{}\n}\n```\n\nThe internal logics for matching and extracting for things like words, regexes, jq, paths, etc. is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.\n\n\n### Template Execution\n\n`pkg/core` provides the engine mechanism which runs the templates/workflows on inputs. It exposes an `Execute` function which does the task of execution while also doing template clustering. The clustering can also be disabled optionally by the user.\n \nAn example of using the core engine is provided below.\n\n```go\nengine := core.New(r.options)\nengine.SetExecuterOptions(executerOpts)\nresults := engine.ExecuteWithOpts(finalTemplates, r.hmapInputProvider, true)\n```\n\n### Adding a New Protocol\n\nProtocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc. are implemented in form of protocol requests.\n\nA protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.\n\nThe code for the websocket protocol is contained in `pkg/protocols/others/websocket`. \n\nBelow a high level skeleton of the websocket implementation is provided with all the important parts present.\n\n```go\npackage websocket\n\n// Request is a request for the Websocket protocol\ntype Request struct {\n\t// Operators for the current request go here.\n\toperators.Operators `yaml:\",inline,omitempty\"`\n\tCompiledOperators   *operators.Operators `yaml:\"-\"`\n\n\t// description: |\n\t//   Address contains address for the request\n\tAddress string `yaml:\"address,omitempty\" jsonschema:\"title=address for the websocket request,description=Address contains address for the request\"`\n\n    // declarations here\n}\n\n// Compile compiles the request generators preparing any requests possible.\nfunc (r *Request) Compile(options *protocols.ExecuterOptions) error {\n\tr.options = options\n\n    // request compilation here as well as client creation\n \n\tif len(r.Matchers) > 0 || len(r.Extractors) > 0 {\n\t\tcompiled := &r.Operators\n\t\tif err := compiled.Compile(); err != nil {\n\t\t\treturn errors.Wrap(err, \"could not compile operators\")\n\t\t}\n\t\tr.CompiledOperators = compiled\n\t}\n\treturn nil\n}\n\n// Requests returns the total number of requests the rule will perform\nfunc (r *Request) Requests() int {\n\tif r.generator != nil {\n\t\treturn r.generator.NewIterator().Total()\n\t}\n\treturn 1\n}\n\n// GetID returns the ID for the request if any.\nfunc (r *Request) GetID() string {\n\treturn \"\"\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) ExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n    // payloads init here\n\tif err := r.executeRequestWithPayloads(input, hostname, value, previous, callback); err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) executeRequestWithPayloads(input, hostname string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n\theader := http.Header{}\n\n    // make the actual request here after setting all options\n\n\tevent := eventcreator.CreateEventWithAdditionalOptions(r, data, r.options.Options.Debug || r.options.Options.DebugResponse, func(internalWrappedEvent *output.InternalWrappedEvent) {\n\t\tinternalWrappedEvent.OperatorsResult.PayloadValues = payloadValues\n\t})\n\tif r.options.Options.Debug || r.options.Options.DebugResponse {\n\t\tresponseOutput := responseBuilder.String()\n\t\tgologger.Debug().Msgf(\"[%s] Dumped Websocket response for %s\", r.options.TemplateID, input)\n\t\tgologger.Print().Msgf(\"%s\", responsehighlighter.Highlight(event.OperatorsResult, responseOutput, r.options.Options.NoColor))\n\t}\n\n\tcallback(event)\n\treturn nil\n}\n\nfunc (r *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent {\n\tdata := &output.ResultEvent{\n\t\tTemplateID:       types.ToString(r.options.TemplateID),\n\t\tTemplatePath:     types.ToString(r.options.TemplatePath),\n\t\t// ... setting more values for result event\n\t}\n\treturn data\n}\n\n// Match performs matching operation for a matcher on model and returns:\n// true and a list of matched snippets if the matcher type is supports it\n// otherwise false and an empty string slice\nfunc (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string) {\n\treturn protocols.MakeDefaultMatchFunc(data, matcher)\n}\n\n// Extract performs extracting operation for an extractor on model and returns true or false.\nfunc (r *Request) Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{} {\n\treturn protocols.MakeDefaultExtractFunc(data, matcher)\n}\n\n// MakeResultEvent creates a result event from internal wrapped event\nfunc (r *Request) MakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent {\n\treturn protocols.MakeDefaultResultEvent(r, wrapped)\n}\n\n// GetCompiledOperators returns a list of the compiled operators\nfunc (r *Request) GetCompiledOperators() []*operators.Operators {\n\treturn []*operators.Operators{r.CompiledOperators}\n}\n\n// Type returns the type of the protocol request\nfunc (r *Request) Type() templateTypes.ProtocolType {\n\treturn templateTypes.WebsocketProtocol\n}\n```\n\nAlmost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, `GetCompiledOperators`, etc. which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.\n\n`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.\n\nStep by step description of how to add a new protocol to Nuclei - \n\n1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with fewer options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `pkg/templates/types/types.go`.\n\n2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.\n\n```go\n\nimport (\n\t...\n\t\"github.com/projectdiscovery/nuclei/v3/pkg/protocols/others/websocket\"\n)\n\n// Template is a YAML input file which defines all the requests and\n// other metadata for a template.\ntype Template struct {\n\t...\n\t// description: |\n\t//   Websocket contains the Websocket request to make in the template.\n\tRequestsWebsocket []*websocket.Request `yaml:\"websocket,omitempty\" json:\"websocket,omitempty\" jsonschema:\"title=websocket requests to make,description=Websocket requests to make for the template\"`\n\t...\n}\n```\n\nAlso add the protocol case to the `Type` function as well as the `TemplateTypes` array in the same `templates.go` file.\n\n```go\n// TemplateTypes is a list of accepted template types\nvar TemplateTypes = []string{\n\t...\n\t\"websocket\",\n}\n\n// Type returns the type of the template\nfunc (t *Template) Type() templateTypes.ProtocolType {\n\t...\n\tcase len(t.RequestsWebsocket) > 0:\n\t\treturn templateTypes.WebsocketProtocol\n\tdefault:\n\t\treturn \"\"\n\t}\n}\n```\n\n3. Add the protocol request to the `Requests` function and `compileProtocolRequests` function in the `compile.go` file in same directory.\n\n```go\n\n// Requests return the total request count for the template\nfunc (template *Template) Requests() int {\n\treturn len(template.RequestsDNS) +\n\t\t...\n\t\tlen(template.RequestsSSL) +\n\t\tlen(template.RequestsWebsocket)\n}\n\n\n// compileProtocolRequests compiles all the protocol requests for the template\nfunc (template *Template) compileProtocolRequests(options protocols.ExecuterOptions) error {\n\t...\n\n\tcase len(template.RequestsWebsocket) > 0:\n\t\trequests = template.convertRequestToProtocolsRequest(template.RequestsWebsocket)\n\t}\n\ttemplate.Executer = executer.NewExecuter(requests, &options)\n\treturn nil\n}\n```\n\nThat's it, you've added a new protocol to Nuclei. The next good step would be to add native integration coverage under `internal/tests/integration` and run it with `go test -tags=integration ./internal/tests/integration`.\n\n\n## Profiling and Tracing\n\nTo analyze Nuclei's performance and resource usage, you can generate CPU & memory profiles and trace files using the `-profile-mem` flag:\n\n```bash\nnuclei -t nuclei-templates/ -u https://example.com -profile-mem=nuclei-$(git describe --tags)\n```\n\nThis command creates three files:\n\n* `nuclei.cpu`: CPU profile\n* `nuclei.mem`: Memory (heap) profile\n* `nuclei.trace`: Execution trace\n\n### Analyzing the CPU/Memory Profiles\n\n* View the profile in the terminal:\n\n```bash\ngo tool pprof nuclei.{cpu,mem}\n```\n\n* Display overall CPU time for processing $$N$$ targets:\n\n```\ngo tool pprof -top nuclei.cpu | grep \"Total samples\"\n```\n\n* Display top memory consumers:\n\n```bash\ngo tool pprof -top nuclei.mem | grep \"$(go list -m)\" | head -10\n```\n\n* Visualize the profile in a web browser:\n\n```bash\ngo tool pprof -http=:$(shuf -i 1000-99999 -n 1) nuclei.{cpu,mem}\n```\n\n### Analyzing the Trace File\n\nTo examine the execution trace:\n\n```bash\ngo tool trace nuclei.trace\n```\n\nThese tools help identify performance bottlenecks and memory leaks, allowing for targeted optimizations of Nuclei's codebase.\n\n## Project Structure\n\n- [pkg/reporting](./pkg/reporting) - Reporting modules for nuclei.\n- [pkg/reporting/exporters/sarif](./pkg/reporting/exporters/sarif) - Sarif Result Exporter\n- [pkg/reporting/exporters/markdown](./pkg/reporting/exporters/markdown) - Markdown Result Exporter\n- [pkg/reporting/exporters/es](./pkg/reporting/exporters/es) - Elasticsearch Result Exporter\n- [pkg/reporting/dedupe](./pkg/reporting/dedupe) - Dedupe module for Results\n- [pkg/reporting/trackers/gitlab](./pkg/reporting/trackers/gitlab) - GitLab Issue Tracker Exporter\n- [pkg/reporting/trackers/jira](./pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter\n- [pkg/reporting/trackers/github](./pkg/reporting/trackers/github) - GitHub Issue Tracker Exporter\n- [pkg/reporting/format](./pkg/reporting/format) - Result Formatting Functions\n- [pkg/parsers](./pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.\n- [pkg/types](./pkg/types) - Contains CLI options as well as misc helper functions.\n- [pkg/progress](./pkg/progress) - Progress tracking\n- [pkg/operators](./pkg/operators) - Operators for Nuclei\n- [pkg/operators/common/dsl](./pkg/operators/common/dsl) - DSL functions for Nuclei YAML Syntax\n- [pkg/operators/matchers](./pkg/operators/matchers) - Matchers implementation\n- [pkg/operators/extractors](./pkg/operators/extractors) - Extractors implementation\n- [pkg/catalog](./pkg/catalog) - Template loading from disk helpers\n- [pkg/catalog/config](./pkg/catalog/config) - Internal configuration management\n- [pkg/catalog/loader](./pkg/catalog/loader) - Implements loading and validation of templates and workflows.\n- [pkg/catalog/loader/filter](./pkg/catalog/loader/filter) - Filter filters templates based on tags and paths\n- [pkg/output](./pkg/output) - Output module for nuclei\n- [pkg/workflows](./pkg/workflows) - Workflow execution logic + declarations\n- [pkg/utils](./pkg/utils) - Utility functions\n- [pkg/model](./pkg/model) - Template Info + misc\n- [pkg/templates](./pkg/templates) - Templates core starting point\n- [pkg/templates/cache](./pkg/templates/cache) - Templates cache\n- [pkg/protocols](./pkg/protocols) - Protocol Specification\n- [pkg/protocols/file](./pkg/protocols/file) - File protocol\n- [pkg/protocols/network](./pkg/protocols/network) - Network protocol\n- [pkg/protocols/common/expressions](./pkg/protocols/common/expressions) - Expression evaluation + Templating Support\n- [pkg/protocols/common/interactsh](./pkg/protocols/common/interactsh) - Interactsh integration\n- [pkg/protocols/common/generators](./pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc.)\n- [pkg/protocols/common/executer](./pkg/protocols/common/executer) - Default Template Executer\n- [pkg/protocols/common/replacer](./pkg/protocols/common/replacer) - Template replacement helpers\n- [pkg/protocols/common/helpers/eventcreator](./pkg/protocols/common/helpers/eventcreator) - Result event creator\n- [pkg/protocols/common/helpers/responsehighlighter](./pkg/protocols/common/helpers/responsehighlighter) - Debug response highlighter\n- [pkg/protocols/common/helpers/deserialization](./pkg/protocols/common/helpers/deserialization) - Deserialization helper functions\n- [pkg/protocols/common/hosterrorscache](./pkg/protocols/common/hosterrorscache) - Host errors cache for tracking erroring hosts\n- [pkg/protocols/offlinehttp](./pkg/protocols/offlinehttp) - Offline http protocol\n- [pkg/protocols/http](./pkg/protocols/http) - HTTP protocol\n- [pkg/protocols/http/race](./pkg/protocols/http/race) - HTTP Race Module\n- [pkg/protocols/http/raw](./pkg/protocols/http/raw) - HTTP Raw Request Support\n- [pkg/protocols/headless](./pkg/protocols/headless) - Headless Module\n- [pkg/protocols/headless/engine](./pkg/protocols/headless/engine) - Internal Headless implementation\n- [pkg/protocols/dns](./pkg/protocols/dns) - DNS protocol\n- [pkg/projectfile](./pkg/projectfile) - Project File Implementation\n\n### Notes\n\n1. The matching as well as interim output functionality is a bit complex, we should simplify it a bit as well.\n","isInternal":false,"tokens":5541,"sizeBytes":25468},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/projectdiscovery/nuclei/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nNuclei is a modern, high-performance vulnerability scanner built in Go that leverages YAML-based templates for customizable vulnerability detection. It supports multiple protocols (HTTP, DNS, TCP, SSL, WebSocket, WHOIS, JavaScript, Code) and is designed for zero false positives through real-world condition simulation.\n\n## Development Commands\n\n### Building and Testing\n- `make build` - Build the main nuclei binary to ./bin/nuclei\n- `make test` - Run unit tests with race detection\n- `make integration` - Run the native integration suite via `go test -tags=integration ./internal/tests/integration`\n- `make functional` - CI-only functional suite entry point\n- `make vet` - Run go vet for code analysis\n- `make tidy` - Clean up go modules\n\n### Validation and Linting\n- `make template-validate` - Validate nuclei templates using the built binary\n- `go fmt ./...` - Format Go code\n- `go vet ./...` - Static analysis\n\n### Development Tools\n- `make devtools-all` - Build all development tools (bindgen, tsgen, scrapefuncs)\n- `make jsupdate-all` - Update JavaScript bindings and TypeScript definitions\n- `make docs` - Generate documentation\n- `make memogen` - Generate memoization code for JavaScript libraries\n\n### Testing Specific Components\n- Run single test: `go test -v ./pkg/path/to/package -run TestName`\n- Integration tests are in `internal/tests/integration/` and run via `go test -tags=integration ./internal/tests/integration`\n\n## Architecture Overview\n\n### Core Components\n- **cmd/nuclei** - Main CLI entry point with flag parsing and configuration\n- **internal/runner** - Core runner that orchestrates the entire scanning process\n- **pkg/core** - Execution engine with work pools and template clustering\n- **pkg/templates** - Template parsing, compilation, and management\n- **pkg/protocols** - Protocol implementations (HTTP, DNS, Network, etc.)\n- **pkg/operators** - Matching and extraction logic (matchers/extractors)\n- **pkg/catalog** - Template discovery and loading from disk/remote sources\n\n### Protocol Architecture\nEach protocol (HTTP, DNS, Network, etc.) implements:\n- Request interface with Compile(), ExecuteWithResults(), Match(), Extract() methods\n- Operators embedding for matching/extraction functionality\n- Protocol-specific request building and execution logic\n\n### Template System\n- Templates are YAML files defining vulnerability detection logic\n- Compiled into executable requests with operators (matchers/extractors)\n- Support for workflows (multistep template execution)\n- Template clustering optimizes identical requests across multiple templates\n\n### Key Execution Flow\n1. Template loading and compilation via pkg/catalog/loader\n2. Input provider setup for targets\n3. Engine creation with work pools for concurrency\n4. Template execution with result collection via operators\n5. Output writing and reporting integration\n\n### JavaScript Integration\n- Custom JavaScript runtime for code protocol templates\n- Auto-generated bindings in pkg/js/generated/\n- Library implementations in pkg/js/libs/\n- Development tools for binding generation in pkg/js/devtools/\n\n## Template Development\n- Templates located in separate nuclei-templates repository\n- YAML format with info, requests, and operators sections  \n- Support for multiple protocol types in single template\n- Built-in DSL functions for dynamic content generation\n- Template validation available via `make template-validate`\n\n## Key Directories\n- **lib/** - SDK for embedding nuclei as a library\n- **examples/** - Usage examples for different scenarios\n- **internal/tests/integration/** - Native integration suite and owned testdata for harness coverage\n- **internal/tests/functional/** - CI-only native functional comparison suite and testcase assets\n- **pkg/fuzz/** - Fuzzing engine and DAST capabilities\n- **pkg/input/** - Input processing for various formats (Burp, OpenAPI, etc.)\n- **pkg/reporting/** - Result export and issue tracking integrations\nSee SECURITY_CONTEXT.md for this repo's known vulnerabilities and recurring weak spots. Check it before writing or reviewing security-sensitive code.\n","isInternal":false,"tokens":873,"sizeBytes":4224},{"name":"README.md","path":"pkg/js/devtools/bindgen/README.md","rawUrl":"https://raw.githubusercontent.com/projectdiscovery/nuclei/HEAD/pkg/js/devtools/bindgen/README.md","title":"bindgen Documentation","category":"plugin-manifest","format":"markdown","content":"## bindgen (aka bindings generator)\n\nbindgen is a tool that automatically generated bindings for native go packages with 'goja'\n\nNative Go packages are available [here](../../libs/)\n\nGenerated Output is available [here](../../generated/)\n\nbindgen generates 3 different types of outputs \n\n- `go` => this directory contains corresponding goja bindings (actual bindings code) ex: [kerberos.go](../../generated/go/libkerberos/kerberos.go)\n- `js` => this is more of a javascript **representation** of all exposed functions and types etc. in javascript ex: [kerberos.js](../../generated/js/libkerberos/kerberos.js) and does not server any functional purpose other than reference\n- `markdown` => autogenerated markdown documentation for each library / package ex: [kerberos.md](../../generated/markdown/libkerberos/kerberos.md)\n\n","isInternal":false,"tokens":179,"sizeBytes":822},{"name":"README.md","path":"pkg/js/devtools/README.md","rawUrl":"https://raw.githubusercontent.com/projectdiscovery/nuclei/HEAD/pkg/js/devtools/README.md","title":"devtools Documentation","category":"plugin-manifest","format":"markdown","content":"## devtools\n\ndevtools contains tools and scripts to automate boring tasks related to javascript layer/ packages.\n\n### bindgen\n\n[bindgen](./bindgen/README.md) is a tool that automatically generated bindings for native go packages with 'goja'\n\n\n### scrapefuncs\n\n[scrapefuncs](./scrapefuncs/README.md) is a tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n\n### Generating API Reference (aka static site using javascript files using jsdoc)\n\n```console\njsdoc -R [Homepage.md] -r -d api_reference -t [optional: jsdoc theme to use] generated/js\n```\n\ngenerated static site will be available at `api_reference/` directory and can be verified using simplehttpserver\n\n```console\nsimplehttpserver\n```\n\nand then open `http://localhost:8000/` in browser\n\n\n### Notes\n\nwe currently use [clean-jsdoc-theme](https://www.npmjs.com/package/clean-jsdoc-theme) demo at [sample-jsproto-docs/](https://projectdiscovery.github.io/js-proto-docs/)","isInternal":false,"tokens":246,"sizeBytes":1014},{"name":"README.md","path":"pkg/js/devtools/scrapefuncs/README.md","rawUrl":"https://raw.githubusercontent.com/projectdiscovery/nuclei/HEAD/pkg/js/devtools/scrapefuncs/README.md","title":"scrapefuncs Documentation","category":"plugin-manifest","format":"markdown","content":"## scrapefuncs\n\nscrapefuncs is go/ast based tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n### Usage\n\n```console\nUsage of ./scrapefuncs:\n  -dir string\n    \tdirectory to process (default \"pkg/js/global\")\n  -key string\n    \topenai api key\n  -keyfile string\n    \topenai api key file\n  -out string\n    \toutput js file with declarations of all global functions\n```\n\n\n### Example\n\n```console\n$ ./scrapefuncs -keyfile ~/.openai.key                                   \n[+] Scraped 7 functions\n\nName: Rand\nSignatures: \"Rand(n int) []byte\"\nDescription: Rand returns a random byte slice of length n\n\nName: RandInt\nSignatures: \"RandInt() int\"\nDescription: RandInt returns a random int\n\nName: log\nSignatures: \"log(msg string)\"\nSignatures: \"log(msg map[string]interface{})\"\nDescription: log prints given input to stdout with [JS] prefix for debugging purposes \n\nName: getNetworkPort\nSignatures: \"getNetworkPort(port string, defaultPort string) string\"\nDescription: getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n\nName: isPortOpen\nSignatures: \"isPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isPortOpen checks if given TCP port is open on host. timeout is optional and defaults to 5 seconds\n\nName: isUDPPortOpen\nSignatures: \"isUDPPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isUDPPortOpen checks if the given UDP port is open on the host. Timeout is optional and defaults to 5 seconds.\n\nName: ToBytes\nSignatures: \"ToBytes(...interface{}) []byte\"\nDescription: ToBytes converts given input to byte slice\n\nName: ToString\nSignatures: \"ToString(...interface{}) string\"\nDescription: ToString converts given input to string\n\n\n[+] Generating jsdoc for all functions\n\n/**\n * Rand returns a random byte slice of length n\n * Rand(n int) []byte\n * @function\n * @param {number} n - The length of the byte slice.\n */\nfunction Rand(n) {\n    // implemented in go\n};\n\n/**\n * RandInt returns a random int\n * RandInt() int\n * @function\n */\nfunction RandInt() {\n    // implemented in go\n};\n\n/**\n * log prints given input to stdout with [JS] prefix for debugging purposes\n * log(msg string)\n * log(msg map[string]interface{})\n * @function\n * @param {string|Object} msg - The message to print.\n */\nfunction log(msg) {\n    // implemented in go\n};\n\n/**\n * getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n * getNetworkPort(port string, defaultPort string) string\n * @function\n * @param {string} port - The port to check.\n * @param {string} defaultPort - The default port to return if the port is colliding.\n */\nfunction getNetworkPort(port, defaultPort) {\n    // implemented in go\n};\n\n/**\n * isPortOpen checks if given port is open on host. timeout is optional and defaults to 5 seconds\n * isPortOpen(host string, port string, [timeout int]) bool\n * @function\n * @param {string} host - The host to check.\n * @param {string} port - The port to check.\n * @param {number} [timeout=5] - The timeout in seconds.\n */\nfunction isPortOpen(host, port, timeout = 5) {\n    // implemented in go\n};\n\n/**\n * ToBytes converts given input to byte slice\n * ToBytes(...interface{}) []byte\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToBytes(...args) {\n    // implemented in go\n};\n\n/**\n * ToString converts given input to string\n * ToString(...interface{}) string\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToString(...args) {\n    // implemented in go\n};\n```","isInternal":false,"tokens":893,"sizeBytes":3623},{"name":"README.md","path":"pkg/js/devtools/tsgen/README.md","rawUrl":"https://raw.githubusercontent.com/projectdiscovery/nuclei/HEAD/pkg/js/devtools/tsgen/README.md","title":"tsgen Documentation","category":"plugin-manifest","format":"markdown","content":"# tsgen\n\ntsgen is devtool to generate dummy typescript code for goja node modules written in golang. These generated typescript code can be compiled to generate .d.ts files to provide intellisense to editors like vscode and documentation for the node modules.","isInternal":false,"tokens":58,"sizeBytes":259}],"systemPromptSnippet":"<yakaai_skills repo=\"projectdiscovery/nuclei\">\n<!-- File: DESIGN.md (Tokens: ~5541 | Category: project-spec) -->\n# Nuclei Architecture Document\n\nA brief overview of Nuclei Engine architecture. This document will be kept updated as the engine progresses.\n\n## pkg/templates\n\n### Template\n\nTemplate is the basic unit of input to the engine which describes the requests to be made, matching to be done, data to extract, etc.\n\nThe template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers. \n\nAny attributes etc. required for the template, engine or requests to function are also set here.\n\nWorkflows are also compiled, their templates are loaded and compiled as well. Any validations etc. on the paths provided are also done here.\n\n`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.\n\n### Preprocessors\n\nPreprocessors are also applied here which can do things at template level. They get data of the template which they can alter at will on runtime. This is used in the engine to do random string generation.\n\nCustom processor can be used if they satisfy the following interface.\n\n```go\ntype Preprocessor interface {\n\tProcess(data []byte) []byte\n}\n```\n\n## pkg/model\n\nModel package implements Information structure for Nuclei Templates. `Info` contains all major metadata information for the template. `Classification` structure can also be used to provide additional context to vulnerability data.\n\nIt also specifies a `WorkflowLoader` interface that is used during workflow loading in template compilation stage.\n\n```go\ntype WorkflowLoader interface {\n\tGetTemplatePathsByTags(tags []string) []string\n\tGetTemplatePaths(templatesList []string, noValidate bool) []string\n}\n```\n\n## pkg/protocols\n\nProtocols package implements all the request protocols supported by Nuclei. This includes http, dns, network, headless and file requests as of now. \n\n### Request\n\nIt exposes a `Request` interface that is implemented by all the request protocols supported.\n\n```go\n// Request is an interface implemented any protocol based request generator.\ntype Request interface {\n\tCompile(options *ExecuterOptions) error\n\tRequests() int\n\tGetID() string\n\tMatch(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\tExtract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\tExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback OutputEventCallback) error\n\tMakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent\n\tMakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent\n\tGetCompiledOperators() []*operators.Operators\n}\n```\n\nMany of these methods are similar across protocols while some are very protocol specific. \n\nA brief overview of the methods is provided below -\n\n- **Compile** - Compiles the request with provided options.\n- **Requests** - Returns total requests made.\n- **GetID** - Returns any ID for the request\n- **Match** - Used to perform matching for patterns using matchers\n- **Extract** - Used to perform extraction for patterns using extractors\n- **ExecuteWithResults** - Request execution function for input.\n- **MakeResultEventItem** - Creates a single result event for the intermediate `InternalWrappedEvent` output structure.\n- **MakeResultEvent** - Returns a slice of results based on an `InternalWrappedEvent` internal output event.\n- **GetCompiledOperators** - Returns the compiled operators for the request.\n\n`MakeDefaultResultEvent` function can be used as a default for `MakeResultEvent` function when no protocol-specific features need to be implemented for result generation. \n\nFor reference protocol requests implementations, one can look at the below packages  - \n\n1. [pkg/protocols/http](./pkg/protocols/http)\n2. [pkg/protocols/dns](./pkg/protocols/dns)\n3. [pkg/protocols/network](./pkg/protocols/network)\n\n### Executer\n\nAll these different requests interfaces are converted to an Executer which is also an interface defined in `pkg/protocols` which is used during final execution of the template.\n\n```go\n// Executer is an interface implemented any protocol based request executer.\ntype Executer interface {\n\tCompile() error\n\tRequests() int\n\tExecute(input string) (bool, error)\n\tExecuteWithResults(input string, callback OutputEventCallback) error\n}\n```\n\nThe `ExecuteWithResults` function accepts a callback, which gets provided with results during execution in form of `*output.InternalWrappedEvent` structure.\n\nThe default executer is provided in `pkg/protocols/common/executer` . It takes a list of Requests and relevant `ExecuterOptions` and implements the Executer interface required for template execution. The executer during Template compilation process is created from this package and used as-is.\n\nA different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates`  We have a single HTTP  request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.\n\nFor Workflow execution, a separate RunWorkflow function is used which executes the workflow independently of the template execution.\n\nWith this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.\n\n## internal/runner\n\n### Template loading\n\nThe first process after all CLI specific initialisation is the loading of template/workflow paths that the user wants to run. This is done by the packages described below.\n\n#### pkg/catalog\n\nThis package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template and current user directory.\n\nThe syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.\n\n\n\nNext step is the initialisation of the reporting modules which is handled in `pkg/reporting`.  \n\n#### pkg/reporting\n\nReporting module contains exporters and trackers as well as a module for deduplication and a module for result formatting. \n\nExporters and Trackers are interfaces defined in pkg/reporting.\n\n```go\n// Tracker is an interface implemented by an issue tracker\ntype Tracker interface {\n\tCreateIssue(event *output.ResultEvent) error\n}\n\n// Exporter is an interface implemented by an issue exporter\ntype Exporter interface {\n\tClose() error\n\tExport(event *output.ResultEvent) error\n}\n```\n\nExporters include `Elasticsearch`, `markdown`, `sarif` . Trackers include `GitHub`, `GitLab` and `Jira`.\n\nEach exporter and trackers implement their own configuration in YAML format and are very modular in nature, so adding new ones is easy.\n\n\n\nAfter reading all the inputs from various sources and initialisation other miscellaneous options, the next bit is the output writing which is done using `pkg/output` module.\n\n#### pkg/output\n\nOutput package implements the output writing functionality for Nuclei.\n\nOutput Writer implements the Writer interface which is called each time a result is found for nuclei.\n\n```go\n// Writer is an interface which writes output to somewhere for nuclei events.\ntype Writer interface {\n\tClose()\n\tColorizer() aurora.Aurora\n\tWrite(*ResultEvent) error\n\tRequest(templateID, url, requestType string, err error)\n}\n```\n\nResultEvent structure is passed to the Nuclei Output Writer which contains the entire detail of a found result. Various intermediary types like `InternalWrappedEvent` and `InternalEvent` are used throughout nuclei protocols and matchers to describe results in various stages of execution.\n\n\n\n Interactsh is also initialised if it is not explicitly disabled. \n\n#### pkg/protocols/common/interactsh\n\nInteractsh module is used to provide automatic Out-of-Band vulnerability identification in Nuclei. \n\nIt uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.\n\nPolling for interactions and server registration only starts when a template uses the interactsh module and is executed by nuclei. After that no registration is required for the entire run.\n\n\n\n### RunEnumeration\n\nNext we arrive in the `RunEnumeration` function of the runner.\n\n`HostErrorsCache` is initialised which is used throughout the run of Nuclei enumeration to keep track of errors per host and skip further requests if the errors are greater than the provided threshold. The functionality for the error tracking cache is defined in [hosterrorscache.go](https://github.com/projectdiscovery/nuclei/blob/main/pkg/protocols/common/hosterrorscache/hosterrorscache.go) and is pretty simplistic in nature.\n\nNext the `WorkflowLoader` is initialised which used to load workflows. It exists in `pkg/parsers/workflow_loader.go`\n\nThe loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc. to return compiled `Templates` and `Workflows`. \n\n#### pkg/catalog/loader\n\nFirst the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module.  Next the path filter module is used to remove the excluded template/workflows paths.\n\n`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc. filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.\n\n`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.\n\nClustering module comes in next whose job is to cluster identical HTTP GET requests together (as a lot of the templates perform the same get requests many times, it's a good way to save many requests on large scans with lots of templates). \n\n### pkg/operators\n\nOperators package implements all the matching and extracting logic of Nuclei. \n\n```go\n// Operators contain the operators that can be applied on protocols\ntype Operators struct {\n\tMatchers []*matchers.Matcher\n\tExtractors []*extractors.Extractor\n\tMatchersCondition string\n}\n```\n\nA protocol only needs to embed the `operators.Operators` type shown above, and it can utilise all the matching/extracting functionality of nuclei.\n\n```go\n// MatchFunc performs matching operation for a matcher on model and returns true or false.\ntype MatchFunc func(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string)\n\n// ExtractFunc performs extracting operation for an extractor on model and returns true or false.\ntype ExtractFunc func(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}\n\n// Execute executes the operators on data and returns a result structure\nfunc (operators *Operators) Execute(data map[string]interface{}, match MatchFunc, extract ExtractFunc, isDebug bool) (*Result, bool) \n```\n\nThe core of this process is the Execute function which takes an input dictionary as well as a Match and Extract function and return a `Result` structure which is used later during nuclei execution to check for results.\n\n```go\n// Result is a result structure created from operators running on data.\ntype Result struct {\n\tMatched bool\n\tExtracted bool\n\tMatches map[string][]string\n\tExtracts map[string][]string\n\tOutputExtracts []string\n\tDynamicValues map[string]interface{}\n\tPayloadValues map[string]interface{}\n}\n```\n\nThe internal logics for matching and extracting for things like words, regexes, jq, paths, etc. is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.\n\n\n### Template Execution\n\n`pkg/core` provides the engine mechanism which runs the templates/workflows on inputs. It exposes an `Execute` function which does the task of execution while also doing template clustering. The clustering can also be disabled optionally by the user.\n \nAn example of using the core engine is provided below.\n\n```go\nengine := core.New(r.options)\nengine.SetExecuterOptions(executerOpts)\nresults := engine.ExecuteWithOpts(finalTemplates, r.hmapInputProvider, true)\n```\n\n### Adding a New Protocol\n\nProtocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc. are implemented in form of protocol requests.\n\nA protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.\n\nThe code for the websocket protocol is contained in `pkg/protocols/others/websocket`. \n\nBelow a high level skeleton of the websocket implementation is provided with all the important parts present.\n\n```go\npackage websocket\n\n// Request is a request for the Websocket protocol\ntype Request struct {\n\t// Operators for the current request go here.\n\toperators.Operators `yaml:\",inline,omitempty\"`\n\tCompiledOperators   *operators.Operators `yaml:\"-\"`\n\n\t// description: |\n\t//   Address contains address for the request\n\tAddress string `yaml:\"address,omitempty\" jsonschema:\"title=address for the websocket request,description=Address contains address for the request\"`\n\n    // declarations here\n}\n\n// Compile compiles the request generators preparing any requests possible.\nfunc (r *Request) Compile(options *protocols.ExecuterOptions) error {\n\tr.options = options\n\n    // request compilation here as well as client creation\n \n\tif len(r.Matchers) > 0 || len(r.Extractors) > 0 {\n\t\tcompiled := &r.Operators\n\t\tif err := compiled.Compile(); err != nil {\n\t\t\treturn errors.Wrap(err, \"could not compile operators\")\n\t\t}\n\t\tr.CompiledOperators = compiled\n\t}\n\treturn nil\n}\n\n// Requests returns the total number of requests the rule will perform\nfunc (r *Request) Requests() int {\n\tif r.generator != nil {\n\t\treturn r.generator.NewIterator().Total()\n\t}\n\treturn 1\n}\n\n// GetID returns the ID for the request if any.\nfunc (r *Request) GetID() string {\n\treturn \"\"\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) ExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n    // payloads init here\n\tif err := r.executeRequestWithPayloads(input, hostname, value, previous, callback); err != nil {\n\t\treturn err\n\t}\n\treturn nil\n}\n\n// ExecuteWithResults executes the protocol requests and returns results instead of writing them.\nfunc (r *Request) executeRequestWithPayloads(input, hostname string, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {\n\theader := http.Header{}\n\n    // make the actual request here after setting all options\n\n\tevent := eventcreator.CreateEventWithAdditionalOptions(r, data, r.options.Options.Debug || r.options.Options.DebugResponse, func(internalWrappedEvent *output.InternalWrappedEvent) {\n\t\tinternalWrappedEvent.OperatorsResult.PayloadValues = payloadValues\n\t})\n\tif r.options.Options.Debug || r.options.Options.DebugResponse {\n\t\tresponseOutput := responseBuilder.String()\n\t\tgologger.Debug().Msgf(\"[%s] Dumped Websocket response for %s\", r.options.TemplateID, input)\n\t\tgologger.Print().Msgf(\"%s\", responsehighlighter.Highlight(event.OperatorsResult, responseOutput, r.options.Options.NoColor))\n\t}\n\n\tcallback(event)\n\treturn nil\n}\n\nfunc (r *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent {\n\tdata := &output.ResultEvent{\n\t\tTemplateID:       types.ToString(r.options.TemplateID),\n\t\tTemplatePath:     types.ToString(r.options.TemplatePath),\n\t\t// ... setting more values for result event\n\t}\n\treturn data\n}\n\n// Match performs matching operation for a matcher on model and returns:\n// true and a list of matched snippets if the matcher type is supports it\n// otherwise false and an empty string slice\nfunc (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string) {\n\treturn protocols.MakeDefaultMatchFunc(data, matcher)\n}\n\n// Extract performs extracting operation for an extractor on model and returns true or false.\nfunc (r *Request) Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{} {\n\treturn protocols.MakeDefaultExtractFunc(data, matcher)\n}\n\n// MakeResultEvent creates a result event from internal wrapped event\nfunc (r *Request) MakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent {\n\treturn protocols.MakeDefaultResultEvent(r, wrapped)\n}\n\n// GetCompiledOperators returns a list of the compiled operators\nfunc (r *Request) GetCompiledOperators() []*operators.Operators {\n\treturn []*operators.Operators{r.CompiledOperators}\n}\n\n// Type returns the type of the protocol request\nfunc (r *Request) Type() templateTypes.ProtocolType {\n\treturn templateTypes.WebsocketProtocol\n}\n```\n\nAlmost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, `GetCompiledOperators`, etc. which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.\n\n`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.\n\nStep by step description of how to add a new protocol to Nuclei - \n\n1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with fewer options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `pkg/templates/types/types.go`.\n\n2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.\n\n```go\n\nimport (\n\t...\n\t\"github.com/projectdiscovery/nuclei/v3/pkg/protocols/others/websocket\"\n)\n\n// Template is a YAML input file which defines all the requests and\n// other metadata for a template.\ntype Template struct {\n\t...\n\t// description: |\n\t//   Websocket contains the Websocket request to make in the template.\n\tRequestsWebsocket []*websocket.Request `yaml:\"websocket,omitempty\" json:\"websocket,omitempty\" jsonschema:\"title=websocket requests to make,description=Websocket requests to make for the template\"`\n\t...\n}\n```\n\nAlso add the protocol case to the `Type` function as well as the `TemplateTypes` array in the same `templates.go` file.\n\n```go\n// TemplateTypes is a list of accepted template types\nvar TemplateTypes = []string{\n\t...\n\t\"websocket\",\n}\n\n// Type returns the type of the template\nfunc (t *Template) Type() templateTypes.ProtocolType {\n\t...\n\tcase len(t.RequestsWebsocket) > 0:\n\t\treturn templateTypes.WebsocketProtocol\n\tdefault:\n\t\treturn \"\"\n\t}\n}\n```\n\n3. Add the protocol request to the `Requests` function and `compileProtocolRequests` function in the `compile.go` file in same directory.\n\n```go\n\n// Requests return the total request count for the template\nfunc (template *Template) Requests() int {\n\treturn len(template.RequestsDNS) +\n\t\t...\n\t\tlen(template.RequestsSSL) +\n\t\tlen(template.RequestsWebsocket)\n}\n\n\n// compileProtocolRequests compiles all the protocol requests for the template\nfunc (template *Template) compileProtocolRequests(options protocols.ExecuterOptions) error {\n\t...\n\n\tcase len(template.RequestsWebsocket) > 0:\n\t\trequests = template.convertRequestToProtocolsRequest(template.RequestsWebsocket)\n\t}\n\ttemplate.Executer = executer.NewExecuter(requests, &options)\n\treturn nil\n}\n```\n\nThat's it, you've added a new protocol to Nuclei. The next good step would be to add native integration coverage under `internal/tests/integration` and run it with `go test -tags=integration ./internal/tests/integration`.\n\n\n## Profiling and Tracing\n\nTo analyze Nuclei's performance and resource usage, you can generate CPU & memory profiles and trace files using the `-profile-mem` flag:\n\n```bash\nnuclei -t nuclei-templates/ -u https://example.com -profile-mem=nuclei-$(git describe --tags)\n```\n\nThis command creates three files:\n\n* `nuclei.cpu`: CPU profile\n* `nuclei.mem`: Memory (heap) profile\n* `nuclei.trace`: Execution trace\n\n### Analyzing the CPU/Memory Profiles\n\n* View the profile in the terminal:\n\n```bash\ngo tool pprof nuclei.{cpu,mem}\n```\n\n* Display overall CPU time for processing $$N$$ targets:\n\n```\ngo tool pprof -top nuclei.cpu | grep \"Total samples\"\n```\n\n* Display top memory consumers:\n\n```bash\ngo tool pprof -top nuclei.mem | grep \"$(go list -m)\" | head -10\n```\n\n* Visualize the profile in a web browser:\n\n```bash\ngo tool pprof -http=:$(shuf -i 1000-99999 -n 1) nuclei.{cpu,mem}\n```\n\n### Analyzing the Trace File\n\nTo examine the execution trace:\n\n```bash\ngo tool trace nuclei.trace\n```\n\nThese tools help identify performance bottlenecks and memory leaks, allowing for targeted optimizations of Nuclei's codebase.\n\n## Project Structure\n\n- [pkg/reporting](./pkg/reporting) - Reporting modules for nuclei.\n- [pkg/reporting/exporters/sarif](./pkg/reporting/exporters/sarif) - Sarif Result Exporter\n- [pkg/reporting/exporters/markdown](./pkg/reporting/exporters/markdown) - Markdown Result Exporter\n- [pkg/reporting/exporters/es](./pkg/reporting/exporters/es) - Elasticsearch Result Exporter\n- [pkg/reporting/dedupe](./pkg/reporting/dedupe) - Dedupe module for Results\n- [pkg/reporting/trackers/gitlab](./pkg/reporting/trackers/gitlab) - GitLab Issue Tracker Exporter\n- [pkg/reporting/trackers/jira](./pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter\n- [pkg/reporting/trackers/github](./pkg/reporting/trackers/github) - GitHub Issue Tracker Exporter\n- [pkg/reporting/format](./pkg/reporting/format) - Result Formatting Functions\n- [pkg/parsers](./pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.\n- [pkg/types](./pkg/types) - Contains CLI options as well as misc helper functions.\n- [pkg/progress](./pkg/progress) - Progress tracking\n- [pkg/operators](./pkg/operators) - Operators for Nuclei\n- [pkg/operators/common/dsl](./pkg/operators/common/dsl) - DSL functions for Nuclei YAML Syntax\n- [pkg/operators/matchers](./pkg/operators/matchers) - Matchers implementation\n- [pkg/operators/extractors](./pkg/operators/extractors) - Extractors implementation\n- [pkg/catalog](./pkg/catalog) - Template loading from disk helpers\n- [pkg/catalog/config](./pkg/catalog/config) - Internal configuration management\n- [pkg/catalog/loader](./pkg/catalog/loader) - Implements loading and validation of templates and workflows.\n- [pkg/catalog/loader/filter](./pkg/catalog/loader/filter) - Filter filters templates based on tags and paths\n- [pkg/output](./pkg/output) - Output module for nuclei\n- [pkg/workflows](./pkg/workflows) - Workflow execution logic + declarations\n- [pkg/utils](./pkg/utils) - Utility functions\n- [pkg/model](./pkg/model) - Template Info + misc\n- [pkg/templates](./pkg/templates) - Templates core starting point\n- [pkg/templates/cache](./pkg/templates/cache) - Templates cache\n- [pkg/protocols](./pkg/protocols) - Protocol Specification\n- [pkg/protocols/file](./pkg/protocols/file) - File protocol\n- [pkg/protocols/network](./pkg/protocols/network) - Network protocol\n- [pkg/protocols/common/expressions](./pkg/protocols/common/expressions) - Expression evaluation + Templating Support\n- [pkg/protocols/common/interactsh](./pkg/protocols/common/interactsh) - Interactsh integration\n- [pkg/protocols/common/generators](./pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc.)\n- [pkg/protocols/common/executer](./pkg/protocols/common/executer) - Default Template Executer\n- [pkg/protocols/common/replacer](./pkg/protocols/common/replacer) - Template replacement helpers\n- [pkg/protocols/common/helpers/eventcreator](./pkg/protocols/common/helpers/eventcreator) - Result event creator\n- [pkg/protocols/common/helpers/responsehighlighter](./pkg/protocols/common/helpers/responsehighlighter) - Debug response highlighter\n- [pkg/protocols/common/helpers/deserialization](./pkg/protocols/common/helpers/deserialization) - Deserialization helper functions\n- [pkg/protocols/common/hosterrorscache](./pkg/protocols/common/hosterrorscache) - Host errors cache for tracking erroring hosts\n- [pkg/protocols/offlinehttp](./pkg/protocols/offlinehttp) - Offline http protocol\n- [pkg/protocols/http](./pkg/protocols/http) - HTTP protocol\n- [pkg/protocols/http/race](./pkg/protocols/http/race) - HTTP Race Module\n- [pkg/protocols/http/raw](./pkg/protocols/http/raw) - HTTP Raw Request Support\n- [pkg/protocols/headless](./pkg/protocols/headless) - Headless Module\n- [pkg/protocols/headless/engine](./pkg/protocols/headless/engine) - Internal Headless implementation\n- [pkg/protocols/dns](./pkg/protocols/dns) - DNS protocol\n- [pkg/projectfile](./pkg/projectfile) - Project File Implementation\n\n### Notes\n\n1. The matching as well as interim output functionality is a bit complex, we should simplify it a bit as well.\n\n<!-- File: CLAUDE.md (Tokens: ~873 | Category: claude-rule) -->\n# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nNuclei is a modern, high-performance vulnerability scanner built in Go that leverages YAML-based templates for customizable vulnerability detection. It supports multiple protocols (HTTP, DNS, TCP, SSL, WebSocket, WHOIS, JavaScript, Code) and is designed for zero false positives through real-world condition simulation.\n\n## Development Commands\n\n### Building and Testing\n- `make build` - Build the main nuclei binary to ./bin/nuclei\n- `make test` - Run unit tests with race detection\n- `make integration` - Run the native integration suite via `go test -tags=integration ./internal/tests/integration`\n- `make functional` - CI-only functional suite entry point\n- `make vet` - Run go vet for code analysis\n- `make tidy` - Clean up go modules\n\n### Validation and Linting\n- `make template-validate` - Validate nuclei templates using the built binary\n- `go fmt ./...` - Format Go code\n- `go vet ./...` - Static analysis\n\n### Development Tools\n- `make devtools-all` - Build all development tools (bindgen, tsgen, scrapefuncs)\n- `make jsupdate-all` - Update JavaScript bindings and TypeScript definitions\n- `make docs` - Generate documentation\n- `make memogen` - Generate memoization code for JavaScript libraries\n\n### Testing Specific Components\n- Run single test: `go test -v ./pkg/path/to/package -run TestName`\n- Integration tests are in `internal/tests/integration/` and run via `go test -tags=integration ./internal/tests/integration`\n\n## Architecture Overview\n\n### Core Components\n- **cmd/nuclei** - Main CLI entry point with flag parsing and configuration\n- **internal/runner** - Core runner that orchestrates the entire scanning process\n- **pkg/core** - Execution engine with work pools and template clustering\n- **pkg/templates** - Template parsing, compilation, and management\n- **pkg/protocols** - Protocol implementations (HTTP, DNS, Network, etc.)\n- **pkg/operators** - Matching and extraction logic (matchers/extractors)\n- **pkg/catalog** - Template discovery and loading from disk/remote sources\n\n### Protocol Architecture\nEach protocol (HTTP, DNS, Network, etc.) implements:\n- Request interface with Compile(), ExecuteWithResults(), Match(), Extract() methods\n- Operators embedding for matching/extraction functionality\n- Protocol-specific request building and execution logic\n\n### Template System\n- Templates are YAML files defining vulnerability detection logic\n- Compiled into executable requests with operators (matchers/extractors)\n- Support for workflows (multistep template execution)\n- Template clustering optimizes identical requests across multiple templates\n\n### Key Execution Flow\n1. Template loading and compilation via pkg/catalog/loader\n2. Input provider setup for targets\n3. Engine creation with work pools for concurrency\n4. Template execution with result collection via operators\n5. Output writing and reporting integration\n\n### JavaScript Integration\n- Custom JavaScript runtime for code protocol templates\n- Auto-generated bindings in pkg/js/generated/\n- Library implementations in pkg/js/libs/\n- Development tools for binding generation in pkg/js/devtools/\n\n## Template Development\n- Templates located in separate nuclei-templates repository\n- YAML format with info, requests, and operators sections  \n- Support for multiple protocol types in single template\n- Built-in DSL functions for dynamic content generation\n- Template validation available via `make template-validate`\n\n## Key Directories\n- **lib/** - SDK for embedding nuclei as a library\n- **examples/** - Usage examples for different scenarios\n- **internal/tests/integration/** - Native integration suite and owned testdata for harness coverage\n- **internal/tests/functional/** - CI-only native functional comparison suite and testcase assets\n- **pkg/fuzz/** - Fuzzing engine and DAST capabilities\n- **pkg/input/** - Input processing for various formats (Burp, OpenAPI, etc.)\n- **pkg/reporting/** - Result export and issue tracking integrations\nSee SECURITY_CONTEXT.md for this repo's known vulnerabilities and recurring weak spots. Check it before writing or reviewing security-sensitive code.\n\n<!-- File: pkg/js/devtools/bindgen/README.md (Tokens: ~179 | Category: plugin-manifest) -->\n## bindgen (aka bindings generator)\n\nbindgen is a tool that automatically generated bindings for native go packages with 'goja'\n\nNative Go packages are available [here](../../libs/)\n\nGenerated Output is available [here](../../generated/)\n\nbindgen generates 3 different types of outputs \n\n- `go` => this directory contains corresponding goja bindings (actual bindings code) ex: [kerberos.go](../../generated/go/libkerberos/kerberos.go)\n- `js` => this is more of a javascript **representation** of all exposed functions and types etc. in javascript ex: [kerberos.js](../../generated/js/libkerberos/kerberos.js) and does not server any functional purpose other than reference\n- `markdown` => autogenerated markdown documentation for each library / package ex: [kerberos.md](../../generated/markdown/libkerberos/kerberos.md)\n\n<!-- File: pkg/js/devtools/README.md (Tokens: ~246 | Category: plugin-manifest) -->\n## devtools\n\ndevtools contains tools and scripts to automate boring tasks related to javascript layer/ packages.\n\n### bindgen\n\n[bindgen](./bindgen/README.md) is a tool that automatically generated bindings for native go packages with 'goja'\n\n\n### scrapefuncs\n\n[scrapefuncs](./scrapefuncs/README.md) is a tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n\n### Generating API Reference (aka static site using javascript files using jsdoc)\n\n```console\njsdoc -R [Homepage.md] -r -d api_reference -t [optional: jsdoc theme to use] generated/js\n```\n\ngenerated static site will be available at `api_reference/` directory and can be verified using simplehttpserver\n\n```console\nsimplehttpserver\n```\n\nand then open `http://localhost:8000/` in browser\n\n\n### Notes\n\nwe currently use [clean-jsdoc-theme](https://www.npmjs.com/package/clean-jsdoc-theme) demo at [sample-jsproto-docs/](https://projectdiscovery.github.io/js-proto-docs/)\n\n<!-- File: pkg/js/devtools/scrapefuncs/README.md (Tokens: ~893 | Category: plugin-manifest) -->\n## scrapefuncs\n\nscrapefuncs is go/ast based tool to scrapes all helper functions exposed in javascript with help of go/ast and generates a js file with jsdoc comments using LLM (OpenAI)\n\n### Usage\n\n```console\nUsage of ./scrapefuncs:\n  -dir string\n    \tdirectory to process (default \"pkg/js/global\")\n  -key string\n    \topenai api key\n  -keyfile string\n    \topenai api key file\n  -out string\n    \toutput js file with declarations of all global functions\n```\n\n\n### Example\n\n```console\n$ ./scrapefuncs -keyfile ~/.openai.key                                   \n[+] Scraped 7 functions\n\nName: Rand\nSignatures: \"Rand(n int) []byte\"\nDescription: Rand returns a random byte slice of length n\n\nName: RandInt\nSignatures: \"RandInt() int\"\nDescription: RandInt returns a random int\n\nName: log\nSignatures: \"log(msg string)\"\nSignatures: \"log(msg map[string]interface{})\"\nDescription: log prints given input to stdout with [JS] prefix for debugging purposes \n\nName: getNetworkPort\nSignatures: \"getNetworkPort(port string, defaultPort string) string\"\nDescription: getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n\nName: isPortOpen\nSignatures: \"isPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isPortOpen checks if given TCP port is open on host. timeout is optional and defaults to 5 seconds\n\nName: isUDPPortOpen\nSignatures: \"isUDPPortOpen(host string, port string, [timeout int]) bool\"\nDescription: isUDPPortOpen checks if the given UDP port is open on the host. Timeout is optional and defaults to 5 seconds.\n\nName: ToBytes\nSignatures: \"ToBytes(...interface{}) []byte\"\nDescription: ToBytes converts given input to byte slice\n\nName: ToString\nSignatures: \"ToString(...interface{}) string\"\nDescription: ToString converts given input to string\n\n\n[+] Generating jsdoc for all functions\n\n/**\n * Rand returns a random byte slice of length n\n * Rand(n int) []byte\n * @function\n * @param {number} n - The length of the byte slice.\n */\nfunction Rand(n) {\n    // implemented in go\n};\n\n/**\n * RandInt returns a random int\n * RandInt() int\n * @function\n */\nfunction RandInt() {\n    // implemented in go\n};\n\n/**\n * log prints given input to stdout with [JS] prefix for debugging purposes\n * log(msg string)\n * log(msg map[string]interface{})\n * @function\n * @param {string|Object} msg - The message to print.\n */\nfunction log(msg) {\n    // implemented in go\n};\n\n/**\n * getNetworkPort registers defaultPort and returns defaultPort if it is a colliding port with other protocols\n * getNetworkPort(port string, defaultPort string) string\n * @function\n * @param {string} port - The port to check.\n * @param {string} defaultPort - The default port to return if the port is colliding.\n */\nfunction getNetworkPort(port, defaultPort) {\n    // implemented in go\n};\n\n/**\n * isPortOpen checks if given port is open on host. timeout is optional and defaults to 5 seconds\n * isPortOpen(host string, port string, [timeout int]) bool\n * @function\n * @param {string} host - The host to check.\n * @param {string} port - The port to check.\n * @param {number} [timeout=5] - The timeout in seconds.\n */\nfunction isPortOpen(host, port, timeout = 5) {\n    // implemented in go\n};\n\n/**\n * ToBytes converts given input to byte slice\n * ToBytes(...interface{}) []byte\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToBytes(...args) {\n    // implemented in go\n};\n\n/**\n * ToString converts given input to string\n * ToString(...interface{}) string\n * @function\n * @param {...any} args - The input to convert.\n */\nfunction ToString(...args) {\n    // implemented in go\n};\n```\n\n<!-- File: pkg/js/devtools/tsgen/README.md (Tokens: ~58 | Category: plugin-manifest) -->\n# tsgen\n\ntsgen is devtool to generate dummy typescript code for goja node modules written in golang. These generated typescript code can be compiled to generate .d.ts files to provide intellisense to editors like vscode and documentation for the node modules.\n</yakaai_skills>"}