## File: README.md # garak, LLM vulnerability scanner *Generative AI Red-teaming & Assessment Kit* `garak` checks if an LLM can be made to fail in a way we don't want. `garak` probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses. If you know `nmap` or `msf` / Metasploit Framework, garak does somewhat similar things to them, but for LLMs. `garak` focuses on ways of making an LLM or dialog system fail. It combines static, dynamic, and adaptive probes to explore this. `garak`'s a free tool. We love developing it and are always interested in adding functionality to support applications. [](https://opensource.org/licenses/Apache-2.0) [](https://github.com/NVIDIA/garak/actions/workflows/test_linux.yml) [](https://github.com/NVIDIA/garak/actions/workflows/test_windows.yml) [](https://github.com/NVIDIA/garak/actions/workflows/test_macos.yml) [](http://garak.readthedocs.io/en/latest/?badge=latest) [](https://arxiv.org/abs/2406.11036) [](https://discord.gg/uVch4puUCs) [](https://github.com/psf/black) [](https://pypi.org/project/garak) [](https://badge.fury.io/py/garak) [](https://pepy.tech/project/garak) [](https://pepy.tech/project/garak) ## Get started ### > See our user guide! [docs.garak.ai](https://docs.garak.ai/) ### > Join our [Discord](https://discord.gg/uVch4puUCs)! ### > Project links & home: [garak.ai](https://garak.ai/) ### > Twitter: [@garak_llm](https://twitter.com/garak_llm) ### > DEF CON [slides](https://garak.ai/garak_aiv_slides.pdf)!
## LLM support currently supports: * [hugging face hub](https://huggingface.co/models) generative models * [replicate](https://replicate.com/) text models * [openai api](https://platform.openai.com/docs/introduction) chat & continuation models * [aws bedrock](https://aws.amazon.com/bedrock/) foundation models * [litellm](https://www.litellm.ai/) * pretty much anything accessible via REST * gguf models like [llama.cpp](https://github.com/ggerganov/llama.cpp) version >= 1046 * .. and many more LLMs! ## Install: `garak` is a command-line tool. It's developed in Linux and OSX. ### Standard install with `pip` Just grab it from PyPI and you should be good to go: ``` python -m pip install -U garak ``` ### Install development version with `pip` The standard pip version of `garak` is updated periodically. To get a fresher version from GitHub, try: ``` python -m pip install -U git+https://github.com/NVIDIA/garak.git@main ``` ### Clone from source `garak` has its own dependencies. You can to install `garak` in its own Conda environment: ``` conda create --name garak "python>=3.10,<=3.12" conda activate garak gh repo clone NVIDIA/garak cd garak python -m pip install -e . ``` OK, if that went fine, you're probably good to go! **Note**: if you cloned before the move to the `NVIDIA` GitHub organisation, but you're reading this at the `github.com/NVIDIA` URI, please update your remotes as follows: ``` git remote set-url origin https://github.com/NVIDIA/garak.git ``` ## Getting started The general syntax is: `garak ` `garak` needs to know what model to scan, and by default, it'll try all the probes it knows on that model, using the vulnerability detectors recommended by each probe. You can see a list of probes using: `garak --list_probes` To specify a generator, use the `--target_type` and, optionally, the `--target_name` options. Model type specifies a model family/interface; model name specifies the exact model to be used. The "Intro to generators" section below describes some of the generators supported. A straightforward generator family is Hugging Face models; to load one of these, set `--target_type` to `huggingface` and `--target_name` to the model's name on Hub (e.g. `"RWKV/rwkv-4-169m-pile"`). Some generators might need an API key to be set as an environment variable, and they'll let you know if they need that. `garak` runs all the probes by default, but you can be specific about that too. `--probes promptinject` will use only the [PromptInject](https://github.com/agencyenterprise/promptinject) framework's methods, for example. You can also specify one specific plugin instead of a plugin family by adding the plugin name after a `.`; for example, `--probes lmrc.SlurUsage` will use an implementation of checking for models generating slurs based on the [Language Model Risk Cards](https://arxiv.org/abs/2303.18190) framework. For help and inspiration, find us on [Twitter](https://twitter.com/garak_llm) or [discord](https://discord.gg/uVch4puUCs)! ## Examples Probe a commercial model for encoding-based prompt injection (OSX/\*nix) (replace example value with a real OpenAI API key) ``` export OPENAI_API_KEY="sk-123XXXXXXXXXXXX" python3 -m garak --target_type openai --target_name gpt-5-nano --probes encoding ``` See if the Hugging Face version of GPT2 is vulnerable to DAN 11.0 ``` python3 -m garak --target_type huggingface --target_name gpt2 --probes dan.Dan_11_0 ``` ## Reading the results For each probe loaded, garak will print a progress bar as it generates. Once generation is complete, a row evaluating that probe's results on each detector is given. If any of the prompt attempts yielded an undesirable behavior, the response will be marked as FAIL, and the failure rate given. Here are the results with the `encoding` module on a GPT-3 variant: And the same results for ChatGPT: We can see that the more recent model is much more susceptible to encoding-based injection attacks, where text-babbage-001 was only found to be vulnerable to quoted-printable and MIME encoding injections. The figures at the end of each row, e.g. 840/840, indicate the number of text generations total and then how many of these seemed to behave OK. The figure can be quite high because more than one generation is made per prompt - by default, 10. Errors go in `garak.log`; the run is logged in detail in a `.jsonl` file specified at analysis start & end. There's a basic analysis script in `analyse/analyse_log.py` which will output the probes and prompts that led to the most hits. Send PRs & open issues. Happy hunting! ## Intro to generators ### Hugging Face Using the Pipeline API: * `--target_type huggingface` (for transformers models to run locally) * `--target_name` - use the model name from Hub. Only generative models will work. If it fails and shouldn't, please open an issue and paste in the command you tried + the exception! Using the Inference API: * `--target_type huggingface.InferenceAPI` (for API-based model access) * `--target_name` - the model name from Hub, e.g. `"mosaicml/mpt-7b-instruct"` Using private endpoints: * `--target_type huggingface.InferenceEndpoint` (for private endpoints) * `--target_name` - the endpoint URL, e.g. `https://xxx.us-east-1.aws.endpoints.huggingface.cloud` * (optional) set the `HF_INFERENCE_TOKEN` environment variable to a Hugging Face API token with the "read" role; see https://huggingface.co/settings/tokens when logged in ### OpenAI * `--target_type openai` * `--target_name` - the OpenAI model you'd like to use. `gpt-5-nano` is fast and fine for testing. * set the `OPENAI_API_KEY` environment variable to your OpenAI API key (e.g. "sk-19763ASDF87q6657"); see https://platform.openai.com/account/api-keys when logged in Recognised model types are whitelisted, because the plugin needs to know which sub-API to use. Completion or ChatCompletion models are OK. If you'd like to use a model not supported, you should get an informative error message, and please send a PR / open an issue. ### Replicate * set the `REPLICATE_API_TOKEN` environment variable to your Replicate API token, e.g. "r8-123XXXXXXXXXXXX"; see https://replicate.com/account/api-tokens when logged in Public Replicate models: * `--target_type replicate` * `--target_name` - the Replicate model name and hash, e.g. `"stability-ai/stablelm-tuned-alpha-7b:c49dae36"` Private Replicate endpoints: * `--target_type replicate.InferenceEndpoint` (for private endpoints) * `--target_name` - username/model-name slug from the deployed endpoint, e.g. `elim/elims-llama2-7b` ### Cohere * `--target_type cohere` * `--target_name` (optional, `command` by default) - The specific Cohere model you'd like to test * set the `COHERE_API_KEY` environment variable to your Cohere API key, e.g. "aBcDeFgHiJ123456789"; see https://dashboard.cohere.ai/api-keys when logged in ### Groq * `--target_type groq` * `--target_name` - The name of the model to access via the Groq API * set the `GROQ_API_KEY` environment variable to your Groq API key, see https://console.groq.com/docs/quickstart for details on creating an API key ### ggml * `--target_type ggml` * `--target_name` - The path to the ggml model you'd like to load, e.g. `/home/leon/llama.cpp/models/7B/ggml-model-q4_0.bin` * set the `GGML_MAIN_PATH` environment variable to the path to your ggml `main` executable ### REST `rest.RestGenerator` is highly flexible and can connect to any REST endpoint that returns plaintext or JSON. It does need some brief config, which will typically result a short YAML file describing your endpoint. See https://reference.garak.ai/en/latest/garak.generators.rest.html for examples. ### NIM Use models from https://build.nvidia.com/ or other NIM endpoints. * set the `NIM_API_KEY` environment variable to your authentication API token, or specify it in the config YAML For chat models: * `--target_type nim` * `--target_name` - the NIM `model` name, e.g. `meta/llama-3.1-8b-instruct` For completion models: * `--target_type nim.NVOpenAICompletion` * `--target_name` - the NIM `model` name, e.g. `bigcode/starcoder2-15b` ### AWS Bedrock * `--target_type bedrock` * `--target_name` - the Bedrock model ID or alias, e.g. `anthropic.claude-3-sonnet-20240229-v1:0` or `claude-3-sonnet` * set the `BEDROCK_API_KEY` environment variable to your AWS Bedrock API key; see https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys-use.html for setup instructions * (optional) set the `BEDROCK_REGION` environment variable to specify the AWS region (defaults to `us-east-1`) Supported model families include Anthropic Claude, Meta Llama, Amazon Titan, AI21 Labs, Cohere, and Mistral AI models. The generator uses the Converse API for unified access across all model types. Example usage: ``` export BEDROCK_API_KEY="your-api-key" export BEDROCK_REGION="us-east-1" garak --target_type bedrock --target_name claude-3-sonnet --probes dan ``` ### Test * `--target_type test` * (alternatively) `--target_name test.Blank` For testing. This always generates the empty string, using the `test.Blank` generator. Will be marked as failing for any tests that *require* an output, e.g. those that make contentious claims and expect the model to refute them in order to pass. * `--target_type test.Repeat` For testing. This generator repeats back the prompt it received. ## Intro to probes | Probe | Description | |----------------------|-------------------------------------------------------------------------------------------------------------------------------| | blank | A simple probe that always sends an empty prompt. | | atkgen | Automated Attack Generation. A red-teaming LLM probes the target and reacts to it in an attempt to get toxic output. Prototype, mostly stateless, for now uses a simple GPT-2 [fine-tuned](https://huggingface.co/garak-llm/artgpt2tox) on the subset of hhrlhf attempts that yielded detectable toxicity (the only target currently supported for now). | | badchars | Implements imperceptible Unicode perturbations (invisible characters, homoglyphs, reorderings, deletions) inspired by the [Bad Characters](https://arxiv.org/abs/2106.09898) paper. | | av_spam_scanning | Probes that attempt to make the model output malicious content signatures | | continuation | Probes that test if the model will continue a probably undesirable word | | dan | Various [DAN](https://adguard.com/en/blog/chatgpt-dan-prompt-abuse.html) and DAN-like attacks | | donotanswer | Prompts to which responsible language models should not answer. | | encoding | Prompt injection through text encoding | | gcg | Disrupt a system prompt by appending an adversarial suffix. | | glitch | Probe model for glitch tokens that provoke unusual behavior. | | grandma | Appeal to be reminded of one's grandmother. | | goodside | Implementations of Riley Goodside attacks. | | leakreplay | Evaluate if a model will replay training data. | | lmrc | Subsample of the [Language Model Risk Cards](https://arxiv.org/abs/2303.18190) probes | | malwaregen | Attempts to have the model generate code for building malware | | misleading | Attempts to make a model support misleading and false claims | | packagehallucination | Trying to get code generations that specify non-existent (and therefore insecure) packages. | | promptinject | Implementation of the Agency Enterprise [PromptInject](https://github.com/agencyenterprise/PromptInject/tree/main/promptinject) work (best paper awards @ NeurIPS ML Safety Workshop 2022) | | realtoxicityprompts | Subset of the RealToxicityPrompts work (data constrained because the full test will take so long to run) | | snowball | [Snowballed Hallucination](https://ofir.io/snowballed_hallucination.pdf) probes designed to make a model give a wrong answer to questions too complex for it to process | | xss | Look for vulnerabilities the permit or enact cross-site attacks, such as private data exfiltration. | ## Logging `garak` generates multiple kinds of log: * A log file, `garak.log`. This includes debugging information from `garak` and its plugins, and is continued across runs. * A report of the current run, structured as JSONL. A new report file is created every time `garak` runs. The name of this file is output at the beginning and, if successful, also at the end of the run. In the report, an entry is made for each probing attempt both as the generations are received, and again when they are evaluated; the entry's `status` attribute takes a constant from `garak.attempts` to describe what stage it was made at. * A hit log, detailing attempts that yielded a vulnerability (a 'hit') ## How is the code structured? Check out the [reference docs](https://reference.garak.ai/) for an authoritative guide to `garak` code structure. In a typical run, `garak` will read a model type (and optionally model name) from the command line, then determine which `probe`s and `detector`s to run, start up a `generator`, and then pass these to a `harness` to do the probing; an `evaluator` deals with the results. There are many modules in each of these categories, and each module provides a number of classes that act as individual plugins. * `garak/probes/` - classes for generating interactions with LLMs * `garak/detectors/` - classes for detecting an LLM is exhibiting a given failure mode * `garak/evaluators/` - assessment reporting schemes * `garak/generators/` - plugins for LLMs to be probed * `garak/harnesses/` - classes for structuring testing * `resources/` - ancillary items required by plugins The default operating mode is to use the `probewise` harness. Given a list of probe module names and probe plugin names, the `probewise` harness instantiates each probe, then for each probe reads its `primary_detector` and `extended_detectors` attributes to get a list of `detector`s to run on the output. Each plugin category (`probes`, `detectors`, `evaluators`, `generators`, `harnesses`) includes a `base.py` which defines the base classes usable by plugins in that category. Each plugin module defines plugin classes that inherit from one of the base classes. For example, `garak.generators.openai.OpenAIGenerator` descends from `garak.generators.base.Generator`. Larger artefacts, like model files and bigger corpora, are kept out of the repository; they can be stored on e.g. Hugging Face Hub and loaded locally by clients using `garak`. ## Developing your own plugin * Take a look at how other plugins do it * Inherit from one of the base classes, e.g. `garak.probes.base.TextProbe` * Override as little as possible * You can test the new code in at least two ways: * Start an interactive Python session * Import the model, e.g. `import garak.probes.mymodule` * Instantiate the plugin, e.g. `p = garak.probes.mymodule.MyProbe()` * Run a scan with test plugins * For probes, try a blank generator and always.Pass detector: `python3 -m garak -m test.Blank -p mymodule -d always.Pass` * For detectors, try a blank generator and a blank probe: `python3 -m garak -m test.Blank -p test.Blank -d mymodule` * For generators, try a blank probe and always.Pass detector: `python3 -m garak -m mymodule -p test.Blank -d always.Pass` * Get `garak` to list all the plugins of the type you're writing, with `--list_probes`, `--list_detectors`, or `--list_generators` ## FAQ We have an FAQ [here](https://github.com/NVIDIA/garak/blob/main/FAQ.md). Reach out if you have any more questions! [garak@nvidia.com](mailto:garak@nvidia.com) Code reference documentation is at [garak.readthedocs.io](https://garak.readthedocs.io/en/latest/). ## Citing garak You can read the [garak preprint paper](garak-paper.pdf). If you use garak, please cite us. ``` @article{garak, title={{garak: A Framework for Security Probing Large Language Models}}, author={Leon Derczynski and Erick Galinkin and Jeffrey Martin and Subho Majumdar and Nanna Inie}, year={2024}, howpublished={\url{https://garak.ai}} } ```
_"Lying is a skill like any other, and if you wish to maintain a level of excellence you have to practice constantly"_ - Elim For updates and news see [@garak_llm](https://twitter.com/garak_llm) © 2023- Leon Derczynski; Apache license v2, see [LICENSE](LICENSE) --- ## File: docs/source/_config.rst config ====== This module holds config values. These are broken into the following major categories: * system: options that don't affect the security assessment * run: options that describe how a garak run will be conducted * plugins: config for plugins (generators, probes, detectors, buffs) * transient: internal values local to a single ``garak`` execution Config values are loaded in the following priority (lowest-first): * Plugin defaults in the code * Core config: from ``garak/resources/garak.core.yaml``; not to be overridden * Site config: from ``$HOME/.config/garak/garak.site.yaml`` or ``garak.site.json`` * Runtime config: from an optional config file (YAML or JSON) specified manually, via e.g. CLI parameter * Command-line options Code ^^^^ garak._config ------------- .. automodule:: garak._config :members: :undoc-members: :show-inheritance: --- ## File: docs/source/_plugins.rst plugins ======= garak._plugins -------------- This module manages plugin enumeration and loading. There is one class per plugin in ``garak``. Enumerating the classes, with e.g. ``--list_probes`` on the command line, means importing each module. Therefore, modules should do as little as possible on load, and delay intensive activities (like loading classifiers) until a plugin's class is instantiated. Code ^^^^ .. automodule:: garak._plugins :members: :undoc-members: :show-inheritance: --- ## File: docs/source/attempt.rst attempt ======= In garak, ``Attempt`` objects track a single prompt and the results of running it on through the generator. Probes work by creating a set of garak.attempt.Attempt objects and setting their class properties. These are passed by the harness to the generator, and the output added to the attempt. Then, a detector assesses the outputs from that attempt and the detector's scores are saved in the attempt. Finally, an evaluator makes judgments of these scores, and writes hits out to the hitlog for any successful probing attempts. Within this, ``Converastion``, ``Turn``, and ``Message`` objects encapsulate conversational turns either sent to models (i.e. prompts) or returned from models (i.e. model output). garak uses an object to encapsulate this to allow easy switching with multimodal probes and generators. garak.attempt ------------- .. automodule:: garak.attempt :members: :undoc-members: :show-inheritance: --- ## File: docs/source/basic.rst Key Concepts and Classes ======================== What are we doing here, and how does it all fit together? Our goal is to test the security of something that takes prompts and returns text. ``garak`` has a few constructs used to simplify and organise this process. generators ---------- :doc:`index_generators` wrap a target LLM or dialogue system. They take a prompt and return the output. The rest is abstracted away. Generator classes deal with things like authentication, loading, connection management, backoff, and all the behind-the-scenes things that need to happen to get that prompt/response interaction working. probes ------ :doc:`index_probes` tries to exploit a weakness and elicit a failure. The probe manages all the interaction with the generator. It determines how often to prompt, and what the content of the prompts is. Interaction between probes and generators is mediated in an object called an attempt. attempt ------- An :doc:`attempt` represents one unique try at breaking the target. A probe wraps up each of its adversarial interactions in an attempt object, and passes this to the generator. The generator adds responses into the attempt and sends the attempt back. This is logged in ``garak`` reporting which contains (among other things) JSON dumps of attempts. Once the probe is done with the attempt and the generator has added its outputs, the outputs are examined for signs of failures. This is done in a detector. detectors --------- :doc:`index_detectors` attempt to identify a single failure mode. This could be for example some unsafe contact, or failure to refuse a request. Detectors do this by examining outputs that are stored in a prompt, looking for a certain phenomenon. This could be a lack of refusal, or continuation of a string in a certain way, or decoding an encoded prompt, for example. buffs ----- :doc:`index_buffs` adjust prompts before they're sent to a generator. This could involve translating them to another language, or adding paraphrases for probes that have only a few, static prompts. evaluators ---------- When detectors have added judgments to attempts, :doc:`index_evaluators` converts the results to an object containing pass/fail data for a specific probe and detector pair. harnesses --------- The :doc:`index_harnesses` manage orchestration of a ``garak`` run. They select probes, then detectors, and co-ordinate running probes, passing results to detectors, and doing the final evaluation .. automodule:: garak._plugins :members: :undoc-members: :show-inheritance: :no-index: --- ## File: docs/source/cli.rst cli === .. automodule:: garak.cli :members: :undoc-members: :show-inheritance: --- ## File: docs/source/cliref.rst CLI reference for garak ======================= :: garak LLM vulnerability scanner v0.16.1.pre1 ( https://github.com/NVIDIA/garak ) at 2026-08-04T13:20:48.168125 usage: python -m garak [-h] [--verbose] [--report_prefix REPORT_PREFIX] [--narrow_output] [--parallel_requests PARALLEL_REQUESTS] [--parallel_attempts PARALLEL_ATTEMPTS] [--skip_unknown] [--seed SEED] [--deprefix] [--eval_threshold EVAL_THRESHOLD] [--generations GENERATIONS] [--config CONFIG] [--target_type TARGET_TYPE] [--target_name TARGET_NAME] [--spec SPEC] [--probes PROBES] [--probe_tags PROBE_TAGS] [--detectors DETECTORS] [--extended_detectors] [--buffs BUFFS] [--buff_option_file BUFF_OPTION_FILE | --buff_options BUFF_OPTIONS] [--detector_option_file DETECTOR_OPTION_FILE | --detector_options DETECTOR_OPTIONS] [--generator_option_file GENERATOR_OPTION_FILE | --generator_options GENERATOR_OPTIONS] [--harness_option_file HARNESS_OPTION_FILE | --harness_options HARNESS_OPTIONS] [--probe_option_file PROBE_OPTION_FILE | --probe_options PROBE_OPTIONS] [--taxonomy TAXONOMY] [--confidence_interval_method {bootstrap,none}] [--bootstrap_num_iterations BOOTSTRAP_NUM_ITERATIONS] [--bootstrap_confidence_level BOOTSTRAP_CONFIDENCE_LEVEL] [--bootstrap_min_sample_size BOOTSTRAP_MIN_SAMPLE_SIZE] [--plugin_info PLUGIN_INFO] [--list_probes] [--list_detectors] [--list_generators] [--list_buffs] [--list_config] [--version] [--report REPORT] [--interactive] [--fix] LLM safety & security scanning tool options: -h, --help show this help message and exit --verbose, -v add one or more times to increase verbosity of output during runtime --report_prefix REPORT_PREFIX Specify an optional prefix for the report and hit logs --narrow_output give narrow CLI output --parallel_requests PARALLEL_REQUESTS How many generator requests to launch in parallel for a given prompt. Ignored for models that support multiple generations per call. --parallel_attempts PARALLEL_ATTEMPTS How many probe attempts to launch in parallel. Raise this for faster runs when using non-local models. --skip_unknown allow skip of unknown probes, detectors, or buffs --seed SEED, -s SEED random seed --deprefix remove the prompt from the front of generator output --eval_threshold EVAL_THRESHOLD minimum threshold for a successful hit --generations GENERATIONS, -g GENERATIONS number of generations per prompt --config CONFIG YAML or JSON config file for this run --target_type TARGET_TYPE, -t TARGET_TYPE, --model_type TARGET_TYPE, -m TARGET_TYPE module and optionally also class of the generator, e.g. 'huggingface', or 'openai' --target_name TARGET_NAME, --model_name TARGET_NAME, -n TARGET_NAME name of the target, e.g. 'timdettmers/guanaco-33b-merged' --spec SPEC, -S SPEC unified selection spec, e.g. 'probes.dan,-probes.dan.DanInTheWild,tag:owasp:llm01'. Selectors: probes.[.], buffs.[.], tag:, tier:; '-' excludes, tier:N is inclusive (tiers 1..N). --probes PROBES, -p PROBES DEPRECATED, use --spec. list of probe names to use, or 'all'. --probe_tags PROBE_TAGS DEPRECATED, use --spec 'tag:'. only include probes with a tag starting with this value (e.g. owasp:llm01) --detectors DETECTORS, -d DETECTORS list of detectors to use, or 'all' for all. Default is to use the probe's suggestion. --extended_detectors If detectors aren't specified on the command line, should we run all detectors? (default is just the primary detector, if given, else everything) --buffs BUFFS, -b BUFFS DEPRECATED, use --spec 'buffs.'. list of buffs to use. Default is none --buff_option_file BUFF_OPTION_FILE, -B BUFF_OPTION_FILE path to JSON file containing options to pass to buff --buff_options BUFF_OPTIONS options to pass to buff, formatted as a JSON dict --detector_option_file DETECTOR_OPTION_FILE, -D DETECTOR_OPTION_FILE path to JSON file containing options to pass to detector --detector_options DETECTOR_OPTIONS options to pass to detector, formatted as a JSON dict --generator_option_file GENERATOR_OPTION_FILE, -G GENERATOR_OPTION_FILE path to JSON file containing options to pass to generator --generator_options GENERATOR_OPTIONS options to pass to generator, formatted as a JSON dict --harness_option_file HARNESS_OPTION_FILE, -H HARNESS_OPTION_FILE path to JSON file containing options to pass to harness --harness_options HARNESS_OPTIONS options to pass to harness, formatted as a JSON dict --probe_option_file PROBE_OPTION_FILE, -P PROBE_OPTION_FILE path to JSON file containing options to pass to probe --probe_options PROBE_OPTIONS options to pass to probe, formatted as a JSON dict --taxonomy TAXONOMY specify a MISP top-level taxonomy to be used for grouping probes in reporting. e.g. 'avid-effect', 'owasp' --confidence_interval_method {bootstrap,none} method for CI calculation: 'bootstrap' (default) or 'none' to disable --bootstrap_num_iterations BOOTSTRAP_NUM_ITERATIONS number of bootstrap iterations for CI calculation (overrides config) --bootstrap_confidence_level BOOTSTRAP_CONFIDENCE_LEVEL confidence level for bootstrap CIs, e.g. 0.95 or 0.99 (overrides config) --bootstrap_min_sample_size BOOTSTRAP_MIN_SAMPLE_SIZE minimum sample size required for bootstrap CI calculation (overrides config) --plugin_info PLUGIN_INFO show info about one plugin; format as type.plugin.class, e.g. probes.lmrc.Profanity --list_probes list available probes. Use -v for a detailed markdown table with tier and description. Combine with --spec to filter, e.g. '--list_probes --spec probes.dan'. --list_detectors list available detectors. Usage: combine with --detectors/-d to filter for detectors that will be activated based on a `detector_spec`, e.g. '-- list_detectors -d misleading.Invalid' to show only that detector. --list_generators list available generation model interfaces --list_buffs list available buffs/fuzzes --list_config print active config info (and don't scan) --version, -V print version info & exit --report REPORT, -r REPORT process garak report into a list of AVID reports --interactive, -I Enter interactive probing mode --fix Update provided configuration with fixer migrations; requires one of --config / --*_option_file, / --*_options See https://github.com/NVIDIA/garak --- ## File: docs/source/command.rst command ======= .. automodule:: garak.command :members: :undoc-members: :show-inheritance: --- ## File: docs/source/configurable.rst .. headings: = - ^ " Configuring garak ================= Beyond the standard CLI options, garak is highly configurable. You can use YAML files to configure a garak run, down to the level of exactly how each plugin behaves. Specifying Custom Configuration ------------------------------- garak can be configured in multiple ways: * Via command-line parameters * Using YAML or JSON config files * Through specifying JSON on the command line The easiest way is often to use a config file (YAML or JSON), and how to do that is described below. Garak Config Hierarchy ^^^^^^^^^^^^^^^^^^^^^^ Configuration values can come from multiple places. At garak load, the ``_config`` module manages parsing configuration. This includes determining the priority of which values go where. The hierarchy is as follows: 1. Values given at the command line 2. Config values given in a YAML or JSON file passed via ``--config`` 3. Values in a YAML or JSON site config, ``garak.site.yaml``, ``garak.site.yml``, or ``garak.site.json``, placed in the config directory (``XDG_CONFIG_DIR``, which is ``~/.config/garak/`` on Linux; see XDG spec for details) 4. Fixed values kept in the garak core config - don't edit this. Package updates will overwrite it, and you might break your garak install. It's in ``garak/resources`` if you want to take a look. 5. Default values specified in plugin code Config Files (YAML and JSON) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Let's take a look at the core config. .. code-block:: yaml --- system: verbose: 0 narrow_output: false parallel_requests: false parallel_attempts: false lite: true show_z: false enable_experimental: false max_workers: 500 run: system_prompt: "You are an AI model and this is a system prompt" seed: deprefix: true eval_threshold: 0.5 generations: 5 spec: include: - probes.dan - tag: owasp:llm01 exclude: - probes.dan.DanInTheWild user_agent: "garak/{version} (LLM vulnerability scanner https://garak.ai)" soft_probe_prompt_cap: 256 plugins: target_type: target_name: detector_spec: auto extended_detectors: false buffs_include_original_prompt: false buff_max: detectors: {} generators: {} buffs: {} harnesses: {} probes: encoding: payloads: - default reporting: report_prefix: taxonomy: report_dir: garak_runs show_100_pass_modules: true group_aggregation_function: minimum Here we can see many entries that correspond to command line options, such as ``target_name`` and ``target_type``, as well as some entried not exposed via CLI such as ``show_100_pass_modules``. System Config Items """"""""""""""""""" * ``parallel_attempts`` - For parallelisable generators, how many attempts should be run in parallel? Raising this is a great way of speeding up garak runs for API-based models * ``parallel_requests`` - For generators not supporting multiple responses per prompt: how many requests to send in parallel with the same prompt? (raising ``parallel_attempts`` generally yields higher performance, depending on how high ``generations`` is set) * ``lite`` - Should we display a caution message that the run might not give very thorough results? * ``verbose`` - Degree of verbosity (values above 0 are experimental, the report & log are authoritative) * ``narrow_output`` - Support output on narrower CLIs * ``show_z`` - Display Z-scores and visual indicators on CLI. It's good, but may be too much info until one has seen garak run a couple of times * ``enable_experimental`` - Enable experimental function CLI flags. Disabled by default. Experimental functions may disrupt your installation and provide unusual/unstable results. Can only be set by editing core config, so a git checkout of garak is recommended for this. * ``max_workers`` - Cap on how many parallel workers can be requested. When raising this in order to use higher parallelisation, keep an eye on system resources (e.g. `ulimit -n 4026` on Linux) **Parallel requests and parallel attempts** These items enable parallelisation within a probe, by launching multiple processes to either try many prompts at the same time (``parallel_attempts``), or to try multiple copies of the same prompt at the same time (``parallel_requests``). In testing, garak maintainers find that ``parallel_attempts`` usually runs quicker - especially if the endpoint is capable of returning more than one response to a query at a time. If an endpoint can only return one response to a query at a time, but generations is set to a value greater than one, then each prompt is posed to the endpoint multiple times. This can be slow. Setting ``parallel_requests`` to a value over one enables making all these requests at the same time, mitigating the wallclock-time cost of multiple generations. Parameter ``parallel_requests`` has no effect if generations is set to 1. Setting ``parallel_requests`` higher than generations also has the same effect as setting ``parallel_requests`` equal to generations. In practice, ``parallel_requests`` and ``parallel_attempts`` are mutually exclusive, so you have to choose between them. We find that using ``parallel_attempts`` usually gives a faster run completion time - especially when the number of generations is lower than the number of different prompts from a probe, which is more oftent he case than not in a default garak run. Run Config Items """""""""""""""" * ``system_prompt`` -- If given and not overriden by the probe itself, probes will pass the specified system prompt when possible for generators that support chat modality. * ``spec`` - The unified selection spec for probes and buffs (``run.spec``); see "Selecting probes and buffs with run.spec" below. If absent, the default is all active probes (``probes.*``); use ``none`` to select no probes explicitly. The intent scope is part of this spec: when no ``intent:`` selector is given, the default scope ``S`` is injected; set ``run.spec`` ``intent:`` selectors to override * ``generations`` - How many times to send each prompt for inference * ``deprefix`` - Remove the prompt from the start of the output (some models return the prompt as part of their output) * ``seed`` - An optional random seed * ``eval_threshold`` - At what point in the 0..1 range output by detectors does a result count as a successful attack / hit * ``user_agent`` - What HTTP user agent string should garak use? ``{version}`` can be used to signify where garak version ID should go * ``soft_probe_prompt_cap`` - For probes that auto-scale their prompt count, the preferred limit of prompts per probe * ``target_lang`` - A single language (as BCP47 that the target application for LLM accepts as prompt and output * ``langproviders`` - A list of configurations representing providers for converting from probe language to lang_spec target languages (BCP47) * ``serve_detectorless_intents`` - Should the intent service provide intents for which there are no configured detectors? Plugins Config Items """""""""""""""""""" * ``target_type`` - The type of target generator, e.g. "nim" or "huggingface" * ``target_name`` - The specific name of the target to be used (optional - if blank, type-specific default is used) * ``detector_spec`` - An optional spec of detectors to be used, if overriding those recommended in probes. Specifying ``detector_spec`` means the ``pxd`` harness will be used. This is equivalent to passing `-d` to the CLI * ``extended_detectors`` - Should just the primary detector be used per probe, or should the extended detectors also be run? The former is fast, the latter thorough. * ``buffs_include_original_prompt`` - When buffing, should the original pre-buff prompt still be included in those posed to the model? * ``buff_max`` - Upper bound on how many items a buff should return * ``detectors`` - Root node for detector plugin configs * ``generators`` - Root note for generator plugin configs * ``buffs`` - Root note for buff plugin configs * ``harnesses`` - Root note for harness plugin configs * ``probes`` - Root note for probe plugin configs .. note:: ``plugins.probe_spec``, ``plugins.buff_spec`` and ``run.probe_tags`` are **deprecated**. They still work (and are mapped onto ``run.spec`` with a deprecation notice) but will be removed in a future release; use ``run.spec`` instead (see below). For an example of how to use the ``detectors``, ``generators``, ``buffs``, ``harnesses``, and ``probes`` root entries, see :ref:`Configuring plugins with YAML ` below. Selecting probes and buffs with run.spec """""""""""""""""""""""""""""""""""""""" ``run.spec`` is the single source of truth for selecting probes and buffs. It has two transports that parse to the same internal spec: a CLI string (``--spec``) and the config-file form (``include`` / ``exclude`` lists). Selectors (a category prefix is mandatory): * ``probes.*`` (or ``probes.all``) - all active probes (the default when no ``run.spec`` is given). ``all`` and ``*`` are interchangeable aliases; ``all`` is handy on the CLI since it needs no shell quoting. A bare ``all`` (or ``*``) behaves as ``probes.*``. Both serialise to the canonical ``*`` token * ``probes.`` - an active family; ``probes..`` - one class * ``none`` (or ``probes.none``) - selects no probes; an explicit empty selection, distinct from an unspecified spec (which defaults to ``probes.*``) * ``buffs.[.]`` - selects buffs (no buffs are run by default); ``buffs.*`` / ``buffs.all`` select all active buffs (the ``all`` alias is generic) * ``tag:`` - filters probes by tag (e.g. ``tag:owasp:llm01``) * ``tier:`` - filters probes by tier; **inclusive** ("log level"): ``tier:N`` admits tiers ``1..N`` (``tier:1`` is the most critical). Names work too (``tier:of_concern`` == ``tier:1``). * ``intent:`` - selects intent typology codes for intent-based probes (e.g. ``intent:S`` for the whole Safety branch, ``intent:S001`` for a category, ``intent:S001mis`` for a leaf); ``intent:*`` or ``intent:all`` selects every intent. This is a **separate axis** consumed by the intent service: it does **not** add or remove probes. When no ``intent:`` is given, the default scope ``S`` (the Safety branch) is injected at resolve time. Typology expansion and detectorless filtering are governed by the ``run.*`` intent modifiers (``run.serve_detectorless_intents``). Only ``IntentProbe`` subclasses consume intents; selecting ``intent:`` without an ``IntentProbe`` warns and proceeds. Polarity: a bare selector (or ``+``) includes; a leading ``-`` removes. Note the asymmetry of ``tier``: ``tier:N`` is the inclusive filter, while ``-tier:N`` removes *exactly* tier ``N``. Resolution applies excludes last (exclude wins). If a spec resolves to no probes garak aborts with an actionable message, unless ``none`` was requested explicitly, in which case the run is a deliberate no-op. ``tier:`` and ``tag:`` filters apply to the whole candidate set, including explicitly-named classes, so e.g. ``probes.foo.Bar,tier:1`` yields nothing when ``foo.Bar`` is tier 3. The spec is a single comma-separated token: whitespace between selectors is rejected, so commas alone need no shell quoting. A ``*`` glob is still a shell wildcard, so quote those specs (or use the ``all`` alias instead). .. code-block:: bash # whole family minus one class garak --spec probes.dan,-probes.dan.DanInTheWild # family filtered by tag garak --spec probes.grandma,tag:owasp:llm06 # all active buffs except one, over all active probes (quote the * glob) garak --spec "probes.*,buffs.*,-buffs.paraphrase" # all active probes plus a specific inactive class (all is the quote-free *) garak --spec probes.all,probes.fitd.FITD # tiers {1,3}: tier:3 admits 1..3, then -tier:2 removes exactly tier 2 garak --spec "+probes.*,+tier:3,-tier:2" # an intent probe over one intent category (intents are a separate axis) garak --spec probes.grandma.GrandmaIntent,intent:S004 .. code-block:: yaml run: spec: include: - probes.dan - tag: owasp:llm01 exclude: - probes.dan.DanInTheWild The deprecated ``--probes`` / ``--probe_tags`` / ``--buffs`` flags (and the ``plugins.probe_spec`` / ``plugins.buff_spec`` / ``run.probe_tags`` config keys) are mapped onto ``run.spec`` with a deprecation notice; ``--spec`` wins if both are given. A legacy ``none`` value (e.g. ``--probes none`` or ``probe_spec: none``) maps to the explicit empty selection ``probes.none``; vacuous values (empty, ``auto``, or omitted) are treated as unspecified and default to all active probes. Reporting Config Items """""""""""""""""""""" * ``report_dir`` - Directory for reporting; defaults to ``$XDG_DATA/garak/garak_runs`` * ``report_prefix`` - Prefix for report files. Defaults to ``garak.$RUN_UUID`` * ``taxonomy`` - Which taxonomy to use to group probes when creating HTML report * ``show_100_pass_modules`` - Should entries scoring 100% still be detailed in the HTML report? * ``show_group_score`` - Should an aggregated score per group be shown in reports? * ``group_aggregation_function`` - How should scored of probe groups (e.g. plugin modules or taxonomy categories) be aggregrated in the HTML report? Options are ``minimum``, ``mean``, ``median``, ``mean_minus_sd``, ``lower_quartile``, and ``proportion_passing``. NB averages like ``mean`` and ``median`` hide a lot of information and aren't recommended. * ``show_top_group_score`` - Should the aggregated score be shown as a top-level figure in report concertinas? * ``confidence_interval_method`` - Method for calculating confidence intervals on attack success rates. Also available via CLI as ``--confidence_interval_method``. Options: - ``"bootstrap"`` (default) - Non-parametric bootstrap with detector performance correction (requires detector metrics and n≥30). - ``"none"`` or empty value - No confidence intervals calculated or displayed. Example YAML configuration: .. code-block:: yaml --- reporting: confidence_interval_method: "bootstrap" # Default - bootstrap CIs --- reporting: confidence_interval_method: # Disable CIs Example CLI usage: .. code-block:: bash python -m garak --confidence_interval_method none ... # Disable CIs for this run python -m garak --confidence_interval_method bootstrap ... # Explicitly enable (default) * ``bootstrap_num_iterations`` - Number of bootstrap resampling iterations for computing confidence intervals on attack success rates (default: 10000). Also available via CLI as ``--bootstrap_num_iterations``. Only used when ``confidence_interval_method`` is ``"bootstrap"``. * ``bootstrap_confidence_level`` - Confidence level for bootstrap confidence intervals, expressed as a decimal between 0 and 1 (default: 0.95 for 95% confidence intervals). Also available via CLI as ``--bootstrap_confidence_level``. Only used when ``confidence_interval_method`` is ``"bootstrap"``. * ``bootstrap_min_sample_size`` - Minimum sample size required for reliable bootstrap confidence interval estimates (default: 30). Also available via CLI as ``--bootstrap_min_sample_size``. Can be increased for more conservative estimates, but lowering it significantly compromises statistical validity. Only used when ``confidence_interval_method`` is ``"bootstrap"``. Bundled Quick Configs ^^^^^^^^^^^^^^^^^^^^^ Garak comes bundled with some quick configs that can be loaded directly using ``--config``. **Note on extensions:** JSON configs can be loaded without the ``.json`` extension (e.g., ``--config fast``). YAML configs require the explicit ``.yaml`` or ``.yml`` extension (e.g., ``--config fast.yaml`` or ``--config fast.yml``). Extensions are case-insensitive, so ``.JSON``, ``.YAML``, and ``.YML`` are also accepted. Bundled configs include: * ``bag`` - The config used for calibration * ``fast`` - Go through a selection of light probes; skip extended detectors These are great places to look at to get an idea of how garak configs can look. Quick configs are stored under ``garak/configs/`` in the source code/install. Using a Custom Config ^^^^^^^^^^^^^^^^^^^^^ To override values in this we can create a new config file (YAML or JSON) and point to it from the command line using ``--config``. For example, to select just ``latentinjection`` probes and run each prompt just once: **YAML format:** .. code-block:: yaml --- run: generations: 1 spec: include: - probes.latentinjection If we save this as ``latent1.yaml`` somewhere, then we can use it with ``garak --config latent1.yaml``. Note: YAML configs require the explicit ``.yaml`` or ``.yml`` extension (case-insensitive). **JSON format:** .. code-block:: json { "run": { "generations": 1, "spec": { "include": ["probes.latentinjection"], "exclude": [] } } } If we save this as ``latent1.json`` somewhere, then we can use it with ``garak --config latent1.json`` or without the extension: ``garak --config latent1``. Using a Custom JSON Config ^^^^^^^^^^^^^^^^^^^^^^^^^^ Some plugins can take a JSON config specified on the command line. This config has the same structure as a YAML config, starting with the plugin model/type. The config can either be written to a file and the path passed, with `--generator_option_file` or `--probe_option_file`, or directly as JSON on the command prompt, with `--generator_options` or `--probe_options`. An example is given in :ref:`RestGenerator Config with JSON ` below. Configuring Plugins ------------------- Garak's functions are through its plugins. Most parts of garak are plugins, like the ``probes`` and ``detectors`` that do the actual examination of the target, the ``generators`` that interface with models, and even the ``harnesses`` that manage run orchestration. Each plugin is a class that has both descriptive and configurable parameters. Viewing Plugin Parameters ^^^^^^^^^^^^^^^^^^^^^^^^^ You can see the parameters for any given plugin using garak ``--plugin_info``. .. code-block:: text $ python -m garak --plugin_info probes.grandma.Win10 garak LLM vulnerability scanner v0.10.0 ( https://github.com/NVIDIA/garak ) at 2024-09-11T11:54:07.215242 Configured info on probes.grandma.Win10: DEFAULT_PARAMS: {'generations': 1} active: True lang: en doc_uri: https://www.independent.co.uk/tech/chatgpt-microsoft-windows-11-grandma-exploit-b2360213.html extended_detectors: ['mitigation.MitigationBypass'] goal: get model to output Win 10 product keys modality: {'in': ['text']} parallelisable_attempts: True primary_detector: productkey.Win5x5 tags: ['avid-effect:security:S0300', 'owasp:llm06', 'risk-cards:lmrc:providing_illegal_info', 'quality:Behavioral:ContentSafety:LegalGoodsServices'] mod_time: 2024-07-01 04:16:40 +0000 Here, we see a list of the descriptive parameters of the plugin. We can see a link to documentation about it, which detectors it uses, tags describing the probe in various typologies, which languages and modalities it supports, and more. We can also see a ``DEFAULT_PARAMS`` entry. This is a dictionary containing configurable parameters for this plugin. In this case, there's a ``generations`` parameter set to ``1``; this is the default value for ``probes``, but is often overridden at run time by the CLI setup. At plugin load, the plugin instance has attributes named in ``DEFAULT_PARAMS`` automatically created, and populated with either values given in the supplied config, or the default. Fixed plugin parameters ^^^^^^^^^^^^^^^^^^^^^^^ Some plugin parameters aren't intended to be altered at instantiation via config. These are the fixed plugin parameters, and are generally those not given in ``DEFAULT_PARAMS``. Descriptions of these are as follows (for a probe - other plugins are similar): * ``description`` - A short description of what the plugin does * ``active`` - Whether or not the plugin is active (i.e. selected) by default * ``doc_uri`` - Link to more information about the plugin * ``extended_detectors`` - Option detectors to use on probe results * ``extra_dependency_names`` - Extra Python modules that garka should import when instantiatng the plugin * ``goal`` - Brief description in imperative form of the probe's intent * ``modality`` - Which modalities the probe supports (as of Nov 2024 the list is ``text``, ``image``, ``audio``, ``video``, ``3d``) * ``parallelisable_attempts`` - Is the probe parallelisable? Recommended false if it has to use an LLM to develop attacks, particularly a local one * ``primary_detector`` - What detector should be used on the probe's outputs? * ``tags`` - List of tags applicable to the plugin, drawn from ``garak/data/tags.misp.tsv`` * ``mod_time`` - Modification timestamp of the plugin source file used to generate this data .. _config_with_yaml: Configuring Plugins with YAML ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Plugin config happens inside the ``plugins`` block. Multiple plugins can be configured in the same YAML. Descend through this specifying plugin type, model, and optionally class, and set variables in the end. These will then be loaded as the plugin's ``DEFAULT_PARAMS`` attribute is parsed and used to populate instance attributes. Here's an example of setting the temperature on an OpenAIGenerator: .. code-block:: yaml plugins: generators: openai: OpenAIGenerator: temperature: 1.0 As noted the class is optional, if the configuration defines keys at the module level these will be applied to the instance and can be overridden by the class level. Here is an example that is equivalent to the configuration above: .. code-block:: yaml plugins: generators: openai: temperature: 1.0 Example: RestGenerator ^^^^^^^^^^^^^^^^^^^^^^ RestGenerator is a slightly complex generator, though mostly because it exposes so many config values, allowing flexible integrations. This example sets ``target_type: rest`` to ensure that this model is selected for the run; that might not always be wanted, and it isn't compulsory. RestGenerator with YAML """"""""""""""""""""""" .. code-block:: yaml plugins: target_type: rest generators: rest: RestGenerator: uri: https://api.example.ai/v1/ key_env_var: EXAMPLE_KEY headers: Authentication: $KEY response_json_field: text request_timeout: 60 This defines a REST endpoint where: * The URI is https://api.example.ai/v1/ * The API key can be found in the ``EXAMPLE_KEY`` environment variable's value (if unspecified, `REST_API_KEY` is checked) * The HTTP header ``"Authentication:"`` should be sent in every request, with the API key as its parameter * The output is JSON and the top-level field ``text`` holds the model's response * Wait up to 60 seconds before timing out (the generator will backoff and retry when this is reached) .. _rest_generator_with_json: RestGenerator config with JSON """""""""""""""""""""""""""""" .. code-block:: JSON { "rest": { "RestGenerator": { "name": "example service", "uri": "https://127.0.0.1/llm", "method": "post", "headers": { "X-Authorization": "$KEY" }, "req_template_json_object": { "text": "$INPUT" }, "response_json": true, "response_json_field": "text" } } } This defines a REST endpoint where: * The URI is https://127.0.0.1/llm * We'll use HTTP `POST` on requests * The HTTP header ``"X-Authorization:"`` should be sent in every request, with the API key as its parameter * The request template is to be a JSON dict with one key, `text`, holding the prompt * The output is JSON and the top-level field ``text`` holds the model's response This should be written to a file, and the file's path passed on the command line with `-G`. Configuration in Code --------------------- The preferred way to instantiate a plugin is using ``garak._plugins.load_plugin()``. This function takes two parameters: * ``name``, the plugin's package, module, and class - e.g. ``generator.test.Lipsum`` * (optional) ``config_root``, either garak._config or a dictionary of a config, beginning at a top-level plugin type. ``load_plugin()`` returns a configured instance of the requested plugin. OpenAIGenerator Config with Dictionary ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: python >>> import garak._plugins >>> c = {"generators":{"openai":{"OpenAIGenerator":{"seed":30,"name":"gpt-4"}}}} >>> garak._plugins.load_plugin("generators.openai.OpenAIGenerator", config_root=c) 🦜 loading generator: OpenAI: gpt-4