h4cker Documentation

This repository is maintained by Omar Santos (@santosomar) and includes thousands of resources related to ethical hacking, bug bounties, digital forensics...

29,872 stars Jupyter Notebook 1 file · ~25,610 tokens #ai#ai-security#artificial-intelligence#awesome-list#awesome-lists#cybersecurity#ethical-hacking#exploit
RAW Doc

File: README.md

This repository is a comprehensive collection of cybersecurity references, scripts, tools, code, labs, and training resources. It is carefully curated and maintained by Omar Santos.

Overview

The repository serves as supplemental material for books, video courses, and live training created by Omar Santos. It includes resources for offensive security, defensive security, cloud and container security, application security, AI security, certifications, labs, and reference material.

Directory Overview

  • Cybersecurity Domains: cybersecurity-domains contains the main domain taxonomy, including fundamentals, offensive security, defensive security, application security, cloud and container security, infrastructure and network security, cryptography and PKI, hardware and embedded security, governance/risk/compliance, and labs/practice.
  • AI: ai contains AI security, LLM engineering, AI-assisted incident response and automation, and AI ethics/privacy/governance resources.
  • Certifications: certifications contains certification roadmaps, Cisco SCOR, cloud certifications, Kubernetes/CNCF certifications, CompTIA, ISC2, offensive security certification references, and supplemental exam topics.
  • Build Your Own Lab: build-your-own-lab remains a root-level section for lab-building and cyber range resources.
  • Training Reference: training-reference contains cheat sheets, O'Reilly resources, curated people/projects to follow, and organized tool indexes.

How to Use

Start with the domain landing page that matches your goal, then follow the links to tools, labs, scripts, and references. Certification resources are grouped separately so exam preparation material is easy to find without mixing it into the broader domain taxonomy.

Contributing

If you wish to contribute, please read the CONTRIBUTING.md file.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

For any inquiries or feedback, please contact Omar Santos.

File: CONTRIBUTING.md

Contributor Covenant Code of Conduct

Pledge

In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, gender identity and expression, level of experience,
education, socio-economic status, nationality, personal appearance, race,
religion, or sexual identity and orientation.

Standards

Examples of behavior that contributes to creating a positive environment
include:

  • Using welcoming and inclusive language
  • Being respectful of differing viewpoints and experiences
  • Gracefully accepting constructive criticism
  • Focusing on what is best for the community
  • Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

  • The use of sexualized language or imagery and unwelcome sexual attention or
    advances
  • Trolling, insulting/derogatory comments, and personal or political attacks
  • Public or private harassment
  • Publishing others' private information, such as a physical or electronic
    address, without explicit permission
  • Other conduct which could reasonably be considered inappropriate in a
    professional setting

Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.

Scope

This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.

Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported. All complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

File: ai/README.md

AI Security and Engineering Resources

This section groups AI resources into security, LLM engineering, incident response and automation, and governance tracks.

  • AI Security - model security testing, prompt injection, AI red teaming, MCP security, risk management, and AI security tools.
  • LLM Engineering - LangChain, LLM frameworks, RAG, vector databases, fine-tuning, Ollama labs, prompt engineering, and ML fundamentals.
  • Incident Response and Automation - AI for incident response, AI coding tools, monitoring, labs, presentations, and training environment security.
  • Ethics, Privacy, and Governance - AI ethics, privacy, and governance resources.

External Resources

File: ai/ai-security/README.md

AI Security

Resources in this track:

File: ai/ai-security/ai-security-overview.md

AI Security Overview

This overview preserves the original AI security navigation notes after the repository was reorganized into AI tracks. Use the current AI Security, LLM Engineering, Incident Response and Automation, and Ethics, Privacy, and Governance indexes for the active structure.

File: ai/ai-security/ai_security_tools.md

AI Security Tools

This is a work in progress, curated list of AI Security tools:

Open Source Tools for AI Red Teaming

Predictive AI

Generative AI

Prompt Firewall and Redaction

Products that intercept prompts and responses and apply security or privacy rules to them. We've blended two categories here because some prompt firewalls just redact private data (and then reidentify in the response) while others focus on identifying and blocking attacks like injection attacks or stopping data leaks. Many of the products in this category do all of the above, which is why they've been combined.

  • Cisco AI Defense - Model Evaluation, monitoring, guardrails, inventory, AI asset discovery, and more.

  • Robust Intelligence AI Firewall - Now part of Cisco.

  • Protect AI Rebuff - A LLM prompt injection detector.

  • Protect AI LLM Guard - Suite of tools to protect LLM applications by helping you detect, redact, and sanitize LLM prompts and responses.

  • HiddenLayer AI Detection and Response - Proactively defend against threats to your LLMs.

  • Vigil LLM - Detect prompt injections, jailbreaks, and other potentially risky Large Language Model (LLM) inputs.

  • Lakera Guard - Protection from prompt injections, data loss, and toxic content.

  • Arthur Shield - Built-in, real-time firewall protection against the biggest LLM risks.

  • Prompt Security - SDK and proxy for protection against common prompt attacks.

  • Private AI - Detect, anonymize, and replace PII with less than half the error rate of alternatives.

  • DynamoGuard - Identify / defend against any type of non-compliance as defined by your specific AI policies and catch attacks.

  • Skyflow LLM Privacy Vault - Redacts PII from prompts flowing to LLMs.

  • Guardrails AI - Guardrails runs Input/Output Guards in your application that detect, quantify and mitigate the presence of specific types of risks.

  • isMalicious Prompt-Injection Scanner - Hosted API that scans untrusted text for prompt injection and checks the reputation of the links it contains before an AI agent acts on it. Also available as an MIT-licensed MCP server.

AI Red Teaming Guidance

AI Red Teaming Datasets

GenAI Security Standards and Solution Landscapes

  • OWASP GenAI Security Project - umbrella project for LLM Top 10, AI security landscape, governance checklist, threat intelligence, agentic app security, secure AI adoption, data security, and AI red teaming initiatives.
    • LLM Top 10 (2025) – updated top risks for LLM/GenAI applications.
    • AI Security Solution Landscape – vendor-agnostic overview of tools mapped to key GenAI risks.
    • Threat Defense COMPASS 1.0 – consolidated view of threats, vulnerabilities, defenses, and mitigations for GenAI systems.
    • Solutions Reference Guide (Q2–Q3 2025) – catalog of commercial and open source controls aligned to OWASP GenAI projects.

AI Risk Maps and Secure Design Patterns

File: ai/ai-security/ai_security_tools_root.md

AI Security Tools

This is a work in progress, curated list of AI Security tools:

Open Source Tools for AI Red Teaming

Predictive AI

Generative AI

Prompt Firewall and Redaction

Products that intercept prompts and responses and apply security or privacy rules to them. We've blended two categories here because some prompt firewalls just redact private data (and then reidentify in the response) while others focus on identifying and blocking attacks like injection attacks or stopping data leaks. Many of the products in this category do all of the above, which is why they've been combined.

  • Cisco AI Defense - Model Evaluation, monitoring, guardrails, inventory, AI asset discovery, and more.

  • Robust Intelligence AI Firewall - Now part of Cisco.

  • Protect AI Rebuff - A LLM prompt injection detector.

  • Protect AI LLM Guard - Suite of tools to protect LLM applications by helping you detect, redact, and sanitize LLM prompts and responses.

  • HiddenLayer AI Detection and Response - Proactively defend against threats to your LLMs.

  • Vigil LLM - Detect prompt injections, jailbreaks, and other potentially risky Large Language Model (LLM) inputs.

  • Lakera Guard - Protection from prompt injections, data loss, and toxic content.

  • Arthur Shield - Built-in, real-time firewall protection against the biggest LLM risks.

  • Prompt Security - SDK and proxy for protection against common prompt attacks.

  • Private AI - Detect, anonymize, and replace PII with less than half the error rate of alternatives.

  • DynamoGuard - Identify / defend against any type of non-compliance as defined by your specific AI policies and catch attacks.

  • Skyflow LLM Privacy Vault - Redacts PII from prompts flowing to LLMs.

  • Guardrails AI - Guardrails runs Input/Output Guards in your application that detect, quantify and mitigate the presence of specific types of risks.

AI Red Teaming Guidance

AI Red Teaming Datasets

Julia AI Security Tools

  • Oracle - AI-powered vulnerability prediction framework using GNN-based pattern recognition, CVE trend analysis, and CVSS scoring. Integrates NVD, MITRE ATT&CK, and EPSS data.

File: ai/ai-security/Coalition-for-Secure-AI-CoSAI-Risk-Map.md

Coalition for Secure AI (CoSAI) Risk Map - Comprehensive Summary

Source: CoSAI Risk Map Documentation, YAML Data, and Tables
GitHub Repository: https://github.com/cosai-oasis/secure-ai-tooling
Operationalizing the CoSAI Risk Map: https://becomingahacker.org/operationalizing-the-cosai-risk-map-cosai-rm-c47a6db128c6

Table of Contents

  1. Overview
  2. Getting Started
  3. Architecture & Components
  4. Security Risks
  5. Security Controls
  6. Personas
  7. Metadata & Frameworks
  8. Development & Contribution
  9. Validation & CI/CD
  10. Quick Reference Tables

1. Overview

The CoSAI Risk Map is a comprehensive framework developed by the Coalition for Secure AI to categorize and address security risks in AI systems. It provides:

  • Structured risk taxonomy covering supply chain, deployment, infrastructure, and runtime security
  • Security controls mapped to specific risks and components
  • Framework mappings to external standards (MITRE ATLAS, NIST AI RMF, STRIDE, OWASP Top 10 for LLM)
  • Visual graph generation for understanding component relationships and control mappings
  • Self-assessment tools for organizations to evaluate their AI security posture

Core Concepts

The Risk Map categorizes AI development into four key areas:

Area Description
Data Training data sources, filtering, processing, and storage
Infrastructure Hardware, serving systems, storage, and development platforms
Model Training, tuning, evaluation, and the model itself
Application User-facing applications, agents, plugins, and integrations

2. Getting Started

Prerequisites

  • Python 3.10+ with dependencies from requirements.txt
  • Node.js 18+ with npm packages (prettier, mermaid-cli)
  • Chrome/Chromium for SVG generation from Mermaid diagrams

Setup Options

bash
# Open repo in VS Code → "Dev Containers: Reopen in Container"
# Then install pre-commit hooks:
bash ./scripts/install-precommit-hook.sh

Option 2: Manual Setup

bash
pip install -r requirements.txt
npm install
bash ./scripts/install-precommit-hook.sh

Validation Commands

bash
# Full validation
python scripts/hooks/validate_riskmap.py --force

# Generate component graph
python scripts/hooks/validate_riskmap.py --to-graph ./preview-graph.md --force

# Generate control graph
python scripts/hooks/validate_riskmap.py --to-controls-graph ./preview-controls.md --force

# Generate risk graph
python scripts/hooks/validate_riskmap.py --to-risk-graph ./preview-risks.md --force

# Format YAML files
npx prettier --write risk-map/yaml/*.yaml

3. Architecture & Components

Component Categories

The Risk Map defines 23 components organized into hierarchical categories:

Infrastructure Components (Data Subcategory)

Component ID Title Description
componentDataSources Data Sources Original sources/repositories for training data (databases, APIs, web scraping, sensors)
componentDataFilteringAndProcessing Data Filtering and Processing Cleaning, transforming, and preparing raw data for training
componentTrainingData Training Data Final curated subset fed into models during training
componentDataStorage Data Storage Infrastructure Storage for training data from ingestion through usage

Infrastructure Components (General)

Component ID Title Description
componentModelStorage Model Storage Local and published storage for model checkpoints
componentModelServing Model Serving Infrastructure Systems for deploying models in production

Model Components (Training)

Component ID Title Description
componentModelFrameworksAndCode Model Frameworks and Code Code/frameworks for training and inference
componentModelTrainingTuning Training and Tuning Teaching models patterns through probability adjustment
componentModelEvaluation Model Evaluation Testing model performance during and after training
componentTheModel The Model Pairing of code and weights from training process

Model Components (Orchestration)

Component ID Title Description
componentOrchestrationInputHandling Input Handling Validating/sanitizing data entering orchestration logic
componentOrchestrationOutputHandling Output Handling Validating/formatting data exiting to downstream components
componentTools External Tools and Services APIs/services agents use to take action
componentMemory Model Memory Context retention across interactions
componentRAGContent RAG Content Curated knowledge for Retrieval-Augmented Generation

Application Components

Component ID Title Description
componentApplication Application User-facing product/feature using AI model
componentApplicationInputHandling Input Handling Filtering/sanitizing model outputs
componentApplicationOutputHandling Output Handling Filtering/sanitizing model inputs

Agent Components

Component ID Title Description
componentAgentUserQuery Agent User Query Processed user request details
componentAgentSystemInstruction Agent System Instructions Agent capabilities, permissions, and limitations
componentAgentInputHandling Input Handling Processing inputs before reasoning core
componentReasoningCore Agent Reasoning Core Planning and reasoning to achieve user goals
componentAgentOutputHandling Output Handling Response rendering and sanitization

4. Security Risks

The Risk Map identifies 26 security risks across 5 categories:

Risk Categories

Category Description Risk Count
risksSupplyChainAndDevelopment Model development, training data, and supply chain 6
risksDeploymentAndInfrastructure Deployment environments and infrastructure 6
risksRuntimeInputSecurity Malicious/adversarial inputs at runtime 4
risksRuntimeDataSecurity Data security during model operation 5
risksRuntimeOutputSecurity Insecure/malicious model outputs 5

Complete Risk Inventory

Supply Chain & Development Risks

ID Title Description
DP Data Poisoning Altering training data to degrade performance or create backdoors
UTD Unauthorized Training Data Using data not authorized for training (privacy/legal issues)
MST Model Source Tampering Tampering with model code, dependencies, or weights
EDH Excessive Data Handling Collection/retention exceeding authorized boundaries
FLP Federated/Distributed Training Privacy Gradient leakage in federated learning systems
MLD Malicious Loader/Deserialization Unsafe loaders causing RCE or integrity compromise

Deployment & Infrastructure Risks

ID Title Description
MXF Model Exfiltration Unauthorized theft of AI models
MDT Model Deployment Tampering Unauthorized changes to deployment components
MRE Model Reverse Engineering Recreating models via input/output analysis
IIC Insecure Integrated Component Vulnerabilities in plugins/libraries interacting with models
ASSC Accelerator and System Side-channels Cross-tenant leakage via hardware side-channels
ADI Adapter/PEFT Injection Trojaned adapters bypassing safety controls

Runtime Input Security Risks

ID Title Description
PIJ Prompt Injection Tricking models to execute injected commands
MEV Model Evasion Perturbing inputs to cause incorrect inferences
DMS Denial of ML Service Overloading systems with resource-intensive queries
EDW Economic Denial of Wallet Cost abuse via token inflation/tool loops

Runtime Data Security Risks

ID Title Description
SDD Sensitive Data Disclosure Disclosure of private/confidential data via queries
ISD Inferred Sensitive Data Inferring personal information not in training data
EDH-I Excessive Data Handling (Inference) Unauthorized handling of user data during inference
EBM Evaluation/Benchmark Manipulation Poisoned evaluation sets misleading safety signals
PCP Prompt/Response Cache Poisoning Cross-user contamination via shared caches

Runtime Output Security Risks

ID Title Description
IMO Insecure Model Output Unvalidated output passed to end users
RA Rogue Actions Unintended actions by model-based agents
COV Covert Channels in Model Outputs Hidden information transmission via outputs
ORH Orchestrator/Route Hijack Silent model swaps via configuration tampering
RVP Retrieval/Vector Store Poisoning Poisoning RAG corpora to steer outputs

5. Security Controls

The Risk Map defines 28 security controls across 6 categories:

Control Categories

Category Description Count
controlsData Data-related controls 6
controlsInfrastructure Infrastructure security 8
controlsModel Model-level controls 3
controlsApplication Application security 6
controlsAssurance Security assurance activities 4
controlsGovernance Governance and policy 4

Complete Control Inventory

Data Controls

ID Title Description
controlTrainingDataSanitization Training Data Sanitization Detect/remove poisoned or sensitive training data
controlTrainingDataManagement Training Data Management Ensure training data is authorized for intended purposes
controlUserDataManagement User Data Management Store/process user data in compliance with consent
controlModelPrivacyEnhancingTechnologies Privacy Enhancing Tech (Training) Differential privacy, federated learning, anonymization
controlRuntimePrivacyEnhancingTechnologies Privacy Enhancing Tech (Inference) Secure MPC, homomorphic encryption, on-device processing
controlRetrievalAndVectorSystemIntegrity Retrieval/Vector Integrity Protect vector databases from poisoning attacks

Infrastructure Controls

ID Title Description
controlModelAndDataInventoryManagement Inventory Management Track all data, code, models, and transformation tools
controlModelAndDataAccessControls Access Controls Minimize internal access to models and datasets
controlModelAndDataIntegrityManagement Integrity Management Ensure integrity protection during development/deployment
controlModelAndDataExecutionIntegrity Execution Integrity Verify provenance and lineage at inference time
controlSecureByDefaultMLTooling Secure-by-Default ML Tooling Use secure frameworks/libraries for AI development
controlIsolatedConfidentialComputing Isolated/Confidential Computing TEEs, secure enclaves, hardware isolation
controlOrchestratorAndRouteIntegrity Orchestrator/Route Integrity Signed route manifests, configuration verification

Model Controls

ID Title Description
controlInputValidationAndSanitization Input Validation Block/restrict adversarial queries
controlOutputValidationAndSanitization Output Validation Block/sanitize insecure model output
controlAdversarialTrainingAndTesting Adversarial Training/Testing Make models robust to adversarial inputs

Application Controls

ID Title Description
controlApplicationAccessManagement Application Access Management Identity, authorization, rate limiting, quotas
controlUserTransparencyAndControls User Transparency Disclosures and data control experiences
controlAgentPluginUserControl Agent User Control User approval for agent actions
controlAgentPluginPermissions Agent Permissions Least-privilege for agentic systems
controlAgentObservability Agent Observability Logging for debugging and security oversight

Assurance Controls

ID Title Scope
controlRedTeaming Red Teaming All components and risks
controlVulnerabilityManagement Vulnerability Management All components and risks
controlThreatDetection Threat Detection All components and risks
controlIncidentResponseManagement Incident Response All components and risks

Governance Controls

ID Title Scope
controlInternalPoliciesAndEducation Internal Policies All risks
controlUserPoliciesAndEducation User Policies Specific risks
controlProductGovernance Product Governance All risks
controlRiskGovernance Risk Governance All risks

6. Personas

The Risk Map defines two key personas:

Persona ID Title Description
personaModelCreator Model Creator Organizations that train/tune foundation models or fine-tune for specific tasks
personaModelConsumer Model Consumer Organizations that build AI applications using models (via API or downloaded) without training

Responsibility Mapping

  • Model Creators are primarily responsible for: Data controls, training security, model integrity
  • Model Consumers are primarily responsible for: Application security, runtime controls, user-facing protections
  • Both are responsible for: Governance, assurance, and universal controls

7. Metadata & Frameworks

Lifecycle Stages

An 8-stage AI lifecycle model:

Order Stage Description
1 planning Initial planning, design, and architecture
2 data-preparation Data collection, cleaning, labeling
3 model-training Training, fine-tuning, optimization
4 development Application development and integration
5 evaluation Testing, validation, performance assessment
6 deployment Production deployment and rollout
7 runtime Active operation in production
8 maintenance Monitoring, updates, retraining

Impact Types

Traditional Security:

  • confidentiality - Protection from unauthorized disclosure
  • integrity - Accuracy and tampering prevention
  • availability - System accessibility
  • privacy - Personal/sensitive information protection
  • compliance - Regulatory adherence

AI-Specific:

  • safety - Prevention of physical harm
  • fairness - Equitable treatment, absence of bias
  • accountability - Traceability and responsibility
  • reliability - Consistency and dependability
  • transparency - Explainability and interpretability

Actor Access Levels

Traditional:

  • external - No direct system access
  • api - API endpoint access
  • user - Authenticated user access
  • privileged - Admin/operator access
  • physical - Physical hardware access

AI-Specific:

  • agent - AI agents with tool execution
  • supply-chain - Position in software/data supply chain
  • infrastructure-provider - Cloud/infrastructure access
  • service-provider - Third-party service access

External Framework Mappings

Framework ID Full Name Description
mitre-atlas MITRE ATLAS Adversarial Threat Landscape for AI Systems
nist-ai-rmf NIST AI RMF AI Risk Management Framework v1.0
stride STRIDE Microsoft threat modeling framework
owasp-top10-llm OWASP Top 10 for LLM Critical security risks for LLM applications (2025)

8. Development & Contribution

Contribution Workflow

  1. Create a GitHub issue to track work
  2. Set up pre-commit hooks (see Setup section)
  3. Make content changes following the guides:
  4. Validate changes against all validation rules
  5. Open PR against develop branch

Best Practices

  • Always run validation before committing: python scripts/hooks/validate_riskmap.py --force
  • Preview graphs visually to see impact of changes
  • Format files: npx prettier --write risk-map/yaml/*.yaml
  • Use meaningful IDs: component[Name], control[Name], persona[Name]
  • Ensure bidirectional edges for components
  • Ensure bidirectional control-risk references
  • Don't list universal controls in individual risks

File Structure

text
risk-map/
├── yaml/                    # Primary data files
│   ├── components.yaml      # Component definitions
│   ├── controls.yaml        # Control definitions
│   ├── risks.yaml           # Risk definitions
│   ├── personas.yaml        # Persona definitions
│   ├── frameworks.yaml      # External framework mappings
│   ├── lifecycle-stage.yaml # Lifecycle stage definitions
│   ├── impact-type.yaml     # Impact type definitions
│   ├── actor-access.yaml    # Actor access level definitions
│   └── mermaid-styles.yaml  # Graph styling configuration
├── schemas/                 # JSON schemas for validation
├── tables/                  # Generated markdown tables
├── diagrams/                # Generated Mermaid diagrams
├── svg/                     # Generated SVG files
└── docs/                    # Documentation

9. Validation & CI/CD

Automated Validation (Pre-commit & CI)

Check Description
YAML Schema Validation Files validated against JSON schemas
Prettier Formatting Consistent YAML formatting
Ruff Linting Python code quality
Edge Consistency Bidirectional component relationships
Control-Risk References Cross-reference validation
Graph Generation All three graph types validated

Manual Validation Commands

bash
# Component edge validation
python scripts/hooks/validate_riskmap.py --force

# Control-risk reference validation
python scripts/hooks/validate_control_risk_references.py --force

# Framework reference validation
python scripts/hooks/validate_framework_references.py --force

# Generate markdown tables
python3 scripts/hooks/yaml_to_markdown.py --all --all-formats

# Run full pre-commit suite
.git/hooks/pre-commit --force

Handling CI Failures

bash
# Regenerate graphs
python scripts/hooks/validate_riskmap.py --to-graph ./risk-map/diagrams/risk-map-graph.md --force
python scripts/hooks/validate_riskmap.py --to-controls-graph ./risk-map/diagrams/controls-graph.md --force
python scripts/hooks/validate_riskmap.py --to-risk-graph ./risk-map/diagrams/controls-to-risk-graph.md --force

# Commit updated graphs
git add risk-map/diagrams/
git commit -m "Update generated graphs"

10. Quick Reference Tables

Risk-to-Control Mapping (Summary)

Risk Primary Controls
Data Poisoning (DP) Training Data Sanitization, Secure ML Tooling, Integrity Management
Prompt Injection (PIJ) Input/Output Validation, Adversarial Training
Model Exfiltration (MXF) Access Controls, Inventory Management, Confidential Computing
Sensitive Data Disclosure (SDD) Privacy Technologies, Output Validation, Agent Controls
Rogue Actions (RA) Agent Permissions, User Control, Output Validation
Model Deployment Tampering (MDT) Execution Integrity, Secure Tooling, Route Integrity

Universal Controls (Apply to All Risks)

  • Red Teaming
  • Vulnerability Management
  • Threat Detection
  • Incident Response Management
  • Internal Policies and Education
  • Product Governance
  • Risk Governance

Self-Assessment Questions

The Risk Map includes a self-assessment with 12 questions to help organizations identify relevant risks:

  1. Do you have robust training data management?
  2. Can you detect/remediate malicious data changes?
  3. Is sensitive user data used in training?
  4. Do you manage user data according to consent?
  5. Do you have a complete inventory of models/datasets?
  6. Do you have robust access controls?
  7. Can you ensure integrity protection?
  8. Are frameworks/libraries analyzed for vulnerabilities?
  9. Do you protect against large-scale malicious queries?
  10. Are you using secure-by-default designs?
  11. Do you perform adversarial testing?
  12. Do you deploy AI-powered agents?

Appendix: Key Files Reference

File Purpose
yaml/components.yaml 23 component definitions with edges
yaml/controls.yaml 28 control definitions
yaml/risks.yaml 26 risk definitions
yaml/frameworks.yaml 4 external framework definitions
yaml/personas.yaml 2 persona definitions
yaml/mermaid-styles.yaml Graph styling configuration
docs/developing.md Main documentation index
docs/validation.md Validation tools reference
docs/guide-*.md Content addition guides

This document was auto-generated from the CoSAI Risk Map documentation, YAML data files, and generated tables.

File: ai/ai-security/model_and_genai_application_security_testing.md

Model and GenAI Application Security Testing

This file is an entry point for testing the security of AI/ML models and GenAI applications. Use it together with:

  • ai_security_tools.md – curated list of offensive and defensive AI security tools.
  • ai_risk_management/README.md – governance, regulatory, and risk management frameworks.
  • prompt_injection/README.md – prompt injection, jailbreak techniques, and defenses.
  • training_environment_security/README.md – securing training and fine‑tuning environments.

1. Core Guidance and Taxonomies

2. Risk Maps and Secure Design Patterns

3. Practical Testing and Tooling

See ai_security_tools.md for:

  • Adversarial robustness tools (ART, Armory, Foolbox, TextAttack, etc.).
  • GenAI red teaming tools (PyRIT, Garak, Promptfoo, Guardrail(s), jailbreak evaluation suites).
  • Prompt firewalls and redaction tools (Cisco AI Defense, Robust Intelligence, Lakera Guard, Rebuff, LLM Guard, etc.).

These support:

  • Pre‑deployment testing – evasion, data leakage, jailbreaks, prompt injection.
  • Runtime testing – continuous red teaming, regression testing for guardrails, and monitoring false positives/negatives (see monitoring.md for observability tools like LangSmith, Langfuse, OpenLLMetry, and Graphsignal).

File: ai/ai-security/model_security_testing.md

Please see https://github.com/The-Art-of-Hacking/h4cker/blob/master/ai/ai_security_tools.md

Julia Adversarial ML Frameworks

  • Mirage - Adversarial machine learning framework in Julia. Implements evasion attacks (FGSM, PGD, C&W), model extraction, membership inference, and robustness testing. Includes defenses like adversarial training and certified robustness. 7,000+ lines of Julia code.

File: ai/ai-security/tools.md

Ai Research Tools

This is a curated list of tools for this category.

File: ai/ai-security/agent-skills/README.md

Agent Skills

An Agent Skill is a self-contained, file-based package that teaches an AI agent how to do a specific job well. Think of it less as code and more as a playbook the agent loads on demand: when the user's request matches the skill's description, the agent reads the skill's instructions, optional reference docs, templates, and helper scripts, and uses them to produce a higher-quality, more consistent output than it could from its training data alone.

The format Anthropic introduced (and the one this repo follows) is essentially a folder containing:

  • A SKILL.md file at the root, with YAML frontmatter (name, description) plus the body of instructions. The frontmatter is what the agent matches on — the description is the trigger.
  • Optional support folders the SKILL.md can reference: references/ (background knowledge the agent reads when needed), templates/ (skeletons to fill in), scripts/ (deterministic Python/Bash the agent can execute instead of "hallucinating" the same logic), and assets/ (schemas, images, etc.).

Why this matters in practice: the agent doesn't load all skills into its context. It scans the descriptions, picks the relevant skill(s), and pulls in only those files. This is called progressive disclosure — it keeps the context window small while making large bodies of expertise available. It's also portable: the same skill folder works in Claude apps, Claude Code, the Anthropic API's Agent SDK, and any other runtime that implements the spec.

The pattern is especially powerful for security work because so much of cyber defense is codified procedure — detection logic, hunt hypotheses, IR runbooks, mitigation checklists — that benefits from being stored as deterministic artifacts the agent can lean on rather than improvise.

Example: `mitre-attack-agent-skills`

In this repo at I created santosomar/mitre-attack-agent-skills, I created one Agent Skill per non-deprecated MITRE ATT&CK technique, sourced from the official mitre-attack/attack-stix-data STIX bundles. The counts break down as 697 enterprise, 97 ICS, and 124 mobile techniques — 918 skills total, organized like this:

text
mitre-attack-agent-skills/
├── enterprise/
├── mobile/
├── ics/
├── manifest.csv
├── manifest.json
└── README.md

Each skill folder is named with the pattern attack-<domain>-<technique-id>-<technique-slug> — so the skill for T1059 (Command and Scripting Interpreter) on enterprise would be something like attack-ent-t1059-command-and-scripting-interpreter/. Inside every skill you get a consistent structure:

  • SKILL.md — frontmatter naming the skill and a description that includes the ATT&CK ID, technique name, domain, and defensive use cases (so the agent can match a user query like "help me hunt for PowerShell abuse" to T1059.001).
  • references/technique-profile.json — structured technique metadata
  • references/detection-and-mitigation.md — defensive guidance
  • references/known-threat-context.md — threat actors and malware observed using the technique
  • templates/detection-brief.md, hunt-plan.md, incident-response-note.md, coverage-assessment.md — fill-in-the-blank artifacts
  • scripts/render_brief.py — deterministic helper for output generation
  • assets/output-schema.json — schema the agent's output should conform to

The repo is explicitly defensive. It's scoped to triage, detection engineering, threat hunting, mitigation, coverage assessment, IR, and authorized validation — not malware development or offensive tradecraft. That scoping lives in the SKILL.md descriptions themselves, which is what shapes how the agent uses them.

What this looks like in use

Imagine you're a SOC analyst and you ask a Claude-powered agent: "We saw suspicious encoded PowerShell from a finance workstation last night. Help me write a hunt plan and a detection brief."

The agent does roughly this:

  1. Scans skill descriptions, matches attack-ent-t1059-001-powershell (and probably parents like T1059).
  2. Loads that skill's SKILL.md, which tells it how to use the bundled resources.
  3. Pulls in references/known-threat-context.md to enumerate actors known to abuse PowerShell (FIN7, APT groups, etc.) and detection-and-mitigation.md for the defensive playbook.
  4. Fills in templates/hunt-plan.md and templates/detection-brief.md with your specific scenario, conforming to assets/output-schema.json.
  5. Returns a structured, ATT&CK-aligned artifact you can drop into your ticket or detection-engineering pipeline.

Without the skill, the agent would still produce something about PowerShell hunting — but it'd be inconsistent, untethered from the canonical ATT&CK profile, and missing the standardized output shape your team expects. The skill turns "ask the LLM for advice" into "execute a vetted procedure with the LLM as the engine."

Why this pattern is interesting for AI-for-cyber

A few things stand out about my approach that are worth stealing for your own security skills:

The one-skill-per-technique granularity maps cleanly to how analysts already think (in ATT&CK IDs), so retrieval is precise. Coarser skills ("help with detections") would force the LLM to do its own taxonomy work and would degrade output quality.

Templates plus schemas equal consistency. When 50 different analysts use the skill, they get 50 detection briefs in the same shape, which is what makes downstream automation (SIEM imports, ticketing, coverage dashboards) feasible.

Scripts inside skills move deterministic work out of the LLM. Rendering a brief, parsing STIX, computing coverage scores — none of that should be left to token generation. The render_brief.py pattern is the right instinct, and you can extend it: a skill could ship a script that queries your SIEM, runs a Sigma converter, or pulls fresh CTI before the LLM composes the final narrative.

Got it — updating to reflect that.

Another Example: Project CodeGuard

Another good example to look at is Project CodeGuard (cosai-oasis/project-codeguard). Full disclosure — I started CodeGuard at Cisco and I'm one of the core maintainers, so this is a project I know intimately. I think it's a useful counterpoint to my MITRE ATT&CK skill pack because it shows a different shape of Agent Skill solving a different problem, but built on the same underlying pattern.

Where my MITRE repo is a library of many narrow skills (one per ATT&CK technique, retrieved on demand based on what the analyst is asking about), CodeGuard is a single broad skill (software-security) that's always in the activation pool whenever code is being written or reviewed. Both are valid Agent Skill designs, and looking at them side-by-side is a good way to understand the design space.

What it is

We started CodeGuard inside Cisco's AI-enabled Security Engineering, Security & Trust team, and then contributed it as an open project under CoSAI (the Coalition for Secure AI, an OASIS Open Project) so the broader community could shape it. The goal is narrow and important: stop AI coding agents from generating insecure code by default. AI assistants will cheerfully produce code that hardcodes secrets, concatenates SQL, picks MD5 for hashing, skips input validation, or disables certificate checks — not because the model "wants" to, but because nothing in the prompt context tells it otherwise.

We address that by shipping a corpus of model-agnostic security rules authored in unified Markdown, plus translators that convert them into the native rule/skill formats of every major AI coding tool — Claude Code, Cursor, Windsurf, GitHub Copilot, Google Antigravity, OpenAI Codex, OpenCode, and others. The same security knowledge, written once, ends up steering whichever agent the developer happens to be using. That portability was a deliberate design call — we didn't want to bet the project on one vendor's skill format, and a developer's choice of IDE shouldn't determine whether their AI assistant follows secure-by-default practices.

How it's packaged as an Agent Skill

The Claude Code distribution of CodeGuard is packaged exactly as an Agent Skill, and we explicitly define the term in our docs: "Agent Skills are model-invoked capabilities that Claude autonomously uses based on task context." That's the same model I described in the MITRE example — the agent matches a request to a skill description, then loads the skill's instructions and supporting files only when relevant.

The plugin layout looks like this:

text
project-codeguard/
├── .claude-plugin/
│   ├── plugin.json
│   └── marketplace.json
├── sources/                     # Authored inputs
│   ├── rules/core/              # Core security rules (Markdown)
│   ├── rules/owasp/             # OWASP supplementary rules
│   └── skills/                  # Authored skill definitions
├── skills/                      # Generated Claude Code skill
│   └── software-security/
│       ├── SKILL.md             # The skill entry point
│       └── rules/               # 23 rule files Claude references
└── src/
    └── convert_to_ide_formats.py  # Translator to other agent formats

The SKILL.md for software-security does what every well-designed skill does: it tells Claude when to activate (any time code is being written, reviewed, or modified — especially when credentials, crypto, input handling, auth, APIs, cloud config, or sensitive data are involved) and what workflow to follow once it's active. We landed on a clean three-step pattern:

  1. Initial security check — figure out which rules apply (language, security domains involved, whether credentials are in scope).
  2. Code generation — apply the secure-by-default patterns from those rules and add comments explaining the security choices.
  3. Security review — run the implementation checklists from each rule, verify no hardcoded secrets, and surface to the developer which rules were applied.

How the rules are organized

This is where some of the design choices we made get interesting. The 22 rules split into two tiers:

Always-apply rules (3) — checked on every code operation, regardless of language or context:

  • codeguard-1-hardcoded-credentials — no secrets, API keys, or tokens in source
  • codeguard-1-crypto-algorithms — ban MD5/SHA-1/DES, require modern algorithms
  • codeguard-1-digital-certificates — validate expiration, key strength, signature algorithms

Context-specific rules (19) — pulled in only when relevant: input validation and injection, authentication/MFA, authorization and access control, session management, API and web service security, client-side web security, data storage, privacy and data protection, logging, additional cryptography, file handling and uploads, XML and serialization, supply chain, DevOps/CI-CD/containers, cloud and Kubernetes, IaC, frameworks and languages, mobile apps, and safe C functions for memory safety.

That two-tier split was a deliberate choice — and it took a couple of iterations to get right. In v1.0.0 we had four always-apply rules; in v1.0.1 we moved safe-c-functions to context-specific because it doesn't make sense to load C/C++ memory-safety guidance into context when someone is writing Python. That tuning matters: every rule that's "always-apply" costs context tokens and adds noise, so we reserve that tier for things that genuinely apply to every line of code an AI agent might write.

Why this is a different Agent Skill pattern than the MITRE pack

Comparing the two designs is the most useful part of looking at them together:

Dimension mitre-attack-agent-skills project-codeguard
Granularity One skill per technique (918 skills) One skill, 22 rules inside it
Activation Triggered by specific ATT&CK-shaped query Activates on any code work
User SOC analyst, detection engineer, IR Developer using an AI coding agent
Output shape Hunt plans, detection briefs, IR notes Secure code + security commentary
Portability Anthropic skill format only Translated to ~8 agent formats
Lifecycle stage Detect / respond Build / shift-left

Both are valid; they're optimized for different problems. When the user's query maps cleanly to a taxonomy with hundreds of nodes, fine-grained skills win because retrieval precision matters. When the skill needs to be ambient — running in the background of every code operation — a single broad skill with internal tiering wins because you can't realistically dispatch to one of 22 micro-skills on every keystroke.

Why this matters for AI security specifically

CodeGuard sits in a part of the AI-security stack I think is underdeveloped: shaping what AI coding agents produce, rather than only scanning what they produce after the fact. SAST/DAST tools and AI code reviewers are necessary, but they catch problems after the agent has already written insecure code, after a human has potentially accepted it, and after it's potentially merged. A security skill that ships with the agent shifts that left to the moment of generation.

It's also a pattern that scales: the same skill format can encode AI/ML-specific guidance (prompt-injection-resistant code, safe LLM-calling patterns, secure agent tool definitions, MCP server hardening). That's a direction we're actively pushing the project, and it's where I see Agent Skills becoming part of the standard secure-development toolkit rather than a niche capability.

If anyone wants to contribute rules — particularly for AI/agent-specific threats — issues and discussions are open on the repo.

The defensive-only framing is intentional and load-bearing. Skill descriptions don't just route requests — they shape what the agent will and won't help with. Putting the scope in the description means the agent self-selects out of misuse cases without the user even seeing a refusal.

If you wanted to extend this idea, natural follow-ups would be a parallel skill pack for D3FEND countermeasures (so the agent can pair every ATT&CK technique with a mapped defense), a pack for Atomic Red Team tests (for purple-team validation), or a pack mapping CWE/CVE to detection guidance. The repo's structure — manifest files, validation summary, per-technique folders — is a reusable scaffold for all of those.

File: ai/ai-security/ai-algorithmic-red-teaming/README.md

🧠🔥 AI Algorithmic Red Teaming

A framework and methodology for proactively testing, validating, and hardening AI systems against adversarial threats, systemic risks, and unintended behaviors.

🚩 What is Algorithmic Red Teaming?

AI Algorithmic Red Teaming is a structured, adversarial testing process that simulates real-world attacks and misuse scenarios against AI models, systems, and infrastructure. It mirrors traditional cybersecurity red teaming — but focuses on probing the behavior, bias, robustness, and resilience of machine learning (ML) and large language model (LLM) systems.

🎯 Objectives

  • Expose vulnerabilities in AI systems through adversarial testing
  • Evaluate robustness to adversarial inputs, data poisoning, and model extraction
  • Test system alignment with security, privacy, and ethical policies
  • Validate controls against overreliance, excessive agency, prompt injection, and insecure plugin design
  • Contribute to AI safety and governance efforts by documenting and mitigating critical risks

OWASP and Cloud Security Alliance (CSA) Guidance

🧩 Key Components

1. Attack Categories

  • Prompt Injection & Jailbreaking
  • Model Evasion (Adversarial Examples)
  • Data Poisoning & Backdoor Attacks
  • Model Extraction (Stealing)
  • Inference Manipulation & Overreliance
  • Sensitive Information Disclosure
  • Insecure Plugin / Tool Use
  • RAG-Specific Attacks (Embedding Manipulation, Vector Leakage)

2. Evaluation Metrics

  • Attack success rate
  • Confidence degradation
  • Output alignment drift
  • Hallucination frequency
  • Guardrail bypass percentage
  • Latency and inference impact

3. Test Surfaces

  • LLM APIs (OpenAI, Claude, Gemini, open-source)
  • Embedding models and vector databases
  • Retrieval-Augmented Generation (RAG) systems
  • Plugin-based LLM architectures
  • Agentic AI frameworks (e.g., CrewAI, AutoGen, LangGraph, and others)
  • Proprietary models in deployment environments

🛠️ Tools & Frameworks

Look under the AI Security Tools section.

File: ai/ai-security/ai-risk-management/README.md

AI Risk Management Frameworks and AI Security Resources

NIST Resources

Europe

Canada

Asia-Pacific

ISO

Cloud Security Alliance

US Government Resources

Additional Securing AI Resources

Industry-Specific Resources

Academia

File: ai/ai-security/ai-risk-management/detecting_ai.md

Detecting AI Usage Within a Company: Strategies and Best Practices

Overview

As AI adoption accelerates across enterprises, organizations need comprehensive strategies to detect, monitor, and govern AI usage. This includes both sanctioned AI tools and "shadow AI" - unauthorized AI applications that employees may be using without IT oversight.

Comprehensive AI Inventory and Discovery

Initial Assessment

You should always perform a comprehensive inventory of existing AI tools and applications within the company. This involves:

  • Departmental Engagement: Identify AI tools across all departments:

    • Customer service (chatbots, sentiment analysis)
    • Marketing (predictive analytics, content generation)
    • HR (resume screening, candidate matching)
    • Finance (fraud detection, automated accounting)
    • Sales (lead scoring, CRM automation)
    • IT/Security (threat detection, log analysis)
    • Legal (contract analysis, compliance monitoring)

  • License and Subscription Audit: Review software licenses and subscriptions for AI-related tools

  • Network and System Audits: Work with IT to audit systems and networks for AI software and services

  • API Usage Analysis: Monitor API calls to AI services (OpenAI, Anthropic, Google AI, etc.)

Shadow AI Detection

  • Network Traffic Analysis: Monitor for connections to known AI service endpoints
  • Browser Extension Monitoring: Detect AI-powered browser extensions and plugins
  • Cloud Service Usage: Audit cloud service logs for AI/ML service consumption
  • Employee Surveys: Conduct regular surveys about AI tool usage
  • Expense Report Analysis: Review expense reports for AI service subscriptions

AI Bill of Materials (AI BOMs)

I published an article that explains AI BOMs in detail.

AI BOM vs SBOM

  • Software Bill of Materials (SBOMs): Document software application components
  • AI Bill of Materials (AI BOMs): Document AI system components including:
    • Model details and architecture
    • Training data sources and characteristics
    • Usage patterns and access controls
    • Performance metrics and limitations
    • Bias and fairness assessments
    • Security and privacy considerations

AI BOM Implementation

  • Model Registry: Maintain a centralized registry of all AI models
  • Data Lineage Tracking: Document data sources and transformations
  • Version Control: Track model versions and updates
  • Risk Assessment: Document potential risks and mitigation strategies

Technical Detection Methods

Code Analysis and Static Detection

The following resources provide rules and patterns for detecting AI usage in code:

  • Semgrep Rules: https://github.com/semgrep/semgrep-rules/tree/develop/ai
  • Custom Detection Patterns:
    • API calls to AI services (OpenAI, Anthropic, Cohere, etc.)
    • ML/AI library imports (transformers, langchain, openai, etc.)
    • Model loading and inference code
    • AI-related configuration files

Network-Based Detection

  • DNS Monitoring: Track queries to AI service domains
  • SSL/TLS Certificate Analysis: Identify connections to AI platforms
  • Bandwidth Analysis: Monitor for unusual data transfer patterns
  • API Gateway Logs: Analyze API usage patterns

Runtime Detection

  • Process Monitoring: Detect AI-related processes and services
  • Memory Usage Patterns: Identify memory-intensive AI workloads
  • GPU Utilization: Monitor GPU usage for AI inference
  • Container and Pod Analysis: Scan for AI-related containers

Enterprise AI Detection Tools

Commercial Solutions

  • Microsoft Purview: AI governance and compliance monitoring
  • IBM Watson OpenScale: AI model monitoring and governance
  • DataRobot MLOps: Model lifecycle management and monitoring
  • Domino Data Lab: Enterprise MLOps platform with governance features

Open Source Tools

  • MLflow: Model tracking and registry
  • Kubeflow: Kubernetes-native ML workflows
  • Apache Airflow: Workflow orchestration with ML capabilities
  • Weights & Biases: Experiment tracking and model monitoring

Security-Focused Tools

  • Protect AI: AI/ML security scanning
  • HiddenLayer: AI model security and monitoring
  • Robust Intelligence: AI model validation and monitoring

AI Content Detection

Text-Based AI Detection

  • GPTZero: Academic and enterprise AI text detection
  • Originality.ai: Content authenticity verification
  • Winston AI: Multi-language AI content detection
  • Turnitin: Academic integrity with AI detection capabilities
  • Copyleaks: AI content detection and plagiarism checking

Image and Media Detection

  • Deepware Scanner: Deepfake detection
  • Sensity: Synthetic media detection
  • Microsoft Video Authenticator: Video manipulation detection
  • Adobe Content Authenticity Initiative: Media provenance tracking

Limitations of AI Detection

  • False Positives: Human-written content flagged as AI-generated
  • Evolving Models: Detection tools lag behind new AI capabilities
  • Adversarial Techniques: Methods to evade detection
  • Language Variations: Reduced accuracy for non-English content

Governance and Policy Framework

AI Usage Policies

  • Acceptable Use Policies: Define approved AI tools and use cases
  • Data Privacy Requirements: Ensure AI tools comply with privacy regulations
  • Security Standards: Establish security requirements for AI tools
  • Vendor Assessment: Evaluate AI service providers for compliance

Monitoring and Compliance

  • Regular Audits: Periodic assessment of AI tool usage
  • Risk Assessments: Evaluate potential risks of AI implementations
  • Incident Response: Procedures for AI-related security incidents
  • Training Programs: Educate employees on AI governance policies

AI Watermarking

  • Google SynthID: Watermarking for AI-generated content
  • OpenAI Watermarking: Text watermarking research
  • Meta Watermarking: Image and video watermarking techniques

Advanced Detection Methods

  • Behavioral Analysis: Detecting AI usage patterns in user behavior
  • Linguistic Forensics: Advanced text analysis for AI detection
  • Multimodal Detection: Combined analysis of text, images, and metadata
  • Blockchain Provenance: Using blockchain for content authenticity

Regulatory Developments

  • EU AI Act: Compliance requirements for AI systems
  • NIST AI Risk Management Framework: Guidelines for AI governance
  • Industry Standards: Emerging standards for AI transparency and accountability

Implementation Roadmap

Phase 1: Discovery and Assessment

  1. Conduct comprehensive AI inventory
  2. Implement basic monitoring tools
  3. Establish baseline policies

Phase 2: Enhanced Monitoring

  1. Deploy advanced detection tools
  2. Implement AI BOMs
  3. Establish governance processes

Phase 3: Continuous Improvement

  1. Regular policy updates
  2. Advanced threat detection
  3. Integration with security operations

Best Practices

Technical Best Practices

  • Defense in Depth: Multiple layers of detection and monitoring
  • Automated Scanning: Regular automated scans for AI usage
  • Integration: Integrate AI detection with existing security tools
  • Documentation: Maintain detailed records of AI systems and usage

Organizational Best Practices

  • Cross-Functional Teams: Include legal, security, and business stakeholders
  • Regular Training: Keep teams updated on AI developments
  • Vendor Management: Establish clear requirements for AI vendors
  • Incident Response: Prepare for AI-related security incidents

Resources and References

File: ai/ai-security/ai_risk_management/README.md

AI Risk Management Frameworks and AI Security Resources

NIST Resources

Europe

Canada

Asia-Pacific

ISO

Cloud Security Alliance

US Government Resources

Additional Securing AI Resources

OWASP GenAI Security Project

  • OWASP GenAI Security Project – global community project providing:
    • LLM Top 10 for 2025
    • AI Security Landscape and Solutions Reference Guide (Q2–Q3 2025)
    • Threat Defense COMPASS 1.0 – cross-mapping of threats, vulnerabilities, and mitigations
    • Initiatives for agentic app security, AI threat intelligence, secure AI adoption, and data security

Coalition for Secure AI (CoSAI)

Industry-Specific Resources

Academia

File: ai/ai-security/MCP-Security/README.md

[Binary file]

File: ai/ai-security/openclaw/README.md

Agentic AI Security Resources: OpenClaw, DefenseClaw, and MAESTRO

This document summarizes three useful resources for an AI security curriculum: OpenClaw as an example of an agent framework with serious security concerns, DefenseClaw as a governance layer for agentic AI, and MAESTRO as a threat modeling framework designed for agentic systems.

Overview

Topic What it is Why it matters
OpenClaw An open-source agentic AI framework and personal AI agent platform with tool use, automation, and integration capabilities. It is a strong case study for agentic AI risk because public reporting and security research described major flaws involving excessive privileges, exposed services, poor secrets handling, and supply-chain concerns.
DefenseClaw A Cisco AI Defense governance layer for OpenClaw and related agentic AI deployments. It shows how scanning, policy enforcement, guardrails, and auditing can be added around agent runtimes to reduce operational risk. It also includes an agent skill scanner, MCP scanner, and Project CodeGuard.
MAESTRO A threat modeling framework for agentic AI developed through the Cloud Security Alliance ecosystem. It helps security teams reason about layered, cross-system, and emergent threats that traditional software threat models do not fully capture.

OpenClaw

OpenClaw is an open-source agent framework designed to let AI agents act on behalf of users, including interacting with tools, files, terminals, cloud services, and communication platforms. That power makes it useful for demonstrations of agentic autonomy, but it also expands the attack surface dramatically because the agent can process untrusted content while holding significant system privileges.

Public reporting and security analysis have described OpenClaw as a high-risk platform when deployed without strong isolation and hardening. Reported issues include exposed administrative interfaces, insecure trust assumptions around localhost and reverse proxies, plaintext secret storage, prompt-injection-driven data theft, and malicious skills distributed through the ecosystem marketplace.

DefenseClaw

DefenseClaw is an open-source security governance layer created for agentic AI, especially for environments built with OpenClaw. It is positioned as the control plane around the agent runtime, enforcing policies and providing visibility into skills, tools, MCP servers, plugins, and LLM traffic.

The main value of DefenseClaw is that it wraps the runtime instead of requiring teams to rewrite their agents from scratch. According to the documentation and repository materials, it combines scanning, enforcement, and auditing so organizations can evaluate agent components before deployment, govern runtime behavior, and maintain an audit trail for security operations and compliance needs.

MAESTRO

MAESTRO stands for Multi-Agent Environment, Security, Threat, Risk, and Outcome, and it was created to support threat modeling for agentic AI systems. The framework was developed to address a gap in traditional security methods, which often do not fully capture risks such as emergent behavior, multi-agent interactions, dynamic tool use, and cross-layer attack propagation.

A central idea in MAESTRO is its layered view of agentic AI, where teams model threats across foundation models, data operations, orchestration layers, tools, infrastructure, observability, and ecosystem interactions. This makes MAESTRO useful for teaching because it gives students a structured way to connect classic security ideas with AI-specific risks such as prompt injection, poisoned retrieval pipelines, goal misalignment, and malicious agent ecosystems.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ai Browse ai catalog → #ai-security Browse ai-security catalog → #artificial-intelligence Browse artificial-intelligence catalog → #awesome-list Browse awesome-list catalog → #awesome-lists Browse awesome-lists catalog → #cybersecurity Browse cybersecurity catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →