# Technical Documentation: chaosblade-io/chaosblade > ℹ️ **Provenance:** Hybrid Fusion: `chaosblade-io/chaosblade` (README + 8 In-Tree Chapters) · [CodeWiki Reference](https://codewiki.google/github.com/chaosblade-io/chaosblade) · Recency: Active (< 180 days) ## 1. Project Overview & Quickstart (chaosblade-io/chaosblade) # Chaosblade: An Easy to Use and Powerful Chaos Engineering Toolkit [](https://travis-ci.org/chaosblade-io/chaosblade) [](https://opencollective.com/chaosblade) [](https://codecov.io/gh/chaosblade-io/chaosblade) [](https://bestpractices.coreinfrastructure.org/projects/5032) 中文版 [README](README_CN.md) Wiki: [DeepWiki](https://deepwiki.com/chaosblade-io/chaosblade-for-deepwiki) ## 🔥🔥🔥 Major UPDATE:Chaosblade Agent Released - Blade AI Blade AI serves as the intelligent agent layer within the ChaosBlade ecosystem: at the foundational level, it invokes ChaosBlade to execute fault injection; at the upper level, it incorporates orchestration capabilities such as intent understanding, security auditing, effect verification, safe recovery, and structured reporting—thereby transforming fault drills from a process of "manually writing commands" into one completed through "conversational interaction." For detailed information, please refer to: [Blade AI README](https://github.com/chaosblade-io/chaosblade/blob/feature/blade-ai/blade-ai/README_en.md) * Release: [blade-ai-v0.1.0](https://github.com/chaosblade-io/chaosblade/releases/tag/blade-ai-v0.1.0) * Code Branch: blade-ai-v0.1.0 ## Introduction ChaosBlade is an Alibaba open source experimental injection tool that follows the principles of chaos engineering and chaos experimental models to help enterprises improve the fault tolerance of distributed systems and ensure business continuity during the process of enterprises going to cloud or moving to cloud native systems. Chaosblade is an internal open source project of MonkeyKing. It is based on Alibaba's nearly ten years of failure testing and drill practice, and combines the best ideas and practices of the Group's businesses. ChaosBlade is not only easy to use, but also supports rich experimental scenarios. The scenarios include: * Basic resources: such as CPU, memory, network, disk, process and other experimental scenarios; * Java applications: such as databases, caches, messages, JVM itself, microservices, etc. You can also specify any class method to inject various complex experimental scenarios; * C ++ applications: such as specifying arbitrary methods or experimental lines of code injection delay, tampering with variables and return values; * container: such as killing the container, the CPU in the container, memory, network, disk, process and other experimental scenarios; * Cloud-native platforms: For example, CPU, memory, network, disk, and process experimental scenarios on Kubernetes platform nodes, Pod network and Pod itself experimental scenarios such as killing Pods, and container experimental scenarios such as the aforementioned Docker container experimental scenario; Encapsulating scenes by domain into individual projects can not only standardize the scenes in the domain, but also facilitate the horizontal and vertical expansion of the scenes. By following the chaos experimental model, the chaosblade cli can be called uniformly. The items currently included are: * [chaosblade](https://github.com/chaosblade-io/chaosblade): Chaos experiment management tool, including commands for creating experiments, destroying experiments, querying experiments, preparing experimental environments, and canceling experimental environments. It is the execution of chaotic experiments. Tools, execution methods include CLI and HTTP. Provides complete commands, experimental scenarios, and scenario parameter descriptions, and the operation is simple and clear. * [chaosblade-spec-go](https://github.com/chaosblade-io/chaosblade-spec-go): Chaos experimental model Golang language definition, scenes implemented using Golang language are easy to implement based on this specification. * [chaosblade-exec-os](https://github.com/chaosblade-io/chaosblade-exec-os): Implementation of basic resource experimental scenarios. * [chaosblade-exec-docker](https://github.com/chaosblade-io/chaosblade-exec-docker): Docker container experimental scenario implementation, standardized by calling the Docker API. * [chaosblade-exec-cri](https://github.com/chaosblade-io/chaosblade-exec-cri): Container experimental scenario implementation, standardized by calling the CRI. * [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator): Kubernetes platform experimental scenario is implemented, chaos experiments are defined by Kubernetes standard CRD method, it is very convenient to use Kubernetes resource operation method To create, update, and delete experimental scenarios, including using kubectl, client-go, etc., and also using the chaosblade cli tool described above. * [chaosblade-exec-jvm](https://github.com/chaosblade-io/chaosblade-exec-jvm): Java application experimental scenario implementation, using Java Agent technology to mount dynamically, without any access, zero-cost use It also supports uninstallation and completely recycles various resources created by the Agent. * [chaosblade-exec-cplus](https://github.com/chaosblade-io/chaosblade-exec-cplus): C ++ application experimental scenario implementation, using GDB technology to implement method and code line level experimental scenario injection. * [chaosblade-box](https://github.com/chaosblade-io/chaosblade-box): Possessing chaos engineering platform and resilience testing platform capabilities.For more information on the resilience testing platform capabilities, see the [main2](https://github.com/chaosblade-io/chaosblade-box/tree/main2) branch. ## Quick Start This guide helps you run your first fault injection on Kubernetes in under 5 minutes using ChaosBlade. We'll inject a CPU stress fault into a Pod as a minimal example. ### Prerequisites Before you begin, make sure you have: - [ ] **kubectl access** to a running Kubernetes cluster (`kubectl cluster-info` returns successfully) - [ ] A **target namespace** with at least one running Pod (default: `default`) - [ ] **ChaosBlade installed**: - [ ] The [chaosblade](https://github.com/chaosblade-io/chaosblade/releases) CLI toolkit downloaded and extracted (the `blade` binary is on your `PATH`) - [ ] The [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator/releases) deployed to your cluster (required for Kubernetes scenarios) Install the operator with Helm: ```shell script helm install chaosblade-operator chaosblade-operator-.tgz --namespace chaosblade --create-namespace ``` Verify the operator is running: ```shell script kubectl get pods -n chaosblade ``` ### Inject Your First Fault (CPU Stress) Run a cpu full-load fault against a Pod in the `default` namespace: ```shell script blade create k8s pod-cpu fullload --cpu-percent 80 --kubeconfig ~/.kube/config --names --namespace default ``` If the injection succeeds, ChaosBlade returns a JSON result containing an experiment `uid`. Save this `uid` to check status or destroy the experiment later: ```json {"code":200,"success":true,"result":""} ``` ### Check the Experiment Status ```shell script blade status ``` ### Recover (Destroy the Fault) Always recover after your drill to restore the target to normal: ```shell script blade destroy ``` That's it! You've completed a full inject-verify-recover cycle. To explore more scenarios, run `blade create k8s -h` or see [Chaos Engineering Practice under Cloud Native](CLOUDNATIVE.md). ## CLI Command You can download the latest chaosblade toolkit from [Releases](https://github.com/chaosblade-io/chaosblade/releases) and extract it and use it. If you want to inject Kubernetes related fault scenarios, you need to install [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator/releases). For detailed Chinese usage documents, please see [chaosblade-help-zh-cn ](https://chaosblade-io.gitbook.io/chaosblade-help-zh-cn/). chaosblade supports CLI and HTTP invocation methods. The supported commands are as follows: * **prepare**: alias is p, preparation before the chaos engineering experiment, such as drilling Java applications, you need to attach the java agent. For example, to drill an application whose application name is business, execute `blade p jvm --process business` on the target host. If the attach is successful, return the uid for status query or agent revoke. * **revoke**: alias is r, undo chaos engineering experiment preparation before, such as detaching java agent. The command is `blade revoke UID` * **create**: alias is c, create a chaos engineering experiment. The command is `blade create [TARGET] [ACTION] [FLAGS]`. For example, if you implement a Dubbo consumer call xxx.xxx.Service interface delay 3s, the command executed is `blade create dubbo delay --consumer --time 3000 --Service xxx.xxx.Service`, if the injection is successful, return the experimental uid for status query and destroy the experiment. * **destroy**: alias is d, destroy a chaos engineering experiment, such as destroying the Dubbo delay experiment mentioned above, the command is `blade destroy UID` * **status**: alias s, query preparation stage or experiment status, the command is `blade status UID` or `blade status --type create` * **server**: start the web server, expose the HTTP service, and call chaosblade through HTTP requests. For example, execute on the target machine xxxx: `blade server start -p 9526` to perform a CPU full load experiment:` curl "http://xxxx:9526/chaosblade?cmd=create%20cpu%20fullload" ` Use the `blade help [COMMAND]` or `blade [COMMAND] -h` command to view help ## Experience Demo Download the chaosblade demo image and experience the use of the blade toolkit Download image command: ```shell script docker pull chaosbladeio/chaosblade-demo ``` Run the demo container: ```shell script docker run -it --privileged chaosbladeio/chaosblade-demo ``` After entering the container, you can read the README.txt file to implement the chaos experiment, Enjoy it. ## Cloud Native [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator) The project is a chaos experiment injection tool for cloud-native platforms. It follows the chaos experiment model to standardize the experimental scenario and defines the experiment as Kubernetes CRD Resources, mapping experimental models to Kubernetes resource attributes, and very friendly combination of chaotic experimental models with Kubernetes declarative design. While relying on chaotic experimental models to conveniently develop scenarios, it can also well integrate Kubernetes design concepts, through kubectl or Write code to directly call the Kubernetes API to create, update, and delete chaotic experiments, and the resource status can clearly indicate the execution status of the experiment, and standardize Kubernetes fault injection. In addition to using the above methods to perform experiments, you can also use the chaosblade cli method to execute kubernetes experimental scenarios and query the experimental status very conveniently. For details, please read the chinese document: [Chaos Engineering Practice under Cloud Native](CLOUDNATIVE.md) ## Compile See [BUILD.md](BUILD.md) for the details. ## Bugs and Feedback For bug report, questions and discussions please submit [GitHub Issues](https://github.com/chaosblade-io/chaosblade/issues). You can also contact us via: * Dingding group (recommended for chinese): 23177705 * Slack group: [chaosblade-io](https://join.slack.com/t/chaosblade-io/shared_invite/zt-f0d3r3f4-TDK13Wr3QRUrAhems28p1w) * Gitter room: [chaosblade community](https://gitter.im/chaosblade-io/community) * Email: chaosblade.io.01@gmail.com * Twitter: [chaosblade.io](https://twitter.com/ChaosbladeI) ## Contributing We welcome every contribution, even if it is just punctuation. See details of [CONTRIBUTING](CONTRIBUTING.md). For the promotion ladder of specific community participation students, see: ([Contributor Ladder](https://github.com/chaosblade-io/community/blob/main/Contributor_Ladder.md)) ## Business Registration The original intention of our open source project is to lower the threshold for chaos engineering to be implemented in enterprises, so we highly value the use of the project in enterprises. Welcome everyone here [ISSUE](https://github.com/chaosblade-io/chaosblade/issues/32). After registration, you will be invited to join the corporate mail group to discuss the problems encountered by Chaos Engineering in the landing of the company and share the landing experience. ## Contributors ### Code Contributors This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)]. [](https://github.com/chaosblade-io/chaosblade/graphs/contributors) ## License Chaosblade is licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full license text. ## 2. In-Tree Documentation Chapters (chaosblade-io/chaosblade) ## File: README.md # Chaosblade: An Easy to Use and Powerful Chaos Engineering Toolkit [](https://travis-ci.org/chaosblade-io/chaosblade) [](https://opencollective.com/chaosblade) [](https://codecov.io/gh/chaosblade-io/chaosblade) [](https://bestpractices.coreinfrastructure.org/projects/5032) 中文版 [README](README_CN.md) Wiki: [DeepWiki](https://deepwiki.com/chaosblade-io/chaosblade-for-deepwiki) ## 🔥🔥🔥 Major UPDATE:Chaosblade Agent Released - Blade AI Blade AI serves as the intelligent agent layer within the ChaosBlade ecosystem: at the foundational level, it invokes ChaosBlade to execute fault injection; at the upper level, it incorporates orchestration capabilities such as intent understanding, security auditing, effect verification, safe recovery, and structured reporting—thereby transforming fault drills from a process of "manually writing commands" into one completed through "conversational interaction." For detailed information, please refer to: [Blade AI README](https://github.com/chaosblade-io/chaosblade/blob/feature/blade-ai/blade-ai/README_en.md) * Release: [blade-ai-v0.1.0](https://github.com/chaosblade-io/chaosblade/releases/tag/blade-ai-v0.1.0) * Code Branch: blade-ai-v0.1.0 ## Introduction ChaosBlade is an Alibaba open source experimental injection tool that follows the principles of chaos engineering and chaos experimental models to help enterprises improve the fault tolerance of distributed systems and ensure business continuity during the process of enterprises going to cloud or moving to cloud native systems. Chaosblade is an internal open source project of MonkeyKing. It is based on Alibaba's nearly ten years of failure testing and drill practice, and combines the best ideas and practices of the Group's businesses. ChaosBlade is not only easy to use, but also supports rich experimental scenarios. The scenarios include: * Basic resources: such as CPU, memory, network, disk, process and other experimental scenarios; * Java applications: such as databases, caches, messages, JVM itself, microservices, etc. You can also specify any class method to inject various complex experimental scenarios; * C ++ applications: such as specifying arbitrary methods or experimental lines of code injection delay, tampering with variables and return values; * container: such as killing the container, the CPU in the container, memory, network, disk, process and other experimental scenarios; * Cloud-native platforms: For example, CPU, memory, network, disk, and process experimental scenarios on Kubernetes platform nodes, Pod network and Pod itself experimental scenarios such as killing Pods, and container experimental scenarios such as the aforementioned Docker container experimental scenario; Encapsulating scenes by domain into individual projects can not only standardize the scenes in the domain, but also facilitate the horizontal and vertical expansion of the scenes. By following the chaos experimental model, the chaosblade cli can be called uniformly. The items currently included are: * [chaosblade](https://github.com/chaosblade-io/chaosblade): Chaos experiment management tool, including commands for creating experiments, destroying experiments, querying experiments, preparing experimental environments, and canceling experimental environments. It is the execution of chaotic experiments. Tools, execution methods include CLI and HTTP. Provides complete commands, experimental scenarios, and scenario parameter descriptions, and the operation is simple and clear. * [chaosblade-spec-go](https://github.com/chaosblade-io/chaosblade-spec-go): Chaos experimental model Golang language definition, scenes implemented using Golang language are easy to implement based on this specification. * [chaosblade-exec-os](https://github.com/chaosblade-io/chaosblade-exec-os): Implementation of basic resource experimental scenarios. * [chaosblade-exec-docker](https://github.com/chaosblade-io/chaosblade-exec-docker): Docker container experimental scenario implementation, standardized by calling the Docker API. * [chaosblade-exec-cri](https://github.com/chaosblade-io/chaosblade-exec-cri): Container experimental scenario implementation, standardized by calling the CRI. * [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator): Kubernetes platform experimental scenario is implemented, chaos experiments are defined by Kubernetes standard CRD method, it is very convenient to use Kubernetes resource operation method To create, update, and delete experimental scenarios, including using kubectl, client-go, etc., and also using the chaosblade cli tool described above. * [chaosblade-exec-jvm](https://github.com/chaosblade-io/chaosblade-exec-jvm): Java application experimental scenario implementation, using Java Agent technology to mount dynamically, without any access, zero-cost use It also supports uninstallation and completely recycles various resources created by the Agent. * [chaosblade-exec-cplus](https://github.com/chaosblade-io/chaosblade-exec-cplus): C ++ application experimental scenario implementation, using GDB technology to implement method and code line level experimental scenario injection. * [chaosblade-box](https://github.com/chaosblade-io/chaosblade-box): Possessing chaos engineering platform and resilience testing platform capabilities.For more information on the resilience testing platform capabilities, see the [main2](https://github.com/chaosblade-io/chaosblade-box/tree/main2) branch. ## Quick Start This guide helps you run your first fault injection on Kubernetes in under 5 minutes using ChaosBlade. We'll inject a CPU stress fault into a Pod as a minimal example. ### Prerequisites Before you begin, make sure you have: - [ ] **kubectl access** to a running Kubernetes cluster (`kubectl cluster-info` returns successfully) - [ ] A **target namespace** with at least one running Pod (default: `default`) - [ ] **ChaosBlade installed**: - [ ] The [chaosblade](https://github.com/chaosblade-io/chaosblade/releases) CLI toolkit downloaded and extracted (the `blade` binary is on your `PATH`) - [ ] The [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator/releases) deployed to your cluster (required for Kubernetes scenarios) Install the operator with Helm: ```shell script helm install chaosblade-operator chaosblade-operator-.tgz --namespace chaosblade --create-namespace ``` Verify the operator is running: ```shell script kubectl get pods -n chaosblade ``` ### Inject Your First Fault (CPU Stress) Run a cpu full-load fault against a Pod in the `default` namespace: ```shell script blade create k8s pod-cpu fullload --cpu-percent 80 --kubeconfig ~/.kube/config --names --namespace default ``` If the injection succeeds, ChaosBlade returns a JSON result containing an experiment `uid`. Save this `uid` to check status or destroy the experiment later: ```json {"code":200,"success":true,"result":""} ``` ### Check the Experiment Status ```shell script blade status ``` ### Recover (Destroy the Fault) Always recover after your drill to restore the target to normal: ```shell script blade destroy ``` That's it! You've completed a full inject-verify-recover cycle. To explore more scenarios, run `blade create k8s -h` or see [Chaos Engineering Practice under Cloud Native](CLOUDNATIVE.md). ## CLI Command You can download the latest chaosblade toolkit from [Releases](https://github.com/chaosblade-io/chaosblade/releases) and extract it and use it. If you want to inject Kubernetes related fault scenarios, you need to install [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator/releases). For detailed Chinese usage documents, please see [chaosblade-help-zh-cn ](https://chaosblade-io.gitbook.io/chaosblade-help-zh-cn/). chaosblade supports CLI and HTTP invocation methods. The supported commands are as follows: * **prepare**: alias is p, preparation before the chaos engineering experiment, such as drilling Java applications, you need to attach the java agent. For example, to drill an application whose application name is business, execute `blade p jvm --process business` on the target host. If the attach is successful, return the uid for status query or agent revoke. * **revoke**: alias is r, undo chaos engineering experiment preparation before, such as detaching java agent. The command is `blade revoke UID` * **create**: alias is c, create a chaos engineering experiment. The command is `blade create [TARGET] [ACTION] [FLAGS]`. For example, if you implement a Dubbo consumer call xxx.xxx.Service interface delay 3s, the command executed is `blade create dubbo delay --consumer --time 3000 --Service xxx.xxx.Service`, if the injection is successful, return the experimental uid for status query and destroy the experiment. * **destroy**: alias is d, destroy a chaos engineering experiment, such as destroying the Dubbo delay experiment mentioned above, the command is `blade destroy UID` * **status**: alias s, query preparation stage or experiment status, the command is `blade status UID` or `blade status --type create` * **server**: start the web server, expose the HTTP service, and call chaosblade through HTTP requests. For example, execute on the target machine xxxx: `blade server start -p 9526` to perform a CPU full load experiment:` curl "http://xxxx:9526/chaosblade?cmd=create%20cpu%20fullload" ` Use the `blade help [COMMAND]` or `blade [COMMAND] -h` command to view help ## Experience Demo Download the chaosblade demo image and experience the use of the blade toolkit Download image command: ```shell script docker pull chaosbladeio/chaosblade-demo ``` Run the demo container: ```shell script docker run -it --privileged chaosbladeio/chaosblade-demo ``` After entering the container, you can read the README.txt file to implement the chaos experiment, Enjoy it. ## Cloud Native [chaosblade-operator](https://github.com/chaosblade-io/chaosblade-operator) The project is a chaos experiment injection tool for cloud-native platforms. It follows the chaos experiment model to standardize the experimental scenario and defines the experiment as Kubernetes CRD Resources, mapping experimental models to Kubernetes resource attributes, and very friendly combination of chaotic experimental models with Kubernetes declarative design. While relying on chaotic experimental models to conveniently develop scenarios, it can also well integrate Kubernetes design concepts, through kubectl or Write code to directly call the Kubernetes API to create, update, and delete chaotic experiments, and the resource status can clearly indicate the execution status of the experiment, and standardize Kubernetes fault injection. In addition to using the above methods to perform experiments, you can also use the chaosblade cli method to execute kubernetes experimental scenarios and query the experimental status very conveniently. For details, please read the chinese document: [Chaos Engineering Practice under Cloud Native](CLOUDNATIVE.md) ## Compile See [BUILD.md](BUILD.md) for the details. ## Bugs and Feedback For bug report, questions and discussions please submit [GitHub Issues](https://github.com/chaosblade-io/chaosblade/issues). You can also contact us via: * Dingding group (recommended for chinese): 23177705 * Slack group: [chaosblade-io](https://join.slack.com/t/chaosblade-io/shared_invite/zt-f0d3r3f4-TDK13Wr3QRUrAhems28p1w) * Gitter room: [chaosblade community](https://gitter.im/chaosblade-io/community) * Email: chaosblade.io.01@gmail.com * Twitter: [chaosblade.io](https://twitter.com/ChaosbladeI) ## Contributing We welcome every contribution, even if it is just punctuation. See details of [CONTRIBUTING](CONTRIBUTING.md). For the promotion ladder of specific community participation students, see: ([Contributor Ladder](https://github.com/chaosblade-io/community/blob/main/Contributor_Ladder.md)) ## Business Registration The original intention of our open source project is to lower the threshold for chaos engineering to be implemented in enterprises, so we highly value the use of the project in enterprises. Welcome everyone here [ISSUE](https://github.com/chaosblade-io/chaosblade/issues/32). After registration, you will be invited to join the corporate mail group to discuss the problems encountered by Chaos Engineering in the landing of the company and share the landing experience. ## Contributors ### Code Contributors This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)]. [](https://github.com/chaosblade-io/chaosblade/graphs/contributors) ## License Chaosblade is licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full license text. --- ## File: docs/beginner_guide_CN.md 欢迎关注 chaosblade 项目!这篇文章将带你快速掌握 chaosblade 工具的使用。 # 下载 chaosblade 获取 chaosblade 最新的 [release](https://github.com/chaosblade-io/chaosblade/releases) 包,目前支持的平台是 linux/amd64 和 darwin/64,下载对应平台的包。 下载完成后解压即可,无需编译。 # 使用 chaosblade 进入解压后的文件夹,可以看到以下内容: ``` ├── bin │   ├── chaos_fuse │   ├── chaos_os │   ├── nsexec │   └── strace ├── blade └── lib │   ├── cplus │   └── sandbox ├── logs └── yaml ``` 其中 blade 是可执行文件,即 chaosblade 工具的 cli,混沌实验执行的工具。执行 `./blade help` 可以查看支持命令有哪些: ``` An easy to use and powerful chaos engineering experiment toolkit Usage: blade [command] Available Commands: create Create a chaos engineering experiment destroy Destroy a chaos experiment help Help about any command prepare Prepare to experiment revoke Undo chaos engineering experiment preparation status Query preparation stage or experiment status version Print version info Flags: -d, --debug Set client to DEBUG mode -h, --help help for blade Use "blade [command] --help" for more information about a command. ``` ## 执行你的第一个混沌实验 我们拿 CPU 满载(CPU 使用率 100%) 演练场景举例(!!**注意,在不清楚影响面的情况下,切勿在生产系统机器上执行**),执行以下命令实施实验: ``` ./blade create cpu fullload ``` 执行结果返回: ``` {"code":200,"success":true,"result":"7c1f7afc281482c8"} ``` 通过 `top` 命令查看 CPU 使用率: ``` CPU usage: 93.79% user, 6.20% sys, 0.0% idle ``` 此时命令已经生效,**停止混沌实验**,执行: ``` ./blade destroy 7c1f7afc281482c8 ``` 返回以下结果,表示停止实验成功: ``` {"code":200,"success":true,"result":"command: cpu fullload --debug false --help false"} ``` 再去观察 CPU 情况,CPU 负载已回到正常状态: ``` CPU usage: 6.36% user, 4.74% sys, 88.88% idle ``` 一次 CPU 满载演练完成。 ## 你的第二个混沌实验 这次实验,我们演练 Dubbo 应用,我们的需求是 consumer 调用 com.alibaba.demo.HelloService 服务下的 hello 接口延迟 3 秒。接下来我们下载所需要的 Dubbo Demo: [dubbo-provider](https://chaosblade.oss-cn-hangzhou.aliyuncs.com/demo/dubbo-provider-1.0-SNAPSHOT.jar) [dubbo-consumer](https://chaosblade.oss-cn-hangzhou.aliyuncs.com/demo/dubbo-consumer-1.0-SNAPSHOT.jar) 下载完成后,执行以下命令启动应用,注意必须先启动 `dubbo-provider`,然后再启动 `dubbo-consumer`: ``` # 启动 dubbo-provider nohup java -Djava.net.preferIPv4Stack=true -Dproject.name=dubbo-provider -jar dubbo-provider-1.0-SNAPSHOT.jar > provider.nohup.log 2>&1 & # 稍等 2 秒,然后启动 dubbo-consumer nohup java -Dserver.port=8080 -Djava.net.preferIPv4Stack=true -Dproject.name=dubbo-consumer -jar dubbo-consumer-1.0-SNAPSHOT.jar > consumer.nohup.log 2>&1 & ``` 访问 `http://localhost:8080/hello?msg=world`,返回以下信息,表示启动成功: ``` { msg: "Dubbo Service: Hello world" } ``` 接下来我们要使用 blade 工具进行混沌实验,在执行实验前,我们需要先执行 prepare 命令,挂载所需要的 java agent: ``` ./blade prepare jvm --process dubbo.consumer ``` 返回以下结果,表示实验准备成功: ``` {"code":200,"success":true,"result":"e669d57f079a00cc"} ``` 我们开始实施混沌实验,我们的需求是 consumer 调用 `com.alibaba.demo.HelloService` 服务下的 `hello` 接口延迟 3 秒。 我们执行 `./blade create dubbo delay -h` 命令查看 dubbo 调用延迟的命令用法: ``` Usage: blade create dubbo delay Flags: --appname string The consumer or provider application name --consumer To tag consumer role experiment. -h, --help help for delay --methodname string The method name in service interface --offset string delay offset for the time --process string Application process name --provider To tag provider experiment --service string The service interface --time string delay time (required) --version string the service version Global Flags: -d, --debug Set client to DEBUG mode ``` 调用 `com.alibaba.demo.HelloService` 服务下的 `hello` 接口延迟 3 秒,我们执行以下命令: ``` ./blade create dubbo delay --time 3000 --service com.alibaba.demo.HelloService --methodname hello --consumer --process dubbo.consumer ``` 返回以下结果,表示执行成功;访问 `http://localhost:8080/hello?msg=world` 验证是否延迟 3 秒。 ``` {"code":200,"success":true,"result":"ec695fee1e458fc6"} ``` 对实施实验的命令进行解析: * `--time`: 3000,表示延迟 3000 ms;单位是 ms * `--service`: com.alibaba.demo.HelloService, 表示调用的服务 * `--methodname`: hello,表示服务接口方法 * `--consumer`: 表示演练的是 dubbo consumer * `--process`: dubbo.consumer,表示对哪个应用进程实施混沌实验 停止当前延迟的混沌实验,再次访问 url 验证是否恢复正常: ``` ./blade destroy ec695fee1e458fc6 ``` 不尽兴的话,我们再实施调用刚才那个服务抛异常,执行 `./blade create dubbo throwCustomException -h` 命令查看帮助: ``` Throw custom exception with --exception option Usage: blade create dubbo throwCustomException Aliases: throwCustomException, tce Flags: --appname string The consumer or provider application name --consumer To tag consumer role experiment. --exception string Exception class inherit java.lang.Exception (required) -h, --help help for throwCustomException --methodname string The method name in service interface --process string Application process name --provider To tag provider experiment --service string The service interface --version string the service version Global Flags: -d, --debug Set client to DEBUG mode ``` 和刚才延迟命令参数差不多,因为相同的参数是演练 dubbo 所需要的,不同的是没有了 `--time`,多了个 `--exception` 参数。 我们模拟调用刚才的服务抛 `java.lang.Exception` 异常: ``` ./blade create dubbo throwCustomException --exception java.lang.Exception --service com.alibaba.demo.HelloService --methodname hello --consumer --process dubbo.consumer ``` 返回以下结果,表示实验执行成功;访问 `http://localhost:8080/hello?msg=world` 验证是否异常 ``` {"code":200,"success":true,"result":"09dd96f4c062df69"} ``` 停止此次试验,再次访问请求,验证是否恢复: ``` ./blade destroy 09dd96f4c062df69 ``` 最后,我们撤销刚才的实验准备,即卸载 Java Agent: ``` ./blade revoke e669d57f079a00cc ``` 如果找不到之前执行 prepare 返回的 UID 的话,执行 `./blade status --type prepare` 命令查询: ``` { "code": 200, "success": true, "result": [ { "Uid": "e669d57f079a00cc", "ProgramType": "jvm", "Process": "dubbo.consumer", "Port": "59688", "Status": "Running", "Error": "", "CreateTime": "2019-03-29T16:19:37.284579975+08:00", "UpdateTime": "2019-03-29T17:05:14.183382945+08:00" } ] } ``` # FAQ ### 如何获取最新版本 chaosblade 每次发布,相关的 changelog 和新版的包都会同步到 RELEASE ,可在此地址下载。 ### Windows 平台是否有支持计划 暂无支持计划,不过欢迎大家提相关支持的 issue,社区会根据大家的需求来决定是否支持。 ### 执行 blade 命令报错: exec format error 或 cannot execute binary file 这个问题是由于 chaosblade 包和运行的平台不兼容造成的,请提 [ISSUE](https://github.com/chaosblade-io/chaosblade/issues)来告知我们您遇到的问题,issue 中标注下载的 chaosblade 包版本和操作系统版本信息。 --- ## File: docs/beginner_guide_EN.md Welcome to the chaosblade project! this article will take you through the chaosblade tool quickly. # Download chaosblade Get the latest [release](https://github.com/chaosblade-io/chaosblade/releases) package of chaosblade, currently supported by linux/amd64 and darwin/64, and download the package for the corresponding platform. Just download and unzip it, no need to compile. # Using chaosblade When you enter the unzipped folder, you can see the following contents: ``` ├── bin │   ├── chaos_fuse │   ├── chaos_os │   ├── nsexec │   └── strace ├── blade └── lib │   ├── cplus │   └── sandbox ├── logs └── yaml ``` where blade is the executable file, the cli of the chaosblade tool, the tool for chaos experiments. Execute `. /blade help` to see what commands are supported: ``` An easy to use and powerful chaos engineering experiment toolkit Usage: blade [command] Available Commands: create Create a chaos engineering experiment destroy Destroy a chaos experiment help Help about any command prepare Prepare to experiment revoke Undo chaos engineering experiment preparation status Query preparation stage or experiment status version Print version info Flags: -d, --debug Set client to DEBUG mode -h, --help help for blade Use "blade [command] --help" for more information about a command. ``` ## Perform your first chaos experiment Let's take a CPU full (100% CPU usage) exercise scenario as an example (!!! **Note, do not execute on the production system machine without knowing the impact surface**), and execute the following command to execute the experiment: ``` ./blade create cpu fullload ``` Execution results return: ``` {"code":200,"success":true,"result":"7c1f7afc281482c8"} ``` View CPU usage with the `top` command: ``` CPU usage: 93.79% user, 6.20% sys, 0.0% idle ``` At this point the command is in effect, **Stop Chaos Experiment** and execute; ``` ./blade destroy 7c1f7afc281482c8 ``` The following result is returned to indicate the success of the stop experiment: ``` CPU usage: 6.36% user, 4.74% sys, 88.88% idle ``` A CPU full load walkthrough is completed. ## Your second chaos experiment For this experiment, we walk through the Dubbo application, and our requirement is that the consumer calls the hello interface under the com.alibaba.demo.HelloService service with a delay of 3 seconds. Next, we download the Dubbo demo we need. [dubbo-provider](https://chaosblade.oss-cn-hangzhou.aliyuncs.com/demo/dubbo-provider-1.0-SNAPSHOT.jar) [dubbo-consumer](https://chaosblade.oss-cn-hangzhou.aliyuncs.com/demo/dubbo-consumer-1.0-SNAPSHOT.jar) After downloading, execute the following command to start the application. Note that you must start `dubbo-provider` first, and then `dubbo-consumer`: ``` # Start dubbo-provider nohup java -Djava.net.preferIPv4Stack=true -Dproject.name=dubbo-provider -jar dubbo-provider-1.0-SNAPSHOT.jar > provider.nohup.log 2>&1 & # Wait 2 seconds, then start dubbo-consumer nohup java -Dserver.port=8080 -Djava.net.preferIPv4Stack=true -Dproject.name=dubbo-consumer -jar dubbo-consumer-1.0-SNAPSHOT.jar > consumer.nohup.log 2>&1 & ``` Visit `http://localhost:8080/hello?msg=world` and return the following message indicating a successful start. ``` { msg: "Dubbo Service: Hello world" } ``` Next, we will use the blade tool to perform chaos experiments. Before we can perform the experiments, we need to execute the prepare command to mount the required java agent. ``` ./blade prepare jvm --process dubbo.consumer ``` The following results are returned to indicate successful experiment preparation: ``` {"code":200,"success":true,"result":"e669d57f079a00cc"} ``` We start implementing chaos experiments, and our requirement is that consumer calls to the `hello` interface under the `com.alibaba.demo.HelloService` service are delayed by 3 seconds. We execute `. /blade create dubbo delay -h` command to see the command usage for the dubbo call delay: ``` Usage: blade create dubbo delay Flags: --appname string The consumer or provider application name --consumer To tag consumer role experiment. -h, --help help for delay --methodname string The method name in service interface --offset string delay offset for the time --process string Application process name --provider To tag provider experiment --service string The service interface --time string delay time (required) --version string the service version Global Flags: -d, --debug Set client to DEBUG mode ``` Calling the `hello` interface under the `com.alibaba.demo.HelloService` service is delayed by 3 seconds and we execute the following command. ``` ./blade create dubbo delay --time 3000 --service com.alibaba.demo.HelloService --methodname hello --consumer --process dubbo.consumer ``` The following result is returned, indicating successful execution; visit `http://localhost:8080/hello?msg=world` to verify that the delay is 3 seconds. ``` {"code":200,"success":true,"result":"ec695fee1e458fc6"} ``` Explanation of the order to perform the experiment: * `--time`: 3000, indicates a delay of 3000 ms; the unit is ms * `--service`: com.alibaba.demo.HelloService, indicating the called service * `-methodname`: hello, indicating the service interface method * `--consumer`: indicates that the walkthrough is a dubbo consumer * `--process`: dubbo.consumer, indicating which application process to implement the chaos experiment on Stop the chaos experiment with the current delay and visit the url again to verify that it is back to normal. ``` ./blade destroy ec695fee1e458fc6 ``` We can also implement a call to that service to throw an exception by running `. /blade create dubbo throwCustomException -h` command to see help: ``` Throw custom exception with --exception option Usage: blade create dubbo throwCustomException Aliases: throwCustomException, tce Flags: --appname string The consumer or provider application name --consumer To tag consumer role experiment. --exception string Exception class inherit java.lang.Exception (required) -h, --help help for throwCustomException --methodname string The method name in service interface --process string Application process name --provider To tag provider experiment --service string The service interface --version string the service version Global Flags: -d, --debug Set client to DEBUG mode ``` The same parameters as the delay command are needed to walk through dubbo, but without the `--time` and with an additional `--exception` parameter. We simulate the call to the service we just made by throwing the `java.lang.Exception` exception: ``` ./blade create dubbo throwCustomException --exception java.lang.Exception --service com.alibaba.demo.HelloService --methodname hello --consumer --process dubbo.consumer ``` The following result is returned, indicating successful execution of the experiment; visit `http://localhost:8080/hello?msg=world` to verify if there is an exception. ``` {"code":200,"success":true,"result":"09dd96f4c062df69"} ``` Stop this trial, access the request again and verify that it is restored. ``` ./blade destroy 09dd96f4c062df69 ``` Finally, we undo the preparation for the experiment we just did, i.e., uninstall the Java Agent. ``` ./blade revoke e669d57f079a00cc ``` If you can't find the UID returned by the previous prepare, execute `. /blade status --type prepare` command to query. ``` { "code": 200, "success": true, "result": [ { "Uid": "e669d57f079a00cc", "ProgramType": "jvm", "Process": "dubbo.consumer", "Port": "59688", "Status": "Running", "Error": "", "CreateTime": "2019-03-29T16:19:37.284579975+08:00", "UpdateTime": "2019-03-29T17:05:14.183382945+08:00" } ] } ``` # FAQ ### How to get the latest version Every time chaosblade is released, the related changelog and the new version of the package will be synchronized to RELEASE, which can be downloaded at [this address](https://github.com/chaosblade-io/chaosblade/releases). ### Is there a support plan for the Windows platform There is no support plan, but you are welcome to raise relevant support issues, the community will decide whether to support according to your needs. ### Executing the blade command reports an error: exec format error or cannot execute binary file This problem is caused by an incompatibility between the chaosblade package and the running platform. Please inform us about the problem by mentioning [ISSUE](https://github.com/chaosblade-io/chaosblade/issues), and mark the issue with the downloaded chaosblade package version and The operating system version information is indicated in the issue. --- ## File: docs/chaos_experiment_model_CN.md 遵循此模型,可以简单明了的执行一次混沌实验,控制实验的最小爆炸半径。并且可以方便快捷的扩展新的实验场景或者增强现有场景。[chaosblade](https://github.com/chaosblade-io/chaosblade) 和 [chaosblade-exec-jvm](https://github.com/chaosblade-io/chaosblade-exec-jvm) 工程都根据此模型实现。 # 模型定义 在给出模型之前先讨论实施一次混沌实验明确的问题: * 对什么做混沌实验 * 混沌实验实施的范围是是什么 * 具体实施什么实验 * 实验生效的匹配条件有哪些 举个例子:一台 ip 是 10.0.0.1 机器上的应用,调用 com.example.HelloService@1.0.0 Dubbo 服务延迟 3s。根据上述的问题列表,先明确的是要对 Dubbo 组件混沌实验,实施实验的范围是 10.0.0.1 单机,对调用 com.example.HelloService@1.0.0 服务模拟 3s 延迟。 明确以上内容,就可以精准的实施一次混沌实验,抽象出以下模型: * Target:实验靶点,指实验发生的组件,例如 容器、应用框架(Dubbo、Redis、Zookeeper)等。 * Scope:实验实施的范围,指具体触发实验的机器或者集群等。 * Matcher:实验规则匹配器,根据所配置的 Target,定义相关的实验匹配规则,可以配置多个。由于每个 Target 可能有各自特殊的匹配条件,比如 RPC 领域的 HSF、Dubbo,可以根据服务提供者提供的服务和服务消费者调用的服务进行匹配,缓存领域的 Redis,可以根据 set、get 操作进行匹配。 * Action:指实验模拟的具体场景,Target 不同,实施的场景也不一样,比如磁盘,可以演练磁盘满,磁盘 IO 读写高,磁盘硬件故障等。如果是应用,可以抽象出延迟、异常、返回指定值(错误码、大对象等)、参数篡改、重复调用等实验场景。 回到上述的例子,可以叙述为对 Dubbo 组件(Target)进行故障演练,演练的是 10.0.0.1 主机(Scope)的应用,调用 com.example.HelloService@1.0.0 (Matcher)服务延迟 3s(Action)。 伪代码可以写成: ```java Toolkit. // 实验靶点 dubbo. // 范围,此处是主机 host("1.0.0.1"). // 组件匹配器,消费者还是服务提供者 consumer(). // 组件匹配器,服务接口 service("com.example.HelloService"). // 组件匹配器,1.0.0 接口版本 version("1.0.0"). // 实验场景,延迟 3s delay(3000); ``` # chaosblade 模型实现 ## chaosblade cli 调用 针对上述例子,chaosblade 调用命令是: ``` blade create dubbo delay --time 3000 --consumer --service com.example.HelloService --version 1.0.0 ``` * `dubbo`: 模型中的 target,对 dubbo 实施实验。 * `delay`: 模型中的 action,执行延迟演练场景。 * `--time`: 模型中 action 参数,指延迟时间。 * `--consumer`、`--service`、`--version`:模型中的 matchers,实验规则匹配器。 **注:** 由于 chaosblade 是在单机执行的工具,所以混沌实验模型中的 scope 默认为本机,不再显示声明。 ## chaosblade 模型结构图 为了有个更加直观的认识,我们先通过以下的模型结构图来大致看一下模型之间的关系。核心接口模型是:ExpModelCommandSpec,由它引申出来的是ExpActionCommandSpec和ExpFlagSpec这两个接口。其中,ExpModelCommandSpec已有的具体实现有:cpu、network、disk等;ExpActionCommandSpec则是如cpu下的fullload之类的;ExpFlagSpec是各类自定义参数,比如--timeout。更加详细的模型定义说明请见后续小节。 ## chaosblade 模型定义 ```go type ExpModelCommandSpec interface { // 组件名称 Name() string // 支持的场景列表 Actions() []ExpActionCommandSpec // ... } ``` **注:** 一个组件混沌实验模型的定义,包含组件名称和所支持的实验场景列表。 ```go type ExpActionCommandSpec interface { // 演练场景名称 Name() string // 规则匹配器列表 Matchers() []ExpFlagSpec // Action 参数列表 Flags() []ExpFlagSpec // Action 执行器 Executor(channel Channel) Executor // ... } ``` **注:** 一个实验场景 action 的定义,包含场景名称,场景所需参数和一些实验规则匹配器 ```go type ExpFlagSpec interface { // 参数名 FlagName() string // 参数描述 FlagDesc() string // 是否需要参数值 FlagNoArgs() bool // 是否是必要参数 FlagRequired() bool } ``` **注:** 实验匹配器定义。 ## chaosblade 模型具体实现 拿 network 组件举例,network 作为混沌实验组件,目前包含网络延迟、网络屏蔽、网络丢包、DNS 篡改演练场景,则依据模型规范,具体实现为: ```go type NetworkCommandSpec struct { } func (*NetworkCommandSpec) Name() string { return "network" } func (*NetworkCommandSpec) Actions() []exec.ExpActionCommandSpec { return []exec.ExpActionCommandSpec{ &DelayActionSpec{}, &DropActionSpec{}, &DnsActionSpec{}, &LossActionSpec{}, } } ``` network target 定义了 `DelayActionSpec`、`DropActionSpec`、`DnsActionSpec`、`LossActionSpec` 四种混沌实验场景,其中 `DelayActionSpec` 定义如下: ```go type DelayActionSpec struct { } func (*DelayActionSpec) Name() string { return "delay" } func (*DelayActionSpec) Matchers() []exec.ExpFlagSpec { return []exec.ExpFlagSpec{ &exec.ExpFlag{ Name: "local-port", Desc: "Port for external service", }, &exec.ExpFlag{ Name: "remote-port", Desc: "Port for invoking", }, &exec.ExpFlag{ Name: "exclude-port", Desc: "Exclude one local port, for example 22 port. This flag is invalid when --local-port or remote-port is specified", }, &exec.ExpFlag{ Name: "device", Desc: "Network device", Required: true, }, } } func (*DelayActionSpec) Flags() []exec.ExpFlagSpec { return []exec.ExpFlagSpec{ &exec.ExpFlag{ Name: "time", Desc: "Delay time, ms", Required: true, }, &exec.ExpFlag{ Name: "offset", Desc: "Delay offset time, ms", }, } } func (*DelayActionSpec) Executor(channel exec.Channel) exec.Executor { return &NetworkDelayExecutor{channel} } ``` * `DelayActionSpec` 包含 2 个场景参数和 4 个规则匹配器。 # 总结 通过以上事例,可以看出此模型简单、易实现,并且可以覆盖目前已知的实验场景。后续可以对此模型进行完善,成为一个混沌实验标准。 # 附录 A 应用级别通用的故障场景: * 延迟 * 异常 * 返回特定值 * 修改参数值 * 重复调用 * try-catch 块异常 # 文档贡献者 [@xcaspar](https://github.com/xcaspar) [@Cenyol](https://github.com/Cenyol) --- ## File: docs/chaos_experiment_model_EN.md Following this model, it is straightforward to perform a chaos experiment and control the minimum explosion radius of the experiment.The [chaosblade](https://github.com/chaosblade-io/chaosblade) and [chaosblade-exec-jvm](https://github.com/chaosblade-io/chaosblade-exec-jvm) projects are implemented according to this model. ## Model Definition Before giving the model discuss the problem of implementing a chaos experiment explicitly. - What to do the chaos experiment on - What is the scope of the chaos experiment - What are the specific experiments to be performed - What are the matching conditions for the experiment to be effective? For example, an application on a 10.0.0.1 machine with an IP address of 10.0.0.1 calls the com.example.HelloService@1.0.0 Dubbo service with a latency of 3s. Based on the list of problems above, the first thing that is clear is that you want to experiment with Dubbo component chaos, implementing the experiment on a single 10.0.0.1 machine, calling com. example.HelloService@1.0.0 service to simulate a 3s delay. By specifying the above, you can precisely implement a chaos experiment by abstracting the following model: * Target: The target of the experiment, the component on which the experiment takes place, such as container, application framework (Dubbo, Redis, Zookeeper), etc. * Scope: The scope of the experiment implementation, referring to the specific machine or cluster that triggers the experiment, etc. * Matcher: Experiment rule matcher, according to the configured Target, define the relevant experiment matching rules, can be configured multiple. As each Target may have its own special matching conditions, for example, HSF and Dubbo in the RPC domain can be matched according to the services provided by the service provider and the services invoked by the service consumer, and Redis in the caching domain can be matched according to set and get operations. * Action: refers to the specific scenario of the experiment simulation, Target is different, the implementation of the scenario is also different, for example, the disk, you can rehearse the disk full, disk IO read and write high, disk hardware failure, etc.. If it is an application, you can abstract the experimental scenarios such as delay, exception, return specified values (error codes, large objects, etc.), parameter tampering, repeated calls, etc. Returning to the above example, it can be described as a failure drill for a Dubbo component (Target), for an application on host 10.0.0.1 (Scope), calling the com.example.HelloService@1.0.0 (Matcher) service with a 3s delay (Action). The pseudo code can be written as follows: ```java Toolkit. // Target dubbo. // Scope host("1.0.0.1"). // Matcher consumer(). // Matcher service("com.example.HelloService"). // Matcher version("1.0.0"). // Action delay(3000); ``` # chaosblade mode implementation ## chaosblade cli call For the above example, the chaosblade call command is: ``` blade create dubbo delay --time 3000 --consumer --service com.example.HelloService --version 1.0.0 ``` - ```dubbo```: Target in the model, which performs experiments on dubbo. - ```delay```: The action in the model that executes the delayed walkthrough scenario. - ```--time```: action argument in the model, referring to the delay time. - ```--consumer```, ```--service```, ```--version```: matchers in the model, experimental rule matchers. **Note**: Since chaosblade is a tool that executes on a single machine, the scope in the chaos experiment model defaults to the local machine and no further declarations are shown. ## Chaosblade model structure diagram In order to have a more intuitive understanding, let's take a general look at the relationship between the models through the following model structure diagram. The core interface model is ExpModelCommandSpec, from which the two interfaces ExpActionCommandSpec and ExpFlagSpec are derived. The ExpModelCommandSpec has specific implementations such as cpu, network, disk, etc.; the ExpActionCommandSpec is such as fullload under cpu, etc.; the ExpFlagSpec is a variety of custom parameters, such as --timeout. For more detailed model definitions, please see the subsequent subsections. ## chaosblade model definition ```go type ExpModelCommandSpec interface { // Component Name Name() string // List of supported scenarios Actions() []ExpActionCommandSpec // ... } ``` **Note**: Definition of a component chaos experiment model with component names and a list of supported experiment scenarios. ``` type ExpActionCommandSpec interface { // Experimental scene name Name() string // Rule Matcher List Matchers() []ExpFlagSpec // Action parameter List Flags() []ExpFlagSpec // Action Executor Executor(channel Channel) Executor // ... } ``` **Note**: Definition of an experimental scenario action, including the scenario name, the parameters required for the scenario and some experimental rule matchers ```go type ExpFlagSpec interface { // Parameter Name FlagName() string // Parameter Description FlagDesc() string // Whether parameter values are required FlagNoArgs() bool // Is it a required parameter FlagRequired() bool } ``` **Note**: Experimental matcher definition. ## Chaosblade model implementation Take the network component as an example, network as a chaos experiment component, currently contains network latency, network shielding, network packet loss, DNS tampering exercise scenarios, then according to the model specification, the specific implementation as follows. ```go type NetworkCommandSpec struct { } func (*NetworkCommandSpec) Name() string { return "network" } func (*NetworkCommandSpec) Actions() []exec.ExpActionCommandSpec { return []exec.ExpActionCommandSpec{ &DelayActionSpec{}, &DropActionSpec{}, &DnsActionSpec{}, &LossActionSpec{}, } } ``` The network target defines four chaotic experiment scenarios, DelayActionSpec, DropActionSpec, DnsActionSpec, and LossActionSpec, where DelayActionSpec is defined as follows. ```go type DelayActionSpec struct { } func (*DelayActionSpec) Name() string { return "delay" } func (*DelayActionSpec) Matchers() []exec.ExpFlagSpec { return []exec.ExpFlagSpec{ &exec.ExpFlag{ Name: "local-port", Desc: "Port for external service", }, &exec.ExpFlag{ Name: "remote-port", Desc: "Port for invoking", }, &exec.ExpFlag{ Name: "exclude-port", Desc: "Exclude one local port, for example 22 port. This flag is invalid when --local-port or remote-port is specified", }, &exec.ExpFlag{ Name: "device", Desc: "Network device", Required: true, }, } } func (*DelayActionSpec) Flags() []exec.ExpFlagSpec { return []exec.ExpFlagSpec{ &exec.ExpFlag{ Name: "time", Desc: "Delay time, ms", Required: true, }, &exec.ExpFlag{ Name: "offset", Desc: "Delay offset time, ms", }, } } func (*DelayActionSpec) Executor(channel exec.Channel) exec.Executor { return &NetworkDelayExecutor{channel} } ``` * `DelayActionSpec` contains 2 scene parameters and 4 rule matchers. ## Summary The above examples show that this model is simple, easy to implement, and can cover the currently known experimental scenarios. This model can be improved later to become a standard for chaotic experiments. ## Appendix A Application-level generic failure scenarios. * Delay * Exceptions * Returning a specific value * Modifying a parameter value * Repeated calls * try-catch block exceptions ## Document Contributors [@xcaspar](https://github.com/xcaspar) [@Cenyol](https://github.com/Cenyol) [@Super-long](https://github.com/Super-long) --- ## File: docs/code_styles.md # Code Style Code style is a set of rules or guidelines when writing source codes of a software project. Following particular code style will definitely help contributors to read and understand source codes very well. In addition, it will help to avoid introducing errors as well. ## Code Style Tools Project chaosblade is written in Golang. And currently we use three tools to help conform code styles in this project. These three tools are: * [gofmt](https://golang.org/cmd/gofmt) * [go vet](https://golang.org/cmd/vet/) And all these tools are used in [Makefile](../Makefile). ## Code Review Comments When collaborating in chaosblade project, we follow the style from [Go Code Review Comments](https://github.com/golang/go/wiki/CodeReviewComments). Before contributing, we treat this as a must-read. ## Additional Style Rules For a project, existing tools and rules may not be sufficient. To align more in styles, we recommend contributors taking a thorough look at the following additional style rules: ### RULE001 - Add blank line between field's comments When constructing a struct, if comments needed for fields in struct, keep a blank line between fields. The encouraged way is as following: ``` golang // correct example // ContainerManager is the default implement of interface ContainerMgr. type ContainerManager struct { // Store stores containers in Backend store. // Element operated in store must has a type of *ContainerMeta. // By default, Store will use local filesystem with json format to store containers. Store *meta.Store // Client is used to interact with containerd. Client ctrd.APIClient // NameToID stores relations between container's name and ID. // It is used to get container ID via container name. NameToID *collect.SafeMap ...... } ``` Rather than: ```golang // wrong example // ContainerManager is the default implement of interface ContainerMgr. type ContainerManager struct { // Store stores containers in Backend store. // Element operated in store must has a type of *ContainerMeta. // By default, Store will use local filesystem with json format to store containers. Store *meta.Store // Client is used to interact with containerd. Client ctrd.APIClient // NameToID stores relations between container's name and ID. // It is used to get container ID via container name. NameToID *collect.SafeMap ...... } ``` ### RULE002 - Add parameter name in interface definition When defining interface functions, we should always explicitly add formal parameters, and this helps a lot to code readability. For example, the following way are preferred: ``` golang // correct example // ContainerMgr is an interface to define all operations against container. type ContainerMgr interface { // Start a container. Start(ctx context.Context, id, detachKeys string) error // Stop a container. Stop(ctx context.Context, name string, timeout int64) error ...... } ``` However, missing formal parameter's name would make interface unreadable, since we would never know what the parameter's real meaning unless turning to one implementation of this interface: ``` golang // wrong example type ContainerMgr interface { // Start a container. Start(context.Context, string, string) error // Stop a container. Stop(context.Context, string, int64) error ...... } ``` In addition, a blank line between function's comments is encouraged to make interface more readable. ### RULE003 - Import Packages When importing packages, to improve readabilities, we should import package by sequence: * Golang's built-in system packages; * project's own packages; * third-party packages. And we should keep a blank line among these three kinds of packages like the following: ``` golang import ( "fmt" "strings" "time" "github.com/chaosblade-io/chaosblade/data" "github.com/chaosblade-io/chaosblade/util" "github.com/sirupsen/logrus" "github.com/spf13/cobra" ) ``` ### RULE004 - Variable declaration position Variable object should be declared at the beginning of the go file following package name and importing. ### RULE005 - Generation of action failure When generating error in one function execution failure, we should generally use the following way to append string "failed to do something" and the specific err instance to construct a new error: ``` golang fmt.Errorf("failed to do something: %v", err) ``` When an err could be thrown out, please remember to add it in the error construction. ### RULE006 - Return fast to indent less chaosblade encourages contributors to take advantages of `return fast` to simply source code and indent less. For example, the following codes are discouraged: ``` golang // wrong example if retry { if t, err := calculateSleepTime(d); err == nil { time.Sleep(t) times++ return retryLoad() } return fmt.Errorf("failed to calculate timeout: %v", err) } return nil ``` In code above, there are some indents which can be avoided. The encouraged way is like the following: ``` golang // correct example if !retry { return nil } t, err := calculateSleepTime(d); if err != nil { return fmt.Errorf("failed to calculate timeout: %v", err) } time.Sleep(t) times++ return retryLoad() ``` ### RULE007 - Lowercase log and error No matter log or error, first letter of the message must be lower-case. So, `log.Debugf("failed to add list: %v", err)` is encouraged. And `log.Debugf("Failed to add list: %v", err)` is not perferred. ### RULE008 - Nested errors When occurring nesting errors, we recommend first considering using package `github.com/pkg/errors`. ### RULE009 - Comment correctly Every comment must begin with `//` plus a whitespace no matter for a variable, struct, function, code block and anything else. Please don't forget the whitespace, and end up all the sentence with a `.`. In addition, it is encouraged to use third person singular to polish the majority of function's comments. For example, the following way ```golang // wrong example // ExecContainer execute a process in container. func (c *Client) ExecContainer(ctx context.Context, process *Process) error { ...... } ``` could be polished to be `executes` rather than `execute`: ```golang // correct example // ExecContainer executes a process in container. func (c *Client) ExecContainer(ctx context.Context, process *Process) error { ...... } ``` ### RULE010 - Always remember DRY We should take `DRY(Don't Repeat Yourself)` into consideration when adding anything. ### RULE011 - Welcome to your addition If you think much more practical code styles should be introduced in chaosblade. Please submit a pull request to make this better. ## Reference [Pouch Code Style](https://github.com/alibaba/pouch/blob/master/docs/contributions/code_styles.md) --- ## File: docs/logic_flow_Introduction_CN.md # 逻辑流程简介 了解完相关模型接口之后,在此之上我们可以继续简单的了解模型之间是如何进行交互,一条命令如:`blade create cpu fullload`输入回车之后,系统是如何一步步解析成对应的模型,并最终执行达到压测负载效果的。话不多说,看下图: ## Cobra框架 首先建议了解一下Cobra,它是go的一个开源工具库,提供简单的接口来创建强大现代的CLI接口,类似于git或者go工具。同时,它也是一个应用,用来生成个人应用框架,组织系统命令、子命令以及相关参数,关于cobra更多具体信息详见[这里官方说明](https://github.com/spf13/cobra),从而开发以Cobra为基础的应用。Docker源码中使用了Cobra。 上图中一开始的添加各种指令,诸如:version、prepare、revoke、create等,在项目源码中就是基于cobra进行实现的,然后再进行二级命令以及相关参数的封装逻辑。 ## chaosblade 流程简介 首先,程序在一开始的时候,会添加各类基础命令,诸如:version、prepare、revoke、create等,这些命令除了create拥有多种二级子命令之外,其余都是原生的Cobra命令模型,只有一级命令搭配参数进行操作。这些命令相对来说都比较简单,具体实现可以直接看源码,在熟悉了Cobra之后看一眼就能明白。 除了项目源码和文档,另一个了解chaosblade用法的方式是通过不断的help提示来获取帮助信息。比如: ```bash blade help blade create help blade create cpu help ``` 在你输入blade之后不知所措的时候,一路help下去会有惊喜不断,感谢大神提供的彩蛋。 ### 各类create子命令 create的子命令在前面的模型章节中对应于:ExpModelCommandSpec接口。主要有如下几类: **OS命令** 即操作系统层级的负载命令,目前只支持*nix系统,提供了cpu、disk、network、mem等负载命令。 **DockerOS命令** 即Docker容器中的操作系统层级的负载命令,这个和上述的OS命令基本是保持一致的,从源码也可以发现,它只是对OS命令对象做了个封装,然后就塞进Docker命令对象中。 还有Jvm、k8s等,可以看出这一级命令名称是名词,也就是呼应了ExpModelCommandSpec这个接口的命名含义,它就是个目标model操作对象。在这些model对象后面的二级子命令才是各种操作action,如fullload。 ### Model对象下的Action 接上Model子命令,这一级二级子命令对应的接口模型为:ExpActionCommandSpec。比如cpu fullload, network delay和drop等。属于真正执行操作的命令,从其命名为动词上可看出。 系统解析到这步之后,就会带上参数ExpFlagSpec去调用具体的bin目录下对应的命令,实现系统负载操作。 ## 总结 本文首先通过逻辑流程简图,展示了系统大致的运作流程。然后简单介绍了Cobra库的功能及其在本项目中的作用。之后从命令的结构上,一级级介绍其运作流程,更多详细信息请参考源码实现。 ## 参考 - [Cobra官方说明](https://github.com/spf13/cobra) ## 文档贡献者 [@Cenyol](https://github.com/Cenyol) --- ## File: docs/logic_flow_Introduction_EN.md # Logical Process Introduction After understanding the relevant model interface, we can continue to understand briefly how the models interact with each other. After entering a command such as `blade create cpu fullload`, how the system resolves into the corresponding model step by step, and finally executes to achieve the effect of pressure test load. Without further ado, see the following figure. ## Cobra It is first recommended to know about Cobra, an open source tool library for go that provides a simple interface to create a powerful modern CLI interface, similar to git or go tools. It is also an application for generating personal application frameworks, organizing system commands, subcommands, and related parameters. For more specific information about cobra see [official description here](https://github.com/spf13/cobra) to develop Cobra-based applications. Cobra is used in the Docker source code. The above diagram begins with adding various commands, such as version, prepare, revoke, create, etc., in the project source code is based on cobra for implementation, and then the secondary commands and related parameters of the packaging logic. ## chaosblade Process Introduction First of all, the program at the beginning, will add all kinds of basic commands, such as: version, prepare, revoke, create, etc., these commands in addition to create has a variety of secondary subcommands, the rest are the native Cobra command model, only one level of command with parameters to operate. These commands are relatively simple, the specific implementation can be seen directly in the source code, after familiar with the Cobra look at a glance to understand. In addition to the project source code and documentation, another way to understand chaosblade usage is to get help information through the constant help prompt. For example: ```bash blade help blade create help blade create cpu help ``` After you type blade overwhelmed, help all the way down will have surprises, thanks to the gods to provide easter eggs! ### Various create subcommands The subcommands of create correspond in the previous model section: ExpModelCommandSpec interface. The main categories are as follows. **OS commands** i.e. OS-level load commands, currently only supported for *nix systems, provide load commands for cpu, disk, network, mem, etc. **DockerOS commands** i.e. OS-level load commands in Docker containers, this is basically consistent with the above OS commands, as you can also find from the source code, it just makes a wrapper around the OS command object, and then it is stuffed into the Docker command object. And Jvm, k8s, etc., you can see that the name of this level of command is a noun, which echoes the meaning of the naming of the interface ExpModelCommandSpec, it is a target model operation object. In these model objects followed by the second-level subcommand is a variety of operational actions, such as fullload. ### Action under Model object The interface model corresponding to this level 2 subcommand is ExpActionCommandSpec, such as cpu fullload, network delay and drop. It is a command that actually performs an action, as can be seen from its naming as a verb. After the system resolves to this step, it will take the parameter ExpFlagSpec to call the corresponding command in the specific bin directory to realize the system load operation. ## Summary This paper first shows the general operation flow of the system through a logical flow sketch. Then it briefly introduces the functions of Cobra library and its role in this project. After that, it introduces the operation flow from the command structure at one level, please refer to the source code implementation for more details. ## Reference - [Cobra Official Description](https://github.com/spf13/cobra) ## Document Contributors [@Cenyol](https://github.com/Cenyol) [@Super-long](https://github.com/Super-long) --- METRICS --- - Files Extracted: 9 - Estimated Token Budget: ~16505 tokens - Recency Window: Active (< 180 days) - Canonical Reference: https://codewiki.google/github.com/chaosblade-io/chaosblade