arthas Documentation

Alibaba Java Diagnostic Tool Arthas/Alibaba Java诊断利器Arthas

37,484 stars Java Java 1 file · ~7,417 tokens #agent#alibaba#arthas#classloader#diagnosis#java#jvm#trace
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy arthas on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run)
docker run -d --name arthas --restart always alibaba/arthas:latest

File: README.md

Arthas





Arthas is a Java Diagnostic tool open sourced by Alibaba.

Arthas allows developers to troubleshoot production issues for Java applications without modifying code or restarting servers.

中文说明/Chinese Documentation

Background

Often times, the production system network is inaccessible from the local development environment. If issues are encountered in production systems, it is impossible to use IDEs to debug the application remotely. More importantly, debugging in production environment is unacceptable, as it will suspend all the threads, resulting in the suspension of business services.

Developers could always try to reproduce the same issue on the test/staging environment. However, this is tricky as some issues cannot be reproduced easily on a different environment, or even disappear once restarted.

And if you're thinking of adding some logs to your code to help troubleshoot the issue, you will have to go through the following lifecycle; test, staging, and then to production. Time is money! This approach is inefficient! Besides, the issue may not be reproducible once the JVM is restarted, as described above.

Arthas was built to solve these issues. A developer can troubleshoot your production issues on-the-fly. No JVM restart, no additional code changes. Arthas works as an observer, which will never suspend your existing threads.

Key features

  • Check whether a class is loaded, or where the class is being loaded. (Useful for troubleshooting jar file conflicts)
  • Decompile a class to ensure the code is running as expected.
  • View classloader statistics, e.g. the number of classloaders, the number of classes loaded per classloader, the classloader hierarchy, possible classloader leaks, etc.
  • View the method invocation details, e.g. method parameter, return object, thrown exception, and etc.
  • Check the stack trace of specified method invocation. This is useful when a developers wants to know the caller of the said method.
  • Trace the method invocation to find slow sub-invocations.
  • Monitor method invocation statistics, e.g. qps, rt, success rate and etc.
  • Monitor system metrics, thread states and cpu usage, gc statistics, and etc.
  • Supports command line interactive mode, with auto-complete feature enabled.
  • Supports telnet and websocket, which enables both local and remote diagnostics with command line and browsers.
  • Supports profiler/Flame Graph
  • Support get objects in the heap that are instances of the specified class.
  • Supports JDK 8+ in version 4.x, including JDK 17, JDK 21, and JDK 25.
  • Supports Linux/Mac/Windows.

Quick start

Downloadarthas-boot.jar,Start with java command:

bash
curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar

Print usage:

bash
java -jar arthas-boot.jar -h

Use `as.sh`

You can install Arthas with one single line command on Linux, Unix, and Mac. Copy the following command and paste it into the command line, then press Enter to run:

bash
curl -L https://arthas.aliyun.com/install.sh | sh

The command above will download the bootstrap script as.sh to the current directory. You can move it any other place you want, or put its location in $PATH.

You can enter its interactive interface by executing as.sh, or execute as.sh -h for more help information.

Documentation

Feature Showcase

Dashboard

Thread

See what is eating your CPU (ranked by top CPU usage) and what is going on there in one glance:

bash
$ thread -n 3
"as-command-execute-daemon" Id=29 cpuUsage=75% RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:440)
    at com.taobao.arthas.core.command.monitor200.ThreadCommand$1.action(ThreadCommand.java:58)
    at com.taobao.arthas.core.command.handler.AbstractCommandHandler.execute(AbstractCommandHandler.java:238)
    at com.taobao.arthas.core.command.handler.DefaultCommandHandler.handleCommand(DefaultCommandHandler.java:67)
    at com.taobao.arthas.core.server.ArthasServer$4.run(ArthasServer.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@6cd0b6f8

"as-session-expire-daemon" Id=25 cpuUsage=24% TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at com.taobao.arthas.core.server.DefaultSessionManager$2.run(DefaultSessionManager.java:85)

"Reference Handler" Id=2 cpuUsage=0% WAITING on java.lang.ref.Reference$Lock@69ba0f27
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.Reference$Lock@69ba0f27
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

jad

Decompile your class with one shot:

java
$ jad javax.servlet.Servlet

ClassLoader:
+-java.net.URLClassLoader@6108b2d7
  +-sun.misc.Launcher$AppClassLoader@18b4aac2
    +-sun.misc.Launcher$ExtClassLoader@1ddf84b8

Location:
/Users/xxx/work/test/lib/servlet-api.jar

/*
 * Decompiled with CFR 0_122.
 */
package javax.servlet;

import java.io.IOException;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

public interface Servlet {
    public void init(ServletConfig var1) throws ServletException;

    public ServletConfig getServletConfig();

    public void service(ServletRequest var1, ServletResponse var2) throws ServletException, IOException;

    public String getServletInfo();

    public void destroy();
}

mc

Memory compiler, compiles .java files into .class files in memory.

bash
$ mc /tmp/Test.java

retransform

Load the external *.class files to retransform/hotswap the loaded classes in JVM.

bash
retransform /tmp/Test.class
retransform -c 327a647b /tmp/Test.class /tmp/Test\$Inner.class

sc

Search any loaded class with detailed information.

bash
$ sc -d org.springframework.web.context.support.XmlWebApplicationContext
 class-info        org.springframework.web.context.support.XmlWebApplicationContext
 code-source       /Users/xxx/work/test/WEB-INF/lib/spring-web-3.2.11.RELEASE.jar
 name              org.springframework.web.context.support.XmlWebApplicationContext
 isInterface       false
 isAnnotation      false
 isEnum            false
 isAnonymousClass  false
 isArray           false
 isLocalClass      false
 isMemberClass     false
 isPrimitive       false
 isSynthetic       false
 simple-name       XmlWebApplicationContext
 modifier          public
 annotation
 interfaces
 super-class       +-org.springframework.web.context.support.AbstractRefreshableWebApplicationContext
                     +-org.springframework.context.support.AbstractRefreshableConfigApplicationContext
                       +-org.springframework.context.support.AbstractRefreshableApplicationContext
                         +-org.springframework.context.support.AbstractApplicationContext
                           +-org.springframework.core.io.DefaultResourceLoader
                             +-java.lang.Object
 class-loader      +-org.apache.catalina.loader.ParallelWebappClassLoader
                     +-java.net.URLClassLoader@6108b2d7
                       +-sun.misc.Launcher$AppClassLoader@18b4aac2
                         +-sun.misc.Launcher$ExtClassLoader@1ddf84b8
 classLoaderHash   25131501

vmtool

Get objects in the heap that are instances of the specified class.

bash
$ vmtool --action getInstances --className java.lang.String --limit 10
@String[][
    @String[com/taobao/arthas/core/shell/session/Session],
    @String[com.taobao.arthas.core.shell.session.Session],
    @String[com/taobao/arthas/core/shell/session/Session],
    @String[com/taobao/arthas/core/shell/session/Session],
    @String[com/taobao/arthas/core/shell/session/Session.class],
    @String[com/taobao/arthas/core/shell/session/Session.class],
    @String[com/taobao/arthas/core/shell/session/Session.class],
    @String[com/],
    @String[java/util/concurrent/ConcurrentHashMap$ValueIterator],
    @String[java/util/concurrent/locks/LockSupport],
]

stack

View the call stack of test.arthas.TestStack#doGet:

text
/* Detailed source-code truncated for AI context efficiency. */

Trace

See what is slowing down your method invocation with trace command:

Watch

Watch the first parameter and thrown exception of test.arthas.TestWatch#doGet only if it throws exception.

bash
$ watch test.arthas.TestWatch doGet {params[0], throwExp} -e
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 65 ms.
ts=2018-09-18 10:26:28;result=@ArrayList[
    @RequestFacade[org.apache.catalina.connector.RequestFacade@79f922b2],
    @NullPointerException[java.lang.NullPointerException],
]

Monitor

Monitor a specific method invocation statistics, including the total number of invocations, average response time, success rate, and every 5 seconds:

bash
$ monitor -c 5 org.apache.dubbo.demo.provider.DemoServiceImpl sayHello
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 109 ms.
 timestamp            class                                           method    total  success  fail  avg-rt(ms)  fail-rate
----------------------------------------------------------------------------------------------------------------------------
 2018-09-20 09:45:32  org.apache.dubbo.demo.provider.DemoServiceImpl  sayHello  5      5        0     0.67        0.00%

 timestamp            class                                           method    total  success  fail  avg-rt(ms)  fail-rate
----------------------------------------------------------------------------------------------------------------------------
 2018-09-20 09:45:37  org.apache.dubbo.demo.provider.DemoServiceImpl  sayHello  5      5        0     1.00        0.00%

 timestamp            class                                           method    total  success  fail  avg-rt(ms)  fail-rate
----------------------------------------------------------------------------------------------------------------------------
 2018-09-20 09:45:42  org.apache.dubbo.demo.provider.DemoServiceImpl  sayHello  5      5        0     0.43        0.00%

Time Tunnel(tt)

Record method invocation data, so that you can check the method invocation parameters, returned value, and thrown exceptions later. It works as if you could come back and replay the past method invocation via time tunnel.

bash
$ tt -t org.apache.dubbo.demo.provider.DemoServiceImpl sayHello
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 75 ms.
 INDEX   TIMESTAMP            COST(ms)  IS-RET  IS-EXP   OBJECT         CLASS                          METHOD
-------------------------------------------------------------------------------------------------------------------------------------
 1000    2018-09-20 09:54:10  1.971195  true    false    0x55965cca     DemoServiceImpl                sayHello
 1001    2018-09-20 09:54:11  0.215685  true    false    0x55965cca     DemoServiceImpl                sayHello
 1002    2018-09-20 09:54:12  0.236303  true    false    0x55965cca     DemoServiceImpl                sayHello
 1003    2018-09-20 09:54:13  0.159598  true    false    0x55965cca     DemoServiceImpl                sayHello
 1004    2018-09-20 09:54:14  0.201982  true    false    0x55965cca     DemoServiceImpl                sayHello
 1005    2018-09-20 09:54:15  0.214205  true    false    0x55965cca     DemoServiceImpl                sayHello
 1006    2018-09-20 09:54:16  0.241863  true    false    0x55965cca     DemoServiceImpl                sayHello
 1007    2018-09-20 09:54:17  0.305747  true    false    0x55965cca     DemoServiceImpl                sayHello
 1008    2018-09-20 09:54:18  0.18468   true    false    0x55965cca     DemoServiceImpl                sayHello

Classloader

bash
$ classloader
 name                                                  numberOfInstances  loadedCountTotal
 BootstrapClassLoader                                  1                  3346
 com.taobao.arthas.agent.ArthasClassloader             1                  1262
 java.net.URLClassLoader                               2                  1033
 org.apache.catalina.loader.ParallelWebappClassLoader  1                  628
 sun.reflect.DelegatingClassLoader                     166                166
 sun.misc.Launcher$AppClassLoader                      1                  31
 com.alibaba.fastjson.util.ASMClassLoader              6                  15
 sun.misc.Launcher$ExtClassLoader                      1                  7
 org.jvnet.hk2.internal.DelegatingClassLoader          2                  2
 sun.reflect.misc.MethodUtil                           1                  1

Web Console

Profiler/FlameGraph

bash
$ profiler start
Started [cpu] profiling
text
$ profiler stop
profiler output file: /tmp/demo/arthas-output/20211207-111550.html
OK

View profiler results under arthas-output via browser:

Arthas Spring Boot Starter

Known Users

Arthas has more than 120 registered users, View All.

Welcome to register the company name in this issue: https://github.com/alibaba/arthas/issues/111 (in order of registration)

Derivative Projects

Credits

Contributors

This project exists, thanks to all the people who contributed.

Projects

  • bytekit Java Bytecode Kit.
  • greys-anatomy: The Arthas code base has derived from Greys, we thank for the excellent work done by Greys.
  • termd: Arthas's terminal implementation is based on termd, an open source library for writing terminal applications in Java.
  • crash: Arthas's text based user interface rendering is based on codes extracted from here
  • cli: Arthas's command line interface implementation is based on cli, open sourced by vert.x
  • compiler Arthas's memory compiler.
  • Apache Commons Net Arthas's telnet client.
  • async-profiler Arthas's profiler command.

File: labs/cluster-management/README.md

Arthas Native Agent - 集群管理

快速开始

启动native-agent

native-agent,启动在需要动态attach的服务器上
启动参数

参数 必填 解释
http-port N http端口 ,默认2671
ws-port N ws端口,默认2672
registration-typ Y 注册中心类型(目前实现的有etcd和zookeeper,推荐etcd)
registration-address Y 注册中心的地址

example:

shell
java -jar native-agent.jar --ip 164.196.97.123 --http-port 2671 --ws-port 2672 --registration-type etcd --registration-address 164.196.97.123:2379

启动native-agent-proxy

做为native-agent和native-agent-management-web的网络中转

参数 必填 解释
port N http/ws端口 ,默认2233
ip Y proxy的ip
management-registration-type Y native-agent-manangement-web的注册中心类型(目前实现的有etcd和zookeeper,推荐etcd)
management-registration-address Y native-agent-manangement-webd的注册中心地址
agent-registration-type Y native-agent的注册中心类型(目前实现的有etcd和zookeeper,推荐etcd)
agent-registration-address Y native-agent的注册中心地址

example:

shell
java -jar native-agent-proxy.jar --ip 164.196.97.123 --management-registration-type etcd --management-registration-address 164.196.97.123:2379 --agent-registration-type etcd --agent-registration-address 164.196.97.123:2379

启动native-agent-management-web

native-agent的管理页面

参数 必填 解释
port N http端口 ,默认3939
registration-typ Y 注册中心类型(目前实现的有etcd和zookeeper,推荐etcd)
registration-address Y 注册中心的地址

example:

shell
java -jar native-agent-management-web.jar  --registration-type etcd --registration-address 164.196.97.123:2379

监控指定JVM

进入native-agent-server管理页面,点击VIEW JAVA PROCESS INFO 按钮,可以查看到当前服务器上的Java进程

进入到Java进程页后,我们可以点击Monitor按钮,Monitor目标Java进程

之后点击MONITOR按钮就可以进入到监控界面了

扩展注册中心

目前实现的有zookeeper和etcd,如果想要扩展注册中心,可以看看下面的实现。下面演示的是native-agent-management-web的扩展,其他也是同样的道理。

需要实现com.alibaba.arthas.nat.agent.management.web.discovery.NativeAgentProxyDiscovery接口,并在META-INF/arthas/com.alibaba.arthas.native.agent.management.web.NativeAgentProxyDiscoveryFactory 添加上你的实现

properties
zookeeper=com.alibaba.arthas.nat.agent.management.web.discovery.impl.ZookeeperNativeAgentProxyDiscovery
etcd=com.alibaba.arthas.nat.agent.management.web.discovery.impl.EtcdNativeAgentProxyDiscovery
注册中心名称=你的实现

添加你的实现

注册中心名称=你实现类的具体路径
之后你启动native-agent-management-web就可以,通过--registration-type参数,来指定你实现的注册中心

shell
java -jar native-agent-management-web.jar --registration-type 注册中心名称 --registration-address 注册中心的地址

File: labs/arthas-grpc-web-proxy/ui/README.md

grpc_web_demo

Project setup

text
npm install

Compiles and hot-reloads for development

text
npm run serve

Compiles and minifies for production

text
npm run build

Lints and fixes files

text
npm run lint

Customize configuration

See Configuration Reference.


File: labs/arthas-grpc-web-proxy/README.md

Arthas-grpc

项目启动流程:

1. grpc-web代理服务配置

  1. 前端grpc-web请求ip和port配置: 配置文件
    js
    app.use(ViewUIPlus)
        .use(router)
        .provide("apiHost","http://localhost:8567")
        .mount('#app')
  2. 后端端口配置: 配置文件, 修改GRPC_WEB_PROXY_PORT变量,即可配置grpc-web代理服务端口。
    若需要配置grpc服务端口和http页面服务端口, 分别修改GRPC_PORT和HTTP_PORT即可
    *注意, 前后端grpc-web代理服务端口需一致(默认使用端口号: 8567)

2. 项目编译

shell
mvn compile

3. 项目运行

启动 com.taobao.arthas.grpcweb.grpc.DemoBootstrap

4. 页面访问

启动后,命令行终端会打印出访问地址

text
Open your web browser and navigate to http://127.0.0.1:{http_port}/index.html

netty grpc web proxy

本项目中使用到的grpc-web代理服务

from: https://github.com/grpc/grpc-web/tree/1.4.2/src/connector

原项目已废弃删除,本项目改用 netty 来做转发。

缺点

原项目需要 .proto 文件编译的 .class才能运行,比如GreeterGrpc,本项目同样有这个问题。

测试

工程导入IDE之后,进入test目录

在 com.taobao.arthas.grpcweb.proxy.server.GrpcWebProxyServerTest 启动测试

也可以用原项目的相关工程来测试

开发验证

可以用其它的 grpc web proxy来抓包辅助验证。

用 envoy

下载envoy 后,可以用本项目里的envoy.yaml

  • envoy --config-path ./envoy.yaml

使用 grpcwebproxy

下载后,启动:

  • grpcwebproxy --backend_addr 127.0.0.1:9090 --run_tls_server=false --allow_all_origins

File: memorycompiler/README.md

Fork from https://github.com/skalogs/SkaETL/tree/master/compiler , f6504fb

  • Support get byte code from DynamicCompiler
  • Change package to com.taobao.arthas

File: site/docs/en/doc/advanced-use.md

Other features

Arthas Async Jobs

If you need to investigate an issue, but you are unsure about the exact time it occurs, you can run the monitoring command in the background and save the output to a log file.

Log the output

All execution records are fully saved in the log file for subsequent analysis.

Docker

Arthas configuration reference for using in Docker containers.

Web Console

Arthas supports living inside a browser. The communication between arthas and browser is via websocket.

Arthas Tunnel

Arthas Tunnel Server/Client enables remote management/connection to Java services across multiple servers.

How to use ognl

IDEA Plugin

Build arthas commands more efficiently in the IntelliJ IDEA compiler.

Arthas Properties

Arthas supports configuration options reference.

Start as a Java Agent

Arthas Spring Boot Starter

Starting with the application.

Load External Commands

Load external command jars when Arthas starts and extend team-specific diagnostic actions as Arthas commands.

HTTP API

The Http API provides structured data and supports more complex interactive functions, making it easier to integrate Arthas into custom interfaces.

Batch Processing

It is convenient for running multiple commands in bulk with custom scripts. It can be used in conjunction with the --select parameter to specify the process name.

as.sh and arthas-boot tips

  • Select the process to be attached via the select option.

Normally, as.sh/arthas-boot.jar needs to a pid, bacause the pid will change.

For example, with math-game.jar already started, use the jps command to see.

bash
$ jps
58883 math-game.jar
58884 Jps

The select option allows you to specify a process name, which is very convenient.

bash
$ ./as.sh --select math-game
Arthas script version: 3.3.6
[INFO] JAVA_HOME: /tmp/java/8.0.222-zulu
Arthas home: /Users/admin/.arthas/lib/3.3.6/arthas
Calculating attach execution time...
Attaching to 59161 using version /Users/admin/.arthas/lib/3.3.6/arthas...

real	0m0.572s
user	0m0.281s
sys	0m0.039s
Attach success.
telnet connecting to arthas server... current timestamp is 1594280799
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
  ,---.  ,------. ,--------.,--.  ,--.  ,---.   ,---.
 /  O  \ |  .--. ''--.  .--'|  '--'  | /  O  \ '   .-'
|  .-.  ||  '--'.'   |  |   |  .--.  ||  .-.  |`.  `-.
|  | |  ||  |\  \    |  |   |  |  |  ||  | |  |.-'    |
`--' `--'`--' '--'   `--'   `--'  `--'`--' `--'`-----'


wiki      https://arthas.aliyun.com/doc
tutorials https://arthas.aliyun.com/doc/arthas-tutorials.html
version   3.3.6
pid       58883

User data report

After the 3.1.4 version, arthas support user data report.

At startup, use the stat-url option, such as: ./as.sh --stat-url 'http://192.168.10.11:8080/api/stat'

There is a sample data report in the tunnel server that users can implement on their own.

StatController.java


File: site/docs/en/doc/advice-class.md

Fundamental Fields in Expressions

There is a very fundamental class Advice for the expressions used in filtering, tracing or monitoring and other aspects in commands.

java
public class Advice {

    private final ClassLoader loader;
    private final Class<?> clazz;
    private final ArthasMethod method;
    private final Object target;
    private final Object[] params;
    private final Object returnObj;
    private final Throwable throwExp;
    private final boolean isBefore;
    private final boolean isThrow;
    private final boolean isReturn;

    // getter/setter
}

Description for the variables in the class Advice:

Name Specification
loader the class loader for the current called class
clazz the reference to the current called class
method the reference to the current called method
target the instance of the current called class
params the parameters for the current call, which is an array (when there's no parameter, it will be an empty array)
returnObj the return value from the current call - only available when the method call returns normally (isReturn==true), and null is for void return value
throwExp the exceptions thrown from the current call - only available when the method call throws exception (isThrow==true)
isBefore flag to indicate the method is about to execute. isBefore==true but isThrow==false and isReturn==false since it's no way to know how the method call will end
isThrow flag to indicate the method call ends with exception thrown
isReturn flag to indicate the method call ends normally without exception thrown

All variables listed above can be used directly in the OGNL expression. The command will not execute and exit if there's illegal OGNL grammar or unexpected variable in the expression.