File: README.md
Arthas
Arthas is a Java Diagnostic tool open sourced by Alibaba.
Arthas allows developers to troubleshoot production issues for Java applications without modifying code or restarting servers.
Background
Often times, the production system network is inaccessible from the local development environment. If issues are encountered in production systems, it is impossible to use IDEs to debug the application remotely. More importantly, debugging in production environment is unacceptable, as it will suspend all the threads, resulting in the suspension of business services.
Developers could always try to reproduce the same issue on the test/staging environment. However, this is tricky as some issues cannot be reproduced easily on a different environment, or even disappear once restarted.
And if you're thinking of adding some logs to your code to help troubleshoot the issue, you will have to go through the following lifecycle; test, staging, and then to production. Time is money! This approach is inefficient! Besides, the issue may not be reproducible once the JVM is restarted, as described above.
Arthas was built to solve these issues. A developer can troubleshoot your production issues on-the-fly. No JVM restart, no additional code changes. Arthas works as an observer, which will never suspend your existing threads.
Key features
- Check whether a class is loaded, or where the class is being loaded. (Useful for troubleshooting jar file conflicts)
- Decompile a class to ensure the code is running as expected.
- View classloader statistics, e.g. the number of classloaders, the number of classes loaded per classloader, the classloader hierarchy, possible classloader leaks, etc.
- View the method invocation details, e.g. method parameter, return object, thrown exception, and etc.
- Check the stack trace of specified method invocation. This is useful when a developers wants to know the caller of the said method.
- Trace the method invocation to find slow sub-invocations.
- Monitor method invocation statistics, e.g. qps, rt, success rate and etc.
- Monitor system metrics, thread states and cpu usage, gc statistics, and etc.
- Supports command line interactive mode, with auto-complete feature enabled.
- Supports telnet and websocket, which enables both local and remote diagnostics with command line and browsers.
- Supports profiler/Flame Graph
- Support get objects in the heap that are instances of the specified class.
- Supports JDK 8+ in version 4.x, including JDK 17, JDK 21, and JDK 25.
- Supports Linux/Mac/Windows.
Online Tutorials(Recommended)
Quick start
Use `arthas-boot`(Recommended)
Downloadarthas-boot.jar,Start with java command:
curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jarPrint usage:
java -jar arthas-boot.jar -hUse `as.sh`
You can install Arthas with one single line command on Linux, Unix, and Mac. Copy the following command and paste it into the command line, then press Enter to run:
curl -L https://arthas.aliyun.com/install.sh | shThe command above will download the bootstrap script as.sh to the current directory. You can move it any other place you want, or put its location in $PATH.
You can enter its interactive interface by executing as.sh, or execute as.sh -h for more help information.
Documentation
- Online Tutorials(Recommended)
- User manual
- Installation
- Download
- Quick start
- Advanced usage
- Commands
- WebConsole
- Docker
- Arthas Spring Boot Starter
- User cases
- FAQ
- Compile and debug/How to contribute
- Release Notes
Feature Showcase
Dashboard
Thread
See what is eating your CPU (ranked by top CPU usage) and what is going on there in one glance:
$ thread -n 3
"as-command-execute-daemon" Id=29 cpuUsage=75% RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:440)
at com.taobao.arthas.core.command.monitor200.ThreadCommand$1.action(ThreadCommand.java:58)
at com.taobao.arthas.core.command.handler.AbstractCommandHandler.execute(AbstractCommandHandler.java:238)
at com.taobao.arthas.core.command.handler.DefaultCommandHandler.handleCommand(DefaultCommandHandler.java:67)
at com.taobao.arthas.core.server.ArthasServer$4.run(ArthasServer.java:276)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@6cd0b6f8
"as-session-expire-daemon" Id=25 cpuUsage=24% TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.taobao.arthas.core.server.DefaultSessionManager$2.run(DefaultSessionManager.java:85)
"Reference Handler" Id=2 cpuUsage=0% WAITING on java.lang.ref.Reference$Lock@69ba0f27
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@69ba0f27
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)jad
Decompile your class with one shot:
$ jad javax.servlet.Servlet
ClassLoader:
+-java.net.URLClassLoader@6108b2d7
+-sun.misc.Launcher$AppClassLoader@18b4aac2
+-sun.misc.Launcher$ExtClassLoader@1ddf84b8
Location:
/Users/xxx/work/test/lib/servlet-api.jar
/*
* Decompiled with CFR 0_122.
*/
package javax.servlet;
import java.io.IOException;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
public interface Servlet {
public void init(ServletConfig var1) throws ServletException;
public ServletConfig getServletConfig();
public void service(ServletRequest var1, ServletResponse var2) throws ServletException, IOException;
public String getServletInfo();
public void destroy();
}mc
Memory compiler, compiles .java files into .class files in memory.
$ mc /tmp/Test.javaretransform
Load the external *.class files to retransform/hotswap the loaded classes in JVM.
retransform /tmp/Test.class
retransform -c 327a647b /tmp/Test.class /tmp/Test\$Inner.classsc
Search any loaded class with detailed information.
$ sc -d org.springframework.web.context.support.XmlWebApplicationContext
class-info org.springframework.web.context.support.XmlWebApplicationContext
code-source /Users/xxx/work/test/WEB-INF/lib/spring-web-3.2.11.RELEASE.jar
name org.springframework.web.context.support.XmlWebApplicationContext
isInterface false
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name XmlWebApplicationContext
modifier public
annotation
interfaces
super-class +-org.springframework.web.context.support.AbstractRefreshableWebApplicationContext
+-org.springframework.context.support.AbstractRefreshableConfigApplicationContext
+-org.springframework.context.support.AbstractRefreshableApplicationContext
+-org.springframework.context.support.AbstractApplicationContext
+-org.springframework.core.io.DefaultResourceLoader
+-java.lang.Object
class-loader +-org.apache.catalina.loader.ParallelWebappClassLoader
+-java.net.URLClassLoader@6108b2d7
+-sun.misc.Launcher$AppClassLoader@18b4aac2
+-sun.misc.Launcher$ExtClassLoader@1ddf84b8
classLoaderHash 25131501vmtool
Get objects in the heap that are instances of the specified class.
$ vmtool --action getInstances --className java.lang.String --limit 10
@String[][
@String[com/taobao/arthas/core/shell/session/Session],
@String[com.taobao.arthas.core.shell.session.Session],
@String[com/taobao/arthas/core/shell/session/Session],
@String[com/taobao/arthas/core/shell/session/Session],
@String[com/taobao/arthas/core/shell/session/Session.class],
@String[com/taobao/arthas/core/shell/session/Session.class],
@String[com/taobao/arthas/core/shell/session/Session.class],
@String[com/],
@String[java/util/concurrent/ConcurrentHashMap$ValueIterator],
@String[java/util/concurrent/locks/LockSupport],
]stack
View the call stack of test.arthas.TestStack#doGet:
/* Detailed source-code truncated for AI context efficiency. */Trace
See what is slowing down your method invocation with trace command:
Watch
Watch the first parameter and thrown exception of test.arthas.TestWatch#doGet only if it throws exception.
$ watch test.arthas.TestWatch doGet {params[0], throwExp} -e
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 65 ms.
ts=2018-09-18 10:26:28;result=@ArrayList[
@RequestFacade[org.apache.catalina.connector.RequestFacade@79f922b2],
@NullPointerException[java.lang.NullPointerException],
]Monitor
Monitor a specific method invocation statistics, including the total number of invocations, average response time, success rate, and every 5 seconds:
$ monitor -c 5 org.apache.dubbo.demo.provider.DemoServiceImpl sayHello
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 109 ms.
timestamp class method total success fail avg-rt(ms) fail-rate
----------------------------------------------------------------------------------------------------------------------------
2018-09-20 09:45:32 org.apache.dubbo.demo.provider.DemoServiceImpl sayHello 5 5 0 0.67 0.00%
timestamp class method total success fail avg-rt(ms) fail-rate
----------------------------------------------------------------------------------------------------------------------------
2018-09-20 09:45:37 org.apache.dubbo.demo.provider.DemoServiceImpl sayHello 5 5 0 1.00 0.00%
timestamp class method total success fail avg-rt(ms) fail-rate
----------------------------------------------------------------------------------------------------------------------------
2018-09-20 09:45:42 org.apache.dubbo.demo.provider.DemoServiceImpl sayHello 5 5 0 0.43 0.00%Time Tunnel(tt)
Record method invocation data, so that you can check the method invocation parameters, returned value, and thrown exceptions later. It works as if you could come back and replay the past method invocation via time tunnel.
$ tt -t org.apache.dubbo.demo.provider.DemoServiceImpl sayHello
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 75 ms.
INDEX TIMESTAMP COST(ms) IS-RET IS-EXP OBJECT CLASS METHOD
-------------------------------------------------------------------------------------------------------------------------------------
1000 2018-09-20 09:54:10 1.971195 true false 0x55965cca DemoServiceImpl sayHello
1001 2018-09-20 09:54:11 0.215685 true false 0x55965cca DemoServiceImpl sayHello
1002 2018-09-20 09:54:12 0.236303 true false 0x55965cca DemoServiceImpl sayHello
1003 2018-09-20 09:54:13 0.159598 true false 0x55965cca DemoServiceImpl sayHello
1004 2018-09-20 09:54:14 0.201982 true false 0x55965cca DemoServiceImpl sayHello
1005 2018-09-20 09:54:15 0.214205 true false 0x55965cca DemoServiceImpl sayHello
1006 2018-09-20 09:54:16 0.241863 true false 0x55965cca DemoServiceImpl sayHello
1007 2018-09-20 09:54:17 0.305747 true false 0x55965cca DemoServiceImpl sayHello
1008 2018-09-20 09:54:18 0.18468 true false 0x55965cca DemoServiceImpl sayHelloClassloader
$ classloader
name numberOfInstances loadedCountTotal
BootstrapClassLoader 1 3346
com.taobao.arthas.agent.ArthasClassloader 1 1262
java.net.URLClassLoader 2 1033
org.apache.catalina.loader.ParallelWebappClassLoader 1 628
sun.reflect.DelegatingClassLoader 166 166
sun.misc.Launcher$AppClassLoader 1 31
com.alibaba.fastjson.util.ASMClassLoader 6 15
sun.misc.Launcher$ExtClassLoader 1 7
org.jvnet.hk2.internal.DelegatingClassLoader 2 2
sun.reflect.misc.MethodUtil 1 1Web Console
Profiler/FlameGraph
$ profiler start
Started [cpu] profiling$ profiler stop
profiler output file: /tmp/demo/arthas-output/20211207-111550.html
OKView profiler results under arthas-output via browser:
Arthas Spring Boot Starter
Known Users
Arthas has more than 120 registered users, View All.
Welcome to register the company name in this issue: https://github.com/alibaba/arthas/issues/111 (in order of registration)
Derivative Projects
Credits
Contributors
This project exists, thanks to all the people who contributed.
Projects
- bytekit Java Bytecode Kit.
- greys-anatomy: The Arthas code base has derived from Greys, we thank for the excellent work done by Greys.
- termd: Arthas's terminal implementation is based on termd, an open source library for writing terminal applications in Java.
- crash: Arthas's text based user interface rendering is based on codes extracted from here
- cli: Arthas's command line interface implementation is based on cli, open sourced by vert.x
- compiler Arthas's memory compiler.
- Apache Commons Net Arthas's telnet client.
- async-profiler Arthas's profiler command.
File: labs/cluster-management/README.md
Arthas Native Agent - 集群管理
快速开始
启动native-agent
native-agent,启动在需要动态attach的服务器上
启动参数
| 参数 | 必填 | 解释 |
|---|---|---|
| http-port | N | http端口 ,默认2671 |
| ws-port | N | ws端口,默认2672 |
| registration-typ | Y | 注册中心类型(目前实现的有etcd和zookeeper,推荐etcd) |
| registration-address | Y | 注册中心的地址 |
example:
java -jar native-agent.jar --ip 164.196.97.123 --http-port 2671 --ws-port 2672 --registration-type etcd --registration-address 164.196.97.123:2379启动native-agent-proxy
做为native-agent和native-agent-management-web的网络中转
| 参数 | 必填 | 解释 |
|---|---|---|
| port | N | http/ws端口 ,默认2233 |
| ip | Y | proxy的ip |
| management-registration-type | Y | native-agent-manangement-web的注册中心类型(目前实现的有etcd和zookeeper,推荐etcd) |
| management-registration-address | Y | native-agent-manangement-webd的注册中心地址 |
| agent-registration-type | Y | native-agent的注册中心类型(目前实现的有etcd和zookeeper,推荐etcd) |
| agent-registration-address | Y | native-agent的注册中心地址 |
example:
java -jar native-agent-proxy.jar --ip 164.196.97.123 --management-registration-type etcd --management-registration-address 164.196.97.123:2379 --agent-registration-type etcd --agent-registration-address 164.196.97.123:2379启动native-agent-management-web
native-agent的管理页面
| 参数 | 必填 | 解释 |
|---|---|---|
| port | N | http端口 ,默认3939 |
| registration-typ | Y | 注册中心类型(目前实现的有etcd和zookeeper,推荐etcd) |
| registration-address | Y | 注册中心的地址 |
example:
java -jar native-agent-management-web.jar --registration-type etcd --registration-address 164.196.97.123:2379监控指定JVM
进入native-agent-server管理页面,点击VIEW JAVA PROCESS INFO 按钮,可以查看到当前服务器上的Java进程
进入到Java进程页后,我们可以点击Monitor按钮,Monitor目标Java进程
之后点击MONITOR按钮就可以进入到监控界面了
扩展注册中心
目前实现的有zookeeper和etcd,如果想要扩展注册中心,可以看看下面的实现。下面演示的是native-agent-management-web的扩展,其他也是同样的道理。
需要实现com.alibaba.arthas.nat.agent.management.web.discovery.NativeAgentProxyDiscovery接口,并在META-INF/arthas/com.alibaba.arthas.native.agent.management.web.NativeAgentProxyDiscoveryFactory 添加上你的实现
zookeeper=com.alibaba.arthas.nat.agent.management.web.discovery.impl.ZookeeperNativeAgentProxyDiscovery
etcd=com.alibaba.arthas.nat.agent.management.web.discovery.impl.EtcdNativeAgentProxyDiscovery
注册中心名称=你的实现添加你的实现
注册中心名称=你实现类的具体路径
之后你启动native-agent-management-web就可以,通过--registration-type参数,来指定你实现的注册中心
java -jar native-agent-management-web.jar --registration-type 注册中心名称 --registration-address 注册中心的地址File: labs/arthas-grpc-web-proxy/ui/README.md
grpc_web_demo
Project setup
npm installCompiles and hot-reloads for development
npm run serveCompiles and minifies for production
npm run buildLints and fixes files
npm run lintCustomize configuration
File: labs/arthas-grpc-web-proxy/README.md
Arthas-grpc
项目启动流程:
1. grpc-web代理服务配置
- 前端grpc-web请求ip和port配置: 配置文件js
app.use(ViewUIPlus) .use(router) .provide("apiHost","http://localhost:8567") .mount('#app') - 后端端口配置: 配置文件, 修改
GRPC_WEB_PROXY_PORT变量,即可配置grpc-web代理服务端口。
若需要配置grpc服务端口和http页面服务端口, 分别修改GRPC_PORT和HTTP_PORT即可
*注意, 前后端grpc-web代理服务端口需一致(默认使用端口号: 8567)
2. 项目编译
mvn compile3. 项目运行
启动 com.taobao.arthas.grpcweb.grpc.DemoBootstrap
4. 页面访问
启动后,命令行终端会打印出访问地址
Open your web browser and navigate to http://127.0.0.1:{http_port}/index.htmlnetty grpc web proxy
本项目中使用到的grpc-web代理服务
from: https://github.com/grpc/grpc-web/tree/1.4.2/src/connector
原项目已废弃删除,本项目改用 netty 来做转发。
缺点
原项目需要 .proto 文件编译的 .class才能运行,比如GreeterGrpc,本项目同样有这个问题。
测试
工程导入IDE之后,进入test目录
在 com.taobao.arthas.grpcweb.proxy.server.GrpcWebProxyServerTest 启动测试
也可以用原项目的相关工程来测试
开发验证
可以用其它的 grpc web proxy来抓包辅助验证。
用 envoy
下载envoy 后,可以用本项目里的envoy.yaml
envoy --config-path ./envoy.yaml
使用 grpcwebproxy
下载后,启动:
grpcwebproxy --backend_addr 127.0.0.1:9090 --run_tls_server=false --allow_all_origins
File: memorycompiler/README.md
Fork from https://github.com/skalogs/SkaETL/tree/master/compiler , f6504fb
- Support get byte code from
DynamicCompiler - Change package to
com.taobao.arthas
File: site/docs/en/doc/advanced-use.md
Other features
Arthas Async Jobs
If you need to investigate an issue, but you are unsure about the exact time it occurs, you can run the monitoring command in the background and save the output to a log file.
Log the output
All execution records are fully saved in the log file for subsequent analysis.
Docker
Arthas configuration reference for using in Docker containers.
Web Console
Arthas supports living inside a browser. The communication between arthas and browser is via websocket.
Arthas Tunnel
Arthas Tunnel Server/Client enables remote management/connection to Java services across multiple servers.
How to use ognl
IDEA Plugin
Build arthas commands more efficiently in the IntelliJ IDEA compiler.
Arthas Properties
Arthas supports configuration options reference.
Start as a Java Agent
Arthas Spring Boot Starter
Starting with the application.
Load External Commands
Load external command jars when Arthas starts and extend team-specific diagnostic actions as Arthas commands.
HTTP API
The Http API provides structured data and supports more complex interactive functions, making it easier to integrate Arthas into custom interfaces.
Batch Processing
It is convenient for running multiple commands in bulk with custom scripts. It can be used in conjunction with the --select parameter to specify the process name.
as.sh and arthas-boot tips
- Select the process to be attached via the
selectoption.
Normally, as.sh/arthas-boot.jar needs to a pid, bacause the pid will change.
For example, with math-game.jar already started, use the jps command to see.
$ jps
58883 math-game.jar
58884 JpsThe select option allows you to specify a process name, which is very convenient.
$ ./as.sh --select math-game
Arthas script version: 3.3.6
[INFO] JAVA_HOME: /tmp/java/8.0.222-zulu
Arthas home: /Users/admin/.arthas/lib/3.3.6/arthas
Calculating attach execution time...
Attaching to 59161 using version /Users/admin/.arthas/lib/3.3.6/arthas...
real 0m0.572s
user 0m0.281s
sys 0m0.039s
Attach success.
telnet connecting to arthas server... current timestamp is 1594280799
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
,---. ,------. ,--------.,--. ,--. ,---. ,---.
/ O \ | .--. ''--. .--'| '--' | / O \ ' .-'
| .-. || '--'.' | | | .--. || .-. |`. `-.
| | | || |\ \ | | | | | || | | |.-' |
`--' `--'`--' '--' `--' `--' `--'`--' `--'`-----'
wiki https://arthas.aliyun.com/doc
tutorials https://arthas.aliyun.com/doc/arthas-tutorials.html
version 3.3.6
pid 58883User data report
After the 3.1.4 version, arthas support user data report.
At startup, use the stat-url option, such as: ./as.sh --stat-url 'http://192.168.10.11:8080/api/stat'
There is a sample data report in the tunnel server that users can implement on their own.
File: site/docs/en/doc/advice-class.md
Fundamental Fields in Expressions
There is a very fundamental class Advice for the expressions used in filtering, tracing or monitoring and other aspects in commands.
public class Advice {
private final ClassLoader loader;
private final Class<?> clazz;
private final ArthasMethod method;
private final Object target;
private final Object[] params;
private final Object returnObj;
private final Throwable throwExp;
private final boolean isBefore;
private final boolean isThrow;
private final boolean isReturn;
// getter/setter
}Description for the variables in the class Advice:
| Name | Specification |
|---|---|
| loader | the class loader for the current called class |
| clazz | the reference to the current called class |
| method | the reference to the current called method |
| target | the instance of the current called class |
| params | the parameters for the current call, which is an array (when there's no parameter, it will be an empty array) |
| returnObj | the return value from the current call - only available when the method call returns normally (isReturn==true), and null is for void return value |
| throwExp | the exceptions thrown from the current call - only available when the method call throws exception (isThrow==true) |
| isBefore | flag to indicate the method is about to execute. isBefore==true but isThrow==false and isReturn==false since it's no way to know how the method call will end |
| isThrow | flag to indicate the method call ends with exception thrown |
| isReturn | flag to indicate the method call ends normally without exception thrown |
All variables listed above can be used directly in the OGNL expression. The command will not execute and exit if there's illegal OGNL grammar or unexpected variable in the expression.