Open-AutoGLM Documentation

File: README.md Copy Chapter

Open-AutoGLM

Readme in English

懒人版快速安装 Copy Chapter

你可以使用Claude Code，配置 GLM Coding Plan 后，输入以下提示词，快速部署本项目。

访问文档，为我安装 AutoGLM
https://raw.githubusercontent.com/zai-org/Open-AutoGLM/refs/heads/main/README.md

项目介绍 Copy Chapter

Phone Agent 是一个基于 AutoGLM 构建的手机端智能助理框架，它能够以多模态方式理解手机屏幕内容，并通过自动化操作帮助用户完成任务。系统通过
ADB(Android Debug Bridge)来控制设备，以视觉语言模型进行屏幕感知，再结合智能规划能力生成并执行操作流程。用户只需用自然语言描述需求，如“打开小红书搜索美食”，Phone
Agent 即可自动解析意图、理解当前界面、规划下一步动作并完成整个流程。系统还内置敏感操作确认机制，并支持在登录或验证码场景下进行人工接管。同时，它提供远程
ADB 调试能力，可通过 WiFi 或网络连接设备，实现灵活的远程控制与开发。

⚠️
本项目仅供研究和学习使用。严禁用于非法获取信息、干扰系统或任何违法活动。请仔细审阅 使用条款。

与其他自动化工具集成 Copy Chapter

Midscene.js Copy Section

Midscene.js 是一款由视觉模型驱动的开源 UI 自动化 SDK，支持通过 JavaScript 或 Yaml 格式的流程语法，实现多平台的自动化。

目前 Midscene.js 已完成对 AutoGLM 模型的适配，你可以通过 Midscene.js 接入指南 快速体验 AutoGLM 在 iOS 和 Android 设备上的自动化效果。

模型下载地址 Copy Chapter

其中，AutoGLM-Phone-9B 是针对中文手机应用优化的模型，而 AutoGLM-Phone-9B-Multilingual 支持英语场景，适用于包含英文等其他语言内容的应用。

Android 环境准备 Copy Chapter

1. Python 环境 Copy Section

建议使用 Python 3.10 及以上版本。

2. 手机调试命令行工具 Copy Section

根据你的设备类型选择相应的工具：

对于 Android 设备 - 使用 ADB

1. 下载官方 ADB 安装包，并解压到自定义路径
2. 配置环境变量

• MacOS 配置方法：在 Terminal 或者任何命令行工具里

  bash Copy

  # 假设解压后的目录为 ~/Downloads/platform-tools。如果不是请自行调整命令。
  export PATH=${PATH}:~/Downloads/platform-tools

• Windows 配置方法：可参考 第三方教程 进行配置。

对于鸿蒙设备 (HarmonyOS NEXT版本以上) - 使用 HDC

1. 下载 HDC 工具：
   • 从 HarmonyOS SDK 下载
2. 配置环境变量

• MacOS/Linux 配置方法：

  bash Copy

  # 假设解压后的目录为 ~/Downloads/harmonyos-sdk/toolchains。请根据实际路径调整。
  export PATH=${PATH}:~/Downloads/harmonyos-sdk/toolchains

• Windows 配置方法：将 HDC 工具所在目录添加到系统 PATH 环境变量

3. Android 7.0+ 或 HarmonyOS 设备，并启用 `开发者模式` 和 `USB 调试` Copy Section

1. 开发者模式启用：通常启用方法是，找到 设置-关于手机-版本号 然后连续快速点击 10
   次左右，直到弹出弹窗显示“开发者模式已启用”。不同手机会有些许差别，如果找不到，可以上网搜索一下教程。
2. USB 调试启用：启用开发者模式之后，会出现 设置-开发者选项-USB 调试，勾选启用
3. 部分机型在设置开发者选项以后, 可能需要重启设备才能生效. 可以测试一下: 将手机用USB数据线连接到电脑后, adb devices
   查看是否有设备信息, 如果没有说明连接失败.

请务必仔细检查相关权限

4. 安装 ADB Keyboard(仅 Android 设备需要，用于文本输入) Copy Section

注意：鸿蒙设备使用原生输入方法，无需安装 ADB Keyboard。

如果你使用的是 Android 设备：

下载 安装包 并在对应的安卓设备中进行安装。
注意，安装完成后还需要到 设置-输入法 或者 设置-键盘列表 中启用 ADB Keyboard 才能生效(或使用命令adb shell ime enable com.android.adbkeyboard/.AdbIMEHow-to-use)

iPhone 环境准备 Copy Chapter

如果你使用的是 iPhone 设备，请参考专门的 iOS 配置文档：

📱 iOS 环境配置指南

该文档详细介绍了如何配置 WebDriverAgent 和 iPhone 设备，以便在 iOS 上使用 AutoGLM。

部署准备工作 Copy Chapter

1. 安装依赖 Copy Section

pip install -r requirements.txt 
pip install -e .

2. 配置 ADB 或 HDC Copy Section

对于 Android 设备

确认 USB数据线具有数据传输功能, 而不是仅有充电功能

确保已安装 ADB 并使用 USB数据线 连接设备：

# 检查已连接的设备
adb devices

# 输出结果应显示你的设备，如：
# List of devices attached
# emulator-5554   device

对于鸿蒙设备

确认 USB数据线具有数据传输功能, 而不是仅有充电功能

确保已安装 HDC 并使用 USB数据线 连接设备：

# 检查已连接的设备
hdc list targets

# 输出结果应显示你的设备，如：
# 7001005458323933328a01bce01c2500

3. 启动模型服务 Copy Section

你可以选择自行部署模型服务，或使用第三方模型服务商。

选项 A: 使用第三方模型服务

如果你不想自行部署模型，可以使用以下已部署我们模型的第三方服务：

1. 智谱 BigModel

• 文档: https://docs.bigmodel.cn/cn/api/introduction
• --base-url: https://open.bigmodel.cn/api/paas/v4
• --model: autoglm-phone
• --apikey: 在智谱平台申请你的 API Key

2. ModelScope(魔搭社区)

• 文档: https://modelscope.cn/models/ZhipuAI/AutoGLM-Phone-9B
• --base-url: https://api-inference.modelscope.cn/v1
• --model: ZhipuAI/AutoGLM-Phone-9B
• --apikey: 在 ModelScope 平台申请你的 API Key

使用第三方服务的示例：

# 使用智谱 BigModel
python main.py --base-url https://open.bigmodel.cn/api/paas/v4 --model "autoglm-phone" --apikey "your-bigmodel-api-key" "打开美团搜索附近的火锅店"

# 使用 ModelScope
python main.py --base-url https://api-inference.modelscope.cn/v1 --model "ZhipuAI/AutoGLM-Phone-9B" --apikey "your-modelscope-api-key" "打开美团搜索附近的火锅店"

选项 B: 自行部署模型

如果你希望在本地或自己的服务器上部署模型：

1. 按照 requirements.txt 中 For Model Deployment 章节自行安装推理引擎框架。

对于SGLang， 除了使用pip安装，你也可以使用官方docker:

shell Copy

docker pull lmsysorg/sglang:v0.5.6.post1

进入容器，执行

text Copy

pip install nvidia-cudnn-cu12==9.16.0.29

对于 vLLM，除了使用pip 安装，你也可以使用官方docker:

shell Copy

docker pull vllm/vllm-openai:v0.12.0

进入容器，执行

text Copy

pip install -U transformers --pre

注意: 上述步骤出现的关于 transformers 的依赖冲突可以忽略。

1. 在对应容器或者实体机中(非容器安装)下载模型，通过 SGlang / vLLM 启动，得到 OpenAI 格式服务。这里提供一个 vLLM部署方案，请严格遵循我们提供的启动参数:

• vLLM:

python3 -m vllm.entrypoints.openai.api_server \
 --served-model-name autoglm-phone-9b \
 --allowed-local-media-path /   \
 --mm-encoder-tp-mode data \
 --mm_processor_cache_type shm \
 --mm_processor_kwargs "{\"max_pixels\":5000000}" \
 --max-model-len 25480  \
 --chat-template-content-format string \
 --limit-mm-per-prompt "{\"image\":10}" \
 --model zai-org/AutoGLM-Phone-9B \
 --port 8000

• SGLang:

python3 -m sglang.launch_server --model-path  zai-org/AutoGLM-Phone-9B \
        --served-model-name autoglm-phone-9b  \
        --context-length 25480  \
        --mm-enable-dp-encoder   \
        --mm-process-config '{"image":{"max_pixels":5000000}}'  \
        --port 8000

• 该模型结构与 GLM-4.1V-9B-Thinking 相同, 关于模型部署的详细内容，你也以查看 GLM-V
  获取模型部署和使用指南。

• 运行成功后，将可以通过 http://localhost:8000/v1 访问模型服务。 如果您在远程服务器部署模型, 使用该服务器的IP访问模型.

4. 检查模型部署 Copy Section

模型服务启动后，可以使用检查脚本验证部署是否成功：

python scripts/check_deployment_cn.py --base-url http://你的IP:你的端口/v1 --model 模型名称

脚本将发送测试请求并展示模型的推理结果，你可以根据输出判断模型部署是否正常工作。

基于给定的任务, 预期输出如下。如果思维链长度很短, 或者出现了乱码, 很可能是模型部署失败, 请仔细检查文档要求的配置和依赖。

用户想要比较这个洗发水在京东和淘宝上的价格，然后选择最便宜的平台下单。当前在小红书app上，显示的是一个关于LUMMI MOOD洗发水的帖子。

我需要：
1. 先启动京东app，搜索这个洗发水
2. 查看京东的价格
3. 再启动淘宝app，搜索这个洗发水
4. 查看淘宝的价格
5. 比较价格后，选择最便宜的京东或淘宝下单

首先，我需要从当前的小红书界面退出，然后启动京东app。
<answer>do(action="Launch", app="京东")

参数说明：

• --base-url: 模型服务地址(根据实际部署地址修改)
• --model: 模型名称
• --messages-file: 可选，指定自定义测试消息文件(默认使用 scripts/sample_messages.json)

使用 AutoGLM Copy Chapter

命令行 Copy Section

根据你部署的模型, 设置 --base-url 和 --model 参数, 设置 --device-type 指定是安卓设备或鸿蒙设备 (默认值 adb 表示安卓设备, hdc 表示鸿蒙设备). 例如:

# Android 设备 - 交互模式
python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b"

# Android 设备 - 指定任务
python main.py --base-url http://localhost:8000/v1 "打开美团搜索附近的火锅店"

# 鸿蒙设备 - 交互模式
python main.py --device-type hdc --base-url http://localhost:8000/v1 --model "autoglm-phone-9b"

# 鸿蒙设备 - 指定任务
python main.py --device-type hdc --base-url http://localhost:8000/v1 "打开美团搜索附近的火锅店"

# 使用 API Key 进行认证
python main.py --apikey sk-xxxxx

# 使用英文 system prompt
python main.py --lang en --base-url http://localhost:8000/v1 "Open Chrome browser"

# 列出支持的应用（Android）
python main.py --list-apps

# 列出支持的应用（鸿蒙）
python main.py --device-type hdc --list-apps

Python API Copy Section

from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

# Configure model
model_config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b",
)

# 创建 Agent
agent = PhoneAgent(model_config=model_config)

# 执行任务
result = agent.run("打开淘宝搜索无线耳机")
print(result)

远程调试 Copy Chapter

Phone Agent 支持通过 WiFi/网络进行远程 ADB/HDC 调试，无需 USB 连接即可控制设备。

配置远程调试 Copy Section

在手机端开启无线调试

Android 设备

确保手机和电脑在同一个WiFi中，如图所示

鸿蒙设备

确保手机和电脑在同一个WiFi中：

1. 进入 设置 > 系统和更新 > 开发者选项
2. 开启 USB 调试 和 无线调试
3. 记录显示的 IP 地址和端口号

在电脑端使用标准 ADB/HDC 命令

# Android 设备 - 通过 WiFi 连接, 改成手机显示的 IP 地址和端口
adb connect 192.168.1.100:5555

# 验证连接
adb devices
# 应显示：192.168.1.100:5555    device

# 鸿蒙设备 - 通过 WiFi 连接
hdc tconn 192.168.1.100:5555

# 验证连接
hdc list targets
# 应显示：192.168.1.100:5555

设备管理命令 Copy Section

Android 设备（ADB）

# 列出所有已连接设备
adb devices

# 连接远程设备
adb connect 192.168.1.100:5555

# 断开指定设备
adb disconnect 192.168.1.100:5555

# 指定设备执行任务
python main.py --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "打开抖音刷视频"

鸿蒙设备（HDC）

# 列出所有已连接设备
hdc list targets

# 连接远程设备
hdc tconn 192.168.1.100:5555

# 断开指定设备
hdc tdisconn 192.168.1.100:5555

# 指定设备执行任务
python main.py --device-type hdc --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "打开抖音刷视频"

Python API 远程连接 Copy Section

Android 设备（ADB）

from phone_agent.adb import ADBConnection, list_devices

# 创建连接管理器
conn = ADBConnection()

# 连接远程设备
success, message = conn.connect("192.168.1.100:5555")
print(f"连接状态: {message}")

# 列出已连接设备
devices = list_devices()
for device in devices:
    print(f"{device.device_id} - {device.connection_type.value}")

# 在 USB 设备上启用 TCP/IP
success, message = conn.enable_tcpip(5555)
ip = conn.get_device_ip()
print(f"设备 IP: {ip}")

# 断开连接
conn.disconnect("192.168.1.100:5555")

鸿蒙设备（HDC）

from phone_agent.hdc import HDCConnection, list_devices

# 创建连接管理器
conn = HDCConnection()

# 连接远程设备
success, message = conn.connect("192.168.1.100:5555")
print(f"连接状态: {message}")

# 列出已连接设备
devices = list_devices()
for device in devices:
    print(f"{device.device_id} - {device.connection_type.value}")

# 断开连接
conn.disconnect("192.168.1.100:5555")

远程连接问题排查 Copy Section

连接被拒绝：

• 确保设备和电脑在同一网络
• 检查防火墙是否阻止 5555 端口
• 确认已启用 TCP/IP 模式：adb tcpip 5555

连接断开：

• WiFi 可能断开了，使用 --connect 重新连接
• 部分设备重启后会禁用 TCP/IP，需要通过 USB 重新启用

多设备：

• 使用 --device-id 指定要使用的设备
• 或使用 --list-devices 查看所有已连接设备

配置 Copy Chapter

自定义SYSTEM PROMPT Copy Section

系统提供中英文两套 prompt，通过 --lang 参数切换：

• --lang cn - 中文 prompt(默认)，配置文件：phone_agent/config/prompts_zh.py
• --lang en - 英文 prompt，配置文件：phone_agent/config/prompts_en.py

可以直接修改对应的配置文件来增强模型在特定领域的能力，或通过注入 app 名称禁用某些 app。

环境变量 Copy Section

模型配置 Copy Section

from phone_agent.model import ModelConfig

config = ModelConfig(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",  # API 密钥(如需要)
    model_name="autoglm-phone-9b",  # 模型名称
    max_tokens=3000,  # 最大输出 token 数
    temperature=0.1,  # 采样温度
    frequency_penalty=0.2,  # 频率惩罚
)

Agent 配置 Copy Section

from phone_agent.agent import AgentConfig

config = AgentConfig(
    max_steps=100,  # 每个任务最大步数
    device_id=None,  # ADB 设备 ID(None 为自动检测)
    lang="cn",  # 语言选择：cn(中文)或 en(英文)
    verbose=True,  # 打印调试信息(包括思考过程和执行动作)
)

Verbose 模式输出 Copy Section

当 verbose=True 时，Agent 会在每一步输出详细信息：

==================================================
💭 思考过程:
--------------------------------------------------
当前在系统桌面，需要先启动小红书应用
--------------------------------------------------
🎯 执行动作:
{
  "_metadata": "do",
  "action": "Launch",
  "app": "小红书"
}
==================================================

... (执行动作后继续下一步)

==================================================
💭 思考过程:
--------------------------------------------------
小红书已打开，现在需要点击搜索框
--------------------------------------------------
🎯 执行动作:
{
  "_metadata": "do",
  "action": "Tap",
  "element": [500, 100]
}
==================================================

🎉 ================================================
✅ 任务完成: 已成功搜索美食攻略
==================================================

这样可以清楚地看到 AI 的推理过程和每一步的具体操作。

支持的应用 Copy Chapter

Android 应用 Copy Section

Phone Agent 支持 50+ 款主流中文应用：

运行 python main.py --list-apps 查看完整列表。

鸿蒙应用 Copy Section

Phone Agent 支持 60+ 款鸿蒙原生应用和系统应用：

运行 python main.py --device-type hdc --list-apps 查看完整列表。

可用操作 Copy Chapter

Agent 可以执行以下操作：

自定义回调 Copy Chapter

处理敏感操作确认和人工接管：

def my_confirmation(message: str) -> bool:
    """敏感操作确认回调"""
    return input(f"确认执行 {message}？(y/n): ").lower() == "y"


def my_takeover(message: str) -> None:
    """人工接管回调"""
    print(f"请手动完成: {message}")
    input("完成后按回车继续...")


agent = PhoneAgent(
    confirmation_callback=my_confirmation,
    takeover_callback=my_takeover,
)

示例 Copy Chapter

查看 examples/ 目录获取更多使用示例：

• basic_usage.py - 基础任务执行
• 单步调试模式
• 批量任务执行
• 自定义回调

二次开发 Copy Chapter

配置开发环境 Copy Section

二次开发需要使用开发依赖：

pip install -e ".[dev]"

运行测试 Copy Section

pytest tests/

完整项目结构 Copy Section

phone_agent/
├── __init__.py          # 包导出
├── agent.py             # PhoneAgent 主类
├── adb/                 # ADB 工具
│   ├── connection.py    # 远程/本地连接管理
│   ├── screenshot.py    # 屏幕截图
│   ├── input.py         # 文本输入 (ADB Keyboard)
│   └── device.py        # 设备控制 (点击、滑动等)
├── actions/             # 操作处理
│   └── handler.py       # 操作执行器
├── config/              # 配置
│   ├── apps.py          # 支持的应用映射
│   ├── prompts_zh.py    # 中文系统提示词
│   └── prompts_en.py    # 英文系统提示词
└── model/               # AI 模型客户端
    └── client.py        # OpenAI 兼容客户端

常见问题 Copy Chapter

我们列举了一些常见的问题，以及对应的解决方案：

设备未找到 Copy Section

尝试通过重启 ADB 服务来解决：

adb kill-server
adb start-server
adb devices

如果仍然无法识别，请检查：

1. USB 调试是否已开启
2. 数据线是否支持数据传输(部分数据线仅支持充电)
3. 手机上弹出的授权框是否已点击「允许」
4. 尝试更换 USB 接口或数据线

能打开应用，但无法点击 Copy Section

部分机型需要同时开启两个调试选项才能正常使用：

• USB 调试
• USB 调试(安全设置)

请在 设置 → 开发者选项 中检查这两个选项是否都已启用。

文本输入不工作 Copy Section

1. 确保设备已安装 ADB Keyboard
2. 在设置 > 系统 > 语言和输入法 > 虚拟键盘 中启用
3. Agent 会在需要输入时自动切换到 ADB Keyboard

截图失败(黑屏) Copy Section

这通常意味着应用正在显示敏感页面(支付、密码、银行类应用)。Agent 会自动检测并请求人工接管。

windows 编码异常问题 Copy Section

报错信息形如 UnicodeEncodeError gbk code

解决办法: 在运行代码的命令前面加上环境变量: PYTHONIOENCODING=utf-8

交互模式非TTY环境无法使用 Copy Section

报错形如: EOF when reading a line

解决办法: 使用非交互模式直接指定任务, 或者切换到 TTY 模式的终端应用.

引用 Copy Section

如果你觉得我们的工作有帮助，请引用以下论文：

@article{liu2024autoglm,
  title={Autoglm: Autonomous foundation agents for guis},
  author={Liu, Xiao and Qin, Bo and Liang, Dongzhu and Dong, Guang and Lai, Hanyu and Zhang, Hanchen and Zhao, Hanlin and Iong, Iat Long and Sun, Jiadai and Wang, Jiaqi and others},
  journal={arXiv preprint arXiv:2411.00820},
  year={2024}
}
@article{xu2025mobilerl,
  title={MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents},
  author={Xu, Yifan and Liu, Xiao and Liu, Xinghan and Fu, Jiaqi and Zhang, Hanchen and Jing, Bohao and Zhang, Shudan and Wang, Yuting and Zhao, Wenyi and Dong, Yuxiao},
  journal={arXiv preprint arXiv:2509.18119},
  year={2025}
}

自动化部署指南(面向 AI) Copy Chapter

本章节专为 AI 助手(如 Claude Code)设计，用于自动化部署 Open-AutoGLM。

如果你是人类读者，可以跳过本章节，按照上面的文档操作即可。

项目概述 Copy Section

Open-AutoGLM 是一个手机 Agent 框架：

• 输入：用户的自然语言指令(如"打开微信发消息给张三")
• 输出：自动操作用户的安卓手机完成任务
• 原理：截图 → 视觉模型理解界面 → 输出点击坐标 → ADB 执行操作 → 循环

架构分为两部分：

1. Agent 代码(本仓库)：运行在用户电脑上，负责调用模型、解析动作、控制手机
2. 视觉模型服务：可以是远程 API，也可以本地部署

部署前置检查 Copy Section

在开始部署前，请逐项向用户确认以下内容：

硬件环境

• 用户有一台安卓手机(Android 7.0+)
• 用户有一根支持数据传输的 USB 数据线(不是仅充电线)
• 手机和电脑可以通过数据线连接

手机端配置

• 手机已开启「开发者模式」(设置 → 关于手机 → 连续点击版本号 7 次)
• 手机已开启「USB 调试」(设置 → 开发者选项 → USB 调试)
• 部分机型需要同时开启「USB 调试(安全设置)」
• 手机已安装 ADB Keyboard 应用(下载地址：https://github.com/senzhk/ADBKeyBoard/blob/master/ADBKeyboard.apk)
• ADB Keyboard 已在系统设置中启用(设置 → 语言和输入法 → 启用 ADB Keyboard)

模型服务确认(二选一)

请明确询问用户：你是否已有可用的 AutoGLM 模型服务？

• 选项 A：使用已部署的模型服务(推荐)

  • 用户提供模型服务的 URL(如 http://xxx.xxx.xxx.xxx:8000/v1)
  • 无需本地 GPU，无需下载模型
  • 直接使用该 URL 作为 --base-url 参数

• 选项 B：本地部署模型(高配置要求)

  • 需要 NVIDIA GPU(建议 24GB+ 显存)
  • 需要安装 vLLM 或 SGLang
  • 需要下载约 20GB 的模型文件
  • 如果用户是新手或不确定，强烈建议选择选项 A

部署流程 Copy Section

阶段一：环境准备

# 1. 安装 ADB 工具
# MacOS:
brew install android-platform-tools
# 或手动下载：https://developer.android.com/tools/releases/platform-tools

# Windows: 下载后解压，添加到 PATH 环境变量

# 2. 验证 ADB 安装
adb version
# 应输出版本信息

# 3. 连接手机并验证
# 用数据线连接手机，手机上点击「允许 USB 调试」
adb devices
# 应输出设备列表，如：
# List of devices attached
# XXXXXXXX    device

如果 adb devices 显示空列表或 unauthorized：

1. 检查手机上是否弹出授权框，点击「允许」
2. 检查 USB 调试是否开启
3. 尝试更换数据线或 USB 接口
4. 执行 adb kill-server && adb start-server 后重试

阶段二：安装 Agent

# 1. 克隆仓库(如果还没有克隆)
git clone https://github.com/zai-org/Open-AutoGLM.git
cd Open-AutoGLM

# 2. 创建虚拟环境(推荐)
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# 3. 安装依赖
pip install -r requirements.txt
pip install -e .

注意：不需要 clone 模型仓库，模型通过 API 调用。

阶段三：配置模型服务

如果用户选择选项 A(使用已部署的模型)：

你可以使用以下第三方模型服务：

1. 智谱 BigModel

   • 文档：https://docs.bigmodel.cn/cn/api/introduction
   • --base-url：https://open.bigmodel.cn/api/paas/v4
   • --model：autoglm-phone
   • --apikey：在智谱平台申请你的 API Key

2. ModelScope(魔搭社区)

   • 文档：https://modelscope.cn/models/ZhipuAI/AutoGLM-Phone-9B
   • --base-url：https://api-inference.modelscope.cn/v1
   • --model：ZhipuAI/AutoGLM-Phone-9B
   • --apikey：在 ModelScope 平台申请你的 API Key

使用示例：

# 使用智谱 BigModel
python main.py --base-url https://open.bigmodel.cn/api/paas/v4 --model "autoglm-phone" --apikey "your-bigmodel-api-key" "打开美团搜索附近的火锅店"

# 使用 ModelScope
python main.py --base-url https://api-inference.modelscope.cn/v1 --model "ZhipuAI/AutoGLM-Phone-9B" --apikey "your-modelscope-api-key" "打开美团搜索附近的火锅店"

或者直接使用用户提供的其他模型服务 URL，跳过本地模型部署步骤。

如果用户选择选项 B(本地部署模型)：

# 1. 安装 vLLM
pip install vllm

# 2. 启动模型服务(会自动下载模型，约 20GB)
python3 -m vllm.entrypoints.openai.api_server \
  --served-model-name autoglm-phone-9b \
  --allowed-local-media-path / \
  --mm-encoder-tp-mode data \
  --mm_processor_cache_type shm \
  --mm_processor_kwargs "{\"max_pixels\":5000000}" \
  --max-model-len 25480 \
  --chat-template-content-format string \
  --limit-mm-per-prompt "{\"image\":10}" \
  --model zai-org/AutoGLM-Phone-9B \
  --port 8000

# 模型服务 URL 为：http://localhost:8000/v1

阶段四：验证部署

# 在 Open-AutoGLM 目录下执行
# 将 {MODEL_URL} 替换为实际的模型服务地址

python main.py --base-url {MODEL_URL} --model "autoglm-phone-9b" "打开微信，对文件传输助手发送消息：部署成功"

预期结果：

• 手机自动打开微信
• 自动搜索「文件传输助手」
• 自动发送消息「部署成功」

异常处理 Copy Section

部署要点 Copy Section

1. 优先确认手机连接：在安装任何代码之前，先确保 adb devices 能看到设备
2. 不要跳过 ADB Keyboard：没有它，中文输入会失败
3. 模型服务是外部依赖：Agent 代码本身不包含模型，需要单独的模型服务
4. 遇到权限问题先检查手机设置：大部分问题都是手机端配置不完整
5. 部署完成后用简单任务测试：建议用「打开微信发消息给文件传输助手」作为验收标准

命令速查 Copy Section

# 检查 ADB 连接
adb devices

# 重启 ADB 服务
adb kill-server && adb start-server

# 安装依赖
pip install -r requirements.txt && pip install -e .

# 运行 Agent(交互模式)
python main.py --base-url {MODEL_URL} --model "autoglm-phone-9b"

# 运行 Agent(单次任务)
python main.py --base-url {MODEL_URL} --model "autoglm-phone-9b" "你的任务描述"

# 查看支持的应用列表
python main.py --list-apps

部署完成的标志：手机能自动执行用户的自然语言指令。

File: docs/ios_setup/ios_setup.md Copy Chapter

iOS 环境配置指南

本文档介绍如何为 Open-AutoGLM 配置 iOS 设备环境。

环境要求 Copy Chapter

• macOS 操作系统
• Xcode（最新版本，在App store中下载）
• 苹果开发者账号（免费账号即可，无需付费）
• iOS 设备（iPhone/iPad）
• USB 数据线或同一 WiFi 网络

WebDriverAgent 配置 Copy Chapter

WebDriverAgent 是 iOS 自动化的核心组件，需要在 iOS 设备上运行。

1. 克隆 WebDriverAgent Copy Section

git clone https://github.com/appium/WebDriverAgent.git
cd WebDriverAgent

直接点击WebDriverAgent.xcodeproj即可使用Xcode打开。

2. 设置 Signing & Capabilities Copy Section

1. 在 Xcode 中选中 WebDriverAgent，出现General、Signing&Capabilities等选项。

2. 进入 Signing & Capabilities 选项卡

3. 勾选 Automatically manage signing。在Team中选择自己的开发者账号

4. 将 Bundle ID 改为唯一标识符，例如：com.yourname.WebDriverAgentRunner

5. TARGETS中，建议将WebDriverAgentLib、WebDriverAgentRunner、IntegrationApp的Signing & Capabilities 都按照相同方式设置。

3. 测试XCode的GUI模式和UI自动化设置 Copy Section

建议先测试GUI模式能否成功安装WebDriverAgent，再进行后续步骤。
Mac和iPhone有USB和WiFi两种连接方式，建议通过USB方式，成功率更高。

通过 WiFi 连接

需要满足以下条件：

1. 通过USB连接。在Finder中选中连接的IPhone，在“通用”中勾选"在 WiFi 中显示这台 iPhone"
2. Mac 与 iPhone 处于同一 WiFi 网络之下

具体步骤

1. 从项目 Target 选择 WebDriverAgentRunner

2. 选择你的设备

3. 长按"▶️"运行按钮，选择 "Test" 后开始编译并部署到你的 iPhone 上

部署成功的标志：1. XCode没有报错。2. 你可以在iPhone上找到名为WebDriverAgentRunner的App

设备信任配置

首次运行时，需要在 iPhone 上完成以下设置，然后重新编译和部署：

1. 输入解锁密码

2. 信任开发者应用

   • 进入：设置 → 通用 → VPN与设备管理
   • 在“开发者 App”中选择对应开发者
   • 点击信任“XXX”

3. 启用 UI 自动化

   • 进入：设置 → 开发者
   • 打开 UI 自动化设置

4. XCode命令行模式部署 Copy Section

1.安装libimobiledevice，用于与 iPhone / iPad 建立连接与通信。

brew install libimobiledevice
# 设备检查
idevice_id -ln

2.使用xcodebuild安装WebAgent。命令行也需要进行“设备信任配置”，参考GUI模式下的方法。

cd WebDriverAgent

xcodebuild -project WebDriverAgent.xcodeproj \
           -scheme WebDriverAgentRunner \
           -destination 'platform=iOS,name=YOUR_PHONE_NAME' \
           test

这里，YOUR_PHONE_NAME可以在xcode的GUI中看到。
WebDriverAgent 成功运行后，会在 Xcode 控制台输出类似以下信息：

ServerURLHere->http://[设备IP]:8100<-ServerURLHere

同时，观察到手机上安装好了WebDriverAgentRunner，屏幕显示Automation Running字样。
其中，http://[设备IP]:8100为WiFi所需的WDA_URL。

使用 AutoGLM Copy Chapter

以上配置完成后，先打开一个新终端，在后台建立端口映射（使用WiFi连接则不需要）：

iproxy 8100 8100

之后，打开一个新终端，通过以下命令使用AutoGLM（WiFi则使用上述获得的WDA_URL）：

python ios.py --base-url "YOUR_BASE_URL" \
    --model  "autoglm-phone" \
    --api-key "YOUR_API_KEY" \
    --wda-url http://localhost:8100 \
    "TASK"

参考资源 Copy Chapter

• WebDriverAgent 官方仓库
• PR141
• Gekowa提供的ios方案

如有其他问题，请参考主项目 README 或提交 Issue。

File: README_coding_agent.md Copy Chapter

Open-AutoGLM Quick Start for Coding Agent

本文专为 AI 助手（如 Claude Code）阅读，用于自动化部署 Open-AutoGLM。

This document is designed for AI assistants (such as Claude Code) to automate the deployment of Open-AutoGLM.

如果你是人类读者，可以跳过本文，按照 README.md 文档操作即可。

If you are a human reader, you can skip this document and follow the README.md instructions instead.

Table of Contents / 目录 Copy Chapter

• English
• 中文

English

Prerequisites Copy Chapter

1. Python Environment Copy Section

Python 3.10 or higher is required.

2. ADB (Android Debug Bridge) Copy Section

1. Download the official ADB installation package
2. Extract and configure environment variables:

macOS:

# Assuming extracted to ~/Downloads/platform-tools
export PATH=${PATH}:~/Downloads/platform-tools

Windows: Add the extracted folder path to your system PATH. Refer to this tutorial if needed.

3. Android Device Setup Copy Section

Requirements:

• Android 7.0+ device or emulator
• Developer Mode enabled
• USB Debugging enabled

Enable Developer Mode:

1. Go to Settings > About Phone > Build Number
2. Tap rapidly about 10 times until "Developer mode enabled" appears

Enable USB Debugging:

1. Go to Settings > Developer Options > USB Debugging
2. Enable the toggle
3. Some devices may require a restart

Important permissions to check:

4. Install ADB Keyboard Copy Section

Download and install ADB Keyboard APK on your device.

After installation, enable it in Settings > Input Method or Settings > Keyboard List.

Installation Copy Chapter

# Install dependencies
pip install -r requirements.txt

# Install package
pip install -e .

ADB Configuration Copy Chapter

Ensure your USB cable supports data transfer (not charging only).

Verify Connection Copy Section

# Check connected devices
adb devices

# Expected output:
# List of devices attached
# emulator-5554   device

Remote Debugging (WiFi) Copy Section

Ensure your phone and computer are on the same WiFi network.

# Connect via WiFi (replace with your phone's IP and port)
adb connect 192.168.1.100:5555

# Verify connection
adb devices

Device Management Copy Section

# List all devices
adb devices

# Connect remote device
adb connect :

# Disconnect device
adb disconnect :

Usage Copy Chapter

Command Line Copy Section

# Interactive mode
python main.py --base-url <MODEL_API_URL> --model <MODEL_NAME>

# Execute specific task
python main.py --base-url <MODEL_API_URL> "Open Chrome browser"

# Use API key authentication
python main.py --apikey sk-xxxxx

# English system prompt
python main.py --lang en --base-url <MODEL_API_URL> "Open Chrome browser"

# List supported apps
python main.py --list-apps

# Specify device
python main.py --device-id 192.168.1.100:5555 --base-url <MODEL_API_URL> "Open TikTok"

Python API Copy Section

from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

# Configure model
model_config = ModelConfig(
    base_url="<MODEL_API_URL>",
    model_name="<MODEL_NAME>",
)

# Create Agent
agent = PhoneAgent(model_config=model_config)

# Execute task
result = agent.run("Open eBay and search for wireless earbuds")
print(result)

Environment Variables Copy Chapter

Troubleshooting Copy Chapter

Device Not Found Copy Section

adb kill-server
adb start-server
adb devices

Check:

1. USB debugging enabled
2. USB cable supports data transfer
3. Authorization popup approved on phone
4. Try different USB port/cable

Can Open Apps but Cannot Tap Copy Section

Enable both in Settings > Developer Options:

• USB Debugging
• USB Debugging (Security Settings)

Text Input Not Working Copy Section

1. Ensure ADB Keyboard is installed
2. Enable in Settings > System > Language & Input > Virtual Keyboard

Windows Encoding Issues Copy Section

Add environment variable before running:

PYTHONIOENCODING=utf-8 python main.py ...

中文

环境要求 Copy Chapter

1. Python 环境 Copy Section

需要 Python 3.10 及以上版本。

2. ADB (Android Debug Bridge) Copy Section

1. 下载官方 ADB 安装包
2. 解压并配置环境变量：

macOS:

# 假设解压到 ~/Downloads/platform-tools
export PATH=${PATH}:~/Downloads/platform-tools

Windows: 将解压后的文件夹路径添加到系统 PATH。可参考此教程。

3. 安卓设备配置 Copy Section

要求：

• Android 7.0+ 设备或模拟器
• 开发者模式已启用
• USB 调试已启用

启用开发者模式：

1. 进入 设置 > 关于手机 > 版本号
2. 连续快速点击约 10 次，直到提示"开发者模式已启用"

启用 USB 调试：

1. 进入 设置 > 开发者选项 > USB 调试
2. 开启开关
3. 部分设备可能需要重启

请务必检查以下权限：

4. 安装 ADB Keyboard Copy Section

在设备上下载并安装 ADB Keyboard APK。

安装后，在 设置 > 输入法 或 设置 > 键盘列表 中启用。

安装 Copy Chapter

# 安装依赖
pip install -r requirements.txt

# 安装包
pip install -e .

ADB 配置 Copy Chapter

请确保 USB 数据线支持数据传输（而非仅充电）。

验证连接 Copy Section

# 检查已连接设备
adb devices

# 预期输出：
# List of devices attached
# emulator-5554   device

远程调试（WiFi） Copy Section

确保手机和电脑在同一 WiFi 网络中。

# 通过 WiFi 连接（替换为手机显示的 IP 和端口）
adb connect 192.168.1.100:5555

# 验证连接
adb devices

设备管理 Copy Section

# 列出所有设备
adb devices

# 连接远程设备
adb connect :

# 断开设备
adb disconnect :

使用方法 Copy Chapter

命令行 Copy Section

# 交互模式
python main.py --base-url <模型API地址> --model <模型名称>

# 执行指定任务
python main.py --base-url <模型API地址> "打开美团搜索附近的火锅店"

# 使用 API Key 认证
python main.py --apikey sk-xxxxx

# 使用英文系统提示词
python main.py --lang en --base-url <模型API地址> "Open Chrome browser"

# 列出支持的应用
python main.py --list-apps

# 指定设备
python main.py --device-id 192.168.1.100:5555 --base-url <模型API地址> "打开抖音刷视频"

Python API Copy Section

from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

# 配置模型
model_config = ModelConfig(
    base_url="<模型API地址>",
    model_name="<模型名称>",
)

# 创建 Agent
agent = PhoneAgent(model_config=model_config)

# 执行任务
result = agent.run("打开淘宝搜索无线耳机")
print(result)

环境变量 Copy Chapter

常见问题 Copy Chapter

设备未找到 Copy Section

adb kill-server
adb start-server
adb devices

检查：

1. USB 调试是否已开启
2. 数据线是否支持数据传输
3. 手机上的授权弹窗是否已点击「允许」
4. 尝试更换 USB 接口或数据线

能打开应用但无法点击 Copy Section

在 设置 > 开发者选项 中同时启用：

• USB 调试
• USB 调试（安全设置）

文本输入不工作 Copy Section

1. 确保已安装 ADB Keyboard
2. 在 设置 > 系统 > 语言和输入法 > 虚拟键盘 中启用

Windows 编码异常 Copy Section

运行代码前添加环境变量：

PYTHONIOENCODING=utf-8 python main.py ...

License Copy Chapter

This project is for research and learning purposes only. See Terms of Use / 使用条款.

File: README_en.md Copy Chapter

Open-AutoGLM

中文阅读.

Quick Start Copy Chapter

You can use Claude Code with GLM Coding Plan and enter the following prompt to quickly deploy this project:

Access the documentation and install AutoGLM for me
https://raw.githubusercontent.com/zai-org/Open-AutoGLM/refs/heads/main/README_en.md

Project Introduction Copy Chapter

Phone Agent is a mobile intelligent assistant framework built on AutoGLM. It understands phone screen content in a multimodal manner and helps users complete tasks through automated operations. The system controls devices via ADB (Android Debug Bridge), perceives screens using vision-language models, and generates and executes operation workflows through intelligent planning. Users simply describe their needs in natural language, such as "Open eBay and search for wireless earphones." and Phone Agent will automatically parse the intent, understand the current interface, plan the next action, and complete the entire workflow. The system also includes a sensitive operation confirmation mechanism and supports manual takeover during login or verification code scenarios. Additionally, it provides remote ADB debugging capabilities, allowing device connection via WiFi or network for flexible remote control and development.

⚠️ This project is for research and learning purposes only. It is strictly prohibited to use for illegal information acquisition, system interference, or any illegal activities. Please carefully review the Terms of Use.

Integration with Other Automation Tools Copy Chapter

Midscene.js Copy Section

Midscene.js is an open-source, vision-model-driven UI automation SDK that supports JavaScript or YAML flow syntax for cross-platform automation.

Midscene.js already supports AutoGLM; see the Midscene.js integration guide to quickly try AutoGLM automation on both iOS and Android devices.

Model Download Links Copy Chapter

AutoGLM-Phone-9B is optimized for Chinese mobile applications, while AutoGLM-Phone-9B-Multilingual supports English scenarios and is suitable for applications containing English or other language content.

Environment Setup Copy Chapter

1. Python Environment Copy Section

Python 3.10 or higher is recommended.

2. Device Debug Tools Copy Section

Choose the appropriate tool based on your device type:

For Android Devices - Using ADB

1. Download the official ADB installation package and extract it to a custom path
2. Configure environment variables

• MacOS configuration: In Terminal or any command line tool

  bash Copy

  # Assuming the extracted directory is ~/Downloads/platform-tools. Adjust the command if different.
  export PATH=${PATH}:~/Downloads/platform-tools

• Windows configuration: Refer to third-party tutorials for configuration.

For HarmonyOS Devices - Using HDC

1. Download HDC tool:
   • From HarmonyOS SDK
2. Configure environment variables

• MacOS/Linux configuration:

  bash Copy

  # Assuming the extracted directory is ~/Downloads/harmonyos-sdk/toolchains. Adjust according to actual path.
  export PATH=${PATH}:~/Downloads/harmonyos-sdk/toolchains

• Windows configuration: Add the HDC tool directory to the system PATH environment variable

3. Android 7.0+ or HarmonyOS Device with `Developer Mode` and `USB Debugging` Enabled Copy Section

1. Enable Developer Mode: The typical method is to find Settings > About Phone > Build Number and tap it rapidly about 10 times until a popup shows "Developer mode has been enabled." This may vary slightly between phones; search online for tutorials if you can't find it.
2. Enable USB Debugging: After enabling Developer Mode, go to Settings > Developer Options > USB Debugging and enable it
3. Some devices may require a restart after setting developer options for them to take effect. You can test by connecting your phone to your computer via USB cable and running adb devices to see if device information appears. If not, the connection has failed.

Please carefully check the relevant permissions

4. Install ADB Keyboard (Required for Android Devices Only, for Text Input) Copy Section

Note: HarmonyOS devices use native input methods and do not require ADB Keyboard.

If you are using an Android device:

Download the installation package and install it on the corresponding Android device.
Note: After installation, you need to enable ADB Keyboard in Settings > Input Method or Settings > Keyboard List for it to work.(or use command adb shell ime enable com.android.adbkeyboard/.AdbIMEHow-to-use)

Deployment Preparation Copy Chapter

1. Install Dependencies Copy Section

pip install -r requirements.txt 
pip install -e .

2. Configure ADB or HDC Copy Section

For Android Devices

Make sure your USB cable supports data transfer, not just charging.

Ensure ADB is installed and connect the device via USB cable:

# Check connected devices
adb devices

# Output should show your device, e.g.:
# List of devices attached
# emulator-5554   device

For HarmonyOS Devices

Make sure your USB cable supports data transfer, not just charging.

Ensure HDC is installed and connect the device via USB cable:

# Check connected devices
hdc list targets

# Output should show your device, e.g.:
# 7001005458323933328a01bce01c2500

3. Start Model Service Copy Section

You can choose to deploy the model service yourself or use a third-party model service provider.

Option A: Use Third-Party Model Services

If you don't want to deploy the model yourself, you can use the following third-party services that have already deployed our model:

1. z.ai

• Documentation: https://docs.z.ai/api-reference/introduction
• --base-url: https://api.z.ai/api/paas/v4
• --model: autoglm-phone-multilingual
• --apikey: Apply for your own API key on the z.ai platform

2. Novita AI

• Documentation: https://novita.ai/models/model-detail/zai-org-autoglm-phone-9b-multilingual
• --base-url: https://api.novita.ai/openai
• --model: zai-org/autoglm-phone-9b-multilingual
• --apikey: Apply for your own API key on the Novita AI platform

3. Parasail

• Documentation: https://www.saas.parasail.io/serverless?name=auto-glm-9b-multilingual
• --base-url: https://api.parasail.io/v1
• --model: parasail-auto-glm-9b-multilingual
• --apikey: Apply for your own API key on the Parasail platform

Example usage with third-party services:

# Using z.ai
python main.py --base-url https://api.z.ai/api/paas/v4 --model "autoglm-phone-multilingual" --apikey "your-z-ai-api-key" "Open Chrome browser"

# Using Novita AI
python main.py --base-url https://api.novita.ai/openai --model "zai-org/autoglm-phone-9b-multilingual" --apikey "your-novita-api-key" "Open Chrome browser"

# Using Parasail
python main.py --base-url https://api.parasail.io/v1 --model "parasail-auto-glm-9b-multilingual" --apikey "your-parasail-api-key" "Open Chrome browser"

Option B: Deploy Model Yourself

If you prefer to deploy the model locally or on your own server:

1. Download the model and install the inference engine framework according to the For Model Deployment section in requirements.txt.
2. Start via SGlang / vLLM to get an OpenAI-format service. Here's a vLLM deployment solution; please strictly follow the startup parameters we provide:

• vLLM:

python3 -m vllm.entrypoints.openai.api_server \
 --served-model-name autoglm-phone-9b-multilingual \
 --allowed-local-media-path /   \
 --mm-encoder-tp-mode data \
 --mm_processor_cache_type shm \
 --mm_processor_kwargs "{\"max_pixels\":5000000}" \
 --max-model-len 25480  \
 --chat-template-content-format string \
 --limit-mm-per-prompt "{\"image\":10}" \
 --model zai-org/AutoGLM-Phone-9B-Multilingual \
 --port 8000

• This model has the same architecture as GLM-4.1V-9B-Thinking. For detailed information about model deployment, you can also check GLM-V for model deployment and usage guides.

• After successful startup, the model service will be accessible at http://localhost:8000/v1. If you deploy the model on a remote server, access it using that server's IP address.

4. Check Model Deployment Copy Section

After starting the model service, you can use the following command to verify the deployment:

python scripts/check_deployment_en.py --base-url http://localhost:8000/v1 --model autoglm-phone-9b-multilingual

If using a third-party model service:

# Novita AI
python scripts/check_deployment_en.py --base-url https://api.novita.ai/openai --model zai-org/autoglm-phone-9b-multilingual --apikey your-novita-api-key

# Parasail
python scripts/check_deployment_en.py --base-url https://api.parasail.io/v1 --model parasail-auto-glm-9b-multilingual --apikey your-parasail-api-key

Upon successful execution, the script will display the model's inference result and token statistics, helping you confirm whether the model deployment is working correctly.

Using AutoGLM Copy Chapter

Command Line Copy Section

Set the --base-url and --model parameters according to your deployed model. For example:

# Android device - Interactive mode
python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual"

# Android device - Specify task
python main.py --base-url http://localhost:8000/v1 "Open Maps and search for nearby coffee shops"

# HarmonyOS device - Interactive mode
python main.py --device-type hdc --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual"

# HarmonyOS device - Specify task
python main.py --device-type hdc --base-url http://localhost:8000/v1 "Open Maps and search for nearby coffee shops"

# Use API key for authentication
python main.py --apikey sk-xxxxx

# Use English system prompt
python main.py --lang en --base-url http://localhost:8000/v1 "Open Chrome browser"

# List supported apps (Android)
python main.py --list-apps

# List supported apps (HarmonyOS)
python main.py --device-type hdc --list-apps

Python API Copy Section

from phone_agent import PhoneAgent
from phone_agent.model import ModelConfig

# Configure model
model_config = ModelConfig(
    base_url="http://localhost:8000/v1",
    model_name="autoglm-phone-9b-multilingual",
)

# Create Agent
agent = PhoneAgent(model_config=model_config)

# Execute task
result = agent.run("Open eBay and search for wireless earphones")
print(result)

Remote Debugging Copy Chapter

Phone Agent supports remote ADB/HDC debugging via WiFi/network, allowing device control without a USB connection.

Configure Remote Debugging Copy Section

Enable Wireless Debugging on Phone

Android Devices

Ensure the phone and computer are on the same WiFi network, as shown below:

HarmonyOS Devices

Ensure the phone and computer are on the same WiFi network:

1. Go to Settings > System & Updates > Developer Options
2. Enable USB Debugging and Wireless Debugging
3. Note the displayed IP address and port number

Use Standard ADB/HDC Commands on Computer

# Android device - Connect via WiFi, replace with the IP address and port shown on your phone
adb connect 192.168.1.100:5555

# Verify connection
adb devices
# Should show: 192.168.1.100:5555    device

# HarmonyOS device - Connect via WiFi
hdc tconn 192.168.1.100:5555

# Verify connection
hdc list targets
# Should show: 192.168.1.100:5555

Device Management Commands Copy Section

Android Devices (ADB)

# List all connected devices
adb devices

# Connect to remote device
adb connect 192.168.1.100:5555

# Disconnect specific device
adb disconnect 192.168.1.100:5555

# Execute task on specific device
python main.py --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual" "Open TikTok and browse videos"

HarmonyOS Devices (HDC)

# List all connected devices
hdc list targets

# Connect to remote device
hdc tconn 192.168.1.100:5555

# Disconnect specific device
hdc tdisconn 192.168.1.100:5555

# Execute task on specific device
python main.py --device-type hdc --device-id 192.168.1.100:5555 --base-url http://localhost:8000/v1 --model "autoglm-phone-9b-multilingual" "Open TikTok and browse videos"

Python API Remote Connection Copy Section

Android Devices (ADB)

from phone_agent.adb import ADBConnection, list_devices

# Create connection manager
conn = ADBConnection()

# Connect to remote device
success, message = conn.connect("192.168.1.100:5555")
print(f"Connection status: {message}")

# List connected devices
devices = list_devices()
for device in devices:
    print(f"{device.device_id} - {device.connection_type.value}")

# Enable TCP/IP on USB device
success, message = conn.enable_tcpip(5555)
ip = conn.get_device_ip()
print(f"Device IP: {ip}")

# Disconnect
conn.disconnect("192.168.1.100:5555")

HarmonyOS Devices (HDC)

from phone_agent.hdc import HDCConnection, list_devices

# Create connection manager
conn = HDCConnection()

# Connect to remote device
success, message = conn.connect("192.168.1.100:5555")
print(f"Connection status: {message}")

# List connected devices
devices = list_devices()
for device in devices:
    print(f"{device.device_id} - {device.connection_type.value}")

# Disconnect
conn.disconnect("192.168.1.100:5555")

Remote Connection Troubleshooting Copy Section

Connection Refused:

• Ensure the device and computer are on the same network
• Check if the firewall is blocking port 5555
• Confirm TCP/IP mode is enabled: adb tcpip 5555

Connection Dropped:

• WiFi may have disconnected; use --connect to reconnect
• Some devices disable TCP/IP after restart; re-enable via USB

Multiple Devices:

• Use --device-id to specify which device to use
• Or use --list-devices to view all connected devices

Configuration Copy Chapter

Custom SYSTEM PROMPT Copy Section

The system provides both Chinese and English prompts, switchable via the --lang parameter:

• --lang cn - Chinese prompt (default), config file: phone_agent/config/prompts_zh.py
• --lang en - English prompt, config file: phone_agent/config/prompts_en.py

You can directly modify the corresponding config files to enhance model capabilities in specific domains or disable certain apps by injecting app names.

Environment Variables Copy Section

Model Configuration Copy Section

from phone_agent.model import ModelConfig

config = ModelConfig(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",  # API key (if required)
    model_name="autoglm-phone-9b-multilingual",  # Model name
    max_tokens=3000,  # Maximum output tokens
    temperature=0.1,  # Sampling temperature
    frequency_penalty=0.2,  # Frequency penalty
)

Agent Configuration Copy Section

from phone_agent.agent import AgentConfig

config = AgentConfig(
    max_steps=100,  # Maximum steps per task
    device_id=None,  # ADB device ID (None for auto-detect)
    lang="en",  # Language: cn (Chinese) or en (English)
    verbose=True,  # Print debug info (including thinking process and actions)
)

Verbose Mode Output Copy Section

When verbose=True, the Agent outputs detailed information at each step:

==================================================
💭 Thinking Process:
--------------------------------------------------
Currently on the system desktop, need to launch eBay app first
--------------------------------------------------
🎯 Executing Action:
{
  "_metadata": "do",
  "action": "Launch",
  "app": "eBay"
}
==================================================

... (continues to next step after executing action)

==================================================
💭 Thinking Process:
--------------------------------------------------
eBay is now open, need to tap the search box
--------------------------------------------------
🎯 Executing Action:
{
  "_metadata": "do",
  "action": "Tap",
  "element": [499, 182]
}
==================================================

🎉 ================================================
✅ Task Completed: Successfully opened eBay and searched for 'wireless earphones'
==================================================

This allows you to clearly see the AI's reasoning process and specific operations at each step.

Supported Apps Copy Chapter

Android Apps Copy Section

Phone Agent supports 50+ mainstream Chinese applications:

Run python main.py --list-apps to see the complete list.

HarmonyOS Apps Copy Section

Phone Agent supports 60+ HarmonyOS native apps and system apps:

Run python main.py --device-type hdc --list-apps to see the complete list.

Available Actions Copy Chapter

The Agent can perform the following actions:

Custom Callbacks Copy Chapter

Handle sensitive operation confirmation and manual takeover:

def my_confirmation(message: str) -> bool:
    """Sensitive operation confirmation callback"""
    return input(f"Confirm execution of {message}? (y/n): ").lower() == "y"


def my_takeover(message: str) -> None:
    """Manual takeover callback"""
    print(f"Please complete manually: {message}")
    input("Press Enter after completion...")


agent = PhoneAgent(
    confirmation_callback=my_confirmation,
    takeover_callback=my_takeover,
)

Examples Copy Chapter

Check the examples/ directory for more usage examples:

• basic_usage.py - Basic task execution
• Single-step debugging mode
• Batch task execution
• Custom callbacks

Development Copy Chapter

Set Up Development Environment Copy Section

Development requires dev dependencies:

pip install -e ".[dev]"

Run Tests Copy Section

pytest tests/

Complete Project Structure Copy Section

phone_agent/
├── __init__.py          # Package exports
├── agent.py             # PhoneAgent main class
├── adb/                 # ADB utilities
│   ├── connection.py    # Remote/local connection management
│   ├── screenshot.py    # Screen capture
│   ├── input.py         # Text input (ADB Keyboard)
│   └── device.py        # Device control (tap, swipe, etc.)
├── actions/             # Action handling
│   └── handler.py       # Action executor
├── config/              # Configuration
│   ├── apps.py          # Supported app mappings
│   ├── prompts_zh.py    # Chinese system prompts
│   └── prompts_en.py    # English system prompts
└── model/               # AI model client
    └── client.py        # OpenAI-compatible client

FAQ Copy Chapter

Here are some common issues and their solutions:

Device Not Found Copy Section

Try resolving by restarting the ADB service:

adb kill-server
adb start-server
adb devices

If the device is still not recognized, please check:

1. Whether USB debugging is enabled
2. Whether the USB cable supports data transfer (some cables only support charging)
3. Whether you have tapped "Allow" on the authorization popup on your phone
4. Try a different USB port or cable

Can Open Apps but Cannot Tap Copy Section

Some devices require both debugging options to be enabled:

• USB Debugging
• USB Debugging (Security Settings)

Please check in Settings → Developer Options that both options are enabled.

Text Input Not Working Copy Section

1. Ensure ADB Keyboard is installed on the device
2. Enable it in Settings > System > Language & Input > Virtual Keyboard
3. The Agent will automatically switch to ADB Keyboard when input is needed

Screenshot Failed (Black Screen) Copy Section

This usually means the app is displaying a sensitive page (payment, password, banking apps). The Agent will automatically detect this and request manual takeover.

Windows Encoding Issues Copy Section

Error message like UnicodeEncodeError gbk code

Solution: Add the environment variable before running the code: PYTHONIOENCODING=utf-8

Interactive Mode Not Working in Non-TTY Environment Copy Section

Error like: EOF when reading a line

Solution: Use non-interactive mode to specify tasks directly, or switch to a TTY-mode terminal application.

Citation Copy Section

If you find our work helpful, please cite the following papers:

@article{liu2024autoglm,
  title={Autoglm: Autonomous foundation agents for guis},
  author={Liu, Xiao and Qin, Bo and Liang, Dongzhu and Dong, Guang and Lai, Hanyu and Zhang, Hanchen and Zhao, Hanlin and Iong, Iat Long and Sun, Jiadai and Wang, Jiaqi and others},
  journal={arXiv preprint arXiv:2411.00820},
  year={2024}
}
@article{xu2025mobilerl,
  title={MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents},
  author={Xu, Yifan and Liu, Xiao and Liu, Xinghan and Fu, Jiaqi and Zhang, Hanchen and Jing, Bohao and Zhang, Shudan and Wang, Yuting and Zhao, Wenyi and Dong, Yuxiao},
  journal={arXiv preprint arXiv:2509.18119},
  year={2025}
}

Automated Deployment Guide (For AI Assistants) Copy Chapter

This section is specifically designed for AI assistants (such as Claude Code) to automate the deployment of Open-AutoGLM.

If you are a human reader, you can skip this section and follow the documentation above.

Project Overview Copy Section

Open-AutoGLM is a phone agent framework:

• Input: User's natural language instructions (e.g., "Open WhatsApp and send a message to John")
• Output: Automatically operates the user's Android phone to complete tasks
• Mechanism: Screenshot → Vision model understands interface → Outputs tap coordinates → ADB executes actions → Loop

The architecture consists of two parts:

1. Agent Code (this repository): Runs on the user's computer, responsible for calling models, parsing actions, and controlling the phone
2. Vision Model Service: Can be a remote API or deployed locally

Pre-Deployment Checklist Copy Section

Before starting deployment, confirm the following items with the user:

Hardware Requirements

• User has an Android phone (Android 7.0+)
• User has a USB cable that supports data transfer (not just charging)
• Phone and computer can be connected via USB cable

Phone Configuration

• Phone has Developer Mode enabled (Settings → About Phone → Tap Build Number 7 times)
• Phone has USB Debugging enabled (Settings → Developer Options → USB Debugging)
• Some models require enabling "USB Debugging (Security Settings)" as well
• ADB Keyboard app is installed (Download: https://github.com/senzhk/ADBKeyBoard/blob/master/ADBKeyboard.apk)
• ADB Keyboard is enabled in system settings (Settings → Language & Input → Enable ADB Keyboard)

Model Service Confirmation (Choose One)

Ask the user explicitly: Do you already have access to an AutoGLM model service?

• Option A: Use an already-deployed model service (Recommended)

  • User provides the model service URL (e.g., http://xxx.xxx.xxx.xxx:8000/v1)
  • No local GPU required, no model download needed
  • Use this URL directly as the --base-url parameter

• Option B: Deploy model locally (High system requirements)

  • Requires NVIDIA GPU (24GB+ VRAM recommended)
  • Requires installation of vLLM or SGLang
  • Requires downloading approximately 20GB of model files
  • If the user is a beginner or unsure, strongly recommend Option A

Deployment Process Copy Section

Phase 1: Environment Setup

# 1. Install ADB tools
# MacOS:
brew install android-platform-tools
# Or download manually: https://developer.android.com/tools/releases/platform-tools

# Windows: Download, extract, and add to PATH environment variable

# 2. Verify ADB installation
adb version
# Should output version information

# 3. Connect phone and verify
# Connect phone via USB cable, tap "Allow USB debugging" on phone
adb devices
# Should output device list, e.g.:
# List of devices attached
# XXXXXXXX    device

If adb devices shows empty list or unauthorized:

1. Check if authorization popup appeared on phone, tap "Allow"
2. Check if USB debugging is enabled
3. Try a different cable or USB port
4. Run adb kill-server && adb start-server and retry

Phase 2: Install Agent

# 1. Clone repository (if not already cloned)
git clone https://github.com/zai-org/Open-AutoGLM.git
cd Open-AutoGLM

# 2. Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt
pip install -e .

Note: No need to clone model repository; models are called via API.

Phase 3: Configure Model Service

If user chooses Option A (using already-deployed model):

You can use the following third-party model services:

1. z.ai

   • Documentation: https://docs.z.ai/api-reference/introduction
   • --base-url: https://api.z.ai/api/paas/v4
   • --model: autoglm-phone-multilingual
   • --apikey: Apply for your own API key on the z.ai platform

2. Novita AI

   • Documentation: https://novita.ai/models/model-detail/zai-org-autoglm-phone-9b-multilingual
   • --base-url: https://api.novita.ai/openai
   • --model: zai-org/autoglm-phone-9b-multilingual
   • --apikey: Apply for your own API key on the Novita AI platform

3. Parasail

   • Documentation: https://www.saas.parasail.io/serverless?name=auto-glm-9b-multilingual
   • --base-url: https://api.parasail.io/v1
   • --model: parasail-auto-glm-9b-multilingual
   • --apikey: Apply for your own API key on the Parasail platform

Example usage:

# Using z.ai
python main.py --base-url https://api.z.ai/api/paas/v4 --model "autoglm-phone-multilingual" --apikey "your-z-ai-api-key" "Open Chrome browser"

# Using Novita AI
python main.py --base-url https://api.novita.ai/openai --model "zai-org/autoglm-phone-9b-multilingual" --apikey "your-novita-api-key" "Open Chrome browser"

# Using Parasail
python main.py --base-url https://api.parasail.io/v1 --model "parasail-auto-glm-9b-multilingual" --apikey "your-parasail-api-key" "Open Chrome browser"

Or use the URL provided by the user directly and skip local model deployment steps.

If user chooses Option B (deploy model locally):

# 1. Install vLLM
pip install vllm

# 2. Start model service (will auto-download model, ~20GB)
python3 -m vllm.entrypoints.openai.api_server \
  --served-model-name autoglm-phone-9b-multilingual \
  --allowed-local-media-path / \
  --mm-encoder-tp-mode data \
  --mm_processor_cache_type shm \
  --mm_processor_kwargs "{\"max_pixels\":5000000}" \
  --max-model-len 25480 \
  --chat-template-content-format string \
  --limit-mm-per-prompt "{\"image\":10}" \
  --model zai-org/AutoGLM-Phone-9B-Multilingual \
  --port 8000

# Model service URL: http://localhost:8000/v1

Phase 4: Verify Deployment

# Execute in the Open-AutoGLM directory
# Replace {MODEL_URL} with the actual model service address

python main.py --base-url {MODEL_URL} --model "autoglm-phone-9b-multilingual" "Open Gmail and send an email to File Transfer Assistant: Deployment successful"

Expected Result:

• Phone automatically opens Gmail
• Automatically searches for recipient
• Automatically sends the message "Deployment successful"

Troubleshooting Copy Section

Deployment Key Points Copy Section

1. Prioritize confirming phone connection: Before installing any code, ensure adb devices can see the device
2. Don't skip ADB Keyboard: Without it, text input will fail
3. Model service is an external dependency: Agent code doesn't include the model; a separate model service is required
4. Check phone settings first for permission issues: Most problems are due to incomplete phone-side configuration
5. Test with simple tasks after deployment: Recommend using "Open Gmail and send message to File Transfer Assistant" as acceptance criteria

Command Quick Reference Copy Section

# Check ADB connection
adb devices

# Restart ADB service
adb kill-server && adb start-server

# Install dependencies
pip install -r requirements.txt && pip install -e .

# Run Agent (interactive mode)
python main.py --base-url {MODEL_URL} --model "autoglm-phone-9b-multilingual"

# Run Agent (single task)
python main.py --base-url {MODEL_URL} --model "autoglm-phone-9b-multilingual" "your task description"

# View supported apps list
python main.py --list-apps

Deployment success indicator: The phone can automatically execute user's natural language instructions.