{"owner":"kvcache-ai","repo":"Mooncake","hasSkills":true,"totalSkillsCount":12,"totalTokensCount":12189,"categories":["marketplace","root-instruction","anthropic-skill","plugin-manifest","claude-rule"],"hasMcp":false,"mcpConfig":null,"found":[".claude-plugin/marketplace.json","AGENTS.md",".claude/skills/mooncake-api/SKILL.md",".claude/skills/mooncake-troubleshoot/SKILL.md","docs/AGENTS.md",".claude/skills/mooncake-ci-local/SKILL.md","mooncake-store/AGENTS.md",".claude/skills/mooncake-api/.claude-plugin/plugin.json",".claude/skills/mooncake-ci-local/scripts/check-prerequisites.sh",".claude/skills/mooncake-ci-local/.claude-plugin/plugin.json",".claude/skills/mooncake-ci-local/examples/minimal.md",".claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json"],"skills":{".claude-plugin/marketplace.json":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-marketplace.json\",\n  \"name\": \"mooncake\",\n  \"owner\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"metadata\": {\n    \"description\": \"Built-in Claude Code skills for working with Mooncake: deployment troubleshooting, local CI validation, and the Python API.\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"mooncake-troubleshoot\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-troubleshoot\"\n      },\n      \"description\": \"Systematically diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, connectivity, logs) and propose fixes.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-troubleshoot.html\"\n    },\n    {\n      \"name\": \"mooncake-ci-local\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-ci-local\"\n      },\n      \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\"\n    },\n    {\n      \"name\": \"mooncake-api\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-api\"\n      },\n      \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\"\n    }\n  ]\n}\n","AGENTS.md":"# AGENTS.md\n\n## `docs/` Directory Changes\n\n- Before modifying files under `docs/`, read `docs/AGENTS.md`.\n\n## Pull Request Guidelines\n\n- Follow `CONTRIBUTING.md` for PR title prefixes, RFC expectations, and\n  contribution workflow.\n- Before opening a PR for nontrivial work, check whether an existing issue or\n  open PR already covers the same change. If the work overlaps, explain the\n  difference instead of duplicating it.\n- Do not open low-value busywork PRs for isolated typo, style, or mechanical\n  changes unless they are part of a substantive requested change.\n- Use `.github/pull_request_template.md` when preparing a PR, and fill in the\n  relevant sections for description, module, type of change, testing,\n  checklist, and AI assistance disclosure.\n- For AI-assisted changes, make sure the human submitter has reviewed every\n  changed line and can defend the change end-to-end.\n- Before handoff, run pre-commit on the files touched by the change when the\n  toolchain is available (see `CONTRIBUTING.md` for the PR-scoped\n  `pre-commit run --files ...` command). Do not use\n  `pre-commit run --all-files` for routine PRs; if it rewrites unrelated\n  files, leave those edits out of the PR.\n- Keep PRs lean: review `git diff` before staging, and include only changes\n  required for the requested task.\n",".claude/skills/mooncake-api/SKILL.md":"---\nname: mooncake-api\ndescription: Help users work with the Mooncake Python APIs for distributed storage and high-performance data transfer. Use when working with Mooncake Store (distributed KV cache), Transfer Engine (RDMA/TCP transfers), service setup (master, metadata server), PyTorch tensors in the Store, zero-copy/buffer management, batch operations and replication, or Mooncake EP / Backend (Expert Parallelism). Trigger on questions about MooncakeDistributedStore, TransferEngine, put/get, put_tensor, register_buffer, ReplicateConfig, or the mooncake.store / mooncake.engine / mooncake.pg Python modules.\n---\n\n# Mooncake Python API Skill\n\nUse this skill to help users work with Mooncake Python APIs for distributed storage and high-performance data transfer.\n\n## When to Use This Skill\n\nUse this skill when users ask about:\n- Using Mooncake Store for distributed KV cache storage\n- Using Transfer Engine for RDMA/TCP data transfers\n- Setting up Mooncake services (master, metadata server)\n- Working with PyTorch tensors in Mooncake Store\n- Zero-copy operations and buffer management\n- Batch operations and replication configuration\n- Mooncake EP (Expert Parallelism) and Mooncake Backend\n- Troubleshooting Mooncake Python API issues\n\n## Routing Guidance\n\n- For most vLLM and SGLang users, start from `docs/source/getting_started/quick-start.md`.\n- For PD disaggregation, direct users to the SGLang/vLLM integration guides listed in Quick Start. Those guides own the serving-framework configuration.\n- For Mooncake Store integrations, direct users to the SGLang/vLLM Store setup guides listed in Quick Start. Do not duplicate `mooncake_master` startup commands in general API answers unless the user is working outside those frameworks.\n- For direct low-level Transfer Engine usage, use `docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects`.\n- For API signatures and method details, use the Python API references under `docs/source/python-api-reference/`.\n\n## Core Components\n\n### 1. Mooncake Store (Distributed KV Cache)\n\n**Import:**\n```python\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n```\n\n**Basic Setup:**\n```python\nstore = MooncakeDistributedStore()\nstore.setup(\n    \"localhost\",                          # local_hostname\n    \"http://localhost:8080/metadata\",     # metadata_server\n    512*1024*1024,                        # global_segment_size (512MB)\n    128*1024*1024,                        # local_buffer_size (128MB)\n    \"tcp\",                                # protocol (\"tcp\" or \"rdma\")\n    \"\",                                   # rdma_devices (empty for auto-select)\n    \"localhost:50051\"                     # master_server_address\n)\n```\n\n**Common Operations:**\n```python\n# Put/Get\nstore.put(\"key\", b\"value\")\ndata = store.get(\"key\")\n\n# Batch operations\nstore.put_batch([\"key1\", \"key2\"], [b\"val1\", b\"val2\"])\nvalues = store.get_batch([\"key1\", \"key2\"])\n\n# Check existence\nexists = store.is_exist(\"key\")  # Returns 1 (exists), 0 (not exists), -1 (error)\n\n# Remove\nstore.remove(\"key\")\nstore.remove_by_regex(\"^prefix_.*\")\nstore.remove_all()\n\n# Cleanup\nstore.close()\n```\n\n**Zero-Copy Operations (Advanced):**\n```python\nimport numpy as np\n\n# Create and register buffer\nbuffer = np.zeros(100*1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nstore.register_buffer(buffer_ptr, buffer.nbytes)\n\n# Zero-copy put\nstore.put_from(\"key\", buffer_ptr, buffer.nbytes)\n\n# Zero-copy get\nrecv_buffer = np.empty(100*1024*1024, dtype=np.uint8)\nrecv_ptr = recv_buffer.ctypes.data\nstore.register_buffer(recv_ptr, recv_buffer.nbytes)\nbytes_read = store.get_into(\"key\", recv_ptr, recv_buffer.nbytes)\n\n# Cleanup\nstore.unregister_buffer(buffer_ptr)\nstore.unregister_buffer(recv_ptr)\n```\n\n**PyTorch Tensor Operations:**\n```python\nimport torch\n\n# Simple tensor operations\ntensor = torch.randn(100, 100)\nstore.put_tensor(\"my_tensor\", tensor)\nretrieved = store.get_tensor(\"my_tensor\")\n\n# Batch tensor operations\ntensors = [torch.randn(100, 100) for _ in range(3)]\nstore.batch_put_tensor([\"t1\", \"t2\", \"t3\"], tensors)\nretrieved_tensors = store.batch_get_tensor([\"t1\", \"t2\", \"t3\"])\n\n# Tensor Parallelism (TP) support\nstore.put_tensor_with_tp(\"model_weights\", tensor, tp_rank=0, tp_size=4, split_dim=0)\nshard = store.get_tensor_with_tp(\"model_weights\", tp_rank=0, tp_size=4)\n```\n\n**Replication Configuration:**\n```python\nconfig = ReplicateConfig()\nconfig.replica_num = 3              # Number of replicas\nconfig.with_soft_pin = True         # Keep in memory longer\nconfig.preferred_segment = \"host:port\"  # Preferred location\n\nstore.put(\"key\", b\"value\", config)\n```\n\n### 2. Transfer Engine (High-Performance Data Transfer)\n\n**Import:**\n```python\nfrom mooncake.engine import TransferEngine, TransferOpcode, TransferNotify\n```\n\n**Basic Setup:**\n```python\nengine = TransferEngine()\nengine.initialize(\n    \"127.0.0.1:12345\",      # local_hostname\n    \"127.0.0.1:2379\",       # metadata_server (or \"etcd://...\")\n    \"tcp\",                  # protocol (\"tcp\" or \"rdma\")\n    \"\"                      # device_name (empty for all devices)\n)\n```\n\n**Buffer Management:**\n```python\n# Allocate managed buffer\nbuffer_size = 1024 * 1024  # 1MB\nbuffer_addr = engine.allocate_managed_buffer(buffer_size)\n\n# Write/read bytes\ndata = b\"Hello, Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_addr, data, len(data))\nread_data = engine.read_bytes_from_buffer(buffer_addr, len(data))\n\n# Free buffer\nengine.free_managed_buffer(buffer_addr, buffer_size)\n```\n\n**Data Transfer Operations:**\n```python\n# Synchronous write\nresult = engine.transfer_sync_write(\n    \"target_host:port\",     # target_hostname\n    local_buffer_addr,      # buffer\n    remote_buffer_addr,     # peer_buffer_address\n    data_length             # length\n)\n\n# Synchronous read\nresult = engine.transfer_sync_read(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Asynchronous write\nbatch_id = engine.transfer_submit_write(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Check status\nstatus = engine.transfer_check_status(batch_id)\n# Returns: 1 (completed), 0 (in progress), -1 (failed), -2 (timeout)\n```\n\n**Batch Transfer Operations:**\n```python\n# Batch synchronous write\nlocal_addrs = [addr1, addr2, addr3]\nremote_addrs = [remote1, remote2, remote3]\nlengths = [len1, len2, len3]\n\nresult = engine.batch_transfer_sync_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Batch asynchronous operations\nbatch_id = engine.batch_transfer_async_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Wait for completion\nresult = engine.get_batch_transfer_status([batch_id])\n```\n\n**Memory Registration (for RDMA):**\n```python\nimport numpy as np\n\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nbuffer_size = buffer.nbytes\n\n# Register memory\nengine.register_memory(buffer_ptr, buffer_size)\n\n# Use buffer for transfers...\n\n# Unregister when done\nengine.unregister_memory(buffer_ptr)\n```\n\n### 3. Mooncake EP & Backend (Expert Parallelism)\n\n**Mooncake Backend (Fault-Tolerant Collectives):**\n```python\nimport torch\nimport torch.distributed as dist\nfrom mooncake import pg\n\n# Initialize with fault tolerance\nactive_ranks = torch.ones((world_size,), dtype=torch.int32, device=\"cuda\")\ndist.init_process_group(\n    backend=\"mooncake\",\n    rank=rank,\n    world_size=world_size,\n    pg_options=pg.MooncakeBackendOptions(active_ranks),\n)\n\n# Use standard PyTorch distributed APIs\ndist.all_gather(...)\ndist.all_reduce(...)\n\n# Check for failures\nassert active_ranks.all()  # Verify no ranks are broken\n```\n\n**Mooncake EP (Expert Parallelism):**\n```python\nfrom mooncake.mooncake_ep_buffer import Buffer\nimport torch.distributed as dist\n\n# Calculate buffer size\nnum_ep_buffer_bytes = Buffer.get_ep_buffer_size_hint(\n    num_max_dispatch_tokens_per_rank=1024,\n    hidden=4096,\n    num_ranks=8,\n    num_experts=64\n)\n\n# Create buffer (must be Mooncake Backend process group)\nbuffer = Buffer(group=dist.group.WORLD, num_ep_buffer_bytes=num_ep_buffer_bytes)\n\n# Dispatch/combine operations\nactive_ranks = torch.ones((num_ranks,), dtype=torch.int32, device=\"cuda\")\nbuffer.dispatch(..., active_ranks=active_ranks, timeout_us=1000000)\nbuffer.combine(..., active_ranks=active_ranks, timeout_us=1000000)\n```\n\n## Starting Services\n\n### Start Master Service (with HTTP metadata server)\n```bash\nmooncake_master \\\n  --enable_http_metadata_server=true \\\n  --http_metadata_server_host=0.0.0.0 \\\n  --http_metadata_server_port=8080 \\\n  --default_kv_lease_ttl=5000\n```\n\n### Using External etcd (Production)\n```bash\n# Start etcd\netcd --listen-client-urls http://0.0.0.0:2379 \\\n     --advertise-client-urls http://0.0.0.0:2379\n\n# Start master\nmooncake_master --default_kv_lease_ttl=5000\n```\n\n## Environment Variables\n\n### Transfer Engine\n- `MC_METADATA_SERVER`: Metadata server URL\n- `MC_FORCE_TCP`: Force TCP transport (set to \"true\")\n- `MC_LOG_LEVEL`: Logging level (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_MS_AUTO_DISC`: Enable RDMA device auto-discovery (set to \"1\")\n- `MC_MS_FILTERS`: Filter RDMA devices (e.g., \"mlx5_0,mlx5_2\")\n- `MC_TRANSFER_TIMEOUT`: Transfer timeout in seconds (default: 30)\n\n### Mooncake Store\n- `MC_STORE_CLUSTER_ID`: Cluster identifier (default: \"mooncake\")\n- `MC_STORE_USE_HUGEPAGE`: Enable hugepage support\n- `MC_STORE_MEMCPY`: Enable local memcpy optimization (set to \"1\")\n- `MC_STORE_CLIENT_METRIC`: Enable client metrics (enabled by default)\n- `MC_YLT_LOG_LEVEL`: Log level (trace/debug/info/warn/error/critical)\n- `MOONCAKE_STORE_CHECKSUM`: Enable diagnostic object-level CRC-64 checks (set to \"1\" before creating any writer or reader client)\n\n## Common Patterns\n\n### Pattern 1: Simple KV Store\n```python\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"tcp\", \"\", \"localhost:50051\")\n\n# Store and retrieve\nstore.put(\"config\", b'{\"model\": \"llama-7b\"}')\nconfig = store.get(\"config\")\n\nstore.close()\n```\n\n### Pattern 2: High-Performance Tensor Storage\n```python\nimport torch\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"rdma\", \"mlx5_0\", \"localhost:50051\")\n\n# Configure replication\nconfig = ReplicateConfig()\nconfig.replica_num = 2\nconfig.with_soft_pin = True\n\n# Store tensor with replication\ntensor = torch.randn(1000, 1000)\nstore.put_tensor(\"weights\", tensor, config)\n\n# Retrieve\nretrieved = store.get_tensor(\"weights\")\n\nstore.close()\n```\n\n### Pattern 3: Zero-Copy Batch Operations\n```python\nimport numpy as np\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 16*1024*1024, \"rdma\", \"\", \"localhost:50051\")\n\n# Prepare buffers\nnum_buffers = 10\nbuffers = [np.random.randn(1024*1024).astype(np.float32) for _ in range(num_buffers)]\nbuffer_ptrs = [buf.ctypes.data for buf in buffers]\nsizes = [buf.nbytes for buf in buffers]\n\n# Register all buffers\nfor ptr, size in zip(buffer_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\n# Batch put\nkeys = [f\"tensor_{i}\" for i in range(num_buffers)]\nresults = store.batch_put_from(keys, buffer_ptrs, sizes)\n\n# Batch get\nrecv_buffers = [np.empty(1024*1024, dtype=np.float32) for _ in range(num_buffers)]\nrecv_ptrs = [buf.ctypes.data for buf in recv_buffers]\n\nfor ptr, size in zip(recv_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\nresults = store.batch_get_into(keys, recv_ptrs, sizes)\n\n# Cleanup\nfor ptr in buffer_ptrs + recv_ptrs:\n    store.unregister_buffer(ptr)\n\nstore.close()\n```\n\n### Pattern 4: Transfer Engine Direct Transfer\n```python\nfrom mooncake.engine import TransferEngine\nimport numpy as np\n\n# Setup engines on both nodes\nengine = TransferEngine()\nengine.initialize(\"127.0.0.1:12345\", \"127.0.0.1:2379\", \"tcp\", \"\")\n\n# Allocate and register buffer\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nengine.register_memory(buffer_ptr, buffer.nbytes)\n\n# Get remote buffer address (from peer via metadata exchange)\nremote_addr = engine.get_first_buffer_address(\"target_host:port\")\n\n# Transfer data\ndata = b\"Hello from Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_ptr, data, len(data))\nresult = engine.transfer_sync_write(\"target_host:port\", buffer_ptr, remote_addr, len(data))\n\n# Cleanup\nengine.unregister_memory(buffer_ptr)\n```\n\n## Error Handling\n\nAll methods return status codes:\n- `0`: Success\n- Negative values: Error codes\n\nCommon checks:\n```python\n# Store operations\nresult = store.put(\"key\", b\"value\")\nif result != 0:\n    print(f\"Put failed with error code: {result}\")\n\n# Existence check\nexists = store.is_exist(\"key\")\nif exists == 1:\n    print(\"Key exists\")\nelif exists == 0:\n    print(\"Key not found\")\nelse:\n    print(\"Error checking existence\")\n\n# Transfer operations\nresult = engine.transfer_sync_write(...)\nif result == 0:\n    print(\"Transfer successful\")\nelse:\n    print(f\"Transfer failed with code: {result}\")\n```\n\n## Troubleshooting\n\n### Connection Issues\n```python\n# Force TCP for testing without RDMA\nimport os\nos.environ[\"MC_FORCE_TCP\"] = \"true\"\n\n# Enable verbose logging\nos.environ[\"MC_LOG_LEVEL\"] = \"0\"\nos.environ[\"MC_YLT_LOG_LEVEL\"] = \"debug\"\n```\n\n### Memory Issues\n```python\n# Check buffer registration before zero-copy ops\nresult = store.register_buffer(buffer_ptr, size)\nif result != 0:\n    raise RuntimeError(f\"Failed to register buffer: {result}\")\n```\n\n### Corrupted Data or Garbled Output\n```python\n# Set this before importing Mooncake or creating any Store client.\nimport os\nos.environ[\"MOONCAKE_STORE_CHECKSUM\"] = \"1\"\n\nfrom mooncake.store import MooncakeDistributedStore\n```\n\nEnable the switch on every writer and reader client process, then reproduce with full-object `put`/`upsert` and `get` operations. Treat `CHECKSUM_MISMATCH` (-801) as a failed read and do not use the destination buffer. Objects without checksum metadata and range reads are not verified. This mode scans object data, stages GPU buffers to host memory, and disables the local hot cache, so use it only for diagnosis.\n\n### Service Connectivity\n```bash\n# Check master is running\ncurl http://localhost:50051\n\n# Check metadata server\ncurl http://localhost:8080/metadata\n```\n\n## Best Practices\n\n1. **Always close stores**: Call `store.close()` when done\n2. **Register buffers for zero-copy**: Required for RDMA operations\n3. **Use batch operations**: Better throughput for multiple operations\n4. **Configure replication**: Use `ReplicateConfig` for important data\n5. **Use soft pinning**: For frequently accessed objects\n6. **Choose protocol wisely**: TCP for dev/test, RDMA for production\n7. **Monitor leases**: Objects have TTL, renew if needed\n8. **Handle errors**: Check return codes and handle failures\n\n## Quick Reference\n\n### Mooncake Store Methods\n- `setup()`: Initialize store\n- `put()`, `get()`: Basic operations\n- `put_batch()`, `get_batch()`: Batch operations\n- `put_from()`, `get_into()`: Zero-copy operations\n- `put_tensor()`, `get_tensor()`: PyTorch tensors\n- `register_buffer()`, `unregister_buffer()`: Buffer management\n- `is_exist()`, `remove()`: Metadata operations\n- `close()`: Cleanup\n\n### Transfer Engine Methods\n- `initialize()`: Setup engine\n- `allocate_managed_buffer()`, `free_managed_buffer()`: Buffer allocation\n- `transfer_sync_write()`, `transfer_sync_read()`: Synchronous transfers\n- `transfer_submit_write()`, `transfer_check_status()`: Async transfers\n- `batch_transfer_sync_write()`, `batch_transfer_sync_read()`: Batch sync\n- `batch_transfer_async_write()`, `get_batch_transfer_status()`: Batch async\n- `register_memory()`, `unregister_memory()`: Memory registration\n- `write_bytes_to_buffer()`, `read_bytes_from_buffer()`: Buffer I/O\n\n## Documentation Links\n\n- Full API Reference: https://kvcache-ai.github.io/Mooncake/\n- Quick Start: docs/source/getting_started/quick-start.md\n- Mooncake Store: docs/source/python-api-reference/mooncake-store.md\n- Transfer Engine: docs/source/python-api-reference/transfer-engine.md\n- Transfer Engine direct usage: docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects\n- EP Backend: docs/source/python-api-reference/ep-backend.md\n",".claude/skills/mooncake-troubleshoot/SKILL.md":"---\nname: mooncake-troubleshoot\ndescription: Automatically diagnose Mooncake deployment and runtime issues. Checks services (mooncake_master, metadata server), RDMA devices, environment variables, connectivity, memory limits, object integrity, and analyzes logs for common error patterns. Use when Mooncake deployment fails, services won't start, connections fail, data is corrupted or garbled, or you encounter runtime errors like \"Error from etcd client\", \"No matched device found\", \"Failed to register memory\", \"NO_AVAILABLE_HANDLE\", \"CHECKSUM_MISMATCH\", or any RDMA/networking issues. Also use when user asks to troubleshoot, debug, diagnose, or fix Mooncake problems.\n---\n\n# Mooncake Deployment Troubleshooting\n\nYou are a Mooncake deployment troubleshooting specialist. Your job is to systematically diagnose issues and provide actionable solutions based on the comprehensive troubleshooting knowledge from Mooncake documentation.\n\n## Diagnostic Strategy\n\nRun checks systematically, reporting findings as you go. Start with simple checks (services, connectivity) before diving into complex issues (RDMA, memory registration).\n\n### 1. Service Status Check\n\nCheck if critical services are running:\n\n```bash\n# Check mooncake_master\nps aux | grep mooncake_master | grep -v grep\n\n# Check port usage\nnetstat -tuln | grep -E '(50051|8080|2379|9003)'\n\n# If using etcd\nps aux | grep etcd | grep -v grep\n```\n\n**Common issues:**\n- `bind address already in use` → Port conflict, use different port with `--rpc_port`\n- Master not running → Check startup logs for errors\n\n### 2. Metadata Server Connectivity\n\nThe metadata server is critical for node discovery and coordination.\n\n```bash\n# Test etcd connectivity\ncurl -s http://127.0.0.1:2379/version\n\n# Or test custom metadata server\ncurl -s $MC_METADATA_SERVER\n\n# Check for proxy interference\necho \"http_proxy: $http_proxy\"\necho \"https_proxy: $https_proxy\"\n```\n\n**Common issues:**\n- `Error from etcd client` → Metadata server unreachable\n  - **Fix:** Ensure etcd is bound to `0.0.0.0` not `127.0.0.1`:\n    ```bash\n    etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n    ```\n  - **Fix:** Disable HTTP proxy:\n    ```bash\n    unset http_proxy https_proxy\n    ```\n\n### 3. Environment Variables Check\n\nVerify critical environment variables are set correctly:\n\n```bash\n# Display Mooncake variables\nenv | grep -E '^(MC_|MOONCAKE_STORE_CHECKSUM=)'\n\n# Key variables to check:\necho \"MC_METADATA_SERVER: $MC_METADATA_SERVER\"\necho \"MC_FORCE_TCP: $MC_FORCE_TCP\"\necho \"MC_LOG_LEVEL: $MC_LOG_LEVEL\"\necho \"MC_YLT_LOG_LEVEL: $MC_YLT_LOG_LEVEL\"\necho \"MC_MS_AUTO_DISC: $MC_MS_AUTO_DISC\"\necho \"MC_MS_FILTERS: $MC_MS_FILTERS\"\necho \"MC_GID_INDEX: $MC_GID_INDEX\"\necho \"MC_MTU: $MC_MTU\"\necho \"MC_IB_PORT: $MC_IB_PORT\"\necho \"MC_ENABLE_DEST_DEVICE_AFFINITY: $MC_ENABLE_DEST_DEVICE_AFFINITY\"\necho \"MOONCAKE_STORE_CHECKSUM: $MOONCAKE_STORE_CHECKSUM\"\n```\n\n**Key variables:**\n- `MC_METADATA_SERVER` - Metadata server URL (required)\n- `MC_FORCE_TCP=true` - Force TCP for testing without RDMA\n- `MC_LOG_LEVEL=0` - Enable verbose logging (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_YLT_LOG_LEVEL=debug` - yalantinglibs log level\n- `MC_MS_AUTO_DISC=1` - Enable topology auto-discovery (default)\n- `MC_MS_FILTERS` - Filter specific RDMA devices (e.g., \"mlx5_1,mlx5_2\")\n- `MC_GID_INDEX` - RDMA GID index (set if GID is all zeros)\n- `MC_MTU` - RDMA MTU size\n- `MC_ENABLE_DEST_DEVICE_AFFINITY=1` - Reduce QP creation (fix \"Failed to create QP\")\n- `MOONCAKE_STORE_CHECKSUM=1` - Enable diagnostic object-level CRC-64 checks; set before starting every writer and reader client process\n\n### 4. RDMA Device Check\n\nOnly run if RDMA is being used (skip if `MC_FORCE_TCP=true`):\n\n```bash\n# List RDMA devices\nibv_devices\n\n# Check device details and status\nibv_devinfo\n\n# Check for ACTIVE ports\nibv_devinfo | grep -A 10 \"state:\"\n\n# Check GID addresses (should NOT be all zeros)\nibv_devinfo | grep -A 20 \"GID\"\n\n# Check peer memory modules\nlsmod | grep peer_mem\nlsmod | grep nvidia_peer_mem\n\n# Check QP count (if \"Failed to create QP\" error)\nrdma resource show qp\n```\n\n**Common issues:**\n- `No matched device found` → RDMA device name in config doesn't exist\n  - **Fix:** Use `ibv_devices` to get correct device names\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n  - **Fix:** Check cable connections, verify with `ibv_devinfo | grep state`\n  - **Fix:** Try different port with `MC_IB_PORT` environment variable\n- GID all zeros → Wrong GID index\n  - **Fix:** Set `MC_GID_INDEX=1` (or 2, 3 depending on network)\n- `Failed to create QP: Cannot allocate memory` → Too many QPs created\n  - **Fix:** Set `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n\n### 5. Memory and Resource Limits\n\nCheck system limits that affect RDMA memory registration:\n\n```bash\n# Check ulimits\nulimit -a\n\n# Focus on max locked memory\nulimit -l\n\n# Check RDMA device memory limits\nibv_devinfo -v | grep max_mr_size\n\n# Check dmesg for memory errors\ndmesg -T | tail -50 | grep -i \"out of mr size\"\n```\n\n**Common issues:**\n- `Failed to register memory: Input/output error` → Memory registration limit exceeded\n  - **Diagnostic:** Check `max_mr_size` with `ibv_devinfo -v`\n  - **Fix:** Reduce memory allocation or split into smaller chunks\n- Cannot allocate memory → ulimit restriction\n  - **Fix:** Set unlimited locked memory:\n    ```bash\n    ulimit -l unlimited\n    ```\n  - **Permanent fix:** Add to `/etc/security/limits.conf`:\n    ```\n    * soft memlock unlimited\n    * hard memlock unlimited\n    ```\n\n### 6. Network Connectivity\n\nTest connectivity between nodes:\n\n```bash\n# Test basic RDMA connectivity\nib_write_bw -d <device_name> -R\n\n# On peer node:\nib_write_bw -d <device_name> -R <server_ip>\n\n# Test GPU Direct RDMA (if CUDA enabled)\nib_write_bw -d <device_name> -R -x gdr\n\n# On peer node:\nib_write_bw -d <device_name> -R -x gdr <server_ip>\n\n# Test DNS resolution\nnslookup <connectable_name>\nping <connectable_name>\n```\n\n**Common issues:**\n- `connection refused` → Incorrect `connectable_name` or `rpc_port`\n  - **Fix:** Ensure `connectable_name` is NOT loopback (127.0.0.1/localhost)\n  - **Fix:** Use actual LAN/WAN IP or valid hostname\n- `Failed to exchange handshake` → RDMA connection setup failure\n  - **Fix:** Verify MTU matches: set `MC_MTU` environment variable\n  - **Fix:** Verify GID is valid (not all zeros)\n  - **Fix:** Test with `ib_send_bw` between nodes first\n\n### 7. Log Analysis\n\nSearch logs for common error patterns and their meanings:\n\n**Metadata/Connectivity Errors:**\n- `Error from etcd client` → Cannot connect to metadata server\n- `ERR_METADATA` → Metadata server communication failed\n- `ERR_DNS` → Invalid `local_server_name` (not valid DNS/IP)\n\n**RDMA Errors:**\n- `No matched device found` → RDMA device name doesn't exist\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n- `Failed to exchange handshake description` → RDMA handshake failed\n- `Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num` → MTU/GID mismatch\n- `Failed to register memory` → Memory registration limit exceeded\n- `Failed to create QP` → Too many QPs, enable `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n- `Worker: Process failed for slice` → Network instability\n- `work request flushed error` → Cascading error (find first error)\n\n**Store Errors:**\n- `NO_AVAILABLE_HANDLE` (-200) → Memory pool exhausted\n  - **Fix:** Increase `global_segment_size` in setup\n  - **Fix:** Check eviction is working (look for eviction logs)\n- `LEASE_EXPIRED` (-707) → Lease expired during transfer\n  - **Fix:** Increase `default_kv_lease_ttl` in master startup\n- `OBJECT_NOT_FOUND` (-704) → Object doesn't exist\n- `SEGMENT_NOT_FOUND` (-101) → No available segments\n- `CHECKSUM_MISMATCH` (-801) → Full-object read differs from the checksum captured before the write\n  - **Action:** Treat the read as failed and do not use the destination buffer\n- `Failed to get description of XXX` → Segment name mismatch\n  - **Fix:** Ensure segment name matches `local_hostname` from peer\n\n**Port/Service Errors:**\n- `bind address already in use` → Port conflict\n  - **Fix:** Use different port: `--rpc_port=50052`\n\n### 8. Configuration Validation\n\nVerify configuration is correct:\n\n```bash\n# Check connectable_name is not loopback\nhostname -I\n\n# Verify master startup flags\nps aux | grep mooncake_master\n\n# Check if using correct protocol\nenv | grep MC_FORCE_TCP\n```\n\n**Critical checks:**\n- `connectable_name` must be non-loopback IP or valid hostname\n- MTU and GID configurations must match network environment\n- RDMA device names must exist on the machine\n- Ports must not be in use by other services\n\n### 9. Object Integrity Diagnostics\n\nWhen the user reports corrupted data or garbled output that may originate from Mooncake Store:\n\n1. Verify that the Store clients, primary master, and standby master use checksum-capable binaries from the same version.\n2. Set `MOONCAKE_STORE_CHECKSUM=1` before starting every writer and reader client process, then restart or recreate existing clients.\n3. Reproduce with full-object `put`/`upsert` and `get` operations. Range reads, including `get_into_ranges`, are not covered.\n4. If `CHECKSUM_MISMATCH` (-801) is returned, the covered read differs from the checksum captured over the writer's source object. Treat the read as failed and do not consume the destination buffer.\n5. If `object_checksum_absent` appears at VLOG(1), metadata has no checksum and verification was skipped. This can occur for existing objects or objects written by clients without the switch.\n6. If verification succeeds, continue investigating corruption outside the covered Store path. A checksum mismatch identifies a difference but does not by itself distinguish transfer corruption from storage corruption.\n7. Warn that this diagnostic mode scans all object data, stages GPU buffers to host memory, and disables the local hot cache. Disable it after diagnosis.\n\n## Error Code Quick Reference\n\n### Transfer Engine Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Normal execution | - |\n| -12 | ERR_ADDRESS_NOT_REGISTERED | Memory not registered | Register memory before use |\n| -14 | ERR_DEVICE_NOT_FOUND | RDMA device not found | Check device name with `ibv_devices` |\n| -16 | ERR_DNS | Invalid local_server_name | Use valid IP/hostname |\n| -19 | ERR_REJECT_HANDSHAKE | Peer rejected handshake | Check peer logs for reason |\n| -20 | ERR_METADATA | Metadata server unreachable | Check etcd/HTTP server |\n\n### Store Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Operation successful | - |\n| -200 | NO_AVAILABLE_HANDLE | Memory pool exhausted | Increase segment size |\n| -707 | LEASE_EXPIRED | Lease expired | Increase lease TTL |\n| -704 | OBJECT_NOT_FOUND | Object doesn't exist | Check object key |\n| -101 | SEGMENT_NOT_FOUND | No available segments | Check segment registration |\n| -801 | CHECKSUM_MISMATCH | Full-object data differs from its stored checksum | Reject the read buffer and investigate the Store data path |\n| -900 | RPC_FAIL | RPC failed | Check network/master |\n| -1000 | ETCD_OPERATION_ERROR | etcd operation failed | Check etcd status |\n\n## Output Format\n\nProvide a structured diagnostic report:\n\n```\n🔍 MOONCAKE DEPLOYMENT DIAGNOSTICS\n==================================\n\n✅ PASSED CHECKS:\n- Service status: mooncake_master running on port 50051\n- Metadata server: etcd accessible at http://127.0.0.1:2379\n- Environment: MC_METADATA_SERVER set correctly\n- [other passing checks]\n\n❌ FAILED CHECKS:\n- RDMA device: mlx5_0 port not ACTIVE (state: PORT_DOWN)\n- Memory limits: max locked memory is 64KB (too low)\n- [other failures with specific error messages]\n\n⚠️  WARNINGS:\n- GID index not set, may cause connection issues\n- HTTP proxy variables set, may interfere with metadata server\n- [other potential issues]\n\n🔧 RECOMMENDED FIXES:\n\n1. Fix RDMA port status:\n   - Check physical cable connections\n   - Verify driver configuration\n   - Command: ibv_devinfo | grep -A 10 \"state:\"\n\n2. Increase memory limits:\n   ulimit -l unlimited\n   # Or permanently in /etc/security/limits.conf:\n   * soft memlock unlimited\n   * hard memlock unlimited\n\n3. Set GID index:\n   export MC_GID_INDEX=1\n\n4. Disable HTTP proxy:\n   unset http_proxy https_proxy\n\n📋 SUMMARY:\n[Brief 2-3 sentence conclusion about deployment health and next steps]\n```\n\n## Troubleshooting Workflow\n\n1. **Start simple**: Check services and basic connectivity first\n2. **Read logs carefully**: First error is usually root cause (subsequent errors cascade)\n3. **Test incrementally**: Use `MC_FORCE_TCP=true` to isolate RDMA issues\n4. **Verify basics**: Check connectable_name, ports, env vars before deep diving\n5. **Use diagnostic tools**: ibv_devices, ibv_devinfo, ib_write_bw, curl\n6. **Reference documentation**: Check error codes and troubleshooting guide\n\n## Quick Fix Commands\n\n**Start metadata server properly:**\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n```\n\n**Enable verbose logging:**\n```bash\nexport MC_LOG_LEVEL=0\nexport MC_YLT_LOG_LEVEL=debug\n```\n\n**Force TCP mode for testing:**\n```bash\nexport MC_FORCE_TCP=true\n```\n\n**Fix memory limits:**\n```bash\nulimit -l unlimited\n```\n\n**Fix too many QPs:**\n```bash\nexport MC_ENABLE_DEST_DEVICE_AFFINITY=1\n```\n\n**Fix GID issues:**\n```bash\nexport MC_GID_INDEX=1  # or 2, 3 depending on network\n```\n\n**Use different port:**\n```bash\nmooncake_master --rpc_port=50052\n```\n\nNow execute the diagnostic checks systematically and provide the structured report.\n","docs/AGENTS.md":"# AGENTS.md - Mooncake Documentation\n\nThis file gives coding agents the repo-local rules for modifying files under\n`docs/`. Keep `README.md` as the human-facing quickstart. Use this file for\nagent workflow, verification, and maintenance guidance.\n\n## Scope\n\n- Applies to changes under `docs/`, especially `docs/source/`.\n- Prefer small, reviewable documentation changes.\n- Do not rewrite unrelated pages, generated files, or formatting-only content.\n- Preserve existing documentation structure unless the user asks for a broader\n  reorganization.\n\n## Build and Preview\n\nRun documentation commands from the `docs` directory:\n\n```\ncd docs\n```\n\nInstall dependencies with `uv` when needed. The requirements file is\n`docs/requirements-docs.txt`; after `cd docs`, use the local filename. If there\nis an existing venv, prefer using the existing one. Otherwise, create one before\ninstalling dependencies:\n\n```\nuv venv\nuv pip install -r requirements-docs.txt\n```\n\nClean stale build output when validating navigation, generated API pages, or\ntheme behavior:\n\n```\nmake clean\n```\n\nBuild HTML before handing off user-visible documentation changes:\n\n```\nmake html\n```\n\nSet `locale` correctly before building when the change depends on localized\ncontent or translated output.\n\nServe the generated site for review:\n\n```\npython -m http.server -d build/html/\n```\n\nThe default URL is `http://localhost:8000`. If port 8000 is busy, choose another\navailable port.\n\n## Editing Guidance\n\n- Source pages live under `docs/source/`.\n- Keep links relative and Sphinx-compatible unless an external URL is required.\n- For navigation changes, inspect `docs/source/index.md` and the relevant\n  toctree before editing individual pages.\n- Use Sphinx-native structure for documentation behavior. Do not use client-side\n  JavaScript or post-render DOM patches for navigation or theme behavior.\n- If the requested behavior is not supported by Sphinx or the active theme,\n  prefer adding a Sphinx extension/plugin instead of patching rendered HTML.\n- When a page should be linked from content but excluded from the main sidebar,\n  use a content link plus appropriate Sphinx metadata such as `orphan: true`\n  instead of hiding rendered sidebar nodes.\n- Keep homepage toctree depth conservative. Do not increase `index.md` maxdepth\n  unless the user explicitly asks for deeper landing-page nesting.\n\n## Validation Checklist\n\n- Run `make html` for docs changes that affect rendered pages, navigation,\n  cross-references, or Sphinx configuration.\n- Check the generated HTML for the changed pages.\n- For sidebar or toctree changes, verify both the article body and left sidebar\n  render the intended entries.\n- If a local preview server is useful for review, start one from `docs/` with\n  `python -m http.server -d build/html/` or an alternate port.\n\n## Pull Request Hygiene\n\n- Keep docs-only changes narrowly scoped.\n- Review `git diff` before staging so generated files or hook-only formatting\n  changes do not leak into the PR.\n- If opening a PR, use the repository pull request template.\n- Use the repository PR title prefix rules from the root `AGENTS.md`.\n",".claude/skills/mooncake-ci-local/SKILL.md":"---\nname: mooncake-ci-local\ndescription: Run Mooncake pre-PR local validation through scripts/run_ci_test.sh. Use this skill whenever the user wants to validate a branch before opening or submitting a PR, run local CI, run ci test, check changes before PR, reproduce GitHub Actions locally, or force a full pre-submit verification. Trigger on phrases like \"提交 PR 前验证\", \"run ci test\", \"run local CI\", \"check my branch\", \"test before PR\", \"pre-submit validation\", and \"reproduce CI locally\".\n---\n\n# Mooncake Pre-PR Local Validation\n\nUse `bash scripts/run_ci_test.sh` as the default entry point. This is the single local lane for PR-before-submit validation, and it already coordinates the reproducible parts of GitHub Actions.\n\n## Default Entry Point\n\nWhen the user asks for any of the following, run the repo script first instead of reconstructing the workflow by hand:\n\n- 提交 PR 前本地验证\n- run ci test\n- run local CI\n- check my branch before PR\n- reproduce CI locally\n\nDefault command:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nWhat this script already covers:\n\n- GitHub-like `paths-filter` against `origin/main`\n- `typos`\n- `scripts/code_format.sh --check`\n- default CMake configure/build/install in `build-ci-local`\n- `ctest`\n- wheel build in `build-wheel-local`\n- wheel installation validation\n- `scripts/run_tests.sh`\n- selected Python API and integration tests\n- per-stage summary and logs under `local_test/run-ci-logs/<timestamp>/`\n\n## Standard Agent Workflow\n\n1. Run `bash scripts/run_ci_test.sh` from the repo root unless the user explicitly asks for a narrower subset.\n2. Read the stage summary instead of dumping raw terminal output.\n3. Report these items back to the user:\n   - passed stages\n   - failed stages\n   - blocked stages\n   - unsupported stages\n   - whether `paths-filter` skipped downstream stages\n   - the log directory under `local_test/run-ci-logs/...`\n4. If there is a failure, inspect the corresponding stage log and summarize the root cause.\n\n## Common Options\n\nForce a full lane even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nUse another base ref:\n\n```bash\nbash scripts/run_ci_test.sh --base origin/main\n```\n\nAuto-install missing dependencies:\n\n```bash\nbash scripts/run_ci_test.sh --install-deps\n```\n\nKeep services running for follow-up debugging:\n\n```bash\nbash scripts/run_ci_test.sh --keep-services\n```\n\n## Minimal Example\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nIf the user wants to ignore changed-path optimization and force the full lane:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nSee also `.claude/skills/mooncake-ci-local/examples/minimal.md`.\n\n## How To Interpret Results\n\n- `passed`: the stage succeeded locally.\n- `failed`: the stage reproduced a real local failure and needs investigation.\n- `blocked`: local environment or dependency issue prevented execution.\n- `unsupported`: intentionally not run in the local lane because it needs external platforms, special hardware, or a non-default build.\n\nIf `paths-filter` skips downstream stages, explain that the current branch changed only non-source paths relative to the selected base.\n\n## Current Local Coverage\n\nIncluded by default:\n\n- spell check\n- code format check\n- default ASan CMake lane in `build-ci-local`\n- `ctest`\n- wheel build and installation test\n- `scripts/run_tests.sh`\n- selected Python API tests\n\nUnsupported by design in the default local lane:\n\n- Ascend jobs\n- T-one integration jobs\n- MUSA jobs\n- Docker image build jobs\n- CUDA 13 wheel jobs\n- PG-backend tests absent from the default wheel build\n- Python drain-http API stage in the local ASan lane\n\n## Targeted Reruns For Debugging\n\nUse targeted reruns only after the full script identifies a failing area, or when the user explicitly asks for a smaller scope.\n\nRerun a specific C++ test pattern:\n\n```bash\ncd build-ci-local\nMC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -R <pattern> --output-on-failure\n```\n\nRerun the Python wheel integration lane:\n\n```bash\nsource test_env/bin/activate\nMC_STORE_MEMCPY=false TEST_SSD_OFFLOAD_IN_EVICT=true ./scripts/run_tests.sh\n```\n\nRerun the safetensor unittest:\n\n```bash\nsource test_env/bin/activate\npython -m unittest mooncake-wheel.tests.test_safetensor_functions\n```\n\n## Notes For The Agent\n\n- Prefer the repo script over rebuilding the CI workflow step by step.\n- Preserve the separation between `build-ci-local` and `build-wheel-local`.\n- Summarize failing stages from their logs instead of pasting raw output.\n- If the user only asks whether the branch is safe before opening a PR, the default answer path is `bash scripts/run_ci_test.sh`.\n","mooncake-store/AGENTS.md":"# Mooncake Store Instructions\n\n## Random Numbers\n\n- Production code in `include/` and `src/` must use `include/random.h`; do not\n  create random engines, seeds, or distributions at call sites.\n- Extend `random.h` and its tests when an operation is missing. Keep bounded\n  sampling unbiased and reject invalid bounds.\n- Tests and benchmarks may use explicit seeds for reproducibility, but should\n  use the shared sampling helpers.\n- Do not migrate `src/cachelib_memory_allocator/` unless explicitly requested.\n- The shared engine is not cryptographically secure; do not use it for secrets\n  or authentication tokens.\n\n## Local-first Host Identity\n\n- Keep the placement host ID separate from the routable transfer endpoint.\n- `MOONCAKE_HOST_ID` takes priority after ASCII whitespace trimming. Unset,\n  empty, or whitespace-only values fall back to the existing `local_hostname`\n  behavior.\n- Normalize endpoint-shaped host IDs before rejecting loopback or wildcard\n  identities.\n",".claude/skills/mooncake-api/.claude-plugin/plugin.json":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-api\",\n  \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"python\", \"api\", \"transfer-engine\", \"store\"]\n}\n",".claude/skills/mooncake-ci-local/scripts/check-prerequisites.sh":"#!/bin/bash\n# Mooncake CI Local Test Prerequisites Check\n# Usage: bash check-prerequisites.sh\n# This script checks and auto-fixes all prerequisites for running Mooncake CI tests locally.\n\nset -e\n\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nYELLOW='\\033[1;33m'\nNC='\\033[0m'\n\necho \"🔍 Checking Mooncake CI test prerequisites...\"\n\n# 1. Check build directory\nif [ ! -f build/CMakeCache.txt ]; then\n  echo -e \"${RED}✗ Build directory not found or not built${NC}\"\n  echo \"  → Run: mkdir build && cd build && cmake .. && cmake --build .\"\n  exit 1\nfi\necho -e \"${GREEN}✓ Build exists${NC}\"\n\n# 2. Check mooncake installation\nif ! python -c \"import mooncake\" 2>/dev/null; then\n  echo -e \"${RED}✗ mooncake package not installed${NC}\"\n  echo \"  → Fixing: Installing mooncake package...\"\n  cd build && sudo cmake --install . && cd - >/dev/null\n  if ! python -c \"import mooncake\" 2>/dev/null; then\n    echo \"  → Alternative: pip install mooncake-wheel/dist/*.whl\"\n    exit 1\n  fi\n  echo -e \"${GREEN}✓ mooncake package installed${NC}\"\nelse\n  echo -e \"${GREEN}✓ mooncake package already installed${NC}\"\nfi\n\n# 3. Check ctest availability\nif ! command -v ctest &> /dev/null; then\n  echo -e \"${RED}✗ ctest not found${NC}\"\n  exit 1\nfi\necho -e \"${GREEN}✓ ctest available${NC}\"\n\n# 4. Kill and restart services (safest approach for local testing)\necho -e \"\\n${YELLOW}Cleaning up and restarting services...${NC}\"\npkill -f \"^etcd\" || true\npkill -f bootstrap_server.py || true\npkill -f mooncake_http_metadata_server || true\nsleep 1\n\n# 5. Start etcd\nif ! command -v etcd &> /dev/null; then\n  echo -e \"${YELLOW}⚠ etcd not found, installing...${NC}\"\n  ETCD_VER=v3.6.1\n  OS=$(uname -s | tr '[:upper:]' '[:lower:]')\n  ARCH=$(uname -m)\n  [ \"$ARCH\" = \"x86_64\" ] && ARCH=\"amd64\"\n  DOWNLOAD_URL=\"https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\"\n  echo \"  Downloading from: $DOWNLOAD_URL\"\n  cd /tmp\n  wget -q \"$DOWNLOAD_URL\" && tar xzf \"etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\" && \\\n    sudo mv \"etcd-${ETCD_VER}-${OS}-${ARCH}\"/etcd* /usr/local/bin/\n  cd - >/dev/null\n  echo -e \"${GREEN}✓ etcd installed${NC}\"\nfi\n\netcd --advertise-client-urls http://127.0.0.1:2379 --listen-client-urls http://127.0.0.1:2379 >/dev/null 2>&1 &\nETCD_PID=$!\nsleep 2\nif ! etcdctl --endpoints=http://127.0.0.1:2379 endpoint health &>/dev/null; then\n  echo -e \"${RED}✗ etcd failed to start${NC}\"\n  kill $ETCD_PID 2>/dev/null || true\n  exit 1\nfi\necho -e \"${GREEN}✓ etcd running (PID: $ETCD_PID)${NC}\"\n\n# 6. Start HTTP metadata server\nif [ -f \"mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py\" ]; then\n  cd mooncake-transfer-engine/example/http-metadata-server-python\n  pip install -q aiohttp 2>/dev/null || true\n  python ./bootstrap_server.py >/dev/null 2>&1 &\n  METADATA_PID=$!\n  cd - >/dev/null\n  sleep 1\n  if curl -s http://127.0.0.1:8080/metadata > /dev/null 2>&1; then\n    echo -e \"${GREEN}✓ HTTP Metadata server running (PID: $METADATA_PID)${NC}\"\n  else\n    echo -e \"${RED}✗ HTTP Metadata server failed to start${NC}\"\n    kill $METADATA_PID $ETCD_PID 2>/dev/null || true\n    exit 1\n  fi\nelse\n  echo -e \"${YELLOW}⚠ Metadata server script not found, skipping${NC}\"\nfi\n\necho -e \"\\n${GREEN}✅ All prerequisites ready!${NC}\"\necho \"Service PIDs: etcd=$ETCD_PID\"\n[ -n \"$METADATA_PID\" ] && echo \"Metadata server PID: $METADATA_PID\"\necho -e \"\\n${YELLOW}To kill services:${NC}\"\necho \"  pkill -f '^etcd'\"\necho \"  pkill -f bootstrap_server\"\n",".claude/skills/mooncake-ci-local/.claude-plugin/plugin.json":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-ci-local\",\n  \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"ci\", \"validation\", \"pre-pr\"]\n}\n",".claude/skills/mooncake-ci-local/examples/minimal.md":"# Minimal Example\n\nGoal: validate the current branch before opening or submitting a PR.\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nForce a full rerun even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nResult location:\n\n- `local_test/run-ci-logs/<timestamp>/`\n\nTypical report format:\n\n- passed stages\n- failed or blocked stages\n- unsupported stages\n- first actionable failure and its log path\n",".claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json":"# claude-plugin Plugin Manifest\n\nPath: `.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json)"},"files":{".claude-plugin/marketplace.json":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-marketplace.json\",\n  \"name\": \"mooncake\",\n  \"owner\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"metadata\": {\n    \"description\": \"Built-in Claude Code skills for working with Mooncake: deployment troubleshooting, local CI validation, and the Python API.\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"mooncake-troubleshoot\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-troubleshoot\"\n      },\n      \"description\": \"Systematically diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, connectivity, logs) and propose fixes.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-troubleshoot.html\"\n    },\n    {\n      \"name\": \"mooncake-ci-local\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-ci-local\"\n      },\n      \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\"\n    },\n    {\n      \"name\": \"mooncake-api\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-api\"\n      },\n      \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\"\n    }\n  ]\n}\n","AGENTS.md":"# AGENTS.md\n\n## `docs/` Directory Changes\n\n- Before modifying files under `docs/`, read `docs/AGENTS.md`.\n\n## Pull Request Guidelines\n\n- Follow `CONTRIBUTING.md` for PR title prefixes, RFC expectations, and\n  contribution workflow.\n- Before opening a PR for nontrivial work, check whether an existing issue or\n  open PR already covers the same change. If the work overlaps, explain the\n  difference instead of duplicating it.\n- Do not open low-value busywork PRs for isolated typo, style, or mechanical\n  changes unless they are part of a substantive requested change.\n- Use `.github/pull_request_template.md` when preparing a PR, and fill in the\n  relevant sections for description, module, type of change, testing,\n  checklist, and AI assistance disclosure.\n- For AI-assisted changes, make sure the human submitter has reviewed every\n  changed line and can defend the change end-to-end.\n- Before handoff, run pre-commit on the files touched by the change when the\n  toolchain is available (see `CONTRIBUTING.md` for the PR-scoped\n  `pre-commit run --files ...` command). Do not use\n  `pre-commit run --all-files` for routine PRs; if it rewrites unrelated\n  files, leave those edits out of the PR.\n- Keep PRs lean: review `git diff` before staging, and include only changes\n  required for the requested task.\n",".claude/skills/mooncake-api/SKILL.md":"---\nname: mooncake-api\ndescription: Help users work with the Mooncake Python APIs for distributed storage and high-performance data transfer. Use when working with Mooncake Store (distributed KV cache), Transfer Engine (RDMA/TCP transfers), service setup (master, metadata server), PyTorch tensors in the Store, zero-copy/buffer management, batch operations and replication, or Mooncake EP / Backend (Expert Parallelism). Trigger on questions about MooncakeDistributedStore, TransferEngine, put/get, put_tensor, register_buffer, ReplicateConfig, or the mooncake.store / mooncake.engine / mooncake.pg Python modules.\n---\n\n# Mooncake Python API Skill\n\nUse this skill to help users work with Mooncake Python APIs for distributed storage and high-performance data transfer.\n\n## When to Use This Skill\n\nUse this skill when users ask about:\n- Using Mooncake Store for distributed KV cache storage\n- Using Transfer Engine for RDMA/TCP data transfers\n- Setting up Mooncake services (master, metadata server)\n- Working with PyTorch tensors in Mooncake Store\n- Zero-copy operations and buffer management\n- Batch operations and replication configuration\n- Mooncake EP (Expert Parallelism) and Mooncake Backend\n- Troubleshooting Mooncake Python API issues\n\n## Routing Guidance\n\n- For most vLLM and SGLang users, start from `docs/source/getting_started/quick-start.md`.\n- For PD disaggregation, direct users to the SGLang/vLLM integration guides listed in Quick Start. Those guides own the serving-framework configuration.\n- For Mooncake Store integrations, direct users to the SGLang/vLLM Store setup guides listed in Quick Start. Do not duplicate `mooncake_master` startup commands in general API answers unless the user is working outside those frameworks.\n- For direct low-level Transfer Engine usage, use `docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects`.\n- For API signatures and method details, use the Python API references under `docs/source/python-api-reference/`.\n\n## Core Components\n\n### 1. Mooncake Store (Distributed KV Cache)\n\n**Import:**\n```python\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n```\n\n**Basic Setup:**\n```python\nstore = MooncakeDistributedStore()\nstore.setup(\n    \"localhost\",                          # local_hostname\n    \"http://localhost:8080/metadata\",     # metadata_server\n    512*1024*1024,                        # global_segment_size (512MB)\n    128*1024*1024,                        # local_buffer_size (128MB)\n    \"tcp\",                                # protocol (\"tcp\" or \"rdma\")\n    \"\",                                   # rdma_devices (empty for auto-select)\n    \"localhost:50051\"                     # master_server_address\n)\n```\n\n**Common Operations:**\n```python\n# Put/Get\nstore.put(\"key\", b\"value\")\ndata = store.get(\"key\")\n\n# Batch operations\nstore.put_batch([\"key1\", \"key2\"], [b\"val1\", b\"val2\"])\nvalues = store.get_batch([\"key1\", \"key2\"])\n\n# Check existence\nexists = store.is_exist(\"key\")  # Returns 1 (exists), 0 (not exists), -1 (error)\n\n# Remove\nstore.remove(\"key\")\nstore.remove_by_regex(\"^prefix_.*\")\nstore.remove_all()\n\n# Cleanup\nstore.close()\n```\n\n**Zero-Copy Operations (Advanced):**\n```python\nimport numpy as np\n\n# Create and register buffer\nbuffer = np.zeros(100*1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nstore.register_buffer(buffer_ptr, buffer.nbytes)\n\n# Zero-copy put\nstore.put_from(\"key\", buffer_ptr, buffer.nbytes)\n\n# Zero-copy get\nrecv_buffer = np.empty(100*1024*1024, dtype=np.uint8)\nrecv_ptr = recv_buffer.ctypes.data\nstore.register_buffer(recv_ptr, recv_buffer.nbytes)\nbytes_read = store.get_into(\"key\", recv_ptr, recv_buffer.nbytes)\n\n# Cleanup\nstore.unregister_buffer(buffer_ptr)\nstore.unregister_buffer(recv_ptr)\n```\n\n**PyTorch Tensor Operations:**\n```python\nimport torch\n\n# Simple tensor operations\ntensor = torch.randn(100, 100)\nstore.put_tensor(\"my_tensor\", tensor)\nretrieved = store.get_tensor(\"my_tensor\")\n\n# Batch tensor operations\ntensors = [torch.randn(100, 100) for _ in range(3)]\nstore.batch_put_tensor([\"t1\", \"t2\", \"t3\"], tensors)\nretrieved_tensors = store.batch_get_tensor([\"t1\", \"t2\", \"t3\"])\n\n# Tensor Parallelism (TP) support\nstore.put_tensor_with_tp(\"model_weights\", tensor, tp_rank=0, tp_size=4, split_dim=0)\nshard = store.get_tensor_with_tp(\"model_weights\", tp_rank=0, tp_size=4)\n```\n\n**Replication Configuration:**\n```python\nconfig = ReplicateConfig()\nconfig.replica_num = 3              # Number of replicas\nconfig.with_soft_pin = True         # Keep in memory longer\nconfig.preferred_segment = \"host:port\"  # Preferred location\n\nstore.put(\"key\", b\"value\", config)\n```\n\n### 2. Transfer Engine (High-Performance Data Transfer)\n\n**Import:**\n```python\nfrom mooncake.engine import TransferEngine, TransferOpcode, TransferNotify\n```\n\n**Basic Setup:**\n```python\nengine = TransferEngine()\nengine.initialize(\n    \"127.0.0.1:12345\",      # local_hostname\n    \"127.0.0.1:2379\",       # metadata_server (or \"etcd://...\")\n    \"tcp\",                  # protocol (\"tcp\" or \"rdma\")\n    \"\"                      # device_name (empty for all devices)\n)\n```\n\n**Buffer Management:**\n```python\n# Allocate managed buffer\nbuffer_size = 1024 * 1024  # 1MB\nbuffer_addr = engine.allocate_managed_buffer(buffer_size)\n\n# Write/read bytes\ndata = b\"Hello, Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_addr, data, len(data))\nread_data = engine.read_bytes_from_buffer(buffer_addr, len(data))\n\n# Free buffer\nengine.free_managed_buffer(buffer_addr, buffer_size)\n```\n\n**Data Transfer Operations:**\n```python\n# Synchronous write\nresult = engine.transfer_sync_write(\n    \"target_host:port\",     # target_hostname\n    local_buffer_addr,      # buffer\n    remote_buffer_addr,     # peer_buffer_address\n    data_length             # length\n)\n\n# Synchronous read\nresult = engine.transfer_sync_read(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Asynchronous write\nbatch_id = engine.transfer_submit_write(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Check status\nstatus = engine.transfer_check_status(batch_id)\n# Returns: 1 (completed), 0 (in progress), -1 (failed), -2 (timeout)\n```\n\n**Batch Transfer Operations:**\n```python\n# Batch synchronous write\nlocal_addrs = [addr1, addr2, addr3]\nremote_addrs = [remote1, remote2, remote3]\nlengths = [len1, len2, len3]\n\nresult = engine.batch_transfer_sync_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Batch asynchronous operations\nbatch_id = engine.batch_transfer_async_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Wait for completion\nresult = engine.get_batch_transfer_status([batch_id])\n```\n\n**Memory Registration (for RDMA):**\n```python\nimport numpy as np\n\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nbuffer_size = buffer.nbytes\n\n# Register memory\nengine.register_memory(buffer_ptr, buffer_size)\n\n# Use buffer for transfers...\n\n# Unregister when done\nengine.unregister_memory(buffer_ptr)\n```\n\n### 3. Mooncake EP & Backend (Expert Parallelism)\n\n**Mooncake Backend (Fault-Tolerant Collectives):**\n```python\nimport torch\nimport torch.distributed as dist\nfrom mooncake import pg\n\n# Initialize with fault tolerance\nactive_ranks = torch.ones((world_size,), dtype=torch.int32, device=\"cuda\")\ndist.init_process_group(\n    backend=\"mooncake\",\n    rank=rank,\n    world_size=world_size,\n    pg_options=pg.MooncakeBackendOptions(active_ranks),\n)\n\n# Use standard PyTorch distributed APIs\ndist.all_gather(...)\ndist.all_reduce(...)\n\n# Check for failures\nassert active_ranks.all()  # Verify no ranks are broken\n```\n\n**Mooncake EP (Expert Parallelism):**\n```python\nfrom mooncake.mooncake_ep_buffer import Buffer\nimport torch.distributed as dist\n\n# Calculate buffer size\nnum_ep_buffer_bytes = Buffer.get_ep_buffer_size_hint(\n    num_max_dispatch_tokens_per_rank=1024,\n    hidden=4096,\n    num_ranks=8,\n    num_experts=64\n)\n\n# Create buffer (must be Mooncake Backend process group)\nbuffer = Buffer(group=dist.group.WORLD, num_ep_buffer_bytes=num_ep_buffer_bytes)\n\n# Dispatch/combine operations\nactive_ranks = torch.ones((num_ranks,), dtype=torch.int32, device=\"cuda\")\nbuffer.dispatch(..., active_ranks=active_ranks, timeout_us=1000000)\nbuffer.combine(..., active_ranks=active_ranks, timeout_us=1000000)\n```\n\n## Starting Services\n\n### Start Master Service (with HTTP metadata server)\n```bash\nmooncake_master \\\n  --enable_http_metadata_server=true \\\n  --http_metadata_server_host=0.0.0.0 \\\n  --http_metadata_server_port=8080 \\\n  --default_kv_lease_ttl=5000\n```\n\n### Using External etcd (Production)\n```bash\n# Start etcd\netcd --listen-client-urls http://0.0.0.0:2379 \\\n     --advertise-client-urls http://0.0.0.0:2379\n\n# Start master\nmooncake_master --default_kv_lease_ttl=5000\n```\n\n## Environment Variables\n\n### Transfer Engine\n- `MC_METADATA_SERVER`: Metadata server URL\n- `MC_FORCE_TCP`: Force TCP transport (set to \"true\")\n- `MC_LOG_LEVEL`: Logging level (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_MS_AUTO_DISC`: Enable RDMA device auto-discovery (set to \"1\")\n- `MC_MS_FILTERS`: Filter RDMA devices (e.g., \"mlx5_0,mlx5_2\")\n- `MC_TRANSFER_TIMEOUT`: Transfer timeout in seconds (default: 30)\n\n### Mooncake Store\n- `MC_STORE_CLUSTER_ID`: Cluster identifier (default: \"mooncake\")\n- `MC_STORE_USE_HUGEPAGE`: Enable hugepage support\n- `MC_STORE_MEMCPY`: Enable local memcpy optimization (set to \"1\")\n- `MC_STORE_CLIENT_METRIC`: Enable client metrics (enabled by default)\n- `MC_YLT_LOG_LEVEL`: Log level (trace/debug/info/warn/error/critical)\n- `MOONCAKE_STORE_CHECKSUM`: Enable diagnostic object-level CRC-64 checks (set to \"1\" before creating any writer or reader client)\n\n## Common Patterns\n\n### Pattern 1: Simple KV Store\n```python\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"tcp\", \"\", \"localhost:50051\")\n\n# Store and retrieve\nstore.put(\"config\", b'{\"model\": \"llama-7b\"}')\nconfig = store.get(\"config\")\n\nstore.close()\n```\n\n### Pattern 2: High-Performance Tensor Storage\n```python\nimport torch\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"rdma\", \"mlx5_0\", \"localhost:50051\")\n\n# Configure replication\nconfig = ReplicateConfig()\nconfig.replica_num = 2\nconfig.with_soft_pin = True\n\n# Store tensor with replication\ntensor = torch.randn(1000, 1000)\nstore.put_tensor(\"weights\", tensor, config)\n\n# Retrieve\nretrieved = store.get_tensor(\"weights\")\n\nstore.close()\n```\n\n### Pattern 3: Zero-Copy Batch Operations\n```python\nimport numpy as np\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 16*1024*1024, \"rdma\", \"\", \"localhost:50051\")\n\n# Prepare buffers\nnum_buffers = 10\nbuffers = [np.random.randn(1024*1024).astype(np.float32) for _ in range(num_buffers)]\nbuffer_ptrs = [buf.ctypes.data for buf in buffers]\nsizes = [buf.nbytes for buf in buffers]\n\n# Register all buffers\nfor ptr, size in zip(buffer_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\n# Batch put\nkeys = [f\"tensor_{i}\" for i in range(num_buffers)]\nresults = store.batch_put_from(keys, buffer_ptrs, sizes)\n\n# Batch get\nrecv_buffers = [np.empty(1024*1024, dtype=np.float32) for _ in range(num_buffers)]\nrecv_ptrs = [buf.ctypes.data for buf in recv_buffers]\n\nfor ptr, size in zip(recv_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\nresults = store.batch_get_into(keys, recv_ptrs, sizes)\n\n# Cleanup\nfor ptr in buffer_ptrs + recv_ptrs:\n    store.unregister_buffer(ptr)\n\nstore.close()\n```\n\n### Pattern 4: Transfer Engine Direct Transfer\n```python\nfrom mooncake.engine import TransferEngine\nimport numpy as np\n\n# Setup engines on both nodes\nengine = TransferEngine()\nengine.initialize(\"127.0.0.1:12345\", \"127.0.0.1:2379\", \"tcp\", \"\")\n\n# Allocate and register buffer\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nengine.register_memory(buffer_ptr, buffer.nbytes)\n\n# Get remote buffer address (from peer via metadata exchange)\nremote_addr = engine.get_first_buffer_address(\"target_host:port\")\n\n# Transfer data\ndata = b\"Hello from Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_ptr, data, len(data))\nresult = engine.transfer_sync_write(\"target_host:port\", buffer_ptr, remote_addr, len(data))\n\n# Cleanup\nengine.unregister_memory(buffer_ptr)\n```\n\n## Error Handling\n\nAll methods return status codes:\n- `0`: Success\n- Negative values: Error codes\n\nCommon checks:\n```python\n# Store operations\nresult = store.put(\"key\", b\"value\")\nif result != 0:\n    print(f\"Put failed with error code: {result}\")\n\n# Existence check\nexists = store.is_exist(\"key\")\nif exists == 1:\n    print(\"Key exists\")\nelif exists == 0:\n    print(\"Key not found\")\nelse:\n    print(\"Error checking existence\")\n\n# Transfer operations\nresult = engine.transfer_sync_write(...)\nif result == 0:\n    print(\"Transfer successful\")\nelse:\n    print(f\"Transfer failed with code: {result}\")\n```\n\n## Troubleshooting\n\n### Connection Issues\n```python\n# Force TCP for testing without RDMA\nimport os\nos.environ[\"MC_FORCE_TCP\"] = \"true\"\n\n# Enable verbose logging\nos.environ[\"MC_LOG_LEVEL\"] = \"0\"\nos.environ[\"MC_YLT_LOG_LEVEL\"] = \"debug\"\n```\n\n### Memory Issues\n```python\n# Check buffer registration before zero-copy ops\nresult = store.register_buffer(buffer_ptr, size)\nif result != 0:\n    raise RuntimeError(f\"Failed to register buffer: {result}\")\n```\n\n### Corrupted Data or Garbled Output\n```python\n# Set this before importing Mooncake or creating any Store client.\nimport os\nos.environ[\"MOONCAKE_STORE_CHECKSUM\"] = \"1\"\n\nfrom mooncake.store import MooncakeDistributedStore\n```\n\nEnable the switch on every writer and reader client process, then reproduce with full-object `put`/`upsert` and `get` operations. Treat `CHECKSUM_MISMATCH` (-801) as a failed read and do not use the destination buffer. Objects without checksum metadata and range reads are not verified. This mode scans object data, stages GPU buffers to host memory, and disables the local hot cache, so use it only for diagnosis.\n\n### Service Connectivity\n```bash\n# Check master is running\ncurl http://localhost:50051\n\n# Check metadata server\ncurl http://localhost:8080/metadata\n```\n\n## Best Practices\n\n1. **Always close stores**: Call `store.close()` when done\n2. **Register buffers for zero-copy**: Required for RDMA operations\n3. **Use batch operations**: Better throughput for multiple operations\n4. **Configure replication**: Use `ReplicateConfig` for important data\n5. **Use soft pinning**: For frequently accessed objects\n6. **Choose protocol wisely**: TCP for dev/test, RDMA for production\n7. **Monitor leases**: Objects have TTL, renew if needed\n8. **Handle errors**: Check return codes and handle failures\n\n## Quick Reference\n\n### Mooncake Store Methods\n- `setup()`: Initialize store\n- `put()`, `get()`: Basic operations\n- `put_batch()`, `get_batch()`: Batch operations\n- `put_from()`, `get_into()`: Zero-copy operations\n- `put_tensor()`, `get_tensor()`: PyTorch tensors\n- `register_buffer()`, `unregister_buffer()`: Buffer management\n- `is_exist()`, `remove()`: Metadata operations\n- `close()`: Cleanup\n\n### Transfer Engine Methods\n- `initialize()`: Setup engine\n- `allocate_managed_buffer()`, `free_managed_buffer()`: Buffer allocation\n- `transfer_sync_write()`, `transfer_sync_read()`: Synchronous transfers\n- `transfer_submit_write()`, `transfer_check_status()`: Async transfers\n- `batch_transfer_sync_write()`, `batch_transfer_sync_read()`: Batch sync\n- `batch_transfer_async_write()`, `get_batch_transfer_status()`: Batch async\n- `register_memory()`, `unregister_memory()`: Memory registration\n- `write_bytes_to_buffer()`, `read_bytes_from_buffer()`: Buffer I/O\n\n## Documentation Links\n\n- Full API Reference: https://kvcache-ai.github.io/Mooncake/\n- Quick Start: docs/source/getting_started/quick-start.md\n- Mooncake Store: docs/source/python-api-reference/mooncake-store.md\n- Transfer Engine: docs/source/python-api-reference/transfer-engine.md\n- Transfer Engine direct usage: docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects\n- EP Backend: docs/source/python-api-reference/ep-backend.md\n",".claude/skills/mooncake-troubleshoot/SKILL.md":"---\nname: mooncake-troubleshoot\ndescription: Automatically diagnose Mooncake deployment and runtime issues. Checks services (mooncake_master, metadata server), RDMA devices, environment variables, connectivity, memory limits, object integrity, and analyzes logs for common error patterns. Use when Mooncake deployment fails, services won't start, connections fail, data is corrupted or garbled, or you encounter runtime errors like \"Error from etcd client\", \"No matched device found\", \"Failed to register memory\", \"NO_AVAILABLE_HANDLE\", \"CHECKSUM_MISMATCH\", or any RDMA/networking issues. Also use when user asks to troubleshoot, debug, diagnose, or fix Mooncake problems.\n---\n\n# Mooncake Deployment Troubleshooting\n\nYou are a Mooncake deployment troubleshooting specialist. Your job is to systematically diagnose issues and provide actionable solutions based on the comprehensive troubleshooting knowledge from Mooncake documentation.\n\n## Diagnostic Strategy\n\nRun checks systematically, reporting findings as you go. Start with simple checks (services, connectivity) before diving into complex issues (RDMA, memory registration).\n\n### 1. Service Status Check\n\nCheck if critical services are running:\n\n```bash\n# Check mooncake_master\nps aux | grep mooncake_master | grep -v grep\n\n# Check port usage\nnetstat -tuln | grep -E '(50051|8080|2379|9003)'\n\n# If using etcd\nps aux | grep etcd | grep -v grep\n```\n\n**Common issues:**\n- `bind address already in use` → Port conflict, use different port with `--rpc_port`\n- Master not running → Check startup logs for errors\n\n### 2. Metadata Server Connectivity\n\nThe metadata server is critical for node discovery and coordination.\n\n```bash\n# Test etcd connectivity\ncurl -s http://127.0.0.1:2379/version\n\n# Or test custom metadata server\ncurl -s $MC_METADATA_SERVER\n\n# Check for proxy interference\necho \"http_proxy: $http_proxy\"\necho \"https_proxy: $https_proxy\"\n```\n\n**Common issues:**\n- `Error from etcd client` → Metadata server unreachable\n  - **Fix:** Ensure etcd is bound to `0.0.0.0` not `127.0.0.1`:\n    ```bash\n    etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n    ```\n  - **Fix:** Disable HTTP proxy:\n    ```bash\n    unset http_proxy https_proxy\n    ```\n\n### 3. Environment Variables Check\n\nVerify critical environment variables are set correctly:\n\n```bash\n# Display Mooncake variables\nenv | grep -E '^(MC_|MOONCAKE_STORE_CHECKSUM=)'\n\n# Key variables to check:\necho \"MC_METADATA_SERVER: $MC_METADATA_SERVER\"\necho \"MC_FORCE_TCP: $MC_FORCE_TCP\"\necho \"MC_LOG_LEVEL: $MC_LOG_LEVEL\"\necho \"MC_YLT_LOG_LEVEL: $MC_YLT_LOG_LEVEL\"\necho \"MC_MS_AUTO_DISC: $MC_MS_AUTO_DISC\"\necho \"MC_MS_FILTERS: $MC_MS_FILTERS\"\necho \"MC_GID_INDEX: $MC_GID_INDEX\"\necho \"MC_MTU: $MC_MTU\"\necho \"MC_IB_PORT: $MC_IB_PORT\"\necho \"MC_ENABLE_DEST_DEVICE_AFFINITY: $MC_ENABLE_DEST_DEVICE_AFFINITY\"\necho \"MOONCAKE_STORE_CHECKSUM: $MOONCAKE_STORE_CHECKSUM\"\n```\n\n**Key variables:**\n- `MC_METADATA_SERVER` - Metadata server URL (required)\n- `MC_FORCE_TCP=true` - Force TCP for testing without RDMA\n- `MC_LOG_LEVEL=0` - Enable verbose logging (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_YLT_LOG_LEVEL=debug` - yalantinglibs log level\n- `MC_MS_AUTO_DISC=1` - Enable topology auto-discovery (default)\n- `MC_MS_FILTERS` - Filter specific RDMA devices (e.g., \"mlx5_1,mlx5_2\")\n- `MC_GID_INDEX` - RDMA GID index (set if GID is all zeros)\n- `MC_MTU` - RDMA MTU size\n- `MC_ENABLE_DEST_DEVICE_AFFINITY=1` - Reduce QP creation (fix \"Failed to create QP\")\n- `MOONCAKE_STORE_CHECKSUM=1` - Enable diagnostic object-level CRC-64 checks; set before starting every writer and reader client process\n\n### 4. RDMA Device Check\n\nOnly run if RDMA is being used (skip if `MC_FORCE_TCP=true`):\n\n```bash\n# List RDMA devices\nibv_devices\n\n# Check device details and status\nibv_devinfo\n\n# Check for ACTIVE ports\nibv_devinfo | grep -A 10 \"state:\"\n\n# Check GID addresses (should NOT be all zeros)\nibv_devinfo | grep -A 20 \"GID\"\n\n# Check peer memory modules\nlsmod | grep peer_mem\nlsmod | grep nvidia_peer_mem\n\n# Check QP count (if \"Failed to create QP\" error)\nrdma resource show qp\n```\n\n**Common issues:**\n- `No matched device found` → RDMA device name in config doesn't exist\n  - **Fix:** Use `ibv_devices` to get correct device names\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n  - **Fix:** Check cable connections, verify with `ibv_devinfo | grep state`\n  - **Fix:** Try different port with `MC_IB_PORT` environment variable\n- GID all zeros → Wrong GID index\n  - **Fix:** Set `MC_GID_INDEX=1` (or 2, 3 depending on network)\n- `Failed to create QP: Cannot allocate memory` → Too many QPs created\n  - **Fix:** Set `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n\n### 5. Memory and Resource Limits\n\nCheck system limits that affect RDMA memory registration:\n\n```bash\n# Check ulimits\nulimit -a\n\n# Focus on max locked memory\nulimit -l\n\n# Check RDMA device memory limits\nibv_devinfo -v | grep max_mr_size\n\n# Check dmesg for memory errors\ndmesg -T | tail -50 | grep -i \"out of mr size\"\n```\n\n**Common issues:**\n- `Failed to register memory: Input/output error` → Memory registration limit exceeded\n  - **Diagnostic:** Check `max_mr_size` with `ibv_devinfo -v`\n  - **Fix:** Reduce memory allocation or split into smaller chunks\n- Cannot allocate memory → ulimit restriction\n  - **Fix:** Set unlimited locked memory:\n    ```bash\n    ulimit -l unlimited\n    ```\n  - **Permanent fix:** Add to `/etc/security/limits.conf`:\n    ```\n    * soft memlock unlimited\n    * hard memlock unlimited\n    ```\n\n### 6. Network Connectivity\n\nTest connectivity between nodes:\n\n```bash\n# Test basic RDMA connectivity\nib_write_bw -d <device_name> -R\n\n# On peer node:\nib_write_bw -d <device_name> -R <server_ip>\n\n# Test GPU Direct RDMA (if CUDA enabled)\nib_write_bw -d <device_name> -R -x gdr\n\n# On peer node:\nib_write_bw -d <device_name> -R -x gdr <server_ip>\n\n# Test DNS resolution\nnslookup <connectable_name>\nping <connectable_name>\n```\n\n**Common issues:**\n- `connection refused` → Incorrect `connectable_name` or `rpc_port`\n  - **Fix:** Ensure `connectable_name` is NOT loopback (127.0.0.1/localhost)\n  - **Fix:** Use actual LAN/WAN IP or valid hostname\n- `Failed to exchange handshake` → RDMA connection setup failure\n  - **Fix:** Verify MTU matches: set `MC_MTU` environment variable\n  - **Fix:** Verify GID is valid (not all zeros)\n  - **Fix:** Test with `ib_send_bw` between nodes first\n\n### 7. Log Analysis\n\nSearch logs for common error patterns and their meanings:\n\n**Metadata/Connectivity Errors:**\n- `Error from etcd client` → Cannot connect to metadata server\n- `ERR_METADATA` → Metadata server communication failed\n- `ERR_DNS` → Invalid `local_server_name` (not valid DNS/IP)\n\n**RDMA Errors:**\n- `No matched device found` → RDMA device name doesn't exist\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n- `Failed to exchange handshake description` → RDMA handshake failed\n- `Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num` → MTU/GID mismatch\n- `Failed to register memory` → Memory registration limit exceeded\n- `Failed to create QP` → Too many QPs, enable `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n- `Worker: Process failed for slice` → Network instability\n- `work request flushed error` → Cascading error (find first error)\n\n**Store Errors:**\n- `NO_AVAILABLE_HANDLE` (-200) → Memory pool exhausted\n  - **Fix:** Increase `global_segment_size` in setup\n  - **Fix:** Check eviction is working (look for eviction logs)\n- `LEASE_EXPIRED` (-707) → Lease expired during transfer\n  - **Fix:** Increase `default_kv_lease_ttl` in master startup\n- `OBJECT_NOT_FOUND` (-704) → Object doesn't exist\n- `SEGMENT_NOT_FOUND` (-101) → No available segments\n- `CHECKSUM_MISMATCH` (-801) → Full-object read differs from the checksum captured before the write\n  - **Action:** Treat the read as failed and do not use the destination buffer\n- `Failed to get description of XXX` → Segment name mismatch\n  - **Fix:** Ensure segment name matches `local_hostname` from peer\n\n**Port/Service Errors:**\n- `bind address already in use` → Port conflict\n  - **Fix:** Use different port: `--rpc_port=50052`\n\n### 8. Configuration Validation\n\nVerify configuration is correct:\n\n```bash\n# Check connectable_name is not loopback\nhostname -I\n\n# Verify master startup flags\nps aux | grep mooncake_master\n\n# Check if using correct protocol\nenv | grep MC_FORCE_TCP\n```\n\n**Critical checks:**\n- `connectable_name` must be non-loopback IP or valid hostname\n- MTU and GID configurations must match network environment\n- RDMA device names must exist on the machine\n- Ports must not be in use by other services\n\n### 9. Object Integrity Diagnostics\n\nWhen the user reports corrupted data or garbled output that may originate from Mooncake Store:\n\n1. Verify that the Store clients, primary master, and standby master use checksum-capable binaries from the same version.\n2. Set `MOONCAKE_STORE_CHECKSUM=1` before starting every writer and reader client process, then restart or recreate existing clients.\n3. Reproduce with full-object `put`/`upsert` and `get` operations. Range reads, including `get_into_ranges`, are not covered.\n4. If `CHECKSUM_MISMATCH` (-801) is returned, the covered read differs from the checksum captured over the writer's source object. Treat the read as failed and do not consume the destination buffer.\n5. If `object_checksum_absent` appears at VLOG(1), metadata has no checksum and verification was skipped. This can occur for existing objects or objects written by clients without the switch.\n6. If verification succeeds, continue investigating corruption outside the covered Store path. A checksum mismatch identifies a difference but does not by itself distinguish transfer corruption from storage corruption.\n7. Warn that this diagnostic mode scans all object data, stages GPU buffers to host memory, and disables the local hot cache. Disable it after diagnosis.\n\n## Error Code Quick Reference\n\n### Transfer Engine Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Normal execution | - |\n| -12 | ERR_ADDRESS_NOT_REGISTERED | Memory not registered | Register memory before use |\n| -14 | ERR_DEVICE_NOT_FOUND | RDMA device not found | Check device name with `ibv_devices` |\n| -16 | ERR_DNS | Invalid local_server_name | Use valid IP/hostname |\n| -19 | ERR_REJECT_HANDSHAKE | Peer rejected handshake | Check peer logs for reason |\n| -20 | ERR_METADATA | Metadata server unreachable | Check etcd/HTTP server |\n\n### Store Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Operation successful | - |\n| -200 | NO_AVAILABLE_HANDLE | Memory pool exhausted | Increase segment size |\n| -707 | LEASE_EXPIRED | Lease expired | Increase lease TTL |\n| -704 | OBJECT_NOT_FOUND | Object doesn't exist | Check object key |\n| -101 | SEGMENT_NOT_FOUND | No available segments | Check segment registration |\n| -801 | CHECKSUM_MISMATCH | Full-object data differs from its stored checksum | Reject the read buffer and investigate the Store data path |\n| -900 | RPC_FAIL | RPC failed | Check network/master |\n| -1000 | ETCD_OPERATION_ERROR | etcd operation failed | Check etcd status |\n\n## Output Format\n\nProvide a structured diagnostic report:\n\n```\n🔍 MOONCAKE DEPLOYMENT DIAGNOSTICS\n==================================\n\n✅ PASSED CHECKS:\n- Service status: mooncake_master running on port 50051\n- Metadata server: etcd accessible at http://127.0.0.1:2379\n- Environment: MC_METADATA_SERVER set correctly\n- [other passing checks]\n\n❌ FAILED CHECKS:\n- RDMA device: mlx5_0 port not ACTIVE (state: PORT_DOWN)\n- Memory limits: max locked memory is 64KB (too low)\n- [other failures with specific error messages]\n\n⚠️  WARNINGS:\n- GID index not set, may cause connection issues\n- HTTP proxy variables set, may interfere with metadata server\n- [other potential issues]\n\n🔧 RECOMMENDED FIXES:\n\n1. Fix RDMA port status:\n   - Check physical cable connections\n   - Verify driver configuration\n   - Command: ibv_devinfo | grep -A 10 \"state:\"\n\n2. Increase memory limits:\n   ulimit -l unlimited\n   # Or permanently in /etc/security/limits.conf:\n   * soft memlock unlimited\n   * hard memlock unlimited\n\n3. Set GID index:\n   export MC_GID_INDEX=1\n\n4. Disable HTTP proxy:\n   unset http_proxy https_proxy\n\n📋 SUMMARY:\n[Brief 2-3 sentence conclusion about deployment health and next steps]\n```\n\n## Troubleshooting Workflow\n\n1. **Start simple**: Check services and basic connectivity first\n2. **Read logs carefully**: First error is usually root cause (subsequent errors cascade)\n3. **Test incrementally**: Use `MC_FORCE_TCP=true` to isolate RDMA issues\n4. **Verify basics**: Check connectable_name, ports, env vars before deep diving\n5. **Use diagnostic tools**: ibv_devices, ibv_devinfo, ib_write_bw, curl\n6. **Reference documentation**: Check error codes and troubleshooting guide\n\n## Quick Fix Commands\n\n**Start metadata server properly:**\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n```\n\n**Enable verbose logging:**\n```bash\nexport MC_LOG_LEVEL=0\nexport MC_YLT_LOG_LEVEL=debug\n```\n\n**Force TCP mode for testing:**\n```bash\nexport MC_FORCE_TCP=true\n```\n\n**Fix memory limits:**\n```bash\nulimit -l unlimited\n```\n\n**Fix too many QPs:**\n```bash\nexport MC_ENABLE_DEST_DEVICE_AFFINITY=1\n```\n\n**Fix GID issues:**\n```bash\nexport MC_GID_INDEX=1  # or 2, 3 depending on network\n```\n\n**Use different port:**\n```bash\nmooncake_master --rpc_port=50052\n```\n\nNow execute the diagnostic checks systematically and provide the structured report.\n","docs/AGENTS.md":"# AGENTS.md - Mooncake Documentation\n\nThis file gives coding agents the repo-local rules for modifying files under\n`docs/`. Keep `README.md` as the human-facing quickstart. Use this file for\nagent workflow, verification, and maintenance guidance.\n\n## Scope\n\n- Applies to changes under `docs/`, especially `docs/source/`.\n- Prefer small, reviewable documentation changes.\n- Do not rewrite unrelated pages, generated files, or formatting-only content.\n- Preserve existing documentation structure unless the user asks for a broader\n  reorganization.\n\n## Build and Preview\n\nRun documentation commands from the `docs` directory:\n\n```\ncd docs\n```\n\nInstall dependencies with `uv` when needed. The requirements file is\n`docs/requirements-docs.txt`; after `cd docs`, use the local filename. If there\nis an existing venv, prefer using the existing one. Otherwise, create one before\ninstalling dependencies:\n\n```\nuv venv\nuv pip install -r requirements-docs.txt\n```\n\nClean stale build output when validating navigation, generated API pages, or\ntheme behavior:\n\n```\nmake clean\n```\n\nBuild HTML before handing off user-visible documentation changes:\n\n```\nmake html\n```\n\nSet `locale` correctly before building when the change depends on localized\ncontent or translated output.\n\nServe the generated site for review:\n\n```\npython -m http.server -d build/html/\n```\n\nThe default URL is `http://localhost:8000`. If port 8000 is busy, choose another\navailable port.\n\n## Editing Guidance\n\n- Source pages live under `docs/source/`.\n- Keep links relative and Sphinx-compatible unless an external URL is required.\n- For navigation changes, inspect `docs/source/index.md` and the relevant\n  toctree before editing individual pages.\n- Use Sphinx-native structure for documentation behavior. Do not use client-side\n  JavaScript or post-render DOM patches for navigation or theme behavior.\n- If the requested behavior is not supported by Sphinx or the active theme,\n  prefer adding a Sphinx extension/plugin instead of patching rendered HTML.\n- When a page should be linked from content but excluded from the main sidebar,\n  use a content link plus appropriate Sphinx metadata such as `orphan: true`\n  instead of hiding rendered sidebar nodes.\n- Keep homepage toctree depth conservative. Do not increase `index.md` maxdepth\n  unless the user explicitly asks for deeper landing-page nesting.\n\n## Validation Checklist\n\n- Run `make html` for docs changes that affect rendered pages, navigation,\n  cross-references, or Sphinx configuration.\n- Check the generated HTML for the changed pages.\n- For sidebar or toctree changes, verify both the article body and left sidebar\n  render the intended entries.\n- If a local preview server is useful for review, start one from `docs/` with\n  `python -m http.server -d build/html/` or an alternate port.\n\n## Pull Request Hygiene\n\n- Keep docs-only changes narrowly scoped.\n- Review `git diff` before staging so generated files or hook-only formatting\n  changes do not leak into the PR.\n- If opening a PR, use the repository pull request template.\n- Use the repository PR title prefix rules from the root `AGENTS.md`.\n",".claude/skills/mooncake-ci-local/SKILL.md":"---\nname: mooncake-ci-local\ndescription: Run Mooncake pre-PR local validation through scripts/run_ci_test.sh. Use this skill whenever the user wants to validate a branch before opening or submitting a PR, run local CI, run ci test, check changes before PR, reproduce GitHub Actions locally, or force a full pre-submit verification. Trigger on phrases like \"提交 PR 前验证\", \"run ci test\", \"run local CI\", \"check my branch\", \"test before PR\", \"pre-submit validation\", and \"reproduce CI locally\".\n---\n\n# Mooncake Pre-PR Local Validation\n\nUse `bash scripts/run_ci_test.sh` as the default entry point. This is the single local lane for PR-before-submit validation, and it already coordinates the reproducible parts of GitHub Actions.\n\n## Default Entry Point\n\nWhen the user asks for any of the following, run the repo script first instead of reconstructing the workflow by hand:\n\n- 提交 PR 前本地验证\n- run ci test\n- run local CI\n- check my branch before PR\n- reproduce CI locally\n\nDefault command:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nWhat this script already covers:\n\n- GitHub-like `paths-filter` against `origin/main`\n- `typos`\n- `scripts/code_format.sh --check`\n- default CMake configure/build/install in `build-ci-local`\n- `ctest`\n- wheel build in `build-wheel-local`\n- wheel installation validation\n- `scripts/run_tests.sh`\n- selected Python API and integration tests\n- per-stage summary and logs under `local_test/run-ci-logs/<timestamp>/`\n\n## Standard Agent Workflow\n\n1. Run `bash scripts/run_ci_test.sh` from the repo root unless the user explicitly asks for a narrower subset.\n2. Read the stage summary instead of dumping raw terminal output.\n3. Report these items back to the user:\n   - passed stages\n   - failed stages\n   - blocked stages\n   - unsupported stages\n   - whether `paths-filter` skipped downstream stages\n   - the log directory under `local_test/run-ci-logs/...`\n4. If there is a failure, inspect the corresponding stage log and summarize the root cause.\n\n## Common Options\n\nForce a full lane even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nUse another base ref:\n\n```bash\nbash scripts/run_ci_test.sh --base origin/main\n```\n\nAuto-install missing dependencies:\n\n```bash\nbash scripts/run_ci_test.sh --install-deps\n```\n\nKeep services running for follow-up debugging:\n\n```bash\nbash scripts/run_ci_test.sh --keep-services\n```\n\n## Minimal Example\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nIf the user wants to ignore changed-path optimization and force the full lane:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nSee also `.claude/skills/mooncake-ci-local/examples/minimal.md`.\n\n## How To Interpret Results\n\n- `passed`: the stage succeeded locally.\n- `failed`: the stage reproduced a real local failure and needs investigation.\n- `blocked`: local environment or dependency issue prevented execution.\n- `unsupported`: intentionally not run in the local lane because it needs external platforms, special hardware, or a non-default build.\n\nIf `paths-filter` skips downstream stages, explain that the current branch changed only non-source paths relative to the selected base.\n\n## Current Local Coverage\n\nIncluded by default:\n\n- spell check\n- code format check\n- default ASan CMake lane in `build-ci-local`\n- `ctest`\n- wheel build and installation test\n- `scripts/run_tests.sh`\n- selected Python API tests\n\nUnsupported by design in the default local lane:\n\n- Ascend jobs\n- T-one integration jobs\n- MUSA jobs\n- Docker image build jobs\n- CUDA 13 wheel jobs\n- PG-backend tests absent from the default wheel build\n- Python drain-http API stage in the local ASan lane\n\n## Targeted Reruns For Debugging\n\nUse targeted reruns only after the full script identifies a failing area, or when the user explicitly asks for a smaller scope.\n\nRerun a specific C++ test pattern:\n\n```bash\ncd build-ci-local\nMC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -R <pattern> --output-on-failure\n```\n\nRerun the Python wheel integration lane:\n\n```bash\nsource test_env/bin/activate\nMC_STORE_MEMCPY=false TEST_SSD_OFFLOAD_IN_EVICT=true ./scripts/run_tests.sh\n```\n\nRerun the safetensor unittest:\n\n```bash\nsource test_env/bin/activate\npython -m unittest mooncake-wheel.tests.test_safetensor_functions\n```\n\n## Notes For The Agent\n\n- Prefer the repo script over rebuilding the CI workflow step by step.\n- Preserve the separation between `build-ci-local` and `build-wheel-local`.\n- Summarize failing stages from their logs instead of pasting raw output.\n- If the user only asks whether the branch is safe before opening a PR, the default answer path is `bash scripts/run_ci_test.sh`.\n","mooncake-store/AGENTS.md":"# Mooncake Store Instructions\n\n## Random Numbers\n\n- Production code in `include/` and `src/` must use `include/random.h`; do not\n  create random engines, seeds, or distributions at call sites.\n- Extend `random.h` and its tests when an operation is missing. Keep bounded\n  sampling unbiased and reject invalid bounds.\n- Tests and benchmarks may use explicit seeds for reproducibility, but should\n  use the shared sampling helpers.\n- Do not migrate `src/cachelib_memory_allocator/` unless explicitly requested.\n- The shared engine is not cryptographically secure; do not use it for secrets\n  or authentication tokens.\n\n## Local-first Host Identity\n\n- Keep the placement host ID separate from the routable transfer endpoint.\n- `MOONCAKE_HOST_ID` takes priority after ASCII whitespace trimming. Unset,\n  empty, or whitespace-only values fall back to the existing `local_hostname`\n  behavior.\n- Normalize endpoint-shaped host IDs before rejecting loopback or wildcard\n  identities.\n",".claude/skills/mooncake-api/.claude-plugin/plugin.json":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-api\",\n  \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"python\", \"api\", \"transfer-engine\", \"store\"]\n}\n",".claude/skills/mooncake-ci-local/scripts/check-prerequisites.sh":"#!/bin/bash\n# Mooncake CI Local Test Prerequisites Check\n# Usage: bash check-prerequisites.sh\n# This script checks and auto-fixes all prerequisites for running Mooncake CI tests locally.\n\nset -e\n\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nYELLOW='\\033[1;33m'\nNC='\\033[0m'\n\necho \"🔍 Checking Mooncake CI test prerequisites...\"\n\n# 1. Check build directory\nif [ ! -f build/CMakeCache.txt ]; then\n  echo -e \"${RED}✗ Build directory not found or not built${NC}\"\n  echo \"  → Run: mkdir build && cd build && cmake .. && cmake --build .\"\n  exit 1\nfi\necho -e \"${GREEN}✓ Build exists${NC}\"\n\n# 2. Check mooncake installation\nif ! python -c \"import mooncake\" 2>/dev/null; then\n  echo -e \"${RED}✗ mooncake package not installed${NC}\"\n  echo \"  → Fixing: Installing mooncake package...\"\n  cd build && sudo cmake --install . && cd - >/dev/null\n  if ! python -c \"import mooncake\" 2>/dev/null; then\n    echo \"  → Alternative: pip install mooncake-wheel/dist/*.whl\"\n    exit 1\n  fi\n  echo -e \"${GREEN}✓ mooncake package installed${NC}\"\nelse\n  echo -e \"${GREEN}✓ mooncake package already installed${NC}\"\nfi\n\n# 3. Check ctest availability\nif ! command -v ctest &> /dev/null; then\n  echo -e \"${RED}✗ ctest not found${NC}\"\n  exit 1\nfi\necho -e \"${GREEN}✓ ctest available${NC}\"\n\n# 4. Kill and restart services (safest approach for local testing)\necho -e \"\\n${YELLOW}Cleaning up and restarting services...${NC}\"\npkill -f \"^etcd\" || true\npkill -f bootstrap_server.py || true\npkill -f mooncake_http_metadata_server || true\nsleep 1\n\n# 5. Start etcd\nif ! command -v etcd &> /dev/null; then\n  echo -e \"${YELLOW}⚠ etcd not found, installing...${NC}\"\n  ETCD_VER=v3.6.1\n  OS=$(uname -s | tr '[:upper:]' '[:lower:]')\n  ARCH=$(uname -m)\n  [ \"$ARCH\" = \"x86_64\" ] && ARCH=\"amd64\"\n  DOWNLOAD_URL=\"https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\"\n  echo \"  Downloading from: $DOWNLOAD_URL\"\n  cd /tmp\n  wget -q \"$DOWNLOAD_URL\" && tar xzf \"etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\" && \\\n    sudo mv \"etcd-${ETCD_VER}-${OS}-${ARCH}\"/etcd* /usr/local/bin/\n  cd - >/dev/null\n  echo -e \"${GREEN}✓ etcd installed${NC}\"\nfi\n\netcd --advertise-client-urls http://127.0.0.1:2379 --listen-client-urls http://127.0.0.1:2379 >/dev/null 2>&1 &\nETCD_PID=$!\nsleep 2\nif ! etcdctl --endpoints=http://127.0.0.1:2379 endpoint health &>/dev/null; then\n  echo -e \"${RED}✗ etcd failed to start${NC}\"\n  kill $ETCD_PID 2>/dev/null || true\n  exit 1\nfi\necho -e \"${GREEN}✓ etcd running (PID: $ETCD_PID)${NC}\"\n\n# 6. Start HTTP metadata server\nif [ -f \"mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py\" ]; then\n  cd mooncake-transfer-engine/example/http-metadata-server-python\n  pip install -q aiohttp 2>/dev/null || true\n  python ./bootstrap_server.py >/dev/null 2>&1 &\n  METADATA_PID=$!\n  cd - >/dev/null\n  sleep 1\n  if curl -s http://127.0.0.1:8080/metadata > /dev/null 2>&1; then\n    echo -e \"${GREEN}✓ HTTP Metadata server running (PID: $METADATA_PID)${NC}\"\n  else\n    echo -e \"${RED}✗ HTTP Metadata server failed to start${NC}\"\n    kill $METADATA_PID $ETCD_PID 2>/dev/null || true\n    exit 1\n  fi\nelse\n  echo -e \"${YELLOW}⚠ Metadata server script not found, skipping${NC}\"\nfi\n\necho -e \"\\n${GREEN}✅ All prerequisites ready!${NC}\"\necho \"Service PIDs: etcd=$ETCD_PID\"\n[ -n \"$METADATA_PID\" ] && echo \"Metadata server PID: $METADATA_PID\"\necho -e \"\\n${YELLOW}To kill services:${NC}\"\necho \"  pkill -f '^etcd'\"\necho \"  pkill -f bootstrap_server\"\n",".claude/skills/mooncake-ci-local/.claude-plugin/plugin.json":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-ci-local\",\n  \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"ci\", \"validation\", \"pre-pr\"]\n}\n",".claude/skills/mooncake-ci-local/examples/minimal.md":"# Minimal Example\n\nGoal: validate the current branch before opening or submitting a PR.\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nForce a full rerun even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nResult location:\n\n- `local_test/run-ci-logs/<timestamp>/`\n\nTypical report format:\n\n- passed stages\n- failed or blocked stages\n- unsupported stages\n- first actionable failure and its log path\n",".claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json":"# claude-plugin Plugin Manifest\n\nPath: `.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\n## `docs/` Directory Changes\n\n- Before modifying files under `docs/`, read `docs/AGENTS.md`.\n\n## Pull Request Guidelines\n\n- Follow `CONTRIBUTING.md` for PR title prefixes, RFC expectations, and\n  contribution workflow.\n- Before opening a PR for nontrivial work, check whether an existing issue or\n  open PR already covers the same change. If the work overlaps, explain the\n  difference instead of duplicating it.\n- Do not open low-value busywork PRs for isolated typo, style, or mechanical\n  changes unless they are part of a substantive requested change.\n- Use `.github/pull_request_template.md` when preparing a PR, and fill in the\n  relevant sections for description, module, type of change, testing,\n  checklist, and AI assistance disclosure.\n- For AI-assisted changes, make sure the human submitter has reviewed every\n  changed line and can defend the change end-to-end.\n- Before handoff, run pre-commit on the files touched by the change when the\n  toolchain is available (see `CONTRIBUTING.md` for the PR-scoped\n  `pre-commit run --files ...` command). Do not use\n  `pre-commit run --all-files` for routine PRs; if it rewrites unrelated\n  files, leave those edits out of the PR.\n- Keep PRs lean: review `git diff` before staging, and include only changes\n  required for the requested task.\n","isInternal":false,"tokens":313,"sizeBytes":1310},{"name":"AGENTS.md","path":"docs/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/docs/AGENTS.md","title":"docs - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md - Mooncake Documentation\n\nThis file gives coding agents the repo-local rules for modifying files under\n`docs/`. Keep `README.md` as the human-facing quickstart. Use this file for\nagent workflow, verification, and maintenance guidance.\n\n## Scope\n\n- Applies to changes under `docs/`, especially `docs/source/`.\n- Prefer small, reviewable documentation changes.\n- Do not rewrite unrelated pages, generated files, or formatting-only content.\n- Preserve existing documentation structure unless the user asks for a broader\n  reorganization.\n\n## Build and Preview\n\nRun documentation commands from the `docs` directory:\n\n```\ncd docs\n```\n\nInstall dependencies with `uv` when needed. The requirements file is\n`docs/requirements-docs.txt`; after `cd docs`, use the local filename. If there\nis an existing venv, prefer using the existing one. Otherwise, create one before\ninstalling dependencies:\n\n```\nuv venv\nuv pip install -r requirements-docs.txt\n```\n\nClean stale build output when validating navigation, generated API pages, or\ntheme behavior:\n\n```\nmake clean\n```\n\nBuild HTML before handing off user-visible documentation changes:\n\n```\nmake html\n```\n\nSet `locale` correctly before building when the change depends on localized\ncontent or translated output.\n\nServe the generated site for review:\n\n```\npython -m http.server -d build/html/\n```\n\nThe default URL is `http://localhost:8000`. If port 8000 is busy, choose another\navailable port.\n\n## Editing Guidance\n\n- Source pages live under `docs/source/`.\n- Keep links relative and Sphinx-compatible unless an external URL is required.\n- For navigation changes, inspect `docs/source/index.md` and the relevant\n  toctree before editing individual pages.\n- Use Sphinx-native structure for documentation behavior. Do not use client-side\n  JavaScript or post-render DOM patches for navigation or theme behavior.\n- If the requested behavior is not supported by Sphinx or the active theme,\n  prefer adding a Sphinx extension/plugin instead of patching rendered HTML.\n- When a page should be linked from content but excluded from the main sidebar,\n  use a content link plus appropriate Sphinx metadata such as `orphan: true`\n  instead of hiding rendered sidebar nodes.\n- Keep homepage toctree depth conservative. Do not increase `index.md` maxdepth\n  unless the user explicitly asks for deeper landing-page nesting.\n\n## Validation Checklist\n\n- Run `make html` for docs changes that affect rendered pages, navigation,\n  cross-references, or Sphinx configuration.\n- Check the generated HTML for the changed pages.\n- For sidebar or toctree changes, verify both the article body and left sidebar\n  render the intended entries.\n- If a local preview server is useful for review, start one from `docs/` with\n  `python -m http.server -d build/html/` or an alternate port.\n\n## Pull Request Hygiene\n\n- Keep docs-only changes narrowly scoped.\n- Review `git diff` before staging so generated files or hook-only formatting\n  changes do not leak into the PR.\n- If opening a PR, use the repository pull request template.\n- Use the repository PR title prefix rules from the root `AGENTS.md`.\n","isInternal":false,"tokens":667,"sizeBytes":3120},{"name":"AGENTS.md","path":"mooncake-store/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/mooncake-store/AGENTS.md","title":"mooncake-store - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# Mooncake Store Instructions\n\n## Random Numbers\n\n- Production code in `include/` and `src/` must use `include/random.h`; do not\n  create random engines, seeds, or distributions at call sites.\n- Extend `random.h` and its tests when an operation is missing. Keep bounded\n  sampling unbiased and reject invalid bounds.\n- Tests and benchmarks may use explicit seeds for reproducibility, but should\n  use the shared sampling helpers.\n- Do not migrate `src/cachelib_memory_allocator/` unless explicitly requested.\n- The shared engine is not cryptographically secure; do not use it for secrets\n  or authentication tokens.\n\n## Local-first Host Identity\n\n- Keep the placement host ID separate from the routable transfer endpoint.\n- `MOONCAKE_HOST_ID` takes priority after ASCII whitespace trimming. Unset,\n  empty, or whitespace-only values fall back to the existing `local_hostname`\n  behavior.\n- Normalize endpoint-shaped host IDs before rejecting loopback or wildcard\n  identities.\n","isInternal":false,"tokens":208,"sizeBytes":977},{"name":"SKILL.md","path":".claude/skills/mooncake-api/SKILL.md","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-api/SKILL.md","title":"mooncake-api","category":"anthropic-skill","format":"markdown","content":"---\nname: mooncake-api\ndescription: Help users work with the Mooncake Python APIs for distributed storage and high-performance data transfer. Use when working with Mooncake Store (distributed KV cache), Transfer Engine (RDMA/TCP transfers), service setup (master, metadata server), PyTorch tensors in the Store, zero-copy/buffer management, batch operations and replication, or Mooncake EP / Backend (Expert Parallelism). Trigger on questions about MooncakeDistributedStore, TransferEngine, put/get, put_tensor, register_buffer, ReplicateConfig, or the mooncake.store / mooncake.engine / mooncake.pg Python modules.\n---\n\n# Mooncake Python API Skill\n\nUse this skill to help users work with Mooncake Python APIs for distributed storage and high-performance data transfer.\n\n## When to Use This Skill\n\nUse this skill when users ask about:\n- Using Mooncake Store for distributed KV cache storage\n- Using Transfer Engine for RDMA/TCP data transfers\n- Setting up Mooncake services (master, metadata server)\n- Working with PyTorch tensors in Mooncake Store\n- Zero-copy operations and buffer management\n- Batch operations and replication configuration\n- Mooncake EP (Expert Parallelism) and Mooncake Backend\n- Troubleshooting Mooncake Python API issues\n\n## Routing Guidance\n\n- For most vLLM and SGLang users, start from `docs/source/getting_started/quick-start.md`.\n- For PD disaggregation, direct users to the SGLang/vLLM integration guides listed in Quick Start. Those guides own the serving-framework configuration.\n- For Mooncake Store integrations, direct users to the SGLang/vLLM Store setup guides listed in Quick Start. Do not duplicate `mooncake_master` startup commands in general API answers unless the user is working outside those frameworks.\n- For direct low-level Transfer Engine usage, use `docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects`.\n- For API signatures and method details, use the Python API references under `docs/source/python-api-reference/`.\n\n## Core Components\n\n### 1. Mooncake Store (Distributed KV Cache)\n\n**Import:**\n```python\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n```\n\n**Basic Setup:**\n```python\nstore = MooncakeDistributedStore()\nstore.setup(\n    \"localhost\",                          # local_hostname\n    \"http://localhost:8080/metadata\",     # metadata_server\n    512*1024*1024,                        # global_segment_size (512MB)\n    128*1024*1024,                        # local_buffer_size (128MB)\n    \"tcp\",                                # protocol (\"tcp\" or \"rdma\")\n    \"\",                                   # rdma_devices (empty for auto-select)\n    \"localhost:50051\"                     # master_server_address\n)\n```\n\n**Common Operations:**\n```python\n# Put/Get\nstore.put(\"key\", b\"value\")\ndata = store.get(\"key\")\n\n# Batch operations\nstore.put_batch([\"key1\", \"key2\"], [b\"val1\", b\"val2\"])\nvalues = store.get_batch([\"key1\", \"key2\"])\n\n# Check existence\nexists = store.is_exist(\"key\")  # Returns 1 (exists), 0 (not exists), -1 (error)\n\n# Remove\nstore.remove(\"key\")\nstore.remove_by_regex(\"^prefix_.*\")\nstore.remove_all()\n\n# Cleanup\nstore.close()\n```\n\n**Zero-Copy Operations (Advanced):**\n```python\nimport numpy as np\n\n# Create and register buffer\nbuffer = np.zeros(100*1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nstore.register_buffer(buffer_ptr, buffer.nbytes)\n\n# Zero-copy put\nstore.put_from(\"key\", buffer_ptr, buffer.nbytes)\n\n# Zero-copy get\nrecv_buffer = np.empty(100*1024*1024, dtype=np.uint8)\nrecv_ptr = recv_buffer.ctypes.data\nstore.register_buffer(recv_ptr, recv_buffer.nbytes)\nbytes_read = store.get_into(\"key\", recv_ptr, recv_buffer.nbytes)\n\n# Cleanup\nstore.unregister_buffer(buffer_ptr)\nstore.unregister_buffer(recv_ptr)\n```\n\n**PyTorch Tensor Operations:**\n```python\nimport torch\n\n# Simple tensor operations\ntensor = torch.randn(100, 100)\nstore.put_tensor(\"my_tensor\", tensor)\nretrieved = store.get_tensor(\"my_tensor\")\n\n# Batch tensor operations\ntensors = [torch.randn(100, 100) for _ in range(3)]\nstore.batch_put_tensor([\"t1\", \"t2\", \"t3\"], tensors)\nretrieved_tensors = store.batch_get_tensor([\"t1\", \"t2\", \"t3\"])\n\n# Tensor Parallelism (TP) support\nstore.put_tensor_with_tp(\"model_weights\", tensor, tp_rank=0, tp_size=4, split_dim=0)\nshard = store.get_tensor_with_tp(\"model_weights\", tp_rank=0, tp_size=4)\n```\n\n**Replication Configuration:**\n```python\nconfig = ReplicateConfig()\nconfig.replica_num = 3              # Number of replicas\nconfig.with_soft_pin = True         # Keep in memory longer\nconfig.preferred_segment = \"host:port\"  # Preferred location\n\nstore.put(\"key\", b\"value\", config)\n```\n\n### 2. Transfer Engine (High-Performance Data Transfer)\n\n**Import:**\n```python\nfrom mooncake.engine import TransferEngine, TransferOpcode, TransferNotify\n```\n\n**Basic Setup:**\n```python\nengine = TransferEngine()\nengine.initialize(\n    \"127.0.0.1:12345\",      # local_hostname\n    \"127.0.0.1:2379\",       # metadata_server (or \"etcd://...\")\n    \"tcp\",                  # protocol (\"tcp\" or \"rdma\")\n    \"\"                      # device_name (empty for all devices)\n)\n```\n\n**Buffer Management:**\n```python\n# Allocate managed buffer\nbuffer_size = 1024 * 1024  # 1MB\nbuffer_addr = engine.allocate_managed_buffer(buffer_size)\n\n# Write/read bytes\ndata = b\"Hello, Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_addr, data, len(data))\nread_data = engine.read_bytes_from_buffer(buffer_addr, len(data))\n\n# Free buffer\nengine.free_managed_buffer(buffer_addr, buffer_size)\n```\n\n**Data Transfer Operations:**\n```python\n# Synchronous write\nresult = engine.transfer_sync_write(\n    \"target_host:port\",     # target_hostname\n    local_buffer_addr,      # buffer\n    remote_buffer_addr,     # peer_buffer_address\n    data_length             # length\n)\n\n# Synchronous read\nresult = engine.transfer_sync_read(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Asynchronous write\nbatch_id = engine.transfer_submit_write(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Check status\nstatus = engine.transfer_check_status(batch_id)\n# Returns: 1 (completed), 0 (in progress), -1 (failed), -2 (timeout)\n```\n\n**Batch Transfer Operations:**\n```python\n# Batch synchronous write\nlocal_addrs = [addr1, addr2, addr3]\nremote_addrs = [remote1, remote2, remote3]\nlengths = [len1, len2, len3]\n\nresult = engine.batch_transfer_sync_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Batch asynchronous operations\nbatch_id = engine.batch_transfer_async_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Wait for completion\nresult = engine.get_batch_transfer_status([batch_id])\n```\n\n**Memory Registration (for RDMA):**\n```python\nimport numpy as np\n\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nbuffer_size = buffer.nbytes\n\n# Register memory\nengine.register_memory(buffer_ptr, buffer_size)\n\n# Use buffer for transfers...\n\n# Unregister when done\nengine.unregister_memory(buffer_ptr)\n```\n\n### 3. Mooncake EP & Backend (Expert Parallelism)\n\n**Mooncake Backend (Fault-Tolerant Collectives):**\n```python\nimport torch\nimport torch.distributed as dist\nfrom mooncake import pg\n\n# Initialize with fault tolerance\nactive_ranks = torch.ones((world_size,), dtype=torch.int32, device=\"cuda\")\ndist.init_process_group(\n    backend=\"mooncake\",\n    rank=rank,\n    world_size=world_size,\n    pg_options=pg.MooncakeBackendOptions(active_ranks),\n)\n\n# Use standard PyTorch distributed APIs\ndist.all_gather(...)\ndist.all_reduce(...)\n\n# Check for failures\nassert active_ranks.all()  # Verify no ranks are broken\n```\n\n**Mooncake EP (Expert Parallelism):**\n```python\nfrom mooncake.mooncake_ep_buffer import Buffer\nimport torch.distributed as dist\n\n# Calculate buffer size\nnum_ep_buffer_bytes = Buffer.get_ep_buffer_size_hint(\n    num_max_dispatch_tokens_per_rank=1024,\n    hidden=4096,\n    num_ranks=8,\n    num_experts=64\n)\n\n# Create buffer (must be Mooncake Backend process group)\nbuffer = Buffer(group=dist.group.WORLD, num_ep_buffer_bytes=num_ep_buffer_bytes)\n\n# Dispatch/combine operations\nactive_ranks = torch.ones((num_ranks,), dtype=torch.int32, device=\"cuda\")\nbuffer.dispatch(..., active_ranks=active_ranks, timeout_us=1000000)\nbuffer.combine(..., active_ranks=active_ranks, timeout_us=1000000)\n```\n\n## Starting Services\n\n### Start Master Service (with HTTP metadata server)\n```bash\nmooncake_master \\\n  --enable_http_metadata_server=true \\\n  --http_metadata_server_host=0.0.0.0 \\\n  --http_metadata_server_port=8080 \\\n  --default_kv_lease_ttl=5000\n```\n\n### Using External etcd (Production)\n```bash\n# Start etcd\netcd --listen-client-urls http://0.0.0.0:2379 \\\n     --advertise-client-urls http://0.0.0.0:2379\n\n# Start master\nmooncake_master --default_kv_lease_ttl=5000\n```\n\n## Environment Variables\n\n### Transfer Engine\n- `MC_METADATA_SERVER`: Metadata server URL\n- `MC_FORCE_TCP`: Force TCP transport (set to \"true\")\n- `MC_LOG_LEVEL`: Logging level (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_MS_AUTO_DISC`: Enable RDMA device auto-discovery (set to \"1\")\n- `MC_MS_FILTERS`: Filter RDMA devices (e.g., \"mlx5_0,mlx5_2\")\n- `MC_TRANSFER_TIMEOUT`: Transfer timeout in seconds (default: 30)\n\n### Mooncake Store\n- `MC_STORE_CLUSTER_ID`: Cluster identifier (default: \"mooncake\")\n- `MC_STORE_USE_HUGEPAGE`: Enable hugepage support\n- `MC_STORE_MEMCPY`: Enable local memcpy optimization (set to \"1\")\n- `MC_STORE_CLIENT_METRIC`: Enable client metrics (enabled by default)\n- `MC_YLT_LOG_LEVEL`: Log level (trace/debug/info/warn/error/critical)\n- `MOONCAKE_STORE_CHECKSUM`: Enable diagnostic object-level CRC-64 checks (set to \"1\" before creating any writer or reader client)\n\n## Common Patterns\n\n### Pattern 1: Simple KV Store\n```python\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"tcp\", \"\", \"localhost:50051\")\n\n# Store and retrieve\nstore.put(\"config\", b'{\"model\": \"llama-7b\"}')\nconfig = store.get(\"config\")\n\nstore.close()\n```\n\n### Pattern 2: High-Performance Tensor Storage\n```python\nimport torch\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"rdma\", \"mlx5_0\", \"localhost:50051\")\n\n# Configure replication\nconfig = ReplicateConfig()\nconfig.replica_num = 2\nconfig.with_soft_pin = True\n\n# Store tensor with replication\ntensor = torch.randn(1000, 1000)\nstore.put_tensor(\"weights\", tensor, config)\n\n# Retrieve\nretrieved = store.get_tensor(\"weights\")\n\nstore.close()\n```\n\n### Pattern 3: Zero-Copy Batch Operations\n```python\nimport numpy as np\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 16*1024*1024, \"rdma\", \"\", \"localhost:50051\")\n\n# Prepare buffers\nnum_buffers = 10\nbuffers = [np.random.randn(1024*1024).astype(np.float32) for _ in range(num_buffers)]\nbuffer_ptrs = [buf.ctypes.data for buf in buffers]\nsizes = [buf.nbytes for buf in buffers]\n\n# Register all buffers\nfor ptr, size in zip(buffer_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\n# Batch put\nkeys = [f\"tensor_{i}\" for i in range(num_buffers)]\nresults = store.batch_put_from(keys, buffer_ptrs, sizes)\n\n# Batch get\nrecv_buffers = [np.empty(1024*1024, dtype=np.float32) for _ in range(num_buffers)]\nrecv_ptrs = [buf.ctypes.data for buf in recv_buffers]\n\nfor ptr, size in zip(recv_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\nresults = store.batch_get_into(keys, recv_ptrs, sizes)\n\n# Cleanup\nfor ptr in buffer_ptrs + recv_ptrs:\n    store.unregister_buffer(ptr)\n\nstore.close()\n```\n\n### Pattern 4: Transfer Engine Direct Transfer\n```python\nfrom mooncake.engine import TransferEngine\nimport numpy as np\n\n# Setup engines on both nodes\nengine = TransferEngine()\nengine.initialize(\"127.0.0.1:12345\", \"127.0.0.1:2379\", \"tcp\", \"\")\n\n# Allocate and register buffer\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nengine.register_memory(buffer_ptr, buffer.nbytes)\n\n# Get remote buffer address (from peer via metadata exchange)\nremote_addr = engine.get_first_buffer_address(\"target_host:port\")\n\n# Transfer data\ndata = b\"Hello from Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_ptr, data, len(data))\nresult = engine.transfer_sync_write(\"target_host:port\", buffer_ptr, remote_addr, len(data))\n\n# Cleanup\nengine.unregister_memory(buffer_ptr)\n```\n\n## Error Handling\n\nAll methods return status codes:\n- `0`: Success\n- Negative values: Error codes\n\nCommon checks:\n```python\n# Store operations\nresult = store.put(\"key\", b\"value\")\nif result != 0:\n    print(f\"Put failed with error code: {result}\")\n\n# Existence check\nexists = store.is_exist(\"key\")\nif exists == 1:\n    print(\"Key exists\")\nelif exists == 0:\n    print(\"Key not found\")\nelse:\n    print(\"Error checking existence\")\n\n# Transfer operations\nresult = engine.transfer_sync_write(...)\nif result == 0:\n    print(\"Transfer successful\")\nelse:\n    print(f\"Transfer failed with code: {result}\")\n```\n\n## Troubleshooting\n\n### Connection Issues\n```python\n# Force TCP for testing without RDMA\nimport os\nos.environ[\"MC_FORCE_TCP\"] = \"true\"\n\n# Enable verbose logging\nos.environ[\"MC_LOG_LEVEL\"] = \"0\"\nos.environ[\"MC_YLT_LOG_LEVEL\"] = \"debug\"\n```\n\n### Memory Issues\n```python\n# Check buffer registration before zero-copy ops\nresult = store.register_buffer(buffer_ptr, size)\nif result != 0:\n    raise RuntimeError(f\"Failed to register buffer: {result}\")\n```\n\n### Corrupted Data or Garbled Output\n```python\n# Set this before importing Mooncake or creating any Store client.\nimport os\nos.environ[\"MOONCAKE_STORE_CHECKSUM\"] = \"1\"\n\nfrom mooncake.store import MooncakeDistributedStore\n```\n\nEnable the switch on every writer and reader client process, then reproduce with full-object `put`/`upsert` and `get` operations. Treat `CHECKSUM_MISMATCH` (-801) as a failed read and do not use the destination buffer. Objects without checksum metadata and range reads are not verified. This mode scans object data, stages GPU buffers to host memory, and disables the local hot cache, so use it only for diagnosis.\n\n### Service Connectivity\n```bash\n# Check master is running\ncurl http://localhost:50051\n\n# Check metadata server\ncurl http://localhost:8080/metadata\n```\n\n## Best Practices\n\n1. **Always close stores**: Call `store.close()` when done\n2. **Register buffers for zero-copy**: Required for RDMA operations\n3. **Use batch operations**: Better throughput for multiple operations\n4. **Configure replication**: Use `ReplicateConfig` for important data\n5. **Use soft pinning**: For frequently accessed objects\n6. **Choose protocol wisely**: TCP for dev/test, RDMA for production\n7. **Monitor leases**: Objects have TTL, renew if needed\n8. **Handle errors**: Check return codes and handle failures\n\n## Quick Reference\n\n### Mooncake Store Methods\n- `setup()`: Initialize store\n- `put()`, `get()`: Basic operations\n- `put_batch()`, `get_batch()`: Batch operations\n- `put_from()`, `get_into()`: Zero-copy operations\n- `put_tensor()`, `get_tensor()`: PyTorch tensors\n- `register_buffer()`, `unregister_buffer()`: Buffer management\n- `is_exist()`, `remove()`: Metadata operations\n- `close()`: Cleanup\n\n### Transfer Engine Methods\n- `initialize()`: Setup engine\n- `allocate_managed_buffer()`, `free_managed_buffer()`: Buffer allocation\n- `transfer_sync_write()`, `transfer_sync_read()`: Synchronous transfers\n- `transfer_submit_write()`, `transfer_check_status()`: Async transfers\n- `batch_transfer_sync_write()`, `batch_transfer_sync_read()`: Batch sync\n- `batch_transfer_async_write()`, `get_batch_transfer_status()`: Batch async\n- `register_memory()`, `unregister_memory()`: Memory registration\n- `write_bytes_to_buffer()`, `read_bytes_from_buffer()`: Buffer I/O\n\n## Documentation Links\n\n- Full API Reference: https://kvcache-ai.github.io/Mooncake/\n- Quick Start: docs/source/getting_started/quick-start.md\n- Mooncake Store: docs/source/python-api-reference/mooncake-store.md\n- Transfer Engine: docs/source/python-api-reference/transfer-engine.md\n- Transfer Engine direct usage: docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects\n- EP Backend: docs/source/python-api-reference/ep-backend.md\n","frontmatter":{"name":"mooncake-api","description":"Help users work with the Mooncake Python APIs for distributed storage and high-performance data transfer. Use when working with Mooncake Store (distributed KV cache), Transfer Engine (RDMA/TCP transfers), service setup (master, metadata server), PyTorch tensors in the Store, zero-copy/buffer management, batch operations and replication, or Mooncake EP / Backend (Expert Parallelism). Trigger on questions about MooncakeDistributedStore, TransferEngine, put/get, put_tensor, register_buffer, ReplicateConfig, or the mooncake.store / mooncake.engine / mooncake.pg Python modules."},"isInternal":false,"tokens":4095,"sizeBytes":16395},{"name":"minimal.md","path":".claude/skills/mooncake-ci-local/examples/minimal.md","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-ci-local/examples/minimal.md","title":"Examples Skill","category":"anthropic-skill","format":"markdown","content":"# Minimal Example\n\nGoal: validate the current branch before opening or submitting a PR.\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nForce a full rerun even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nResult location:\n\n- `local_test/run-ci-logs/<timestamp>/`\n\nTypical report format:\n\n- passed stages\n- failed or blocked stages\n- unsupported stages\n- first actionable failure and its log path\n","isInternal":false,"tokens":136,"sizeBytes":552},{"name":"SKILL.md","path":".claude/skills/mooncake-ci-local/SKILL.md","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-ci-local/SKILL.md","title":"mooncake-ci-local","category":"anthropic-skill","format":"markdown","content":"---\nname: mooncake-ci-local\ndescription: Run Mooncake pre-PR local validation through scripts/run_ci_test.sh. Use this skill whenever the user wants to validate a branch before opening or submitting a PR, run local CI, run ci test, check changes before PR, reproduce GitHub Actions locally, or force a full pre-submit verification. Trigger on phrases like \"提交 PR 前验证\", \"run ci test\", \"run local CI\", \"check my branch\", \"test before PR\", \"pre-submit validation\", and \"reproduce CI locally\".\n---\n\n# Mooncake Pre-PR Local Validation\n\nUse `bash scripts/run_ci_test.sh` as the default entry point. This is the single local lane for PR-before-submit validation, and it already coordinates the reproducible parts of GitHub Actions.\n\n## Default Entry Point\n\nWhen the user asks for any of the following, run the repo script first instead of reconstructing the workflow by hand:\n\n- 提交 PR 前本地验证\n- run ci test\n- run local CI\n- check my branch before PR\n- reproduce CI locally\n\nDefault command:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nWhat this script already covers:\n\n- GitHub-like `paths-filter` against `origin/main`\n- `typos`\n- `scripts/code_format.sh --check`\n- default CMake configure/build/install in `build-ci-local`\n- `ctest`\n- wheel build in `build-wheel-local`\n- wheel installation validation\n- `scripts/run_tests.sh`\n- selected Python API and integration tests\n- per-stage summary and logs under `local_test/run-ci-logs/<timestamp>/`\n\n## Standard Agent Workflow\n\n1. Run `bash scripts/run_ci_test.sh` from the repo root unless the user explicitly asks for a narrower subset.\n2. Read the stage summary instead of dumping raw terminal output.\n3. Report these items back to the user:\n   - passed stages\n   - failed stages\n   - blocked stages\n   - unsupported stages\n   - whether `paths-filter` skipped downstream stages\n   - the log directory under `local_test/run-ci-logs/...`\n4. If there is a failure, inspect the corresponding stage log and summarize the root cause.\n\n## Common Options\n\nForce a full lane even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nUse another base ref:\n\n```bash\nbash scripts/run_ci_test.sh --base origin/main\n```\n\nAuto-install missing dependencies:\n\n```bash\nbash scripts/run_ci_test.sh --install-deps\n```\n\nKeep services running for follow-up debugging:\n\n```bash\nbash scripts/run_ci_test.sh --keep-services\n```\n\n## Minimal Example\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nIf the user wants to ignore changed-path optimization and force the full lane:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nSee also `.claude/skills/mooncake-ci-local/examples/minimal.md`.\n\n## How To Interpret Results\n\n- `passed`: the stage succeeded locally.\n- `failed`: the stage reproduced a real local failure and needs investigation.\n- `blocked`: local environment or dependency issue prevented execution.\n- `unsupported`: intentionally not run in the local lane because it needs external platforms, special hardware, or a non-default build.\n\nIf `paths-filter` skips downstream stages, explain that the current branch changed only non-source paths relative to the selected base.\n\n## Current Local Coverage\n\nIncluded by default:\n\n- spell check\n- code format check\n- default ASan CMake lane in `build-ci-local`\n- `ctest`\n- wheel build and installation test\n- `scripts/run_tests.sh`\n- selected Python API tests\n\nUnsupported by design in the default local lane:\n\n- Ascend jobs\n- T-one integration jobs\n- MUSA jobs\n- Docker image build jobs\n- CUDA 13 wheel jobs\n- PG-backend tests absent from the default wheel build\n- Python drain-http API stage in the local ASan lane\n\n## Targeted Reruns For Debugging\n\nUse targeted reruns only after the full script identifies a failing area, or when the user explicitly asks for a smaller scope.\n\nRerun a specific C++ test pattern:\n\n```bash\ncd build-ci-local\nMC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -R <pattern> --output-on-failure\n```\n\nRerun the Python wheel integration lane:\n\n```bash\nsource test_env/bin/activate\nMC_STORE_MEMCPY=false TEST_SSD_OFFLOAD_IN_EVICT=true ./scripts/run_tests.sh\n```\n\nRerun the safetensor unittest:\n\n```bash\nsource test_env/bin/activate\npython -m unittest mooncake-wheel.tests.test_safetensor_functions\n```\n\n## Notes For The Agent\n\n- Prefer the repo script over rebuilding the CI workflow step by step.\n- Preserve the separation between `build-ci-local` and `build-wheel-local`.\n- Summarize failing stages from their logs instead of pasting raw output.\n- If the user only asks whether the branch is safe before opening a PR, the default answer path is `bash scripts/run_ci_test.sh`.\n","frontmatter":{"name":"mooncake-ci-local","description":"Run Mooncake pre-PR local validation through scripts/run_ci_test.sh. Use this skill whenever the user wants to validate a branch before opening or submitting a PR, run local CI, run ci test, check changes before PR, reproduce GitHub Actions locally, or force a full pre-submit verification. Trigger on phrases like \"提交 PR 前验证\", \"run ci test\", \"run local CI\", \"check my branch\", \"test before PR\", \"pre-submit validation\", and \"reproduce CI locally\"."},"isInternal":false,"tokens":1113,"sizeBytes":4788},{"name":"SKILL.md","path":".claude/skills/mooncake-troubleshoot/SKILL.md","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-troubleshoot/SKILL.md","title":"mooncake-troubleshoot","category":"anthropic-skill","format":"markdown","content":"---\nname: mooncake-troubleshoot\ndescription: Automatically diagnose Mooncake deployment and runtime issues. Checks services (mooncake_master, metadata server), RDMA devices, environment variables, connectivity, memory limits, object integrity, and analyzes logs for common error patterns. Use when Mooncake deployment fails, services won't start, connections fail, data is corrupted or garbled, or you encounter runtime errors like \"Error from etcd client\", \"No matched device found\", \"Failed to register memory\", \"NO_AVAILABLE_HANDLE\", \"CHECKSUM_MISMATCH\", or any RDMA/networking issues. Also use when user asks to troubleshoot, debug, diagnose, or fix Mooncake problems.\n---\n\n# Mooncake Deployment Troubleshooting\n\nYou are a Mooncake deployment troubleshooting specialist. Your job is to systematically diagnose issues and provide actionable solutions based on the comprehensive troubleshooting knowledge from Mooncake documentation.\n\n## Diagnostic Strategy\n\nRun checks systematically, reporting findings as you go. Start with simple checks (services, connectivity) before diving into complex issues (RDMA, memory registration).\n\n### 1. Service Status Check\n\nCheck if critical services are running:\n\n```bash\n# Check mooncake_master\nps aux | grep mooncake_master | grep -v grep\n\n# Check port usage\nnetstat -tuln | grep -E '(50051|8080|2379|9003)'\n\n# If using etcd\nps aux | grep etcd | grep -v grep\n```\n\n**Common issues:**\n- `bind address already in use` → Port conflict, use different port with `--rpc_port`\n- Master not running → Check startup logs for errors\n\n### 2. Metadata Server Connectivity\n\nThe metadata server is critical for node discovery and coordination.\n\n```bash\n# Test etcd connectivity\ncurl -s http://127.0.0.1:2379/version\n\n# Or test custom metadata server\ncurl -s $MC_METADATA_SERVER\n\n# Check for proxy interference\necho \"http_proxy: $http_proxy\"\necho \"https_proxy: $https_proxy\"\n```\n\n**Common issues:**\n- `Error from etcd client` → Metadata server unreachable\n  - **Fix:** Ensure etcd is bound to `0.0.0.0` not `127.0.0.1`:\n    ```bash\n    etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n    ```\n  - **Fix:** Disable HTTP proxy:\n    ```bash\n    unset http_proxy https_proxy\n    ```\n\n### 3. Environment Variables Check\n\nVerify critical environment variables are set correctly:\n\n```bash\n# Display Mooncake variables\nenv | grep -E '^(MC_|MOONCAKE_STORE_CHECKSUM=)'\n\n# Key variables to check:\necho \"MC_METADATA_SERVER: $MC_METADATA_SERVER\"\necho \"MC_FORCE_TCP: $MC_FORCE_TCP\"\necho \"MC_LOG_LEVEL: $MC_LOG_LEVEL\"\necho \"MC_YLT_LOG_LEVEL: $MC_YLT_LOG_LEVEL\"\necho \"MC_MS_AUTO_DISC: $MC_MS_AUTO_DISC\"\necho \"MC_MS_FILTERS: $MC_MS_FILTERS\"\necho \"MC_GID_INDEX: $MC_GID_INDEX\"\necho \"MC_MTU: $MC_MTU\"\necho \"MC_IB_PORT: $MC_IB_PORT\"\necho \"MC_ENABLE_DEST_DEVICE_AFFINITY: $MC_ENABLE_DEST_DEVICE_AFFINITY\"\necho \"MOONCAKE_STORE_CHECKSUM: $MOONCAKE_STORE_CHECKSUM\"\n```\n\n**Key variables:**\n- `MC_METADATA_SERVER` - Metadata server URL (required)\n- `MC_FORCE_TCP=true` - Force TCP for testing without RDMA\n- `MC_LOG_LEVEL=0` - Enable verbose logging (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_YLT_LOG_LEVEL=debug` - yalantinglibs log level\n- `MC_MS_AUTO_DISC=1` - Enable topology auto-discovery (default)\n- `MC_MS_FILTERS` - Filter specific RDMA devices (e.g., \"mlx5_1,mlx5_2\")\n- `MC_GID_INDEX` - RDMA GID index (set if GID is all zeros)\n- `MC_MTU` - RDMA MTU size\n- `MC_ENABLE_DEST_DEVICE_AFFINITY=1` - Reduce QP creation (fix \"Failed to create QP\")\n- `MOONCAKE_STORE_CHECKSUM=1` - Enable diagnostic object-level CRC-64 checks; set before starting every writer and reader client process\n\n### 4. RDMA Device Check\n\nOnly run if RDMA is being used (skip if `MC_FORCE_TCP=true`):\n\n```bash\n# List RDMA devices\nibv_devices\n\n# Check device details and status\nibv_devinfo\n\n# Check for ACTIVE ports\nibv_devinfo | grep -A 10 \"state:\"\n\n# Check GID addresses (should NOT be all zeros)\nibv_devinfo | grep -A 20 \"GID\"\n\n# Check peer memory modules\nlsmod | grep peer_mem\nlsmod | grep nvidia_peer_mem\n\n# Check QP count (if \"Failed to create QP\" error)\nrdma resource show qp\n```\n\n**Common issues:**\n- `No matched device found` → RDMA device name in config doesn't exist\n  - **Fix:** Use `ibv_devices` to get correct device names\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n  - **Fix:** Check cable connections, verify with `ibv_devinfo | grep state`\n  - **Fix:** Try different port with `MC_IB_PORT` environment variable\n- GID all zeros → Wrong GID index\n  - **Fix:** Set `MC_GID_INDEX=1` (or 2, 3 depending on network)\n- `Failed to create QP: Cannot allocate memory` → Too many QPs created\n  - **Fix:** Set `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n\n### 5. Memory and Resource Limits\n\nCheck system limits that affect RDMA memory registration:\n\n```bash\n# Check ulimits\nulimit -a\n\n# Focus on max locked memory\nulimit -l\n\n# Check RDMA device memory limits\nibv_devinfo -v | grep max_mr_size\n\n# Check dmesg for memory errors\ndmesg -T | tail -50 | grep -i \"out of mr size\"\n```\n\n**Common issues:**\n- `Failed to register memory: Input/output error` → Memory registration limit exceeded\n  - **Diagnostic:** Check `max_mr_size` with `ibv_devinfo -v`\n  - **Fix:** Reduce memory allocation or split into smaller chunks\n- Cannot allocate memory → ulimit restriction\n  - **Fix:** Set unlimited locked memory:\n    ```bash\n    ulimit -l unlimited\n    ```\n  - **Permanent fix:** Add to `/etc/security/limits.conf`:\n    ```\n    * soft memlock unlimited\n    * hard memlock unlimited\n    ```\n\n### 6. Network Connectivity\n\nTest connectivity between nodes:\n\n```bash\n# Test basic RDMA connectivity\nib_write_bw -d <device_name> -R\n\n# On peer node:\nib_write_bw -d <device_name> -R <server_ip>\n\n# Test GPU Direct RDMA (if CUDA enabled)\nib_write_bw -d <device_name> -R -x gdr\n\n# On peer node:\nib_write_bw -d <device_name> -R -x gdr <server_ip>\n\n# Test DNS resolution\nnslookup <connectable_name>\nping <connectable_name>\n```\n\n**Common issues:**\n- `connection refused` → Incorrect `connectable_name` or `rpc_port`\n  - **Fix:** Ensure `connectable_name` is NOT loopback (127.0.0.1/localhost)\n  - **Fix:** Use actual LAN/WAN IP or valid hostname\n- `Failed to exchange handshake` → RDMA connection setup failure\n  - **Fix:** Verify MTU matches: set `MC_MTU` environment variable\n  - **Fix:** Verify GID is valid (not all zeros)\n  - **Fix:** Test with `ib_send_bw` between nodes first\n\n### 7. Log Analysis\n\nSearch logs for common error patterns and their meanings:\n\n**Metadata/Connectivity Errors:**\n- `Error from etcd client` → Cannot connect to metadata server\n- `ERR_METADATA` → Metadata server communication failed\n- `ERR_DNS` → Invalid `local_server_name` (not valid DNS/IP)\n\n**RDMA Errors:**\n- `No matched device found` → RDMA device name doesn't exist\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n- `Failed to exchange handshake description` → RDMA handshake failed\n- `Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num` → MTU/GID mismatch\n- `Failed to register memory` → Memory registration limit exceeded\n- `Failed to create QP` → Too many QPs, enable `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n- `Worker: Process failed for slice` → Network instability\n- `work request flushed error` → Cascading error (find first error)\n\n**Store Errors:**\n- `NO_AVAILABLE_HANDLE` (-200) → Memory pool exhausted\n  - **Fix:** Increase `global_segment_size` in setup\n  - **Fix:** Check eviction is working (look for eviction logs)\n- `LEASE_EXPIRED` (-707) → Lease expired during transfer\n  - **Fix:** Increase `default_kv_lease_ttl` in master startup\n- `OBJECT_NOT_FOUND` (-704) → Object doesn't exist\n- `SEGMENT_NOT_FOUND` (-101) → No available segments\n- `CHECKSUM_MISMATCH` (-801) → Full-object read differs from the checksum captured before the write\n  - **Action:** Treat the read as failed and do not use the destination buffer\n- `Failed to get description of XXX` → Segment name mismatch\n  - **Fix:** Ensure segment name matches `local_hostname` from peer\n\n**Port/Service Errors:**\n- `bind address already in use` → Port conflict\n  - **Fix:** Use different port: `--rpc_port=50052`\n\n### 8. Configuration Validation\n\nVerify configuration is correct:\n\n```bash\n# Check connectable_name is not loopback\nhostname -I\n\n# Verify master startup flags\nps aux | grep mooncake_master\n\n# Check if using correct protocol\nenv | grep MC_FORCE_TCP\n```\n\n**Critical checks:**\n- `connectable_name` must be non-loopback IP or valid hostname\n- MTU and GID configurations must match network environment\n- RDMA device names must exist on the machine\n- Ports must not be in use by other services\n\n### 9. Object Integrity Diagnostics\n\nWhen the user reports corrupted data or garbled output that may originate from Mooncake Store:\n\n1. Verify that the Store clients, primary master, and standby master use checksum-capable binaries from the same version.\n2. Set `MOONCAKE_STORE_CHECKSUM=1` before starting every writer and reader client process, then restart or recreate existing clients.\n3. Reproduce with full-object `put`/`upsert` and `get` operations. Range reads, including `get_into_ranges`, are not covered.\n4. If `CHECKSUM_MISMATCH` (-801) is returned, the covered read differs from the checksum captured over the writer's source object. Treat the read as failed and do not consume the destination buffer.\n5. If `object_checksum_absent` appears at VLOG(1), metadata has no checksum and verification was skipped. This can occur for existing objects or objects written by clients without the switch.\n6. If verification succeeds, continue investigating corruption outside the covered Store path. A checksum mismatch identifies a difference but does not by itself distinguish transfer corruption from storage corruption.\n7. Warn that this diagnostic mode scans all object data, stages GPU buffers to host memory, and disables the local hot cache. Disable it after diagnosis.\n\n## Error Code Quick Reference\n\n### Transfer Engine Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Normal execution | - |\n| -12 | ERR_ADDRESS_NOT_REGISTERED | Memory not registered | Register memory before use |\n| -14 | ERR_DEVICE_NOT_FOUND | RDMA device not found | Check device name with `ibv_devices` |\n| -16 | ERR_DNS | Invalid local_server_name | Use valid IP/hostname |\n| -19 | ERR_REJECT_HANDSHAKE | Peer rejected handshake | Check peer logs for reason |\n| -20 | ERR_METADATA | Metadata server unreachable | Check etcd/HTTP server |\n\n### Store Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Operation successful | - |\n| -200 | NO_AVAILABLE_HANDLE | Memory pool exhausted | Increase segment size |\n| -707 | LEASE_EXPIRED | Lease expired | Increase lease TTL |\n| -704 | OBJECT_NOT_FOUND | Object doesn't exist | Check object key |\n| -101 | SEGMENT_NOT_FOUND | No available segments | Check segment registration |\n| -801 | CHECKSUM_MISMATCH | Full-object data differs from its stored checksum | Reject the read buffer and investigate the Store data path |\n| -900 | RPC_FAIL | RPC failed | Check network/master |\n| -1000 | ETCD_OPERATION_ERROR | etcd operation failed | Check etcd status |\n\n## Output Format\n\nProvide a structured diagnostic report:\n\n```\n🔍 MOONCAKE DEPLOYMENT DIAGNOSTICS\n==================================\n\n✅ PASSED CHECKS:\n- Service status: mooncake_master running on port 50051\n- Metadata server: etcd accessible at http://127.0.0.1:2379\n- Environment: MC_METADATA_SERVER set correctly\n- [other passing checks]\n\n❌ FAILED CHECKS:\n- RDMA device: mlx5_0 port not ACTIVE (state: PORT_DOWN)\n- Memory limits: max locked memory is 64KB (too low)\n- [other failures with specific error messages]\n\n⚠️  WARNINGS:\n- GID index not set, may cause connection issues\n- HTTP proxy variables set, may interfere with metadata server\n- [other potential issues]\n\n🔧 RECOMMENDED FIXES:\n\n1. Fix RDMA port status:\n   - Check physical cable connections\n   - Verify driver configuration\n   - Command: ibv_devinfo | grep -A 10 \"state:\"\n\n2. Increase memory limits:\n   ulimit -l unlimited\n   # Or permanently in /etc/security/limits.conf:\n   * soft memlock unlimited\n   * hard memlock unlimited\n\n3. Set GID index:\n   export MC_GID_INDEX=1\n\n4. Disable HTTP proxy:\n   unset http_proxy https_proxy\n\n📋 SUMMARY:\n[Brief 2-3 sentence conclusion about deployment health and next steps]\n```\n\n## Troubleshooting Workflow\n\n1. **Start simple**: Check services and basic connectivity first\n2. **Read logs carefully**: First error is usually root cause (subsequent errors cascade)\n3. **Test incrementally**: Use `MC_FORCE_TCP=true` to isolate RDMA issues\n4. **Verify basics**: Check connectable_name, ports, env vars before deep diving\n5. **Use diagnostic tools**: ibv_devices, ibv_devinfo, ib_write_bw, curl\n6. **Reference documentation**: Check error codes and troubleshooting guide\n\n## Quick Fix Commands\n\n**Start metadata server properly:**\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n```\n\n**Enable verbose logging:**\n```bash\nexport MC_LOG_LEVEL=0\nexport MC_YLT_LOG_LEVEL=debug\n```\n\n**Force TCP mode for testing:**\n```bash\nexport MC_FORCE_TCP=true\n```\n\n**Fix memory limits:**\n```bash\nulimit -l unlimited\n```\n\n**Fix too many QPs:**\n```bash\nexport MC_ENABLE_DEST_DEVICE_AFFINITY=1\n```\n\n**Fix GID issues:**\n```bash\nexport MC_GID_INDEX=1  # or 2, 3 depending on network\n```\n\n**Use different port:**\n```bash\nmooncake_master --rpc_port=50052\n```\n\nNow execute the diagnostic checks systematically and provide the structured report.\n","frontmatter":{"name":"mooncake-troubleshoot","description":"Automatically diagnose Mooncake deployment and runtime issues. Checks services (mooncake_master, metadata server), RDMA devices, environment variables, connectivity, memory limits, object integrity, and analyzes logs for common error patterns. Use when Mooncake deployment fails, services won't start, connections fail, data is corrupted or garbled, or you encounter runtime errors like \"Error from etcd client\", \"No matched device found\", \"Failed to register memory\", \"NO_AVAILABLE_HANDLE\", \"CHECKSUM_MISMATCH\", or any RDMA/networking issues. Also use when user asks to troubleshoot, debug, diagnose, or fix Mooncake problems."},"isInternal":false,"tokens":3487,"sizeBytes":13599},{"name":"check-prerequisites.sh","path":".claude/skills/mooncake-ci-local/scripts/check-prerequisites.sh","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-ci-local/scripts/check-prerequisites.sh","title":"Claude Directive: check-prerequisites.sh","category":"claude-rule","format":"markdown","content":"#!/bin/bash\n# Mooncake CI Local Test Prerequisites Check\n# Usage: bash check-prerequisites.sh\n# This script checks and auto-fixes all prerequisites for running Mooncake CI tests locally.\n\nset -e\n\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nYELLOW='\\033[1;33m'\nNC='\\033[0m'\n\necho \"🔍 Checking Mooncake CI test prerequisites...\"\n\n# 1. Check build directory\nif [ ! -f build/CMakeCache.txt ]; then\n  echo -e \"${RED}✗ Build directory not found or not built${NC}\"\n  echo \"  → Run: mkdir build && cd build && cmake .. && cmake --build .\"\n  exit 1\nfi\necho -e \"${GREEN}✓ Build exists${NC}\"\n\n# 2. Check mooncake installation\nif ! python -c \"import mooncake\" 2>/dev/null; then\n  echo -e \"${RED}✗ mooncake package not installed${NC}\"\n  echo \"  → Fixing: Installing mooncake package...\"\n  cd build && sudo cmake --install . && cd - >/dev/null\n  if ! python -c \"import mooncake\" 2>/dev/null; then\n    echo \"  → Alternative: pip install mooncake-wheel/dist/*.whl\"\n    exit 1\n  fi\n  echo -e \"${GREEN}✓ mooncake package installed${NC}\"\nelse\n  echo -e \"${GREEN}✓ mooncake package already installed${NC}\"\nfi\n\n# 3. Check ctest availability\nif ! command -v ctest &> /dev/null; then\n  echo -e \"${RED}✗ ctest not found${NC}\"\n  exit 1\nfi\necho -e \"${GREEN}✓ ctest available${NC}\"\n\n# 4. Kill and restart services (safest approach for local testing)\necho -e \"\\n${YELLOW}Cleaning up and restarting services...${NC}\"\npkill -f \"^etcd\" || true\npkill -f bootstrap_server.py || true\npkill -f mooncake_http_metadata_server || true\nsleep 1\n\n# 5. Start etcd\nif ! command -v etcd &> /dev/null; then\n  echo -e \"${YELLOW}⚠ etcd not found, installing...${NC}\"\n  ETCD_VER=v3.6.1\n  OS=$(uname -s | tr '[:upper:]' '[:lower:]')\n  ARCH=$(uname -m)\n  [ \"$ARCH\" = \"x86_64\" ] && ARCH=\"amd64\"\n  DOWNLOAD_URL=\"https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\"\n  echo \"  Downloading from: $DOWNLOAD_URL\"\n  cd /tmp\n  wget -q \"$DOWNLOAD_URL\" && tar xzf \"etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\" && \\\n    sudo mv \"etcd-${ETCD_VER}-${OS}-${ARCH}\"/etcd* /usr/local/bin/\n  cd - >/dev/null\n  echo -e \"${GREEN}✓ etcd installed${NC}\"\nfi\n\netcd --advertise-client-urls http://127.0.0.1:2379 --listen-client-urls http://127.0.0.1:2379 >/dev/null 2>&1 &\nETCD_PID=$!\nsleep 2\nif ! etcdctl --endpoints=http://127.0.0.1:2379 endpoint health &>/dev/null; then\n  echo -e \"${RED}✗ etcd failed to start${NC}\"\n  kill $ETCD_PID 2>/dev/null || true\n  exit 1\nfi\necho -e \"${GREEN}✓ etcd running (PID: $ETCD_PID)${NC}\"\n\n# 6. Start HTTP metadata server\nif [ -f \"mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py\" ]; then\n  cd mooncake-transfer-engine/example/http-metadata-server-python\n  pip install -q aiohttp 2>/dev/null || true\n  python ./bootstrap_server.py >/dev/null 2>&1 &\n  METADATA_PID=$!\n  cd - >/dev/null\n  sleep 1\n  if curl -s http://127.0.0.1:8080/metadata > /dev/null 2>&1; then\n    echo -e \"${GREEN}✓ HTTP Metadata server running (PID: $METADATA_PID)${NC}\"\n  else\n    echo -e \"${RED}✗ HTTP Metadata server failed to start${NC}\"\n    kill $METADATA_PID $ETCD_PID 2>/dev/null || true\n    exit 1\n  fi\nelse\n  echo -e \"${YELLOW}⚠ Metadata server script not found, skipping${NC}\"\nfi\n\necho -e \"\\n${GREEN}✅ All prerequisites ready!${NC}\"\necho \"Service PIDs: etcd=$ETCD_PID\"\n[ -n \"$METADATA_PID\" ] && echo \"Metadata server PID: $METADATA_PID\"\necho -e \"\\n${YELLOW}To kill services:${NC}\"\necho \"  pkill -f '^etcd'\"\necho \"  pkill -f bootstrap_server\"\n","isInternal":false,"tokens":1129,"sizeBytes":3485},{"name":"marketplace.json","path":".claude-plugin/marketplace.json","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude-plugin/marketplace.json","title":"mooncake","category":"marketplace","format":"json","content":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-marketplace.json\",\n  \"name\": \"mooncake\",\n  \"owner\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"metadata\": {\n    \"description\": \"Built-in Claude Code skills for working with Mooncake: deployment troubleshooting, local CI validation, and the Python API.\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"mooncake-troubleshoot\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-troubleshoot\"\n      },\n      \"description\": \"Systematically diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, connectivity, logs) and propose fixes.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-troubleshoot.html\"\n    },\n    {\n      \"name\": \"mooncake-ci-local\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-ci-local\"\n      },\n      \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\"\n    },\n    {\n      \"name\": \"mooncake-api\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-api\"\n      },\n      \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\"\n    }\n  ]\n}\n","frontmatter":{"$schema":"https://json.schemastore.org/claude-code-marketplace.json","name":"mooncake","owner":{"name":"Mooncake Team"},"metadata":{"description":"Built-in Claude Code skills for working with Mooncake: deployment troubleshooting, local CI validation, and the Python API."},"plugins":[{"name":"mooncake-troubleshoot","source":{"source":"git-subdir","url":"https://github.com/kvcache-ai/Mooncake.git","path":".claude/skills/mooncake-troubleshoot"},"description":"Systematically diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, connectivity, logs) and propose fixes.","homepage":"https://kvcache-ai.github.io/Mooncake/skills/mooncake-troubleshoot.html"},{"name":"mooncake-ci-local","source":{"source":"git-subdir","url":"https://github.com/kvcache-ai/Mooncake.git","path":".claude/skills/mooncake-ci-local"},"description":"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.","homepage":"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html"},{"name":"mooncake-api","source":{"source":"git-subdir","url":"https://github.com/kvcache-ai/Mooncake.git","path":".claude/skills/mooncake-api"},"description":"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.","homepage":"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html"}]},"isInternal":false,"tokens":442,"sizeBytes":1643},{"name":"plugin.json","path":".claude/skills/mooncake-api/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-api/.claude-plugin/plugin.json","title":"mooncake-api","category":"plugin-manifest","format":"json","content":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-api\",\n  \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"python\", \"api\", \"transfer-engine\", \"store\"]\n}\n","frontmatter":{"$schema":"https://json.schemastore.org/claude-code-plugin-manifest.json","name":"mooncake-api","description":"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.","author":{"name":"Mooncake Team"},"homepage":"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html","repository":"https://github.com/kvcache-ai/Mooncake","license":"Apache-2.0","keywords":["mooncake","python","api","transfer-engine","store"]},"isInternal":false,"tokens":146,"sizeBytes":487},{"name":"plugin.json","path":".claude/skills/mooncake-ci-local/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-ci-local/.claude-plugin/plugin.json","title":"mooncake-ci-local","category":"plugin-manifest","format":"json","content":"{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-ci-local\",\n  \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"ci\", \"validation\", \"pre-pr\"]\n}\n","frontmatter":{"$schema":"https://json.schemastore.org/claude-code-plugin-manifest.json","name":"mooncake-ci-local","description":"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.","author":{"name":"Mooncake Team"},"homepage":"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html","repository":"https://github.com/kvcache-ai/Mooncake","license":"Apache-2.0","keywords":["mooncake","ci","validation","pre-pr"]},"isInternal":false,"tokens":153,"sizeBytes":526},{"name":"plugin.json","path":".claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json","title":"claude-plugin Plugin Manifest","category":"plugin-manifest","format":"json","content":"# claude-plugin Plugin Manifest\n\nPath: `.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json)","tokens":300,"sizeBytes":258}],"systemPromptSnippet":"<yakaai_skills repo=\"kvcache-ai/Mooncake\">\n<!-- File: AGENTS.md (Tokens: ~313 | Category: root-instruction) -->\n# AGENTS.md\n\n## `docs/` Directory Changes\n\n- Before modifying files under `docs/`, read `docs/AGENTS.md`.\n\n## Pull Request Guidelines\n\n- Follow `CONTRIBUTING.md` for PR title prefixes, RFC expectations, and\n  contribution workflow.\n- Before opening a PR for nontrivial work, check whether an existing issue or\n  open PR already covers the same change. If the work overlaps, explain the\n  difference instead of duplicating it.\n- Do not open low-value busywork PRs for isolated typo, style, or mechanical\n  changes unless they are part of a substantive requested change.\n- Use `.github/pull_request_template.md` when preparing a PR, and fill in the\n  relevant sections for description, module, type of change, testing,\n  checklist, and AI assistance disclosure.\n- For AI-assisted changes, make sure the human submitter has reviewed every\n  changed line and can defend the change end-to-end.\n- Before handoff, run pre-commit on the files touched by the change when the\n  toolchain is available (see `CONTRIBUTING.md` for the PR-scoped\n  `pre-commit run --files ...` command). Do not use\n  `pre-commit run --all-files` for routine PRs; if it rewrites unrelated\n  files, leave those edits out of the PR.\n- Keep PRs lean: review `git diff` before staging, and include only changes\n  required for the requested task.\n\n<!-- File: docs/AGENTS.md (Tokens: ~667 | Category: root-instruction) -->\n# AGENTS.md - Mooncake Documentation\n\nThis file gives coding agents the repo-local rules for modifying files under\n`docs/`. Keep `README.md` as the human-facing quickstart. Use this file for\nagent workflow, verification, and maintenance guidance.\n\n## Scope\n\n- Applies to changes under `docs/`, especially `docs/source/`.\n- Prefer small, reviewable documentation changes.\n- Do not rewrite unrelated pages, generated files, or formatting-only content.\n- Preserve existing documentation structure unless the user asks for a broader\n  reorganization.\n\n## Build and Preview\n\nRun documentation commands from the `docs` directory:\n\n```\ncd docs\n```\n\nInstall dependencies with `uv` when needed. The requirements file is\n`docs/requirements-docs.txt`; after `cd docs`, use the local filename. If there\nis an existing venv, prefer using the existing one. Otherwise, create one before\ninstalling dependencies:\n\n```\nuv venv\nuv pip install -r requirements-docs.txt\n```\n\nClean stale build output when validating navigation, generated API pages, or\ntheme behavior:\n\n```\nmake clean\n```\n\nBuild HTML before handing off user-visible documentation changes:\n\n```\nmake html\n```\n\nSet `locale` correctly before building when the change depends on localized\ncontent or translated output.\n\nServe the generated site for review:\n\n```\npython -m http.server -d build/html/\n```\n\nThe default URL is `http://localhost:8000`. If port 8000 is busy, choose another\navailable port.\n\n## Editing Guidance\n\n- Source pages live under `docs/source/`.\n- Keep links relative and Sphinx-compatible unless an external URL is required.\n- For navigation changes, inspect `docs/source/index.md` and the relevant\n  toctree before editing individual pages.\n- Use Sphinx-native structure for documentation behavior. Do not use client-side\n  JavaScript or post-render DOM patches for navigation or theme behavior.\n- If the requested behavior is not supported by Sphinx or the active theme,\n  prefer adding a Sphinx extension/plugin instead of patching rendered HTML.\n- When a page should be linked from content but excluded from the main sidebar,\n  use a content link plus appropriate Sphinx metadata such as `orphan: true`\n  instead of hiding rendered sidebar nodes.\n- Keep homepage toctree depth conservative. Do not increase `index.md` maxdepth\n  unless the user explicitly asks for deeper landing-page nesting.\n\n## Validation Checklist\n\n- Run `make html` for docs changes that affect rendered pages, navigation,\n  cross-references, or Sphinx configuration.\n- Check the generated HTML for the changed pages.\n- For sidebar or toctree changes, verify both the article body and left sidebar\n  render the intended entries.\n- If a local preview server is useful for review, start one from `docs/` with\n  `python -m http.server -d build/html/` or an alternate port.\n\n## Pull Request Hygiene\n\n- Keep docs-only changes narrowly scoped.\n- Review `git diff` before staging so generated files or hook-only formatting\n  changes do not leak into the PR.\n- If opening a PR, use the repository pull request template.\n- Use the repository PR title prefix rules from the root `AGENTS.md`.\n\n<!-- File: mooncake-store/AGENTS.md (Tokens: ~208 | Category: root-instruction) -->\n# Mooncake Store Instructions\n\n## Random Numbers\n\n- Production code in `include/` and `src/` must use `include/random.h`; do not\n  create random engines, seeds, or distributions at call sites.\n- Extend `random.h` and its tests when an operation is missing. Keep bounded\n  sampling unbiased and reject invalid bounds.\n- Tests and benchmarks may use explicit seeds for reproducibility, but should\n  use the shared sampling helpers.\n- Do not migrate `src/cachelib_memory_allocator/` unless explicitly requested.\n- The shared engine is not cryptographically secure; do not use it for secrets\n  or authentication tokens.\n\n## Local-first Host Identity\n\n- Keep the placement host ID separate from the routable transfer endpoint.\n- `MOONCAKE_HOST_ID` takes priority after ASCII whitespace trimming. Unset,\n  empty, or whitespace-only values fall back to the existing `local_hostname`\n  behavior.\n- Normalize endpoint-shaped host IDs before rejecting loopback or wildcard\n  identities.\n\n<!-- File: .claude/skills/mooncake-api/SKILL.md (Tokens: ~4095 | Category: anthropic-skill) -->\n---\nname: mooncake-api\ndescription: Help users work with the Mooncake Python APIs for distributed storage and high-performance data transfer. Use when working with Mooncake Store (distributed KV cache), Transfer Engine (RDMA/TCP transfers), service setup (master, metadata server), PyTorch tensors in the Store, zero-copy/buffer management, batch operations and replication, or Mooncake EP / Backend (Expert Parallelism). Trigger on questions about MooncakeDistributedStore, TransferEngine, put/get, put_tensor, register_buffer, ReplicateConfig, or the mooncake.store / mooncake.engine / mooncake.pg Python modules.\n---\n\n# Mooncake Python API Skill\n\nUse this skill to help users work with Mooncake Python APIs for distributed storage and high-performance data transfer.\n\n## When to Use This Skill\n\nUse this skill when users ask about:\n- Using Mooncake Store for distributed KV cache storage\n- Using Transfer Engine for RDMA/TCP data transfers\n- Setting up Mooncake services (master, metadata server)\n- Working with PyTorch tensors in Mooncake Store\n- Zero-copy operations and buffer management\n- Batch operations and replication configuration\n- Mooncake EP (Expert Parallelism) and Mooncake Backend\n- Troubleshooting Mooncake Python API issues\n\n## Routing Guidance\n\n- For most vLLM and SGLang users, start from `docs/source/getting_started/quick-start.md`.\n- For PD disaggregation, direct users to the SGLang/vLLM integration guides listed in Quick Start. Those guides own the serving-framework configuration.\n- For Mooncake Store integrations, direct users to the SGLang/vLLM Store setup guides listed in Quick Start. Do not duplicate `mooncake_master` startup commands in general API answers unless the user is working outside those frameworks.\n- For direct low-level Transfer Engine usage, use `docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects`.\n- For API signatures and method details, use the Python API references under `docs/source/python-api-reference/`.\n\n## Core Components\n\n### 1. Mooncake Store (Distributed KV Cache)\n\n**Import:**\n```python\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n```\n\n**Basic Setup:**\n```python\nstore = MooncakeDistributedStore()\nstore.setup(\n    \"localhost\",                          # local_hostname\n    \"http://localhost:8080/metadata\",     # metadata_server\n    512*1024*1024,                        # global_segment_size (512MB)\n    128*1024*1024,                        # local_buffer_size (128MB)\n    \"tcp\",                                # protocol (\"tcp\" or \"rdma\")\n    \"\",                                   # rdma_devices (empty for auto-select)\n    \"localhost:50051\"                     # master_server_address\n)\n```\n\n**Common Operations:**\n```python\n# Put/Get\nstore.put(\"key\", b\"value\")\ndata = store.get(\"key\")\n\n# Batch operations\nstore.put_batch([\"key1\", \"key2\"], [b\"val1\", b\"val2\"])\nvalues = store.get_batch([\"key1\", \"key2\"])\n\n# Check existence\nexists = store.is_exist(\"key\")  # Returns 1 (exists), 0 (not exists), -1 (error)\n\n# Remove\nstore.remove(\"key\")\nstore.remove_by_regex(\"^prefix_.*\")\nstore.remove_all()\n\n# Cleanup\nstore.close()\n```\n\n**Zero-Copy Operations (Advanced):**\n```python\nimport numpy as np\n\n# Create and register buffer\nbuffer = np.zeros(100*1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nstore.register_buffer(buffer_ptr, buffer.nbytes)\n\n# Zero-copy put\nstore.put_from(\"key\", buffer_ptr, buffer.nbytes)\n\n# Zero-copy get\nrecv_buffer = np.empty(100*1024*1024, dtype=np.uint8)\nrecv_ptr = recv_buffer.ctypes.data\nstore.register_buffer(recv_ptr, recv_buffer.nbytes)\nbytes_read = store.get_into(\"key\", recv_ptr, recv_buffer.nbytes)\n\n# Cleanup\nstore.unregister_buffer(buffer_ptr)\nstore.unregister_buffer(recv_ptr)\n```\n\n**PyTorch Tensor Operations:**\n```python\nimport torch\n\n# Simple tensor operations\ntensor = torch.randn(100, 100)\nstore.put_tensor(\"my_tensor\", tensor)\nretrieved = store.get_tensor(\"my_tensor\")\n\n# Batch tensor operations\ntensors = [torch.randn(100, 100) for _ in range(3)]\nstore.batch_put_tensor([\"t1\", \"t2\", \"t3\"], tensors)\nretrieved_tensors = store.batch_get_tensor([\"t1\", \"t2\", \"t3\"])\n\n# Tensor Parallelism (TP) support\nstore.put_tensor_with_tp(\"model_weights\", tensor, tp_rank=0, tp_size=4, split_dim=0)\nshard = store.get_tensor_with_tp(\"model_weights\", tp_rank=0, tp_size=4)\n```\n\n**Replication Configuration:**\n```python\nconfig = ReplicateConfig()\nconfig.replica_num = 3              # Number of replicas\nconfig.with_soft_pin = True         # Keep in memory longer\nconfig.preferred_segment = \"host:port\"  # Preferred location\n\nstore.put(\"key\", b\"value\", config)\n```\n\n### 2. Transfer Engine (High-Performance Data Transfer)\n\n**Import:**\n```python\nfrom mooncake.engine import TransferEngine, TransferOpcode, TransferNotify\n```\n\n**Basic Setup:**\n```python\nengine = TransferEngine()\nengine.initialize(\n    \"127.0.0.1:12345\",      # local_hostname\n    \"127.0.0.1:2379\",       # metadata_server (or \"etcd://...\")\n    \"tcp\",                  # protocol (\"tcp\" or \"rdma\")\n    \"\"                      # device_name (empty for all devices)\n)\n```\n\n**Buffer Management:**\n```python\n# Allocate managed buffer\nbuffer_size = 1024 * 1024  # 1MB\nbuffer_addr = engine.allocate_managed_buffer(buffer_size)\n\n# Write/read bytes\ndata = b\"Hello, Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_addr, data, len(data))\nread_data = engine.read_bytes_from_buffer(buffer_addr, len(data))\n\n# Free buffer\nengine.free_managed_buffer(buffer_addr, buffer_size)\n```\n\n**Data Transfer Operations:**\n```python\n# Synchronous write\nresult = engine.transfer_sync_write(\n    \"target_host:port\",     # target_hostname\n    local_buffer_addr,      # buffer\n    remote_buffer_addr,     # peer_buffer_address\n    data_length             # length\n)\n\n# Synchronous read\nresult = engine.transfer_sync_read(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Asynchronous write\nbatch_id = engine.transfer_submit_write(\n    \"target_host:port\",\n    local_buffer_addr,\n    remote_buffer_addr,\n    data_length\n)\n\n# Check status\nstatus = engine.transfer_check_status(batch_id)\n# Returns: 1 (completed), 0 (in progress), -1 (failed), -2 (timeout)\n```\n\n**Batch Transfer Operations:**\n```python\n# Batch synchronous write\nlocal_addrs = [addr1, addr2, addr3]\nremote_addrs = [remote1, remote2, remote3]\nlengths = [len1, len2, len3]\n\nresult = engine.batch_transfer_sync_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Batch asynchronous operations\nbatch_id = engine.batch_transfer_async_write(\n    \"target_host:port\",\n    local_addrs,\n    remote_addrs,\n    lengths\n)\n\n# Wait for completion\nresult = engine.get_batch_transfer_status([batch_id])\n```\n\n**Memory Registration (for RDMA):**\n```python\nimport numpy as np\n\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nbuffer_size = buffer.nbytes\n\n# Register memory\nengine.register_memory(buffer_ptr, buffer_size)\n\n# Use buffer for transfers...\n\n# Unregister when done\nengine.unregister_memory(buffer_ptr)\n```\n\n### 3. Mooncake EP & Backend (Expert Parallelism)\n\n**Mooncake Backend (Fault-Tolerant Collectives):**\n```python\nimport torch\nimport torch.distributed as dist\nfrom mooncake import pg\n\n# Initialize with fault tolerance\nactive_ranks = torch.ones((world_size,), dtype=torch.int32, device=\"cuda\")\ndist.init_process_group(\n    backend=\"mooncake\",\n    rank=rank,\n    world_size=world_size,\n    pg_options=pg.MooncakeBackendOptions(active_ranks),\n)\n\n# Use standard PyTorch distributed APIs\ndist.all_gather(...)\ndist.all_reduce(...)\n\n# Check for failures\nassert active_ranks.all()  # Verify no ranks are broken\n```\n\n**Mooncake EP (Expert Parallelism):**\n```python\nfrom mooncake.mooncake_ep_buffer import Buffer\nimport torch.distributed as dist\n\n# Calculate buffer size\nnum_ep_buffer_bytes = Buffer.get_ep_buffer_size_hint(\n    num_max_dispatch_tokens_per_rank=1024,\n    hidden=4096,\n    num_ranks=8,\n    num_experts=64\n)\n\n# Create buffer (must be Mooncake Backend process group)\nbuffer = Buffer(group=dist.group.WORLD, num_ep_buffer_bytes=num_ep_buffer_bytes)\n\n# Dispatch/combine operations\nactive_ranks = torch.ones((num_ranks,), dtype=torch.int32, device=\"cuda\")\nbuffer.dispatch(..., active_ranks=active_ranks, timeout_us=1000000)\nbuffer.combine(..., active_ranks=active_ranks, timeout_us=1000000)\n```\n\n## Starting Services\n\n### Start Master Service (with HTTP metadata server)\n```bash\nmooncake_master \\\n  --enable_http_metadata_server=true \\\n  --http_metadata_server_host=0.0.0.0 \\\n  --http_metadata_server_port=8080 \\\n  --default_kv_lease_ttl=5000\n```\n\n### Using External etcd (Production)\n```bash\n# Start etcd\netcd --listen-client-urls http://0.0.0.0:2379 \\\n     --advertise-client-urls http://0.0.0.0:2379\n\n# Start master\nmooncake_master --default_kv_lease_ttl=5000\n```\n\n## Environment Variables\n\n### Transfer Engine\n- `MC_METADATA_SERVER`: Metadata server URL\n- `MC_FORCE_TCP`: Force TCP transport (set to \"true\")\n- `MC_LOG_LEVEL`: Logging level (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_MS_AUTO_DISC`: Enable RDMA device auto-discovery (set to \"1\")\n- `MC_MS_FILTERS`: Filter RDMA devices (e.g., \"mlx5_0,mlx5_2\")\n- `MC_TRANSFER_TIMEOUT`: Transfer timeout in seconds (default: 30)\n\n### Mooncake Store\n- `MC_STORE_CLUSTER_ID`: Cluster identifier (default: \"mooncake\")\n- `MC_STORE_USE_HUGEPAGE`: Enable hugepage support\n- `MC_STORE_MEMCPY`: Enable local memcpy optimization (set to \"1\")\n- `MC_STORE_CLIENT_METRIC`: Enable client metrics (enabled by default)\n- `MC_YLT_LOG_LEVEL`: Log level (trace/debug/info/warn/error/critical)\n- `MOONCAKE_STORE_CHECKSUM`: Enable diagnostic object-level CRC-64 checks (set to \"1\" before creating any writer or reader client)\n\n## Common Patterns\n\n### Pattern 1: Simple KV Store\n```python\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"tcp\", \"\", \"localhost:50051\")\n\n# Store and retrieve\nstore.put(\"config\", b'{\"model\": \"llama-7b\"}')\nconfig = store.get(\"config\")\n\nstore.close()\n```\n\n### Pattern 2: High-Performance Tensor Storage\n```python\nimport torch\nfrom mooncake.store import MooncakeDistributedStore, ReplicateConfig\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 128*1024*1024, \"rdma\", \"mlx5_0\", \"localhost:50051\")\n\n# Configure replication\nconfig = ReplicateConfig()\nconfig.replica_num = 2\nconfig.with_soft_pin = True\n\n# Store tensor with replication\ntensor = torch.randn(1000, 1000)\nstore.put_tensor(\"weights\", tensor, config)\n\n# Retrieve\nretrieved = store.get_tensor(\"weights\")\n\nstore.close()\n```\n\n### Pattern 3: Zero-Copy Batch Operations\n```python\nimport numpy as np\nfrom mooncake.store import MooncakeDistributedStore\n\nstore = MooncakeDistributedStore()\nstore.setup(\"localhost\", \"http://localhost:8080/metadata\",\n            512*1024*1024, 16*1024*1024, \"rdma\", \"\", \"localhost:50051\")\n\n# Prepare buffers\nnum_buffers = 10\nbuffers = [np.random.randn(1024*1024).astype(np.float32) for _ in range(num_buffers)]\nbuffer_ptrs = [buf.ctypes.data for buf in buffers]\nsizes = [buf.nbytes for buf in buffers]\n\n# Register all buffers\nfor ptr, size in zip(buffer_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\n# Batch put\nkeys = [f\"tensor_{i}\" for i in range(num_buffers)]\nresults = store.batch_put_from(keys, buffer_ptrs, sizes)\n\n# Batch get\nrecv_buffers = [np.empty(1024*1024, dtype=np.float32) for _ in range(num_buffers)]\nrecv_ptrs = [buf.ctypes.data for buf in recv_buffers]\n\nfor ptr, size in zip(recv_ptrs, sizes):\n    store.register_buffer(ptr, size)\n\nresults = store.batch_get_into(keys, recv_ptrs, sizes)\n\n# Cleanup\nfor ptr in buffer_ptrs + recv_ptrs:\n    store.unregister_buffer(ptr)\n\nstore.close()\n```\n\n### Pattern 4: Transfer Engine Direct Transfer\n```python\nfrom mooncake.engine import TransferEngine\nimport numpy as np\n\n# Setup engines on both nodes\nengine = TransferEngine()\nengine.initialize(\"127.0.0.1:12345\", \"127.0.0.1:2379\", \"tcp\", \"\")\n\n# Allocate and register buffer\nbuffer = np.ones(1024*1024, dtype=np.uint8)\nbuffer_ptr = buffer.ctypes.data\nengine.register_memory(buffer_ptr, buffer.nbytes)\n\n# Get remote buffer address (from peer via metadata exchange)\nremote_addr = engine.get_first_buffer_address(\"target_host:port\")\n\n# Transfer data\ndata = b\"Hello from Transfer Engine!\"\nengine.write_bytes_to_buffer(buffer_ptr, data, len(data))\nresult = engine.transfer_sync_write(\"target_host:port\", buffer_ptr, remote_addr, len(data))\n\n# Cleanup\nengine.unregister_memory(buffer_ptr)\n```\n\n## Error Handling\n\nAll methods return status codes:\n- `0`: Success\n- Negative values: Error codes\n\nCommon checks:\n```python\n# Store operations\nresult = store.put(\"key\", b\"value\")\nif result != 0:\n    print(f\"Put failed with error code: {result}\")\n\n# Existence check\nexists = store.is_exist(\"key\")\nif exists == 1:\n    print(\"Key exists\")\nelif exists == 0:\n    print(\"Key not found\")\nelse:\n    print(\"Error checking existence\")\n\n# Transfer operations\nresult = engine.transfer_sync_write(...)\nif result == 0:\n    print(\"Transfer successful\")\nelse:\n    print(f\"Transfer failed with code: {result}\")\n```\n\n## Troubleshooting\n\n### Connection Issues\n```python\n# Force TCP for testing without RDMA\nimport os\nos.environ[\"MC_FORCE_TCP\"] = \"true\"\n\n# Enable verbose logging\nos.environ[\"MC_LOG_LEVEL\"] = \"0\"\nos.environ[\"MC_YLT_LOG_LEVEL\"] = \"debug\"\n```\n\n### Memory Issues\n```python\n# Check buffer registration before zero-copy ops\nresult = store.register_buffer(buffer_ptr, size)\nif result != 0:\n    raise RuntimeError(f\"Failed to register buffer: {result}\")\n```\n\n### Corrupted Data or Garbled Output\n```python\n# Set this before importing Mooncake or creating any Store client.\nimport os\nos.environ[\"MOONCAKE_STORE_CHECKSUM\"] = \"1\"\n\nfrom mooncake.store import MooncakeDistributedStore\n```\n\nEnable the switch on every writer and reader client process, then reproduce with full-object `put`/`upsert` and `get` operations. Treat `CHECKSUM_MISMATCH` (-801) as a failed read and do not use the destination buffer. Objects without checksum metadata and range reads are not verified. This mode scans object data, stages GPU buffers to host memory, and disables the local hot cache, so use it only for diagnosis.\n\n### Service Connectivity\n```bash\n# Check master is running\ncurl http://localhost:50051\n\n# Check metadata server\ncurl http://localhost:8080/metadata\n```\n\n## Best Practices\n\n1. **Always close stores**: Call `store.close()` when done\n2. **Register buffers for zero-copy**: Required for RDMA operations\n3. **Use batch operations**: Better throughput for multiple operations\n4. **Configure replication**: Use `ReplicateConfig` for important data\n5. **Use soft pinning**: For frequently accessed objects\n6. **Choose protocol wisely**: TCP for dev/test, RDMA for production\n7. **Monitor leases**: Objects have TTL, renew if needed\n8. **Handle errors**: Check return codes and handle failures\n\n## Quick Reference\n\n### Mooncake Store Methods\n- `setup()`: Initialize store\n- `put()`, `get()`: Basic operations\n- `put_batch()`, `get_batch()`: Batch operations\n- `put_from()`, `get_into()`: Zero-copy operations\n- `put_tensor()`, `get_tensor()`: PyTorch tensors\n- `register_buffer()`, `unregister_buffer()`: Buffer management\n- `is_exist()`, `remove()`: Metadata operations\n- `close()`: Cleanup\n\n### Transfer Engine Methods\n- `initialize()`: Setup engine\n- `allocate_managed_buffer()`, `free_managed_buffer()`: Buffer allocation\n- `transfer_sync_write()`, `transfer_sync_read()`: Synchronous transfers\n- `transfer_submit_write()`, `transfer_check_status()`: Async transfers\n- `batch_transfer_sync_write()`, `batch_transfer_sync_read()`: Batch sync\n- `batch_transfer_async_write()`, `get_batch_transfer_status()`: Batch async\n- `register_memory()`, `unregister_memory()`: Memory registration\n- `write_bytes_to_buffer()`, `read_bytes_from_buffer()`: Buffer I/O\n\n## Documentation Links\n\n- Full API Reference: https://kvcache-ai.github.io/Mooncake/\n- Quick Start: docs/source/getting_started/quick-start.md\n- Mooncake Store: docs/source/python-api-reference/mooncake-store.md\n- Transfer Engine: docs/source/python-api-reference/transfer-engine.md\n- Transfer Engine direct usage: docs/source/design/transfer-engine/index.md#using-transfer-engine-in-your-projects\n- EP Backend: docs/source/python-api-reference/ep-backend.md\n\n<!-- File: .claude/skills/mooncake-ci-local/examples/minimal.md (Tokens: ~136 | Category: anthropic-skill) -->\n# Minimal Example\n\nGoal: validate the current branch before opening or submitting a PR.\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nForce a full rerun even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nResult location:\n\n- `local_test/run-ci-logs/<timestamp>/`\n\nTypical report format:\n\n- passed stages\n- failed or blocked stages\n- unsupported stages\n- first actionable failure and its log path\n\n<!-- File: .claude/skills/mooncake-ci-local/SKILL.md (Tokens: ~1113 | Category: anthropic-skill) -->\n---\nname: mooncake-ci-local\ndescription: Run Mooncake pre-PR local validation through scripts/run_ci_test.sh. Use this skill whenever the user wants to validate a branch before opening or submitting a PR, run local CI, run ci test, check changes before PR, reproduce GitHub Actions locally, or force a full pre-submit verification. Trigger on phrases like \"提交 PR 前验证\", \"run ci test\", \"run local CI\", \"check my branch\", \"test before PR\", \"pre-submit validation\", and \"reproduce CI locally\".\n---\n\n# Mooncake Pre-PR Local Validation\n\nUse `bash scripts/run_ci_test.sh` as the default entry point. This is the single local lane for PR-before-submit validation, and it already coordinates the reproducible parts of GitHub Actions.\n\n## Default Entry Point\n\nWhen the user asks for any of the following, run the repo script first instead of reconstructing the workflow by hand:\n\n- 提交 PR 前本地验证\n- run ci test\n- run local CI\n- check my branch before PR\n- reproduce CI locally\n\nDefault command:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nWhat this script already covers:\n\n- GitHub-like `paths-filter` against `origin/main`\n- `typos`\n- `scripts/code_format.sh --check`\n- default CMake configure/build/install in `build-ci-local`\n- `ctest`\n- wheel build in `build-wheel-local`\n- wheel installation validation\n- `scripts/run_tests.sh`\n- selected Python API and integration tests\n- per-stage summary and logs under `local_test/run-ci-logs/<timestamp>/`\n\n## Standard Agent Workflow\n\n1. Run `bash scripts/run_ci_test.sh` from the repo root unless the user explicitly asks for a narrower subset.\n2. Read the stage summary instead of dumping raw terminal output.\n3. Report these items back to the user:\n   - passed stages\n   - failed stages\n   - blocked stages\n   - unsupported stages\n   - whether `paths-filter` skipped downstream stages\n   - the log directory under `local_test/run-ci-logs/...`\n4. If there is a failure, inspect the corresponding stage log and summarize the root cause.\n\n## Common Options\n\nForce a full lane even if `paths-filter` would skip downstream stages:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nUse another base ref:\n\n```bash\nbash scripts/run_ci_test.sh --base origin/main\n```\n\nAuto-install missing dependencies:\n\n```bash\nbash scripts/run_ci_test.sh --install-deps\n```\n\nKeep services running for follow-up debugging:\n\n```bash\nbash scripts/run_ci_test.sh --keep-services\n```\n\n## Minimal Example\n\nUser prompt:\n\n- 提交 PR 前，帮我跑一遍本地 CI 验证当前分支。\n\nExpected action:\n\n```bash\nbash scripts/run_ci_test.sh\n```\n\nIf the user wants to ignore changed-path optimization and force the full lane:\n\n```bash\nbash scripts/run_ci_test.sh --skip-path-filter\n```\n\nSee also `.claude/skills/mooncake-ci-local/examples/minimal.md`.\n\n## How To Interpret Results\n\n- `passed`: the stage succeeded locally.\n- `failed`: the stage reproduced a real local failure and needs investigation.\n- `blocked`: local environment or dependency issue prevented execution.\n- `unsupported`: intentionally not run in the local lane because it needs external platforms, special hardware, or a non-default build.\n\nIf `paths-filter` skips downstream stages, explain that the current branch changed only non-source paths relative to the selected base.\n\n## Current Local Coverage\n\nIncluded by default:\n\n- spell check\n- code format check\n- default ASan CMake lane in `build-ci-local`\n- `ctest`\n- wheel build and installation test\n- `scripts/run_tests.sh`\n- selected Python API tests\n\nUnsupported by design in the default local lane:\n\n- Ascend jobs\n- T-one integration jobs\n- MUSA jobs\n- Docker image build jobs\n- CUDA 13 wheel jobs\n- PG-backend tests absent from the default wheel build\n- Python drain-http API stage in the local ASan lane\n\n## Targeted Reruns For Debugging\n\nUse targeted reruns only after the full script identifies a failing area, or when the user explicitly asks for a smaller scope.\n\nRerun a specific C++ test pattern:\n\n```bash\ncd build-ci-local\nMC_METADATA_SERVER=http://127.0.0.1:8080/metadata DEFAULT_KV_LEASE_TTL=500 ctest -R <pattern> --output-on-failure\n```\n\nRerun the Python wheel integration lane:\n\n```bash\nsource test_env/bin/activate\nMC_STORE_MEMCPY=false TEST_SSD_OFFLOAD_IN_EVICT=true ./scripts/run_tests.sh\n```\n\nRerun the safetensor unittest:\n\n```bash\nsource test_env/bin/activate\npython -m unittest mooncake-wheel.tests.test_safetensor_functions\n```\n\n## Notes For The Agent\n\n- Prefer the repo script over rebuilding the CI workflow step by step.\n- Preserve the separation between `build-ci-local` and `build-wheel-local`.\n- Summarize failing stages from their logs instead of pasting raw output.\n- If the user only asks whether the branch is safe before opening a PR, the default answer path is `bash scripts/run_ci_test.sh`.\n\n<!-- File: .claude/skills/mooncake-troubleshoot/SKILL.md (Tokens: ~3487 | Category: anthropic-skill) -->\n---\nname: mooncake-troubleshoot\ndescription: Automatically diagnose Mooncake deployment and runtime issues. Checks services (mooncake_master, metadata server), RDMA devices, environment variables, connectivity, memory limits, object integrity, and analyzes logs for common error patterns. Use when Mooncake deployment fails, services won't start, connections fail, data is corrupted or garbled, or you encounter runtime errors like \"Error from etcd client\", \"No matched device found\", \"Failed to register memory\", \"NO_AVAILABLE_HANDLE\", \"CHECKSUM_MISMATCH\", or any RDMA/networking issues. Also use when user asks to troubleshoot, debug, diagnose, or fix Mooncake problems.\n---\n\n# Mooncake Deployment Troubleshooting\n\nYou are a Mooncake deployment troubleshooting specialist. Your job is to systematically diagnose issues and provide actionable solutions based on the comprehensive troubleshooting knowledge from Mooncake documentation.\n\n## Diagnostic Strategy\n\nRun checks systematically, reporting findings as you go. Start with simple checks (services, connectivity) before diving into complex issues (RDMA, memory registration).\n\n### 1. Service Status Check\n\nCheck if critical services are running:\n\n```bash\n# Check mooncake_master\nps aux | grep mooncake_master | grep -v grep\n\n# Check port usage\nnetstat -tuln | grep -E '(50051|8080|2379|9003)'\n\n# If using etcd\nps aux | grep etcd | grep -v grep\n```\n\n**Common issues:**\n- `bind address already in use` → Port conflict, use different port with `--rpc_port`\n- Master not running → Check startup logs for errors\n\n### 2. Metadata Server Connectivity\n\nThe metadata server is critical for node discovery and coordination.\n\n```bash\n# Test etcd connectivity\ncurl -s http://127.0.0.1:2379/version\n\n# Or test custom metadata server\ncurl -s $MC_METADATA_SERVER\n\n# Check for proxy interference\necho \"http_proxy: $http_proxy\"\necho \"https_proxy: $https_proxy\"\n```\n\n**Common issues:**\n- `Error from etcd client` → Metadata server unreachable\n  - **Fix:** Ensure etcd is bound to `0.0.0.0` not `127.0.0.1`:\n    ```bash\n    etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n    ```\n  - **Fix:** Disable HTTP proxy:\n    ```bash\n    unset http_proxy https_proxy\n    ```\n\n### 3. Environment Variables Check\n\nVerify critical environment variables are set correctly:\n\n```bash\n# Display Mooncake variables\nenv | grep -E '^(MC_|MOONCAKE_STORE_CHECKSUM=)'\n\n# Key variables to check:\necho \"MC_METADATA_SERVER: $MC_METADATA_SERVER\"\necho \"MC_FORCE_TCP: $MC_FORCE_TCP\"\necho \"MC_LOG_LEVEL: $MC_LOG_LEVEL\"\necho \"MC_YLT_LOG_LEVEL: $MC_YLT_LOG_LEVEL\"\necho \"MC_MS_AUTO_DISC: $MC_MS_AUTO_DISC\"\necho \"MC_MS_FILTERS: $MC_MS_FILTERS\"\necho \"MC_GID_INDEX: $MC_GID_INDEX\"\necho \"MC_MTU: $MC_MTU\"\necho \"MC_IB_PORT: $MC_IB_PORT\"\necho \"MC_ENABLE_DEST_DEVICE_AFFINITY: $MC_ENABLE_DEST_DEVICE_AFFINITY\"\necho \"MOONCAKE_STORE_CHECKSUM: $MOONCAKE_STORE_CHECKSUM\"\n```\n\n**Key variables:**\n- `MC_METADATA_SERVER` - Metadata server URL (required)\n- `MC_FORCE_TCP=true` - Force TCP for testing without RDMA\n- `MC_LOG_LEVEL=0` - Enable verbose logging (0=INFO, 1=WARNING, 2=ERROR)\n- `MC_YLT_LOG_LEVEL=debug` - yalantinglibs log level\n- `MC_MS_AUTO_DISC=1` - Enable topology auto-discovery (default)\n- `MC_MS_FILTERS` - Filter specific RDMA devices (e.g., \"mlx5_1,mlx5_2\")\n- `MC_GID_INDEX` - RDMA GID index (set if GID is all zeros)\n- `MC_MTU` - RDMA MTU size\n- `MC_ENABLE_DEST_DEVICE_AFFINITY=1` - Reduce QP creation (fix \"Failed to create QP\")\n- `MOONCAKE_STORE_CHECKSUM=1` - Enable diagnostic object-level CRC-64 checks; set before starting every writer and reader client process\n\n### 4. RDMA Device Check\n\nOnly run if RDMA is being used (skip if `MC_FORCE_TCP=true`):\n\n```bash\n# List RDMA devices\nibv_devices\n\n# Check device details and status\nibv_devinfo\n\n# Check for ACTIVE ports\nibv_devinfo | grep -A 10 \"state:\"\n\n# Check GID addresses (should NOT be all zeros)\nibv_devinfo | grep -A 20 \"GID\"\n\n# Check peer memory modules\nlsmod | grep peer_mem\nlsmod | grep nvidia_peer_mem\n\n# Check QP count (if \"Failed to create QP\" error)\nrdma resource show qp\n```\n\n**Common issues:**\n- `No matched device found` → RDMA device name in config doesn't exist\n  - **Fix:** Use `ibv_devices` to get correct device names\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n  - **Fix:** Check cable connections, verify with `ibv_devinfo | grep state`\n  - **Fix:** Try different port with `MC_IB_PORT` environment variable\n- GID all zeros → Wrong GID index\n  - **Fix:** Set `MC_GID_INDEX=1` (or 2, 3 depending on network)\n- `Failed to create QP: Cannot allocate memory` → Too many QPs created\n  - **Fix:** Set `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n\n### 5. Memory and Resource Limits\n\nCheck system limits that affect RDMA memory registration:\n\n```bash\n# Check ulimits\nulimit -a\n\n# Focus on max locked memory\nulimit -l\n\n# Check RDMA device memory limits\nibv_devinfo -v | grep max_mr_size\n\n# Check dmesg for memory errors\ndmesg -T | tail -50 | grep -i \"out of mr size\"\n```\n\n**Common issues:**\n- `Failed to register memory: Input/output error` → Memory registration limit exceeded\n  - **Diagnostic:** Check `max_mr_size` with `ibv_devinfo -v`\n  - **Fix:** Reduce memory allocation or split into smaller chunks\n- Cannot allocate memory → ulimit restriction\n  - **Fix:** Set unlimited locked memory:\n    ```bash\n    ulimit -l unlimited\n    ```\n  - **Permanent fix:** Add to `/etc/security/limits.conf`:\n    ```\n    * soft memlock unlimited\n    * hard memlock unlimited\n    ```\n\n### 6. Network Connectivity\n\nTest connectivity between nodes:\n\n```bash\n# Test basic RDMA connectivity\nib_write_bw -d <device_name> -R\n\n# On peer node:\nib_write_bw -d <device_name> -R <server_ip>\n\n# Test GPU Direct RDMA (if CUDA enabled)\nib_write_bw -d <device_name> -R -x gdr\n\n# On peer node:\nib_write_bw -d <device_name> -R -x gdr <server_ip>\n\n# Test DNS resolution\nnslookup <connectable_name>\nping <connectable_name>\n```\n\n**Common issues:**\n- `connection refused` → Incorrect `connectable_name` or `rpc_port`\n  - **Fix:** Ensure `connectable_name` is NOT loopback (127.0.0.1/localhost)\n  - **Fix:** Use actual LAN/WAN IP or valid hostname\n- `Failed to exchange handshake` → RDMA connection setup failure\n  - **Fix:** Verify MTU matches: set `MC_MTU` environment variable\n  - **Fix:** Verify GID is valid (not all zeros)\n  - **Fix:** Test with `ib_send_bw` between nodes first\n\n### 7. Log Analysis\n\nSearch logs for common error patterns and their meanings:\n\n**Metadata/Connectivity Errors:**\n- `Error from etcd client` → Cannot connect to metadata server\n- `ERR_METADATA` → Metadata server communication failed\n- `ERR_DNS` → Invalid `local_server_name` (not valid DNS/IP)\n\n**RDMA Errors:**\n- `No matched device found` → RDMA device name doesn't exist\n- `Device XXX port not active` → RDMA port not in ACTIVE state\n- `Failed to exchange handshake description` → RDMA handshake failed\n- `Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num` → MTU/GID mismatch\n- `Failed to register memory` → Memory registration limit exceeded\n- `Failed to create QP` → Too many QPs, enable `MC_ENABLE_DEST_DEVICE_AFFINITY=1`\n- `Worker: Process failed for slice` → Network instability\n- `work request flushed error` → Cascading error (find first error)\n\n**Store Errors:**\n- `NO_AVAILABLE_HANDLE` (-200) → Memory pool exhausted\n  - **Fix:** Increase `global_segment_size` in setup\n  - **Fix:** Check eviction is working (look for eviction logs)\n- `LEASE_EXPIRED` (-707) → Lease expired during transfer\n  - **Fix:** Increase `default_kv_lease_ttl` in master startup\n- `OBJECT_NOT_FOUND` (-704) → Object doesn't exist\n- `SEGMENT_NOT_FOUND` (-101) → No available segments\n- `CHECKSUM_MISMATCH` (-801) → Full-object read differs from the checksum captured before the write\n  - **Action:** Treat the read as failed and do not use the destination buffer\n- `Failed to get description of XXX` → Segment name mismatch\n  - **Fix:** Ensure segment name matches `local_hostname` from peer\n\n**Port/Service Errors:**\n- `bind address already in use` → Port conflict\n  - **Fix:** Use different port: `--rpc_port=50052`\n\n### 8. Configuration Validation\n\nVerify configuration is correct:\n\n```bash\n# Check connectable_name is not loopback\nhostname -I\n\n# Verify master startup flags\nps aux | grep mooncake_master\n\n# Check if using correct protocol\nenv | grep MC_FORCE_TCP\n```\n\n**Critical checks:**\n- `connectable_name` must be non-loopback IP or valid hostname\n- MTU and GID configurations must match network environment\n- RDMA device names must exist on the machine\n- Ports must not be in use by other services\n\n### 9. Object Integrity Diagnostics\n\nWhen the user reports corrupted data or garbled output that may originate from Mooncake Store:\n\n1. Verify that the Store clients, primary master, and standby master use checksum-capable binaries from the same version.\n2. Set `MOONCAKE_STORE_CHECKSUM=1` before starting every writer and reader client process, then restart or recreate existing clients.\n3. Reproduce with full-object `put`/`upsert` and `get` operations. Range reads, including `get_into_ranges`, are not covered.\n4. If `CHECKSUM_MISMATCH` (-801) is returned, the covered read differs from the checksum captured over the writer's source object. Treat the read as failed and do not consume the destination buffer.\n5. If `object_checksum_absent` appears at VLOG(1), metadata has no checksum and verification was skipped. This can occur for existing objects or objects written by clients without the switch.\n6. If verification succeeds, continue investigating corruption outside the covered Store path. A checksum mismatch identifies a difference but does not by itself distinguish transfer corruption from storage corruption.\n7. Warn that this diagnostic mode scans all object data, stages GPU buffers to host memory, and disables the local hot cache. Disable it after diagnosis.\n\n## Error Code Quick Reference\n\n### Transfer Engine Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Normal execution | - |\n| -12 | ERR_ADDRESS_NOT_REGISTERED | Memory not registered | Register memory before use |\n| -14 | ERR_DEVICE_NOT_FOUND | RDMA device not found | Check device name with `ibv_devices` |\n| -16 | ERR_DNS | Invalid local_server_name | Use valid IP/hostname |\n| -19 | ERR_REJECT_HANDSHAKE | Peer rejected handshake | Check peer logs for reason |\n| -20 | ERR_METADATA | Metadata server unreachable | Check etcd/HTTP server |\n\n### Store Error Codes\n\n| Code | Name | Meaning | Fix |\n|------|------|---------|-----|\n| 0 | Success | Operation successful | - |\n| -200 | NO_AVAILABLE_HANDLE | Memory pool exhausted | Increase segment size |\n| -707 | LEASE_EXPIRED | Lease expired | Increase lease TTL |\n| -704 | OBJECT_NOT_FOUND | Object doesn't exist | Check object key |\n| -101 | SEGMENT_NOT_FOUND | No available segments | Check segment registration |\n| -801 | CHECKSUM_MISMATCH | Full-object data differs from its stored checksum | Reject the read buffer and investigate the Store data path |\n| -900 | RPC_FAIL | RPC failed | Check network/master |\n| -1000 | ETCD_OPERATION_ERROR | etcd operation failed | Check etcd status |\n\n## Output Format\n\nProvide a structured diagnostic report:\n\n```\n🔍 MOONCAKE DEPLOYMENT DIAGNOSTICS\n==================================\n\n✅ PASSED CHECKS:\n- Service status: mooncake_master running on port 50051\n- Metadata server: etcd accessible at http://127.0.0.1:2379\n- Environment: MC_METADATA_SERVER set correctly\n- [other passing checks]\n\n❌ FAILED CHECKS:\n- RDMA device: mlx5_0 port not ACTIVE (state: PORT_DOWN)\n- Memory limits: max locked memory is 64KB (too low)\n- [other failures with specific error messages]\n\n⚠️  WARNINGS:\n- GID index not set, may cause connection issues\n- HTTP proxy variables set, may interfere with metadata server\n- [other potential issues]\n\n🔧 RECOMMENDED FIXES:\n\n1. Fix RDMA port status:\n   - Check physical cable connections\n   - Verify driver configuration\n   - Command: ibv_devinfo | grep -A 10 \"state:\"\n\n2. Increase memory limits:\n   ulimit -l unlimited\n   # Or permanently in /etc/security/limits.conf:\n   * soft memlock unlimited\n   * hard memlock unlimited\n\n3. Set GID index:\n   export MC_GID_INDEX=1\n\n4. Disable HTTP proxy:\n   unset http_proxy https_proxy\n\n📋 SUMMARY:\n[Brief 2-3 sentence conclusion about deployment health and next steps]\n```\n\n## Troubleshooting Workflow\n\n1. **Start simple**: Check services and basic connectivity first\n2. **Read logs carefully**: First error is usually root cause (subsequent errors cascade)\n3. **Test incrementally**: Use `MC_FORCE_TCP=true` to isolate RDMA issues\n4. **Verify basics**: Check connectable_name, ports, env vars before deep diving\n5. **Use diagnostic tools**: ibv_devices, ibv_devinfo, ib_write_bw, curl\n6. **Reference documentation**: Check error codes and troubleshooting guide\n\n## Quick Fix Commands\n\n**Start metadata server properly:**\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379\n```\n\n**Enable verbose logging:**\n```bash\nexport MC_LOG_LEVEL=0\nexport MC_YLT_LOG_LEVEL=debug\n```\n\n**Force TCP mode for testing:**\n```bash\nexport MC_FORCE_TCP=true\n```\n\n**Fix memory limits:**\n```bash\nulimit -l unlimited\n```\n\n**Fix too many QPs:**\n```bash\nexport MC_ENABLE_DEST_DEVICE_AFFINITY=1\n```\n\n**Fix GID issues:**\n```bash\nexport MC_GID_INDEX=1  # or 2, 3 depending on network\n```\n\n**Use different port:**\n```bash\nmooncake_master --rpc_port=50052\n```\n\nNow execute the diagnostic checks systematically and provide the structured report.\n\n<!-- File: .claude/skills/mooncake-ci-local/scripts/check-prerequisites.sh (Tokens: ~1129 | Category: claude-rule) -->\n#!/bin/bash\n# Mooncake CI Local Test Prerequisites Check\n# Usage: bash check-prerequisites.sh\n# This script checks and auto-fixes all prerequisites for running Mooncake CI tests locally.\n\nset -e\n\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nYELLOW='\\033[1;33m'\nNC='\\033[0m'\n\necho \"🔍 Checking Mooncake CI test prerequisites...\"\n\n# 1. Check build directory\nif [ ! -f build/CMakeCache.txt ]; then\n  echo -e \"${RED}✗ Build directory not found or not built${NC}\"\n  echo \"  → Run: mkdir build && cd build && cmake .. && cmake --build .\"\n  exit 1\nfi\necho -e \"${GREEN}✓ Build exists${NC}\"\n\n# 2. Check mooncake installation\nif ! python -c \"import mooncake\" 2>/dev/null; then\n  echo -e \"${RED}✗ mooncake package not installed${NC}\"\n  echo \"  → Fixing: Installing mooncake package...\"\n  cd build && sudo cmake --install . && cd - >/dev/null\n  if ! python -c \"import mooncake\" 2>/dev/null; then\n    echo \"  → Alternative: pip install mooncake-wheel/dist/*.whl\"\n    exit 1\n  fi\n  echo -e \"${GREEN}✓ mooncake package installed${NC}\"\nelse\n  echo -e \"${GREEN}✓ mooncake package already installed${NC}\"\nfi\n\n# 3. Check ctest availability\nif ! command -v ctest &> /dev/null; then\n  echo -e \"${RED}✗ ctest not found${NC}\"\n  exit 1\nfi\necho -e \"${GREEN}✓ ctest available${NC}\"\n\n# 4. Kill and restart services (safest approach for local testing)\necho -e \"\\n${YELLOW}Cleaning up and restarting services...${NC}\"\npkill -f \"^etcd\" || true\npkill -f bootstrap_server.py || true\npkill -f mooncake_http_metadata_server || true\nsleep 1\n\n# 5. Start etcd\nif ! command -v etcd &> /dev/null; then\n  echo -e \"${YELLOW}⚠ etcd not found, installing...${NC}\"\n  ETCD_VER=v3.6.1\n  OS=$(uname -s | tr '[:upper:]' '[:lower:]')\n  ARCH=$(uname -m)\n  [ \"$ARCH\" = \"x86_64\" ] && ARCH=\"amd64\"\n  DOWNLOAD_URL=\"https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\"\n  echo \"  Downloading from: $DOWNLOAD_URL\"\n  cd /tmp\n  wget -q \"$DOWNLOAD_URL\" && tar xzf \"etcd-${ETCD_VER}-${OS}-${ARCH}.tar.gz\" && \\\n    sudo mv \"etcd-${ETCD_VER}-${OS}-${ARCH}\"/etcd* /usr/local/bin/\n  cd - >/dev/null\n  echo -e \"${GREEN}✓ etcd installed${NC}\"\nfi\n\netcd --advertise-client-urls http://127.0.0.1:2379 --listen-client-urls http://127.0.0.1:2379 >/dev/null 2>&1 &\nETCD_PID=$!\nsleep 2\nif ! etcdctl --endpoints=http://127.0.0.1:2379 endpoint health &>/dev/null; then\n  echo -e \"${RED}✗ etcd failed to start${NC}\"\n  kill $ETCD_PID 2>/dev/null || true\n  exit 1\nfi\necho -e \"${GREEN}✓ etcd running (PID: $ETCD_PID)${NC}\"\n\n# 6. Start HTTP metadata server\nif [ -f \"mooncake-transfer-engine/example/http-metadata-server-python/bootstrap_server.py\" ]; then\n  cd mooncake-transfer-engine/example/http-metadata-server-python\n  pip install -q aiohttp 2>/dev/null || true\n  python ./bootstrap_server.py >/dev/null 2>&1 &\n  METADATA_PID=$!\n  cd - >/dev/null\n  sleep 1\n  if curl -s http://127.0.0.1:8080/metadata > /dev/null 2>&1; then\n    echo -e \"${GREEN}✓ HTTP Metadata server running (PID: $METADATA_PID)${NC}\"\n  else\n    echo -e \"${RED}✗ HTTP Metadata server failed to start${NC}\"\n    kill $METADATA_PID $ETCD_PID 2>/dev/null || true\n    exit 1\n  fi\nelse\n  echo -e \"${YELLOW}⚠ Metadata server script not found, skipping${NC}\"\nfi\n\necho -e \"\\n${GREEN}✅ All prerequisites ready!${NC}\"\necho \"Service PIDs: etcd=$ETCD_PID\"\n[ -n \"$METADATA_PID\" ] && echo \"Metadata server PID: $METADATA_PID\"\necho -e \"\\n${YELLOW}To kill services:${NC}\"\necho \"  pkill -f '^etcd'\"\necho \"  pkill -f bootstrap_server\"\n\n<!-- File: .claude-plugin/marketplace.json (Tokens: ~442 | Category: marketplace) -->\n{\n  \"$schema\": \"https://json.schemastore.org/claude-code-marketplace.json\",\n  \"name\": \"mooncake\",\n  \"owner\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"metadata\": {\n    \"description\": \"Built-in Claude Code skills for working with Mooncake: deployment troubleshooting, local CI validation, and the Python API.\"\n  },\n  \"plugins\": [\n    {\n      \"name\": \"mooncake-troubleshoot\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-troubleshoot\"\n      },\n      \"description\": \"Systematically diagnose Mooncake deployment and runtime issues (services, RDMA, env vars, connectivity, logs) and propose fixes.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-troubleshoot.html\"\n    },\n    {\n      \"name\": \"mooncake-ci-local\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-ci-local\"\n      },\n      \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\"\n    },\n    {\n      \"name\": \"mooncake-api\",\n      \"source\": {\n        \"source\": \"git-subdir\",\n        \"url\": \"https://github.com/kvcache-ai/Mooncake.git\",\n        \"path\": \".claude/skills/mooncake-api\"\n      },\n      \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n      \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\"\n    }\n  ]\n}\n\n<!-- File: .claude/skills/mooncake-api/.claude-plugin/plugin.json (Tokens: ~146 | Category: plugin-manifest) -->\n{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-api\",\n  \"description\": \"Work with the Mooncake Store, Transfer Engine, and EP/Backend Python APIs.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-api.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"python\", \"api\", \"transfer-engine\", \"store\"]\n}\n\n<!-- File: .claude/skills/mooncake-ci-local/.claude-plugin/plugin.json (Tokens: ~153 | Category: plugin-manifest) -->\n{\n  \"$schema\": \"https://json.schemastore.org/claude-code-plugin-manifest.json\",\n  \"name\": \"mooncake-ci-local\",\n  \"description\": \"Run Mooncake pre-PR local validation via scripts/run_ci_test.sh, reproducing the reproducible parts of GitHub Actions.\",\n  \"author\": {\n    \"name\": \"Mooncake Team\"\n  },\n  \"homepage\": \"https://kvcache-ai.github.io/Mooncake/skills/mooncake-ci-local.html\",\n  \"repository\": \"https://github.com/kvcache-ai/Mooncake\",\n  \"license\": \"Apache-2.0\",\n  \"keywords\": [\"mooncake\", \"ci\", \"validation\", \"pre-pr\"]\n}\n\n<!-- File: .claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json (Tokens: ~300 | Category: plugin-manifest) -->\n# claude-plugin Plugin Manifest\n\nPath: `.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/kvcache-ai/Mooncake/HEAD/.claude/skills/mooncake-troubleshoot/.claude-plugin/plugin.json)\n</yakaai_skills>"}