{"owner":"apache","repo":"skywalking","hasSkills":true,"totalSkillsCount":10,"totalTokensCount":25341,"categories":["anthropic-skill","claude-rule"],"hasMcp":false,"mcpConfig":null,"found":[".claude/skills/gh-pull-request/SKILL.md",".claude/skills/generate-classes/SKILL.md",".claude/skills/test/SKILL.md",".claude/skills/license/SKILL.md",".claude/skills/package/SKILL.md",".claude/skills/run-e2e/SKILL.md",".claude/skills/new-monitoring-feature/SKILL.md",".claude/skills/ci-e2e-debug/SKILL.md",".claude/skills/compile/SKILL.md","CLAUDE.md"],"skills":{".claude/skills/gh-pull-request/SKILL.md":"---\nname: gh-pull-request\ndescription: Verify, commit, and push changes on a PR branch. Runs pre-flight checks (compile, checkstyle, license headers) before every push. Also creates the PR if one doesn't exist yet.\n---\n\n# PR Branch Workflow\n\nRun pre-flight checks, commit, push, and optionally create a PR.\n\n## Pre-flight checks\n\nRun these checks before every commit+push and fix any failures:\n\n### 1. Compile and checkstyle\n\n```bash\n# Checkstyle\n./mvnw -B -q clean checkstyle:check\n\n# Full build (compile + javadoc)\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### 2. License header check\n\n```bash\nlicense-eye header check\n```\n\nIf invalid files are found, fix with `license-eye header fix` and re-check.\n\n### 3. Unnecessary fully-qualified class names\n\nThe project checkstyle forbids inline FQCNs — every type reference in code should resolve\nthrough an `import`, not a fully-qualified name. Checkstyle does not always catch this (it\nmisses cases like inline `java.util.HashMap`, `java.util.concurrent.TimeUnit`, or\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` used as a local\nvariable type, generic parameter, or `new` target). Audit the files the branch touched\nbefore pushing:\n\nUse the `Grep` tool (ripgrep) rather than BSD `grep` on macOS — the scan below relies on a\nnegative lookahead that BSD `grep` doesn't support and GNU `grep -P` does:\n\n```\npattern: ^(?!\\s*(import |package |\\s*\\*)).*\\b(java\\.util\\.|java\\.io\\.|java\\.nio\\.|java\\.util\\.concurrent\\.|javassist\\.|org\\.apache\\.skywalking\\.)[A-Z][A-Za-z0-9_]*\nglob:    *.java\noutput_mode: content\n-n: true\n```\n\nScope the scan to files the branch touched, not the whole tree — pre-existing FQDNs on\nunrelated files generate noise. Use `git diff --name-only master...HEAD -- '*.java'` to get\nthe changed list, then run the ripgrep pattern against each.\n\nAcceptable exceptions (same as the `CLAUDE.md` rule):\n - Two classes with the same simple name would collide if both imported.\n - A Javadoc `{@link}` where the short name would be ambiguous to the reader.\n - Inside a string literal (e.g., a class name passed to `Class.forName`).\n\nFix every other hit — add an `import` and switch to the short name. This includes\n`new java.util.HashMap<>()`, `java.util.Set<String>` parameter types, and\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` as a local\nvariable type. Field declarations, method signatures, local variables, and generic\ntype arguments should all use the imported short name.\n\nRe-run checkstyle after the fix — a sloppy `sed`/`replace_all` can corrupt the `import`\nline itself (e.g., turning `import java.util.concurrent.locks.ReentrantLock;` into\n`import ReentrantLock;`), which causes a cryptic checkstyle `Range [0, -1) out of\nbounds for length N` error, not a normal violation line. If you see that error, inspect\nthe imports block first.\n\n## Commit and push\n\nAfter checks pass, commit and push:\n\n```bash\ngit add <files>\ngit commit -m \"<message>\"\ngit push -u origin <branch-name>\n```\n\n### Branch strategy\n- **Never work directly on master branch**\n- If on master, create a new branch first: `git checkout -b feature/<name>` or `git checkout -b fix/<name>`\n\n## Create PR (if not yet created)\n\nCheck whether a PR already exists for the current branch:\n\n```bash\ngh pr view --json number 2>/dev/null\n```\n\nIf no PR exists, create one:\n\n### PR title\nSummarize the changes concisely. Examples:\n- `Fix BanyanDB query timeout issue`\n- `Add support for OpenTelemetry metrics`\n\n### PR description\n\nRead `.github/PULL_REQUEST_TEMPLATE` and use its **exact format with checkboxes**. Do NOT use a custom summary format.\n\nKey template sections — uncomment the relevant one:\n\n**For Bug Fixes:**\n```\n### Fix <bug description or issue link>\n- [ ] Add a unit test to verify that the fix works.\n- [ ] Explain briefly why the bug exists and how to fix it.\n```\n\n**For New Features:**\n```\n### <Feature description>\n- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.\n- [ ] Update the documentation to include this new feature.\n- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.\n- [ ] If it's UI related, attach the screenshots below.\n```\n\n**For Performance Improvements:**\n```\n### Improve the performance of <class or module or ...>\n- [ ] Add a benchmark for the improvement.\n- [ ] The benchmark result.\n- [ ] Links/URLs to the theory proof or discussion articles/blogs.\n```\n\n**Always include:**\n```\n- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.\n- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).\n```\n\n### Create command\n\n```bash\ngh pr create --title \"<title>\" --body \"$(cat <<'EOF'\n<PR body from template>\nEOF\n)\"\n```\n\n### Post-creation\n- Add `copilot` as a reviewer: `gh pr edit <number> --add-reviewer copilot`\n- Do NOT add AI assistant as co-author. Code responsibility is on the committer's hands.\n- Return the PR URL when done.\n\n## After the PR is merged\n\nOnce the PR is merged, sync the default branch and clean up the feature branch:\n\n```bash\n# 1. Prune stale remote refs. GitHub auto-deletes the PR's branch on merge, so\n#    the remote feature branch is usually already gone; --prune removes the\n#    dangling local tracking ref.\ngit fetch origin --prune\n\n# 2. Switch back to the default branch and fast-forward it to include the merge.\ngit checkout master\ngit pull --ff-only origin master\n\n# 3. Confirm the change actually landed in master before deleting anything —\n#    `git log --oneline -1` should show the merge/squash commit with the PR\n#    number, or grep for a symbol the PR introduced.\ngit log --oneline -1\n\n# 4. Delete the local feature branch. SkyWalking SQUASH-merges PRs, so the\n#    feature branch's commit is NOT an ancestor of master (master gets a new\n#    squash commit instead). `git branch -d` therefore reports \"not fully\n#    merged\" — that is expected, not an error. After confirming the content is\n#    in master (step 3), force-delete:\ngit branch -d <branch> 2>/dev/null || git branch -D <branch>\n```\n\nNotes:\n- `git branch -d` failing with \"not fully merged\" on a squash-merged PR is normal — the squash commit has a different SHA than the feature commit. Verify via step 3, then `-D`.\n- If the remote branch was not auto-deleted (some repo settings), remove it explicitly: `git push origin --delete <branch>`.\n- Do NOT skip step 3. Force-deleting a local branch whose work didn't actually merge loses it.\n",".claude/skills/generate-classes/SKILL.md":"---\nname: generate-classes\ndescription: Generate bytecode classes from DSL scripts (MAL, OAL, LAL, Hierarchy). Runs the compiler and dumps .class files for inspection.\nargument-hint: \"<mal|oal|lal|hierarchy|all>\"\n---\n\n# Generate DSL Classes\n\nRun the v2 compiler (ANTLR4 + Javassist) to generate bytecode classes from DSL scripts and dump `.class` files to disk for inspection.\n\n## Commands by argument\n\n### `mal` — MAL expression classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=MALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/mal/**/*.generated-classes/`\n\n### `oal` — OAL metrics/dispatcher/builder classes\n\n```bash\n./mvnw test -pl oap-server/oal-rt \\\n  -Dtest=RuntimeOALGenerationTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/oal-rt/target/test-classes/metrics/`, `metrics/builder/`, `dispatcher/`\n\n### `lal` — LAL filter/extractor classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=LALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/lal/**/*.generated-classes/`\n\n### `hierarchy` — Hierarchy rule classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=HierarchyRuleExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/hierarchy-rule/*.generated-classes/`\n\n### `all` or no argument — generate all DSLs\n\nRun all four commands above sequentially.\n\n## After generation\n\nPrint the output location for the requested DSL so the user knows where to find the generated `.class` files. Use `javap` to decompile:\n\n```bash\njavap -c -p <path-to-class-file>\n```\n\n## Cleaning generated classes\n\n```bash\n./mvnw clean -pl oap-server/analyzer/dsl-scripts-test\n```\n",".claude/skills/test/SKILL.md":"---\nname: test\ndescription: Run unit tests, integration tests, or slow integration tests matching CI. Use to validate changes before submitting a PR.\nargument-hint: \"[unit|integration|slow|module-name]\"\n---\n\n# Tests\n\nRun tests matching CI configuration.\n\n## Commands by argument\n\n### `unit` or no argument — unit tests\n\n```bash\n./mvnw clean test -q -B -D\"checkstyle.skip\"\n```\n\nCI runs this on:\n- JDK 11: ubuntu, macOS, Windows\n- JDK 17, 21, 25: ubuntu only\n\n### `integration` — integration tests (excludes slow)\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow\n```\n\nCI runs this on JDK 11, 17, 21, 25 (ubuntu only).\n\n### `slow` — slow integration tests\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true \\\n  -Dit.test=org.apache.skywalking.library.elasticsearch.ElasticSearchIT \\\n  -Dfailsafe.failIfNoSpecifiedTests=false\n```\n\nCI runs on JDK 11 (ubuntu only). Currently only ElasticSearch/OpenSearch IT is in the slow matrix.\n\n### Module name — single module tests\n\n```bash\n# Unit tests for a specific module\n./mvnw clean test -pl oap-server/analyzer/<module-name> -D\"checkstyle.skip\"\n\n# Integration tests for a specific module\n./mvnw -B clean integration-test -pl oap-server/analyzer/<module-name> -Dcheckstyle.skip -DskipUTs=true\n```\n\n## Test frameworks\n\n- JUnit 5 (`org.junit.jupiter`)\n- Mockito for mocking\n- AssertJ for assertions\n- PowerMock for reflection utilities\n\n## Test naming conventions\n\n| Type | Pattern | Maven phase |\n|------|---------|-------------|\n| Unit tests | `*Test.java` | `test` |\n| Integration tests | `IT*.java` or `*IT.java` | `integration-test` |\n\n## Slow test tagging\n\nTests tagged with `@Tag(\"slow\")` are excluded from the normal integration-test job and run separately in the slow-integration-test job. Use this tag for tests that spin up external services (Elasticsearch, etc.) and take significant time.\n\n## CI retry behavior\n\nAll three CI jobs retry on failure (run the same command twice with `||`). This handles flaky tests but masks intermittent issues — if a test fails locally, investigate rather than relying on retries.\n\n## CI reference\n\nCI workflow: `.github/workflows/skywalking.yaml`\n\n| Job | JDK | OS | Timeout |\n|-----|-----|----|---------|\n| `unit-test` | 11 (3 OS) + 17, 21, 25 (ubuntu) | ubuntu, macOS, Windows | 30 min |\n| `integration-test` | 11, 17, 21, 25 | ubuntu | 60 min |\n| `slow-integration-test` | 11 | ubuntu | 60 min |\n",".claude/skills/license/SKILL.md":"---\nname: license\ndescription: Check and fix Apache 2.0 license headers and dependency licenses using skywalking-eyes. Use before submitting a PR.\nargument-hint: \"[check|fix|deps]\"\n---\n\n# License Checks\n\nCheck and fix license compliance using [skywalking-eyes](https://github.com/apache/skywalking-eyes). Two CI jobs use this tool:\n\n1. **license-header** — verifies all source files have Apache 2.0 headers\n2. **dependency-license** — verifies the LICENSE file matches current dependencies\n\n## Steps\n\n### Header check (default, or `check` argument)\n\n```bash\nlicense-eye header check\n```\n\n- **All valid**: Output shows `valid: N, invalid: 0` — nothing to do.\n- **Invalid files found**: Fix with `license-eye header fix`, then re-check.\n\n### Header fix (`fix` argument)\n\n```bash\nlicense-eye header fix\nlicense-eye header check\n```\n\n### Dependency license check (`deps` argument)\n\nThis regenerates the LICENSE file from dependency metadata and checks for drift:\n\n```bash\nlicense-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl || exit 1\nif [ ! -z \"$(git diff -U0 ./dist-material/release-docs/LICENSE)\" ]; then\n  echo \"LICENSE file is not updated correctly\"\n  git diff -U0 ./dist-material/release-docs/LICENSE\nfi\n```\n\nIf the LICENSE file changed, review the diff and commit it. **Important**: CI runs on Linux — some dependencies have platform-specific variants. If you're on macOS/Windows, the LICENSE diff may be a platform artifact. Verify before committing.\n\n## Rules\n\nConfiguration is in `.licenserc.yaml`:\n- Java, XML, YAML/YML files require Apache 2.0 headers\n- JSON and Markdown files are excluded (JSON doesn't support comments)\n- Generated files and certain vendor paths are excluded\n- SPI service files (`META-INF/services/`) require headers (use `#` comment style)\n\n## Installation\n\n```bash\n# Same version as CI (pinned commit)\ngo install github.com/apache/skywalking-eyes/cmd/license-eye@5b7ee1731d036b5aac68f8bd3fc9e6f98ada082e\n\n# Or via Homebrew (macOS)\nbrew install license-eye\n```\n",".claude/skills/package/SKILL.md":"---\nname: package\ndescription: Rebuild the SkyWalking distribution and OAP Docker image after source changes. Use before running e2e tests so the image reflects your code changes. Avoids the \"image looks updated but runtime has stale jars\" trap.\nargument-hint: \"[oap|all|dist-only]\"\n---\n\n# Package OAP Distribution & Docker Image\n\nRebuilding the Docker image after a source-code change has a **two-step dependency**: rebuild the dist tarball, then rebuild the image. Skipping the first step silently produces an image with stale jars — the Docker build still \"succeeds,\" the image has a fresh timestamp, but the embedded jar is the one from your previous build.\n\n## Why this matters\n\n`make docker.oap` in the root `Makefile` is defined as:\n\n```\ndocker.% push.docker.%: $(CONTEXT)/$(DIST) $(SW_ROOT)/docker/%/*\n    $(DOCKER_RULE)\n```\n\nIt depends on `$(CONTEXT)/$(DIST)` (the dist tarball) **as a file prerequisite**. If that tarball already exists, make does not regenerate it — it just copies whatever is on disk into `dist/docker_build/oap/` and runs `docker buildx build`. There is **no** dependency that rebuilds the tarball from source.\n\nOnly `make docker` triggers the full chain (`init → build.all → docker.all`). So:\n\n| Command | Rebuilds source? | Rebuilds tarball? | Rebuilds image? |\n|---------|:--:|:--:|:--:|\n| `./mvnw -pl <module> package` | only that module | no | no |\n| `./mvnw -pl apm-dist -am package -Pbackend,dist` | all backend deps + dist | yes | no |\n| `make docker.oap` | **no** | **no** (uses whatever's on disk) | yes |\n| `make docker` | yes (`build.all`) | yes | yes (`docker.all`) |\n\n## `flatten:flatten` — always run it before `package`/`install`\n\nSkyWalking's poms use `${revision}` as a placeholder version (e.g., `10.5.0-SNAPSHOT`). The `flatten-maven-plugin` resolves `${revision}` into concrete versions and writes a `.flattened-pom.xml`. Without it:\n\n- Installed artifacts carry the literal string `${revision}` in their coordinates and cannot be resolved as dependencies.\n- Downstream modules (including `apm-dist`) see an inconsistent dependency graph.\n- Symptoms are subtle: `-pl <module> -am` may succeed in isolation but fail or pull in stale transitive artifacts when the same module is consumed by another build invocation in the same session.\n\n**Always run `flatten:flatten` in the same goal chain as `package` or `install`.** The CI `dist-tar` job and the `compile` skill both do this. Example:\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -DskipTests\n```\n\nNot optional — treat it as part of `package`.\n\n## Pick the right command\n\n### Changed OAP source → want a new image\n\n```bash\n# Recommended: full chain\nmake docker\n```\n\nor, faster and equivalent for backend-only changes (skips UI):\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true\nmake docker.oap\n```\n\n**Do not** just run `make docker.oap` after a code edit — the image will look rebuilt but your change will not be in the jars.\n**Do not** skip `flatten:flatten` — see the section above.\n\n### Changed only the dist packaging (e.g., `apm-dist/`, log4j2.xml)\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests\nmake docker.oap\n```\n\n### Changed only Dockerfile / entrypoint (`docker/oap/*`)\n\nThe tarball is untouched, so `make docker.oap` alone is correct:\n\n```bash\nmake docker.oap\n```\n\n## Verify the fix reached the image\n\nDocker's \"image created\" timestamp updates even when the content is stale (because `--no-cache` is used). **Don't trust the timestamp.** Verify the jar contents directly:\n\n```bash\n# Copy the jar out of the container\ndocker cp <container-name>:/skywalking/oap-libs/<module>-<version>.jar /tmp/verify.jar\n\n# Extract the specific class\ncd /tmp && jar -xf verify.jar <path/to/YourClass.class>\n\n# Grep for a string your fix introduced (unique literal, method name, etc.)\ngrep -oa \"myFixSignature\" <path/to/YourClass.class>\n```\n\nIf grep finds nothing, the image does not contain your fix — you forgot to rebuild the dist. Re-run with the correct chain.\n\n## Common pitfalls\n\n- **`make docker.oap` after `./mvnw package`**: The module jar is fresh in `oap-server/.../target/`, but the dist tarball at `dist/apache-skywalking-apm-bin.tar.gz` is untouched. Image uses the old jar.\n- **Cancelling a `make docker` mid-flight**: leaves the dist half-rebuilt. Re-run `make docker` or delete `dist/` first.\n- **Stale buildx cache**: not usually the issue (the Makefile passes `--no-cache`), but if you suspect it, run `docker buildx prune`.\n- **Trusting image timestamps**: `docker images` shows when the image was *tagged*, not when its content actually changed. A no-op rebuild still updates the timestamp.\n- **Interactive buildx hangs**: if `docker buildx build` sits silent for minutes with no stdout, kill it and retry. The buildx container driver occasionally stalls; a restart is faster than waiting.\n\n## The golden rule\n\nAfter any Java source edit that affects code shipped in `oap-libs/`, run one of:\n\n1. `make docker` (full, safest)\n2. `./mvnw -pl apm-dist -am -o package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true && make docker.oap` (fast)\n\nThen **verify the fix is in the image's jar** before running e2e. Don't assume.\n",".claude/skills/run-e2e/SKILL.md":"---\nname: run-e2e\ndescription: Run SkyWalking E2E tests locally\ndisable-model-invocation: true\nargument-hint: \"[test-case-path]\"\n---\n\n# Run SkyWalking E2E Test\n\nRun an E2E test case using `skywalking-infra-e2e`. The user provides a test case path (e.g., `simple/jdk`, `storage/banyandb`, `alarm`).\n\n## Prerequisites\n\nAll tools require **Go** installed. Check `.github/workflows/` for the exact `e2e` commit used in CI.\n\n### e2e CLI\n\nBuilt from [apache/skywalking-infra-e2e](https://github.com/apache/skywalking-infra-e2e), pinned by commit in CI:\n\n```bash\n# Install the pinned commit\ngo install github.com/apache/skywalking-infra-e2e/cmd/e2e@<commit-id>\n\n# Or clone and build locally (useful when debugging the e2e tool itself)\ngit clone https://github.com/apache/skywalking-infra-e2e.git\ncd skywalking-infra-e2e\ngit checkout <commit-id>\nmake build\n# binary is in bin/e2e — add to PATH or copy to $GOPATH/bin\n```\n\n### swctl, yq, and other tools\n\nE2E test cases run pre-install steps (see `setup.steps` in each `e2e.yaml`) that install tools into `/tmp/skywalking-infra-e2e/bin`. When running locally, you need these tools on your PATH.\n\n**swctl** — SkyWalking CLI, used in verify cases to query OAP's GraphQL API. Pinned at `SW_CTL_COMMIT` in `test/e2e-v2/script/env`:\n\n```bash\n# Option 1: Use the install script (same as CI)\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: Build from source\ngo install github.com/apache/skywalking-cli/cmd/swctl@<SW_CTL_COMMIT>\n```\n\n**yq** — YAML processor, used in verify cases:\n\n```bash\n# Option 1: Use the install script\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: brew install yq (macOS)\n```\n\n**Other tools** (only needed for specific test cases):\n\n| Tool | Install script | Used by |\n|------|---------------|---------|\n| `kubectl` | `install.sh kubectl` | Kubernetes-based tests |\n| `helm` | `install.sh helm` | Helm chart tests |\n| `istioctl` | `install.sh istioctl` | Istio/service mesh tests |\n| `etcdctl` | `install.sh etcdctl` | etcd cluster tests |\n\nAll install scripts are at `test/e2e-v2/script/prepare/setup-e2e-shell/`.\n\n## Steps\n\n### 1. Determine the test case\n\nResolve the user's argument to a full path under `test/e2e-v2/cases/`. If ambiguous, list matching directories and ask.\n\n```bash\nls test/e2e-v2/cases/<argument>/e2e.yaml\n```\n\n### 2. Check if rebuild is needed\n\nCompare source file timestamps against the last build:\n\n```bash\n# OAP server changes since last build\nfind oap-server apm-protocol -type f \\( \\\n  -name \"*.java\" -o -name \"*.yaml\" -o -name \"*.yml\" -o \\\n  -name \"*.json\" -o -name \"*.xml\" -o -name \"*.properties\" -o \\\n  -name \"*.proto\" \\\n\\) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null | head -5\n\n# Test service changes since last build\nfind test/e2e-v2/java-test-service -type f \\( \\\n  -name \"*.java\" -o -name \"*.xml\" -o -name \"*.yaml\" -o -name \"*.yml\" \\\n\\) -newer test/e2e-v2/java-test-service/e2e-service-provider/target/*.jar 2>/dev/null | head -5\n```\n\nIf files are found, warn the user and suggest rebuilding before running.\n\n### 3. Rebuild if needed (only with user confirmation)\n\n```bash\n# Rebuild OAP\n./mvnw clean flatten:flatten package -Pall -Dmaven.test.skip && make docker\n\n# Rebuild test services\n./mvnw -f test/e2e-v2/java-test-service/pom.xml clean flatten:flatten package\n```\n\n### 4. Run the E2E test\n\nSet required environment variables and run:\n\n```bash\nexport SW_AGENT_JDK_VERSION=8\ne2e run -c test/e2e-v2/cases/<case-path>/e2e.yaml\n```\n\n### 5. If the test fails\n\nDo NOT run cleanup immediately. Instead:\n\n1. Check container logs:\n   ```bash\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs oap\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs provider\n   ```\n\n2. Run verify separately (can retry after investigation):\n   ```bash\n   e2e verify -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n3. Only cleanup when done debugging:\n   ```bash\n   e2e cleanup -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n### 6. Manually fire each verify query (fast triage)\n\nThe `e2e verify` retry loop runs in sequence and stops at the first failing case, so a single bad query hides every case after it. When a verify fails, **run each verify case directly against the still-running OAP** before editing anything — you'll see the real error (bad flag, missing data, wrong expected), not the progress spinner. This is also the right way to author new verify cases: craft the query against live OAP, confirm the actual YAML, then write the expected file.\n\n```bash\n# Find the host-side port that infra-e2e bound to OAP's container port 12800.\n# (Each run picks a new random port; the trigger log prints it too.)\ndocker ps --filter \"name=skywalking_e2e-oap\" --format \"{{.Ports}}\" \\\n  | grep -oE \"[0-9]+->12800\" | head -1\n# => e.g. 56381->12800\n\nURL=http://localhost:56381/graphql\nSWCTL=/tmp/skywalking-infra-e2e/bin/swctl\n\n# Copy the query from e2e.yaml verbatim, then substitute ${oap_host} → localhost\n# and ${oap_12800} → the port you just found:\n$SWCTL --display yaml --base-url=$URL service ly IOS\n$SWCTL --display yaml --base-url=$URL logs list --service-name=MyiOSApp\n$SWCTL --display yaml --base-url=$URL metrics exec --expression=service_cpm --service-name=MyiOSApp\n```\n\nWhen a `swctl` subcommand rejects a flag (`Incorrect Usage: flag provided but not defined: -layer`), the e2e config is using syntax the pinned `swctl` commit doesn't support. Find the right syntax with `swctl <cmd> --help` and update the e2e config. Common cases encountered:\n\n| Broken flag/form | Working form |\n|---|---|\n| `service ls --layer IOS` | `service ly IOS` |\n| `metrics exec ... --is-normal=true` | drop `--is-normal` (default behavior) |\n\nFor queries that *don't* use `swctl` (raw `curl` against `/loki/...`, Zipkin, PromQL), hit the matching exposed port:\n\n```bash\ncurl \"http://localhost:$(docker ps --filter name=skywalking_e2e-oap --format '{{.Ports}}' | grep -oE '[0-9]+->3100' | head -1 | cut -d'-' -f1)/loki/api/v1/labels\"\n```\n\n### 7. UI template changes require a fresh DB\n\n`UITemplateInitializer.initTemplate()` (in `oap-server/server-core`) calls `uiTemplateManagementService.addIfNotExist(setting)` — keyed by the `id` field in each `ui-initialized-templates/**/*.json`. Same ID → skipped. So edits to an *existing* template JSON (adding widgets, relabeling, changing expressions) will **not** be applied on an already-initialized OAP, even after a container restart, because the old copy still lives in storage.\n\nTo pick up dashboard JSON changes:\n\n```bash\n# Remove both containers — BanyanDB stores state inside the container FS in the\n# e2e compose (no named volume), so removing the container wipes state cleanly.\ndocker rm -f skywalking_e2e-oap-1 skywalking_e2e-banyandb-1\n\n# For compose setups that use a named volume, also:\n# docker volume rm <volume-name>\n\n# Then re-run — OAP sees empty storage, loads the new template JSON.\ne2e run -c test/e2e-v2/cases/<case>/e2e.yaml\n```\n\nSymptom to watch for: you edit the JSON, rebuild, redeploy — dashboard in the UI still shows the pre-edit layout. That's not a caching bug; that's `addIfNotExist` doing exactly what its name says.\n\n### 8. Author the expected YAML from live output\n\nFor a new verify case, the workflow is:\n\n1. Fire the query manually (see step 6) and capture the YAML.\n2. Pick which fields are *meaningful domain values* (must match exactly) vs *dynamic runtime values* (`notEmpty` / `gt` / `ge`). See `test/e2e-v2/CLAUDE.md` for the decision guide.\n3. Write the expected file. If the response is a list, wrap the items in `{{- contains . }} ... {{- end }}` so ordering and extra actual items don't fail the match.\n4. Re-run `e2e verify` alone (the containers are still up from the previous run); iterate on the expected file without rebuilding.\n\n### 9. Expected-file authoring traps\n\nThese burn CI cycles and pass locally. Each was learned the hard way.\n\n**Unquoted `content: {{ notEmpty .content }}` with `:` inside the value.** Sim-generated or real log content routinely includes colons (`POST https://api.example.com/cart failed: 500`, `HTTP/1.1 500: Internal Server Error`). Without quoting, the template renders to invalid YAML (snakeyaml parses `failed:` as a nested key) and the *whole log entry* marshals to `nil`. Symptom: diff shows `- nil` at *every* position in the expected logs list vs real maps in actual. Fix: wrap in single quotes — `content: '{{ notEmpty .content }}'`. Single-quoted YAML preserves `:` in the scalar; only fails on embedded `'`. Double quotes also work unless the content has `\"`.\n\n**Nested `contains` with multiple per-element pattern assertions against a varied stream.** The template renders the block body once per actual element; when the outer block body has *multiple* inner `contains` patterns asserting specific tag key/value pairs, and only *some* actuals satisfy all the inner patterns, `go-cmp` with `contains` can end up comparing `[rendered_for_A0, nil, nil, ...]` vs `[A0, A1, A2, ...]` and fail despite `contains` being permissive on extras. Specifically: outer `contains .logs` with a single log pattern + inner `contains .tags` asserting two distinct key/value pairs. On a simulator emitting heterogeneous errors (js + promise + ajax + pageNotFound), only a subset satisfy the inner assertion. Passes locally with 1–2 logs, fails in CI with 6+.\n- Keep the outer `contains` body lenient: field-shape checks (`notEmpty`, `gt`), one discriminator tag that *every* element in the stream carries.\n- Cover per-category assertions via separate labeled-metric verify cases, not inside the nested template.\n- Rule of thumb: \"at least one log exists with the right layer routing\" inside the logs expected; per-category coverage via `meter_*_count{label=X}` verify cases.\n\n**Hand-crafted OTLP curl payloads drift from real SDK output.** When the upstream SDK ships a published simulator image (mini-program-monitor's `sim-wechat` / `sim-alipay`, browser-client-js sim, etc.), prefer driving the e2e with that image in `MODE=timed` with a bounded `DURATION_MS` over hand-rolling the OTLP JSON. Hand-crafted payloads miss real-world shape issues: delta-vs-cumulative temporality, label-cardinality surprises, stacktrace formatting variance, attribute key names that changed between SDK versions. Pin to a released tag (`v0.4.0`), not `:latest` or HEAD SHA — reproducibility.\n\n**`timeUnixNano: \"0\"` in an OTLP metric datapoint.** The receiver propagates this into MAL's bucket computation and the metric lands in the 1970 time bucket — `swctl metrics exec` over the \"last N minutes\" window won't find it. Either use `$(date +%s)000000000` at setup time or omit the field if the receiver accepts \"now\" as default.\n\n**Setup-step curl loop with `|| sleep` pattern.** The shell line `for ... do curl && break || sleep 5; done` exits 0 when every attempt connection-refused because the final `sleep 5` returns 0. OAP takes ~50 s to start in CI, so all attempts fail before OAP is ready, and the setup step silently succeeds with zero traffic ingested. Fix: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e` at step top.\n\n**`swctl` flag rejected.** If a verify case uses a flag the pinned `swctl` commit doesn't support (`service ls --layer` vs `service ly`), the whole case fails 20× before CI gives up. Fire each verify query by hand once before pushing (step 6 above).\n\n**Published image cache miss in CI.** `docker compose pull` sometimes hits rate limits or unreachable registries; the test spins until timeout with \"dependency failed to start\". Look at the CI log for `Error response from daemon: pull access denied` or `manifest unknown`. If you see that, pin a different image tag that's definitely published (check `docker manifest inspect <tag>` locally), not a floating one.\n\n## Common test cases\n\n| Shorthand | Path |\n|-----------|------|\n| `simple/jdk` | `test/e2e-v2/cases/simple/jdk/` |\n| `storage/banyandb` | `test/e2e-v2/cases/storage/banyandb/` |\n| `storage/elasticsearch` | `test/e2e-v2/cases/storage/elasticsearch/` |\n| `alarm` | `test/e2e-v2/cases/alarm/` |\n| `log` | `test/e2e-v2/cases/log/` |\n| `profiling/trace` | `test/e2e-v2/cases/profiling/trace/` |\n",".claude/skills/new-monitoring-feature/SKILL.md":"---\nname: new-monitoring-feature\ndescription: Add a new monitoring target / layer to SkyWalking OAP. Orients you to the OAL / MAL / LAL / SpanListener / SegmentListener extension points, the UI template + submodule touchpoints, the docs set that must move together, and the cross-cutting traps that don't live in any one skill.\n---\n\n# Adding a New Monitoring Feature / Layer\n\nSkyWalking monitoring for a new target (a new `Layer` — mobile OS, gateway, DB, language runtime, …) is composed from the same set of reusable extension points. This skill is the index: which extension point fits which signal, where its contract lives, which docs must be updated together, and which harness-wide traps the extension-specific docs don't cover.\n\nThere is one skill per narrow concern. This one is the wiring map.\n\n---\n\n## Working posture\n\n- **Default scope for a new monitoring feature is narrow.** For most new monitoring targets, traces are **re-used as-is** — you do not write or modify a trace analyzer. Metrics and log extensions via OTLP (MAL + LAL) are the common shape. OTLP / Zipkin span-based metrics analysis (`SpanListener`) already exists and only needs a new implementation if your feature introduces genuinely new semantic-convention attributes to extract (e.g. `gen_ai.*`, `mcp.*` for the GenAI layer). Do **not** touch `CommonAnalysisListener` / `RPCAnalysisListener` / `SegmentAnalysisListener` unless the OAP core protocol gains a new `SpanLayer` enum value — details in §2.\n- **If you find yourself editing more than this skill suggests, stop and confirm.** Scope creep on trace analysis, span-layer mapping, receiver handlers, or protobuf/proto contracts is a strong signal the framing is wrong. Ask the developer before committing that direction.\n- **Verify every new feature locally before pushing to CI.** Compile + checkstyle + license-eye are minimums, but they only tell you the code builds — they do **not** tell you the new MAL rule compiles its closures at startup, the LAL layer:auto dispatch works, the OTLP payload shape matches the MAL labels, the UI template renders against real data, or the layer shows up in `swctl service ly <LAYER>`. Run the new e2e case end-to-end locally against a live OAP (see [run-e2e skill](../run-e2e/SKILL.md)) and fire every verify step with `swctl` by hand at least once. Pushing to CI without a local green run wastes shared CI cycles and slow-loops review.\n\n---\n\n## 0. Register the `Layer` — the feature's entry point\n\nA `Layer` is how OAP slices services / instances / endpoints by data source. The UI, storage partitioning, menu navigation, and OAL aggregation all key off it.\n\n`Layer` is a registry-backed value type (no longer a closed enum). Built-in layers are declared as `public static final Layer` constants in `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java`; external layers are registered through the `Layer.register(name, ordinal, normal)` API at boot. **Pick the registration path that matches the scope of your feature:**\n\n| Your feature ships as | Registration path |\n|---|---|\n| Part of the OAP distribution (in-tree, the common case for new SkyWalking-supported targets) | Add a `public static final Layer` constant to `Layer.java` with the next sequential ordinal in `0–49`. Examples: `IOS = register(\"IOS\", 47, true)`, `APISIX = register(\"APISIX\", 21, true)`, `VIRTUAL_DATABASE = register(\"VIRTUAL_DATABASE\", 14, false)` for inferred/non-real services. |\n| An out-of-tree MAL or LAL rule file | Add a top-level `layerDefinitions:` block to the rule file. The DSL loader funnels each entry through `Layer.register` before compiling the rule. One file ships the layer + the rules that produce its telemetry. |\n| An out-of-tree plugin module (jar) | Implement `org.apache.skywalking.oap.server.core.analysis.LayerExtension` and register via `META-INF/services/`. Discovered by `LayerExtensionLoader` during `CoreModuleProvider.prepare()`. |\n| Operator-deployed config (no code, no DSL) | Add an entry to `oap-server/server-starter/src/main/resources/layer-extensions.yml` (or override on the OAP node's classpath). |\n\n**Ordinal conventions:** `0–49` is in active use by built-ins. `50–999` is reserved by convention for future built-in layers. External layers are recommended (not required) to start at `>= 1000` to avoid colliding with future built-ins on OAP upgrade. Collisions in either direction are detected at boot via the ordinal-uniqueness check, which fails OAP startup loudly.\n\n**Storage encoding is the ordinal int**, persisted in BanyanDB / Elasticsearch / JDBC. Every OAP node that reads or writes a given layer must agree on its `(name, ordinal)` mapping — deploy `layer-extensions.yml` and any `layerDefinitions:` rule files identically across all nodes. The registry is sealed at the start of `Core.notifyAfterCompleted()`; later registration attempts throw.\n\n**UI template folders are auto-discovered by file scan, not by `Layer.values()`.** `UITemplateInitializer` walks `ui-initialized-templates/**/*.json` recursively (depth 2) and trusts each template's own `configuration.layer` field. Drop a folder of dashboard JSONs on disk and the initializer picks them up on the next boot — folder name is purely organizational.\n\n**Component ID lookup in Java code**: IDs declared in `component-libraries.yml` are loaded at runtime into `ComponentLibraryCatalogService`'s `componentName2Id` map — they are **not** exposed as Java enum constants. To look up by name in listener code, inject the catalog service and resolve once at construction:\n```java\nIComponentLibraryCatalogService catalog = moduleManager.find(CoreModule.NAME)\n    .provider().getService(IComponentLibraryCatalogService.class);\nint myComponentId = catalog.getComponentId(\"My-Component-Name\");\n```\nCache as an `int` field; runtime comparisons are then plain `componentId == myComponentId`. **Trap:** there is a `ComponentsDefine` class under `skywalking-trace-receiver-plugin/src/test/java/.../mock/ComponentsDefine.java` — it is a test-only mock holding five hand-picked constants (Tomcat, Dubbo, RocketMQ, MongoDB). Do not import or extend it from production code.\n\nEmit the layer from every source object your feature produces. Built-in layers have a static-field accessor; external layers are looked up by name through the registry:\n\n```java\n// Built-in layer (constant)\nservice.setLayer(Layer.IOS);\n\n// External layer (registered via yaml / SPI / layerDefinitions:)\nservice.setLayer(Layer.nameOf(\"IOT_FLEET\"));\n```\n\nDownstream (the core OAL, `service ly <LAYER>` swctl query, topology filters, UI root dashboard's layer selector) all work off this single enum value.\n\n---\n\n## 1. Pick the right extension point\n\n| Signal you want to process | Extension point | Register via |\n|---|---|---|\n| **SkyWalking native trace segments** (from language agents) | `AnalysisListener` (TraceSegmentListener / EntryAnalysisListener / ExitAnalysisListener / LocalAnalysisListener / FirstAnalysisListener) + factory in `agent-analyzer` | `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory` |\n| **OTLP / Zipkin trace spans** (OTel SDKs, Istio, Envoy, iOS/Android, …) | `SpanListener` in `core.trace` | `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener` |\n| **OTel metrics** (Prometheus scrape, Envoy stats, Istio, …) | MAL rules — `otel-rules/<namespace>/*.yaml` | Append the rule path to `enabledOtelMetricsRules` in `application.yml` |\n| **Logs** (OTLP, SkyWalking native, Fluent Bit, Envoy ALS) | LAL rules — `lal/<rule>.yaml` | Append to `SW_LOG_LAL_FILES` (env) or `lalFiles` (YAML) under `log-analyzer` |\n| **Pre-computed meter / custom meter** (custom agent protocol, meter receiver) | MAL `meter-analyzer-config/*.yaml` | Append to `meter-analyzer-config` entries |\n| **Aggregation from Source entities** (service, instance, endpoint, relation) | OAL rules — `oal/*.oal` | Already loaded by `OAL grammar`; just add lines |\n\nAll five DSLs (OAL, MAL, LAL, Hierarchy + SpanListener SPI) compile via ANTLR4 + Javassist at startup. Source-of-truth details are in:\n- [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md)\n- [`docs/en/concepts-and-designs/mal.md`](../../../docs/en/concepts-and-designs/mal.md)\n- [`docs/en/concepts-and-designs/lal.md`](../../../docs/en/concepts-and-designs/lal.md)\n- [`docs/en/concepts-and-designs/scope-definitions.md`](../../../docs/en/concepts-and-designs/scope-definitions.md)\n- [`oap-server/analyzer/meter-analyzer/CLAUDE.md`](../../../oap-server/analyzer/meter-analyzer/CLAUDE.md) — MAL compiler internals\n- [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — LAL compiler internals\n\n---\n\n## 2. Trace ingestion — two parallel pipelines\n\nSkyWalking OAP ingests traces through two distinct entry points. Pick based on source.\n\n> **Before writing any trace extension — ask \"do I actually need one?\"**\n>\n> For **most new layers you do not touch trace analysis at all.** Segments (native or\n> OTLP) flow through the existing pipeline, `RPCAnalysisListener.parseExit` produces\n> the outbound `ServiceRelation` edges, the backend-side agent produces the inbound\n> metrics via its own `parseEntry`, and the new-layer Service / Instance / Endpoint\n> entities come from **MAL + LAL**, not from trace analysis. This is how browser\n> (`Layer.BROWSER`), iOS (`Layer.IOS`), mobile mini-programs\n> (`Layer.WECHAT_MINI_PROGRAM` / `Layer.ALIPAY_MINI_PROGRAM`), and most\n> scraped-metric layers work.\n>\n> Extend trace analysis **only** when the feature genuinely needs per-span\n> interpretation that the existing pipeline cannot express:\n>\n> - **A new `SpanLayer`** — OAP core protobuf has a new `SpanLayer` enum value that\n>   callers of `identifyServiceLayer` / `identifyRemoteServiceLayer` must dispatch on.\n>   That means touching `CommonAnalysisListener` / `RPCAnalysisListener`.\n> - **A genuinely new signal class not covered by existing listeners** — e.g. the GenAI\n>   work added `GenAISpanListener` for LLM / MCP instrumentation because OTLP spans\n>   carry semantic-convention attributes (`gen_ai.*`, `mcp.*`) that no prior listener\n>   extracted. That is rare.\n>\n> **Anti-pattern we've been bitten by:** using a componentId mapping in\n> `CommonAnalysisListener` to assign a client-side layer to exit-only segments. This\n> does not produce inbound metrics anyway (no entry spans → no\n> `service_cpm` / `endpoint_cpm` from OAL), and it is orthogonal to the Service entity\n> layer (which MAL / LAL sets). Mirror the browser pattern instead: ship `Layer` +\n> `component-libraries.yml` + MAL + LAL + dashboards, and leave the listener chain\n> alone.\n\n### 2.1 SkyWalking native segments — `AnalysisListener`\n\n**Source protocol**: `apm-protocol/apm-network/src/main/proto/language-agent/Tracing.proto` (`TraceSegmentObject`).\n\n**Receiver**: `oap-server/server-receiver-plugin/skywalking-trace-receiver-plugin`. After wire decoding, segments are handed to `SegmentParserServiceImpl` which fans out to registered `AnalysisListenerFactory` implementations.\n\n**Extension**: implement one or more of\n- `FirstAnalysisListener` — per-segment, before per-span phase\n- `EntryAnalysisListener` — per-entry-span\n- `ExitAnalysisListener` — per-exit-span\n- `LocalAnalysisListener` — per-local-span\n- `TraceSegmentListener` — whole-segment, after per-span phase\n\nRegister the **factory** in `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory`. Existing examples: `RPCAnalysisListener`, `MultiScopesAnalysisListener`, `SegmentAnalysisListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`.\n\nUse when: the data arrives on the SW native wire (language agents — Java, Python, Go, Node.js, .NET, PHP, Lua, etc.).\n\n### 2.2 OTLP / Zipkin spans — `SpanListener`\n\n**Source protocol**: OTLP proto (`opentelemetry/proto/trace/v1/trace.proto`) or Zipkin v2 JSON.\n\n**Receiver**: `oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/.../otlp/OpenTelemetryTraceHandler.java` (OTLP) and `oap-server/server-receiver-plugin/zipkin-receiver-plugin/.../ITZipkinTraceCollector` (Zipkin). After decoding, spans are iterated and passed to every registered `SpanListener` through `SpanListenerManager`.\n\n**Extension**: implement `org.apache.skywalking.oap.server.core.trace.SpanListener`:\n\n```java\npublic interface SpanListener {\n    String[] requiredModules();\n    void init(ModuleManager moduleManager);\n    SpanListenerResult onOTLPSpan(OTLPSpanReader span, String serviceName, Map<String,String> resourceAttributes,\n                                  String scopeName, String scopeVersion);\n}\n```\n\nName the service by `serviceName`, not the resource's `service.name`: it is the service the span is analyzed and\nstored under, which `receiver-otel.otlpServiceGroupKeys` may have grouped (`shop::cart` for `service.name=cart`).\n\nRegister in `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener`. Reference implementations:\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSHTTPSpanListener.java` — OTel Swift URLSession → OAL (Service / ServiceInstance / Endpoint)\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSMetricKitSpanListener.java` — OTel Swift MetricKit → MAL samples via `OpenTelemetryMetricRequestProcessor.toMeter`\n- `oap-server/analyzer/gen-ai-analyzer/src/main/java/.../GenAISpanListener.java` — LLM / MCP instrumentation\n\nReturn `SpanListenerResult.builder().shouldPersist(false).build()` to prevent a span being stored as a trace (useful for e.g. 24-hour aggregate spans that are not meaningful as traces).\n\n**When both native and OTLP should feed the same metric**, write your extension against the downstream scope (Service / ServiceInstance / Endpoint + OAL) and emit those entities from **both** a SegmentListener and a SpanListener. The OAL layer is protocol-agnostic.\n\n---\n\n## 3. Metrics — MAL / OAL rules\n\n### 3.1 OAL (for entities already produced by core sources)\n\n`oap-server/server-starter/src/main/resources/oal/core.oal` already defines the standard `service_cpm`, `service_resp_time`, `service_sla`, `endpoint_cpm`, etc. — keyed off the `Service`, `ServiceInstance`, `Endpoint` source scopes. If your new layer emits those entities with the right `Layer`, you get the core metrics for free.\n\n**Who emits those entities matters.** `service_cpm` / `service_resp_time` / `service_sla` / `endpoint_cpm` / `endpoint_resp_time` come from `RPCAnalysisListener.parseEntry` → `callingIn.toService()` / `toEndpoint()`. That only fires on **inbound entry spans**. Client-side / edge layers (browser, iOS, mini-programs) only emit exit spans, so these OAL metrics do **not** populate under those layers — their Service entities are produced by MAL / LAL, and their dashboards use MAL metrics (`meter_<layer>_*`) for load / latency, not `service_cpm` / `service_resp_time`. Don't wire dashboards to OAL-derived metrics that the layer never produces.\n\nAdditional OAL rules go in the same file. OAL grammar: `oap-server/oal-grammar/src/main/antlr4/OALParser.g4`. Syntax and examples: [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md).\n\n### 3.2 MAL (for meter / metric samples)\n\n**File location**: `oap-server/server-starter/src/main/resources/otel-rules/<namespace>/*.yaml` (OTel scrape) or `meter-analyzer-config/*.yaml` (SW native meter).\n\n**Registration** (MANDATORY): append the rule path to the default in `oap-server/server-starter/src/main/resources/application.yml`:\n```yaml\nreceiver-otel:\n  default:\n    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:\"...,<your-namespace>/*\"}\n```\nIf you skip this, the rule isn't loaded in the default starter and downstream `converters.forEach(...)` is a silent no-op — samples just disappear.\n\n**Also update** `test/e2e-v2/cases/storage/expected/config-dump.yml`. This e2e diffs `/debugging/config/dump` output against a static file; every default-value change in `application.yml` must mirror here or the storage e2e fails.\n\n**Naming**: keep attribute scope in the metric name. If a metric reads from `foo.bar.foreground.exit_count`, call it `foo_foreground_exit_count`, not `foo_exit_count`. When two attributes (foreground vs background, ingress vs egress) share a theme, keep them as separate distinguishable metrics — don't flatten.\n\n### 3.3 Histogram pitfalls (both OAL and MAL)\n\n- **Per-bucket counts, not cumulative.** `histogram_percentile` accumulates internally. Feeding cumulative counts doubles the accumulator and shifts percentiles up. Each observation contributes `1` to exactly one `le` bucket.\n- **`le` label scale.** `SampleFamily.histogram()` rescales `le` by `defaultHistogramBucketUnit.toMillis(1)` (default `SECONDS` ⇒ ×1000). If your raw labels are already in ms, use `SampleFamilyBuilder.newBuilder(samples).defaultHistogramBucketUnit(TimeUnit.MILLISECONDS).build()`.\n- **Overflow bucket.** MAL parses `le=\"Infinity\"` as `Double.POSITIVE_INFINITY` → stored as `Long.MAX_VALUE`. If a percentile lands there, the UI renders ≈ 9.2 × 10¹⁸. Prefer a finite sentinel (e.g., 30 s for hang/launch) so outliers remain visible without crashing the chart.\n- **`NamingControl`** — every `service` / `serviceInstance` / `endpoint` label that ends up as an entity key must go through `namingControl.formatServiceName()` / `formatInstanceName()` / `formatEndpointName()`. Raw resource attributes (long git-SHA build metadata, path-style endpoints) will otherwise exceed storage limits.\n\n---\n\n## 4. Logs — LAL\n\n**File location**: `oap-server/server-starter/src/main/resources/lal/<feature>.yaml`.\n\n**Registration**: append to `SW_LOG_LAL_FILES` env var, or `log-analyzer.default.lalFiles` in `application.yml`.\n\n**Grammar**: `oap-server/analyzer/log-analyzer/src/main/antlr4/LALParser.g4`. The compiler docs are thorough: [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — covers:\n- `filter { … }` block with `parsed.*`, `tag()`, `sourceAttribute()`, `log.*`, `def` locals, `extractor {}`, `metrics {}`, `sink {}`\n- `layer: auto` mode (script decides layer from source attrs)\n- `inputType` / `outputType` SPI (`LALSourceTypeProvider`) to bind typed proto to `parsed.*`\n- Test harness at `oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/oap-cases/` — ship YAML + `.data.yaml` pair; `LALScriptExecutionTest` picks them up automatically.\n\n**`layer: auto` gotcha**: the extractor's `layer \"X\"` assignment must propagate to `LogMetadata.layer` (not just the output builder) — `FilterSpec.doSink` reads metadata to decide drop/route. The current codegen (`LALBlockCodegen.generateFieldToOutput`) handles this correctly; if you extend codegen, preserve that invariant.\n\n---\n\n## 5. UI — templates, submodule, and menu\n\n### 5.1 Dashboard templates\n\nJSON files under `oap-server/server-starter/src/main/resources/ui-initialized-templates/<feature>/`. Conventional fileset:\n\n- `<feature>-root.json` — `ServiceList` widget, layered service index.\n- `<feature>-service.json` — per-service dashboard, usually a `Tab` with Overview / Instance / Endpoint / Log sub-tabs.\n- `<feature>-instance.json` — per-instance dashboard.\n- `<feature>-endpoint.json` — per-endpoint dashboard.\n- Menu link in `oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml`.\n\n> **Collaborate with the developer on dashboard design.** Dashboard layout and metric selection are product decisions, not derivable from the MAL rules alone. Before authoring the JSON, check in with the developer on: which metric goes on which panel, which percentiles / aggregations to show, what's on the root list vs. per-service vs. per-instance vs. per-endpoint pages, what stays in the Overview tab vs. a sub-tab, widget sizes / ordering, and whether Trace / Log tabs belong at all (depends on whether the feature emits traces and logs). Then **manually set up the feature against a live OAP** — emit a few real OTLP payloads, open each dashboard in the UI, confirm every widget renders non-empty, and walk the drill-down from root → service → instance → endpoint. Only after that eyes-on pass is the dashboard ready to ship.\n\nDetails that always bite:\n\n- **Tab chrome eats ~4 rows.** Set outer `Tab.h = content_max_y + 4` or you get an inner scrollbar on top of the page scrollbar.\n- **Widget heights consistent.** Use `h=15` per row for `Line` / `Bar` widgets to match mesh / general peer templates. Don't mix h=12 / h=15 / h=24 across a feature's dashboards.\n- **List widgets (ServiceList / InstanceList / EndpointList)** render two pieces per cell:\n  - `expressions` → the aggregated summary number (right side). Required. Wrap raw metrics in `avg(...)` / `sum(...)`. Omitting it leaves the column empty.\n  - `subExpressions` → the sparkline trend (left side). Optional, falls back to `expressions` if absent.\n  - `metricConfig` pairs 1-to-1 with `expressions` (not `subExpressions`). Use for `label` (column header) and `unit`.\n  - Compound columns via MQE addition: `avg(metric_a + metric_b)` in `expressions`, `metric_a + metric_b` in `subExpressions`.\n  - Normal widgets with multiple expressions on one chart — always give a human label per expression in `metricConfig`, otherwise legend renders raw metric names.\n\n### 5.2 Hot-reload (edits to shipped templates take effect on restart)\n\n`UITemplateInitializer` normally uses `addIfNotExist` (keyed on JSON `id`), so on a plain restart your edits to shipped templates are ignored — the seeded copy in storage wins. For dev / extension work, set the environment variable and restart:\n\n```bash\nSW_UI_TEMPLATE_FORCE_RELOAD=true\n# then restart OAP — shipped templates overwrite the seeded copies via addOrReplace.\n# Hard-refresh the browser (Cmd+Shift+R) because the booster UI caches config in memory.\n```\n\nThis is a dev / extension knob only — read directly from the OS environment, not wired through `application.yml`. Unset or `false` preserves the default production behavior where operators' UI edits persist across restarts.\n\n### 5.3 New top-level menu group (e.g. \"Mobile\", \"Gateway\")\n\nNew **child entries** only need template JSON + `menu.yaml`. A new **top-level group** needs submodule work in `skywalking-ui` (git submodule at `skywalking-ui/`):\n\n1. `skywalking-ui/src/assets/icons/<name>.svg` — menu icon.\n2. `skywalking-ui/src/locales/lang/menus/{en,es,zh}.ts` — i18n labels (`<name>`, `<name>_desc`, `<name>_<child>`, `<name>_<child>_desc`). Missing i18n falls back to raw keys on screen.\n3. Commit those upstream in `apache/skywalking-booster-ui`, bump the submodule pointer here.\n4. Rebuild with `-Pui,dist` (not `-Pbackend,dist`) and `make docker.ui`, otherwise the new assets aren't in the image. See [package skill](../package/SKILL.md).\n\n---\n\n## 6. Docs set — everything that moves together\n\n| File | What to add |\n|---|---|\n| `docs/en/setup/backend/<feature>-monitoring.md` | User-facing setup guide. Required SDK / collector, required env vars, sample OTLP payload, metrics / log tags list. |\n| `docs/menu.yml` | Add the guide to the docs navigation tree. |\n| `docs/en/security/README.md` | **Client-side monitoring**: mobile / browser / mini-program endpoints accept data from the public internet without auth by default. Add a rate-limit + abuse-protection note. Agent-side monitoring over private VPC usually doesn't need this. |\n| `docs/en/changes/changes.md` | Per-release changelog. Add under `#### OAP Server` for features/bug-fixes, `#### UI` if the submodule bumps, `#### Documentation` if new guides. |\n| `docs/en/swip/SWIP-<N>.md` | Design doc for non-trivial features; keep reconciled with final code after review. |\n| `docs/en/swip/readme.md` | Move SWIP from \"Proposed\" to \"Accepted\" once merged. |\n\n---\n\n## 7. E2E test\n\nFolder: `test/e2e-v2/cases/<feature>/`. See [run-e2e skill](../run-e2e/SKILL.md) for harness basics (setup / trigger / verify phases, template functions, expected-file authoring via `contains`).\n\nFeature-specific extras:\n\n1. **Docker-compose**: extend `../../script/docker-compose/base-compose.yml` oap + storage. Set `SW_OTEL_RECEIVER_ENABLED_HANDLERS`, `SW_LOG_LAL_FILES`, and any feature-specific env. Don't list default rules here — rely on `application.yml` so the config-dump test stays consistent.\n2. **Register in CI**: add an entry under the e2e matrix in `.github/workflows/skywalking.yaml`.\n3. **OTLP data hygiene**:\n   - `spanId` is exactly 16 hex chars; `traceId` is exactly 32. Protobuf-JSON decodes them as base64; other lengths return HTTP 400.\n   - Setup-step curl loops must fail loudly: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e`. The pattern `for …; do curl && break || sleep 5; done` silently succeeds when every attempt connection-refused (the final `sleep` returns 0) because OAP takes ~50 s to start in CI.\n4. **Verify each case by hand before pushing** and use a published simulator image where the upstream SDK has one. See [`run-e2e` skill §6–§9](../run-e2e/SKILL.md) for verify-query triage, expected-file authoring, and the list of expected-file traps (YAML quoting on colons, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl loops that swallow failure).\n5. **Config dump**: when you modify any `application.yml` default (rule list, handler list, ...), also update `test/e2e-v2/cases/storage/expected/config-dump.yml`.\n\n---\n\n## 8. Pre-submit checklist\n\nRun in order; each has a dedicated skill:\n\n1. [`compile`](../compile/SKILL.md) — `./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall -Dmaven.test.skip -Dcheckstyle.skip -Dgpg.skip`. Javadoc `[ERROR] ... error:` must be fixed; `[ERROR] ... warning:` is noise.\n2. Tests: `./mvnw -pl oap-server/analyzer/log-analyzer -am test -Dtest=LALScriptExecutionTest` for LAL changes, equivalent for MAL, OAL unit tests where the rule lives.\n3. [`license`](../license/SKILL.md) — `license-eye header check` (Apache 2.0 headers). If the UI submodule bumped npm deps, also `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit the `LICENSE` diff.\n4. [`package`](../package/SKILL.md) — full `make docker` (or dist-rebuild + `make docker.oap` / `make docker.ui`). Don't trust `docker.oap` alone to pick up code changes; the dist tarball is a file-level prereq.\n5. [`run-e2e`](../run-e2e/SKILL.md) — run the new case end-to-end locally before pushing.\n6. [`gh-pull-request`](../gh-pull-request/SKILL.md) — commit and open PR with the template, attach UI screenshots.\n\n---\n\n## 9. Cross-cutting traps collected from real extensions\n\n| Trap | Symptom | Fix |\n|---|---|---|\n| Layer folder named with hyphens | `ui-initialized-templates/my-layer/*.json` on disk but dashboard empty | Folder must be `Layer.name().toLowerCase()` (underscores). `wechat_mini_program/`, not `wechat-mini-program/` |\n| Extending `CommonAnalysisListener` for a client-side layer | Added componentId → layer mapping \"to get topology for free\"; then `service_cpm` etc. don't populate anyway | Client-side (exit-only) layers don't need trace-analysis changes. Ship `Layer` + `component-libraries.yml` + MAL + LAL. See §2 header for when extending trace analysis is actually justified |\n| Dashboards wired to OAL-derived metrics on a client-side layer | Charts stay empty — `service_cpm` / `service_resp_time` / `endpoint_cpm` never populate | Those come from inbound `parseEntry`, which mini-program / browser / iOS don't emit. Use MAL metrics (`meter_<layer>_*`) instead |\n| LAL `#` comment in DSL body | OAP boot fails with `extraneous input 'metrics'` at the `#` line | LAL grammar accepts `//` and `/* */` only. No `#` comments |\n| LAL bare def-var value access — `tag 'x': myvar` | OAP boot fails with `Cannot resolve expression '[myvar]'` | `LALValueCodegen` only treats def-vars as valid when there's a method chain (`myvar?.foo`). Inline `sourceAttribute(...)` or `tag(...)` in the three value-access sites instead of storing to a def, or add a no-op chain like `myvar.toString()`. Conditions (`if (myvar == ...)`) go through a different codegen path and DO handle bare def refs |\n| MAL rule file with multi-doc YAML (`---` separator) | `MALFilterExecutionTest` fails with `expected a single document`; OAP would also fail at startup via `Rules.loadRules` (calls `Yaml.loadAs`, single-doc) | Split into one file per top-level `expSuffix` / `filter` / `metricPrefix` combo. Reference each file by name in `application.yml` (`lalFiles`, `malFiles`, `enabledOtelMetricsRules`) |\n| Histogram bucket bounds in ms with default `.histogram()` call | Percentile values come out 1000× inflated (`P50 = 100000` for a 100 ms request) | MAL's default `defaultHistogramBucketUnit = SECONDS` multiplies `le` by 1000 when converting to stored ms. For SDKs that emit ms bounds directly (mini-program, most in-house JS SDKs), use `.histogram(\"le\", TimeUnit.MILLISECONDS)`. `TimeUnit` is registered in `MALCodegenHelper.ENUM_FQCN` so rule YAML can reference it |\n| `.rate()` / `.increase()` on a DELTA counter / histogram `_count` | Compute delta-of-delta — nonsense values | OTLP DELTA means each flush is already the per-flush count. MAL's per-minute aggregator sums directly to requests-per-minute. For CUMULATIVE (Prometheus default), use `.rate(\"PT1M\")` for per-sec or `.increase(\"PT1M\")` for per-min; they are **not** equivalent (differ by 60×) |\n| MAL bucket-family name with `_histogram` suffix | Rule compiles, metric stays empty | `PrometheusMetricConverter` emits the bucket family as `<metric_name>` (no suffix), with `<metric_name>_count` and `<metric_name>_sum` as the companion families. Reference the bucket family by its bare name in the rule |\n| LAL `tag(\"key\") == null` check on a missing tag | Check never triggers — LAL's `tag()` returns empty string `\"\"` for absent tags, not `null`, so conditional abort is a no-op | Guard on both: `tag(\"key\") == null || tag(\"key\") == \"\"`. Same for `sourceAttribute(\"key\")` |\n| MAL rule file with `expSuffix: service(...)` mixing service-scoped and endpoint-scoped rules | Endpoint rules' metrics appear under the service entity (no endpoint_traffic rows created); `swctl endpoint ls` returns empty. `MetricConvert.formatExp:132` appends `(exp).expSuffix` to every rule, so `.endpoint(...)` in the rule's expression is followed by `.service(...)` and the service scope wins. | Follow the APISIX pattern — leave `expSuffix:` empty and explicitly chain `.service(...)` / `.endpoint(...)` on each rule. If every rule in a file is same-scope (e.g. all instance-scoped in `*-instance.yaml`), `expSuffix` is still fine |\n\nE2E-side authoring traps (log-content YAML quoting, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl swallowing failure) live in the [`run-e2e` skill §9](../run-e2e/SKILL.md).\n| Missing layer-root template | Menu item lands on empty \"no dashboard\" view | `Layer.vue:41-44` requires a template with `isRoot: true`; ship a `<layer>-root.json` (precedent: `ios/ios-root.json`) |\n| New layer not selectable in UI / swctl | `service ly <LAYER>` returns empty | Add the enum in `Layer.java` with a fresh id; see §0 |\n| Stale dist in image | `make docker.oap` succeeds but behavior is old | See [`package` skill](../package/SKILL.md); rebuild dist first, then image |\n| Shipped template edits don't render after restart | Edited JSON on disk, but OAP shows old copy | Set `SW_UI_TEMPLATE_FORCE_RELOAD=true` (environment variable, not `application.yml`) and restart; OAP writes `addOrReplace`. Unset in production so operator UI edits aren't clobbered. |\n| MAL service-scoped rule fragmented by instance | \"Overall app\" metric has N series per version | Keep `service_instance_id` out of service-scoped `.sum`/`.avg`. Put instance-scoped metrics in a separate `<feature>-instance.yaml` with `expSuffix: instance(...)` — see `otel-rules/ios/` pattern. |\n| Config dump drift | Storage e2e fails after `application.yml` edit | Mirror the change in `test/e2e-v2/cases/storage/expected/config-dump.yml` |\n| LAL `layer:auto` WARN drop | Every log silently dropped | Codegen must set `metadata.layer` alongside output builder's layer |\n| Histogram 1000× off | p50 = 2 000 000 ms | `defaultHistogramBucketUnit(MILLISECONDS)` on the `SampleFamily` |\n| Histogram shifted up | p50 = 5 000 on 500–2500 ms data | Emit per-bucket, not cumulative |\n| Empty list column | Dashboard instance table has no numbers | Add `expressions` (not just `subExpressions`) |\n| Double scroll | Inner tab scroll + page scroll | `Tab.h = content_max + 4` chrome buffer |\n| Raw metric name in legend | Multiple expressions on one widget | Add parallel `metricConfig[].label` entries |\n| `spanKind` mismatch | Span skipped silently | `OTLPSpanReaderImpl.spanKind()` returns enum `.name()` like `SPAN_KIND_CLIENT`, not `CLIENT` |\n| Negative latency | `*_sla` inflated | Clamp `rawLatency < 0 ? 0 : (int) min(Integer.MAX_VALUE, rawLatency)` |\n| SLA too optimistic | Client-side DNS / timeout / connection-refused counted as success | Require a real response code for success: `status = code > 0 && code < 400`. Do not treat missing (0) as success. |\n| Silent drop with no rules | `converters == null` NPE elsewhere | Init to empty list at declaration |\n| `+Inf` bucket surfaces as `9.2e18` | UI shows garbage when a sample lands in overflow | Use finite sentinel bucket (e.g. 30 s) |\n| OTel submodule bump misses UI image | New icons / i18n not visible | Rebuild with `-Pui,dist` + `make docker.ui` |\n| LICENSE drift after submodule bump | CI `Dependency licenses` fails | `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit |\n| Setup-step curl loop swallows failure | e2e passes metrics but fails log verify | `curl -f --retry-connrefused --retry-all-errors` + `set -e` |\n| Verify case flag mismatch | CI fails early on `swctl` flag unknown to pinned commit | Fire every verify command locally first; bump `SW_CTL_COMMIT` if needed |\n\n---\n\n## 10. Reference extensions (read their code)\n\nBrowse these when you need a template to copy from. Grouped by the kind of data source, because the extension points differ significantly:\n\n### SW native trace (language agents) — the \"General\" layer and friends\n\n`Layer.GENERAL` is the default layer for services reporting via the SkyWalking native trace protocol (Java / Python / Go / Node.js / .NET / PHP / Lua / Rust agents). It's the richest reference because the native pipeline is where the **full topology** lives:\n\n- **Service → service dependency** (topology): derived by `MultiScopesAnalysisListener` and `NetworkAddressAliasMappingListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`. Entry/exit spans produce `ServiceRelation`, `ServiceInstanceRelation` sources.\n- **Endpoint → endpoint dependency**: `RPCAnalysisListener` emits `EndpointRelation` sources from cross-process refs and exit spans. This is unique to the native protocol — OTLP spans don't preserve upstream entry-span metadata the same way, so OTLP-only layers generally lose endpoint-level topology.\n- OAL aggregation: `oap-server/server-starter/src/main/resources/oal/core.oal` — all `service_relation_*`, `service_instance_relation_*`, `endpoint_relation_*` metrics are already defined there; a new native-tracing layer just needs to produce the right sources with the right `Layer`.\n\nRead order: `AnalysisListenerFactory` SPI registration → `SegmentParserServiceImpl` fan-out → individual listeners → OAL rules.\n\n### Service-mesh (Istio / Envoy ALS) — topology from proxy-side signals\n\nNo agent in the service; Envoy emits HTTP/TCP access logs (ALS) and metrics. Service-to-service topology is reconstructed from ALS request/response pairs.\n\n- `oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/` — ALS receiver + `EnvoyAccessLogBuilder` persistence (LAL output type).\n- `otel-rules/istio-*`, `otel-rules/envoy/*` — MAL rules for proxy-emitted OTel metrics.\n- `lal/envoy-als.yaml` — LAL rule that extracts topology-generating fields from ALS.\n- Layer: `Layer.MESH`, plus `Layer.MESH_CP` (control plane) and `Layer.MESH_DP` (data plane) for split-plane topology.\n\nTopology parity with native traces: ALS gives you service-level relations; endpoint-level relations are approximate (proxy sees inbound + outbound, not the full call-chain that a native agent observes).\n\n### OTLP-only layers (OTel SDKs, iOS/Android, AI gateway, …)\n\nOTLP spans feed a `SpanListener`. Topology is **partial by construction** — an OTel Swift SDK on a mobile device only knows about its own outbound requests; there's no upstream caller. So these layers typically produce only `Service` / `ServiceInstance` / `Endpoint` sources (no `*Relation`) and the Topology view shows isolated islands, by design.\n\n- **iOS** (`Layer.IOS`, OTLP traces + MetricKit + LAL diagnostics): `oap-server/analyzer/ios-analyzer/` + `docs/en/swip/SWIP-11.md`.\n- **Envoy AI Gateway** (`Layer.ENVOY_AI_GATEWAY`, LLM / MCP, OTLP metrics + Zipkin sampled traces + LAL access-log sampling): `oap-server/analyzer/gen-ai-analyzer/`, `otel-rules/envoy-ai-gateway/*`, `lal/envoy-ai-gateway.yaml`.\n\n### OTel-metrics-only layers (scraped exporters)\n\nNo spans, no logs — just MAL rules converting OTel metric scrapes.\n\n- **MySQL / PostgreSQL**: `otel-rules/mysql/*`, `otel-rules/postgresql/*`.\n- **Kubernetes**: `otel-rules/k8s/*` — paired with `virtual-k8s-service` (`Layer.K8S_SERVICE`) sources for synthetic services.\n- **AWS**: `otel-rules/aws-eks/*`, `otel-rules/aws-s3/*`, `otel-rules/aws-dynamodb/*`.\n\n### SW native meter (custom protocol)\n\nNo OTel; the agent / plugin sends pre-built meter samples on the SW `MeterProtocol`.\n\n- **Nginx Lua** (`Layer.NGINX`): `meter-analyzer-config/nginx-service.yaml`, with the Lua agent in `apache/skywalking-nginx-lua`.\n- **APISIX** (`Layer.APISIX`): `meter-analyzer-config/apisix-service.yaml` + LAL trace extraction.\n\n### Which to copy from\n\n| You're adding | Closest reference |\n|---|---|\n| Language agent (new JVM / runtime) | The General / Java layer flow; extend `AnalysisListener` SPI. |\n| Service mesh / proxy | `mesh-*` + `envoy-*` rules; LAL on ALS. |\n| Mobile / browser / mini-program SDK (OTLP) | `ios-analyzer` (OTLP client-side). Expect no topology relations. |\n| Database / middleware (metrics only) | `mysql`, `postgresql` — pure MAL rules. |\n| Cloud service (no agent, just metrics) | `aws-eks` / `aws-s3`. |\n| Custom protocol / home-grown telemetry | `apisix` / `nginx` — SW native meter. |\n",".claude/skills/ci-e2e-debug/SKILL.md":"---\nname: ci-e2e-debug\ndescription: Download and inspect CI e2e test logs from GitHub Actions artifacts. Use when investigating e2e test failures in CI.\nargument-hint: \"<RUN_ID or GitHub Actions URL>\"\n---\n\n# CI E2E Debug Skill\n\nDownload test log artifacts from a GitHub Actions CI run and inspect them for errors.\n\n## Steps\n\n1. **Find the artifact**: Use `gh api` to list artifacts for the given CI run:\n   ```bash\n   gh api repos/apache/skywalking/actions/runs/<RUN_ID>/artifacts --jq '.artifacts[] | {id: .id, name: .name}'\n   ```\n\n2. **Download and extract**: Download the artifact zip and extract it:\n   ```bash\n   cd /tmp && rm -rf e2e-debug-logs && mkdir e2e-debug-logs && cd e2e-debug-logs\n   gh api repos/apache/skywalking/actions/artifacts/<ARTIFACT_ID>/zip > artifact.zip\n   unzip -o artifact.zip\n   ```\n\n3. **Inspect OAP logs**: Look for errors in the OAP server logs:\n   ```bash\n   # Find OAP log files\n   find /tmp/e2e-debug-logs -name \"skywalking-oap-*.log\" -o -name \"oap.log\"\n   # Check for errors\n   grep -E \"ERROR|Exception|FATAL|CannotCompileException\" <log_file> | head -30\n   ```\n\n4. **Inspect other component logs**: Check BanyanDB, UI, and other pod logs as needed.\n\n5. **Report findings**: Summarize the root cause error from the logs.\n\n## Notes\n- CI artifacts are automatically uploaded by the e2e test framework to `$SW_INFRA_E2E_LOG_DIR`\n- Log files are organized by namespace/pod name\n- OAP init pods may have different errors than the main OAP pod — check all of them\n- Common errors: MAL/LAL/OAL compilation failures, storage connection issues, module initialization errors\n- **Profile exporter tests** (e.g., Trace Profiling ES): the exporter runs as a separate process with `MockCoreModuleProvider`. If a new service is added to `CoreModule.services()` but not registered in the mock, the exporter fails at startup with `requiredCheck()` error — but the OAP logs will show no errors (OAP is fine, the exporter subprocess is what fails). Check the test step that runs `profile_exporter.sh`.\n- **When OAP logs are clean but tests fail**: look at which specific test step failed. Some e2e tests invoke standalone tools (profile exporter, data generator) that boot with mock providers and may fail independently of OAP.\n",".claude/skills/compile/SKILL.md":"---\nname: compile\ndescription: Build SkyWalking OAP server, run javadoc checks, and verify checkstyle. Use to validate changes before submitting a PR.\nargument-hint: \"[all|backend|javadoc|checkstyle|module-name]\"\n---\n\n# Compile & Verify\n\nBuild the project and run static checks matching CI.\n\n## Prerequisites\n\n- JDK 11, 17, or 21 (LTS versions)\n- Maven 3.6+ (use `./mvnw` wrapper)\n\n## Maven profiles\n\n- `backend` (default): Builds OAP server modules\n- `ui` (default): Builds web application\n- `dist` (default): Creates distribution packages\n- `all`: Builds everything including submodule initialization\n\n## Commands by argument\n\n### `all` or no argument — full CI build\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### `backend` — backend only (faster)\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -Dmaven.test.skip\n```\n\n### `javadoc` — javadoc check only\n\nJavadoc requires delombok output, so `install` must run first:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nRunning `javadoc:javadoc` alone without `install` will miss errors because `${delombok.output.dir}` won't be populated.\n\n### `checkstyle` — checkstyle only\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n\n### Module name — single module build\n\n```bash\n./mvnw clean flatten:flatten package -pl oap-server/analyzer/<module-name> -Dmaven.test.skip\n```\n\n## Reading javadoc output\n\nMaven prefixes all javadoc output with `[ERROR]`, but the actual severity is in the message after the line number. Only lines containing `error:` fail the build; lines with `warning:` do not.\n\n```\n[ERROR] Foo.java:42: error: bad use of '>'        ← ACTUAL ERROR (must fix)\n[ERROR] Foo.java:50: warning: no @param for <T>   ← WARNING (does not fail build)\n```\n\n### Common javadoc errors\n\n| Error | Cause | Fix |\n|-------|-------|-----|\n| `bad use of '>'` | Bare `>` in javadoc HTML (e.g., `->` in `<pre>` blocks) | Use `{@code ->}` or `-&gt;` |\n| heading out of sequence | Heading level skips the expected hierarchy | See heading rules below |\n| reference not found | `{@link Foo#bar()}` with wrong signature | Match exact parameter types: `{@link Foo#bar(ArgType)}` |\n\n### Javadoc heading rules (JDK 13+)\n\nStrict heading validation was introduced in JDK 13. JDK 11 does **not** enforce it, but JDK 17/21/25 do. Write headings correctly for forward compatibility:\n\n| Javadoc location | Start heading at |\n|---|---|\n| Class, interface, enum, package, module | `<h2>` |\n| Constructor, method, field | `<h4>` |\n| Standalone HTML files (`doc-files/`) | `<h1>` |\n\nThe generated javadoc page uses `<h1>` for the class name and `<h3>` for member sections (Methods, Fields, etc.), so class-level subsections must use `<h2>` and method-level subsections must use `<h4>` to maintain proper nesting.\n\n## CI reference\n\nCI uses JDK 11 on Linux. The `dist-tar` job runs:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nThe `code-style` job runs:\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n","CLAUDE.md":"# CLAUDE.md - AI Assistant Guide for Apache SkyWalking\n\nThis file provides guidance for AI assistants working with the Apache SkyWalking codebase.\n\n## Project Overview\n\nApache SkyWalking is an open-source APM (Application Performance Monitoring) system designed for microservices, cloud-native, and container-based architectures. It provides distributed tracing, service mesh telemetry analysis, metrics aggregation, alerting, and observability capabilities.\n\n## Repository Structure\n\n```\nskywalking/\n├── oap-server/                    # OAP (Observability Analysis Platform) backend server\n│   ├── server-core/               # Core module with fundamental services\n│   ├── server-library/            # Shared libraries (module system, util, etc.)\n│   ├── server-receiver-plugin/    # Data receivers (gRPC, HTTP, Kafka, etc.)\n│   ├── server-storage-plugin/     # Storage implementations (BanyanDB, Elasticsearch, etc.)\n│   ├── server-cluster-plugin/     # Cluster coordination (Zookeeper, K8s, etc.)\n│   ├── server-query-plugin/       # Query interfaces (GraphQL)\n│   ├── server-alarm-plugin/       # Alerting system\n│   ├── server-fetcher-plugin/     # Data fetchers\n│   ├── server-configuration/      # Dynamic configuration providers\n│   ├── oal-grammar/               # OAL (Observability Analysis Language) grammar\n│   ├── oal-rt/                    # OAL runtime\n│   ├── mqe-grammar/               # MQE (Metrics Query Engine) grammar\n│   ├── mqe-rt/                    # MQE runtime\n│   ├── server-testing/             # Shared test utilities (DSL test framework)\n│   ├── analyzer/                  # Log and trace analyzers\n│   ├── ai-pipeline/               # AI/ML pipeline components\n│   ├── exporter/                  # Data export functionality\n│   └── server-tools/              # Standalone tools (profile exporter) with mock providers\n├── apm-protocol/                  # Protocol definitions (submodule)\n│   └── apm-network/               # gRPC/Protobuf network protocols\n├── skywalking-ui/                 # Web UI (submodule - skywalking-booster-ui)\n├── apm-webapp/                    # Web application packaging\n├── apm-dist/                      # Distribution packaging\n├── docs/                          # Documentation\n├── docker/                        # Docker build files\n├── test/                          # E2E and integration tests\n└── tools/                         # Development tools\n```\n\n## Architecture & Key Concepts\n\n### Module System\nSkyWalking uses a custom module/provider architecture based on Java SPI:\n\n- **ModuleDefine**: Declares a module and its required services\n- **ModuleProvider**: Implements a module with specific technology/approach\n- **Service**: Interface that modules expose to other modules\n\nKey pattern:\n```java\npublic class XxxModule extends ModuleDefine {\n    public Class[] services() {\n        return new Class[] { XxxService.class };\n    }\n}\n\npublic class XxxModuleProvider extends ModuleProvider {\n    public void prepare() { /* initialize */ }\n    public void start() { /* start services */ }\n}\n```\n\n### Core Concepts\n- **OAL (Observability Analysis Language)**: DSL for defining metrics aggregation rules\n- **MQE (Metrics Query Engine)**: DSL for querying metrics\n- **LAL (Log Analysis Language)**: DSL for log processing\n- **MAL (Meter Analysis Language)**: DSL for meter data processing\n- **Source/Scope**: Data model definitions for telemetry data\n- **Stream Processing**: Metrics, Records, and TopN processing pipelines\n\n### Data Flow\n1. Agents/Collectors send data via gRPC/HTTP/Kafka\n2. Receiver plugins parse and validate data\n3. Analysis engine processes data using OAL/LAL/MAL\n4. Storage plugins persist aggregated data\n5. Query plugins serve data to UI/API\n\n## Code Style & Conventions\n\n### Checkstyle Rules (enforced via `apm-checkstyle/checkStyle.xml`)\n\n**Prohibited patterns:**\n- No `System.out.println` - use proper logging (SLF4J)\n- No `@author` tags - ASF projects don't use author annotations\n- No Chinese characters in source files\n- No tab characters (use 4 spaces)\n- No star imports (`import xxx.*`)\n- No unused or redundant imports\n- No empty statements (standalone `;`)\n- No fully-qualified class names inline in code — always add an `import` statement and\n  use the short name. Acceptable exceptions: (a) two classes with the same simple name\n  would collide if both imported, (b) the class appears exactly once in a Javadoc\n  `{@link}` where the short name would be ambiguous to the reader, (c) the name is inside\n  GENERATED source — a codegen string literal or a `.ftl` template. Field declarations,\n  method signatures, local variables, and generic type arguments should always use the\n  imported short name — `private RemoteClientManager rcm;`, not `private\n  org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager rcm;`.\n  - Exception (c) is a Javassist constraint, not a preference: its compiler has no\n    `import` statement and resolves simple names only against `java.lang`\n    (`ClassPool.importedPackages`; widening it needs `importPackage`, which is pool-wide\n    mutable state and is never called here). So generated source must fully qualify\n    everything, `java.util.Map` included. Shortening an FQCN in a codegen string or a\n    template compiles as Java and then fails at runtime — audit real Java only.\n- No one-line delegate methods. A wrapper whose only body is a single forwarding call\n  to another class (`return Other.foo(a, b);`) adds a hop without value. Inline the\n  call at the use site, or call the underlying object directly (including via method\n  reference: `obj::foo` instead of `this::wrapperOfFoo`).\n\n**Required patterns:**\n- `@Override` annotation required for overridden methods\n- `equals()` and `hashCode()` must be overridden together\n- Javadoc `@param`, `@return`, `@throws` must have descriptions\n- Long constants must use uppercase `L` (e.g., `100L` not `100l`)\n- `default` case must come last in switch statements\n- One statement per line\n\n**Naming conventions:**\n- Constants/static variables: `UPPER_CASE_WITH_UNDERSCORES`\n- Type parameters: `UPPER_CASE` (e.g., `TYPE`, `KEY`, `VALUE`)\n- Package names: `org.apache.skywalking.*` or `test.apache.skywalking.*`\n- Type names: `PascalCase` or `UPPER_CASE_WITH_UNDERSCORES`\n- Local variables/parameters/members: `camelCase`\n- **Function-oriented naming, not abstract metaphor**: classes and methods are named for\n  what they do, not for an abstract concept. Prefer concrete verbs (`load`, `apply`,\n  `unregister`, `compile`, `verify`, `commit`, `rollback`) over metaphorical ones\n  (`seed`, `hydrate`, `bootstrap`, `prime`). Class names follow the same rule —\n  `StaticRuleLoader` (loads static rules), not `StaticBundleSeeder`; `DSLSyncTimer` (syncs\n  DB → state on a timer), not `TickRunner`. If you can't name a method without reaching\n  for a metaphor, the method is probably doing too much; split it.\n\n**File limits:**\n- Max file length: 3000 lines\n\n**Whitespace:**\n- Whitespace required after commas, semicolons, type casts\n- Whitespace required around operators\n- No multiple consecutive blank lines\n- Empty line separators between class members (fields can be grouped)\n\n### Code Style (via `codeStyle.xml` for IntelliJ IDEA)\n\n**Indentation:**\n- 4-space indentation\n- 4-space continuation indent\n\n**Imports:**\n- No star imports (threshold set to 999)\n- Import order: regular imports, blank line, static imports\n\n**Formatting:**\n- `while` in do-while on new line\n- Align multiline chained method calls\n- Align multiline parameters in calls\n- Array initializer braces on new lines\n- Wrap long method call chains\n\n**General:**\n- Use `final` for local variables and parameters\n- Use Lombok annotations (`@Getter`, `@Setter`, `@Builder`, `@Data`, `@Slf4j`, etc.)\n- Follow existing patterns in similar files\n\n**Comments:**\n- Don't explain what the code already says. Comment only when reading the code is\n  not enough — invariants that aren't visible in this method, cross-cutting\n  ordering, why a non-obvious branch exists, why a seemingly redundant call is\n  required. If the code is clear, no comment.\n- Connect logic that lives in different places. A check here that exists only\n  because of a guarantee made elsewhere deserves a one-line pointer to that\n  guarantee. A workaround for a known bug elsewhere deserves a link.\n- Trim restated javadoc. Method-level docs should describe the contract, not\n  re-narrate the body. Multi-paragraph rationale belongs in the design doc, not\n  the source.\n\n**Tests:**\n- Always use JUnit assertions (`assertTrue`, `assertEquals`, `assertThrows`, …).\n  Never write bare Java `assert` statements in tests. Although surefire defaults\n  `enableAssertions=true` so bare `assert` does fire in this project's tests,\n  JUnit assertions are the convention everywhere else in the codebase, survive\n  any test-runner configuration, and produce a structured failure that names\n  the actual value instead of a context-free `AssertionError`.\n\n### License Header\nJava, XML, and YAML/YML files must include the Apache 2.0 license header (see `HEADER` file).\nJSON and Markdown files are excluded (JSON doesn't support comments, see `.licenserc.yaml`).\n\n### JDK 11 Compatibility\n\nAll code must be compatible with JDK 11 (LTS). The project supports JDK 11, 17, and 21.\n\n**Prohibited Java features (post-JDK 11):**\n\n| Feature | JDK Version | Use Instead |\n|---------|-------------|-------------|\n| Switch expressions (`->`) | 14+ | Traditional `switch` with `case:` and `break` |\n| `Stream.toList()` | 16+ | `.collect(Collectors.toList())` |\n| Text blocks (`\"\"\"...\"\"\"`) | 15+ | String concatenation or `+` |\n| Records | 14+ | Regular classes with Lombok `@Data` |\n| Pattern matching for `instanceof` | 14+ | Traditional cast after `instanceof` |\n| Sealed classes/interfaces | 15+ | Regular classes/interfaces |\n\n**Allowed Java features (JDK 11 compatible):**\n- `List.of()`, `Set.of()`, `Map.of()` - Immutable collections (Java 9+)\n- `Optional` methods - `orElseThrow()`, `ifPresentOrElse()` (Java 9+)\n- Lambda expressions and method references (Java 8+)\n- Stream API (Java 8+)\n- Lombok annotations (`@Getter`, `@Builder`, `@Data`, `@Slf4j`)\n\n**Verification commands:**\n```bash\n# Check for switch expressions (should return no matches)\ngrep -r \"switch.*->\" src/ --include=\"*.java\"\n\n# Check for Stream.toList() (should return no matches)\ngrep -r \"\\.toList()\" src/ --include=\"*.java\"\n\n# Check for text blocks (should return no matches)\ngrep -r '\"\"\"' src/ --include=\"*.java\"\n```\n\n## Git Submodules\n\nThe project uses submodules for protocol definitions and UI:\n- `apm-protocol/apm-network/src/main/proto` - skywalking-data-collect-protocol\n- `oap-server/server-query-plugin/.../query-protocol` - skywalking-query-protocol\n- `skywalking-ui` - skywalking-booster-ui\n- `oap-server/server-library/library-banyandb-client/src/main/proto` - banyandb-client-proto\n\nAlways use `--recurse-submodules` when cloning or update submodules manually.\n\n## Key Files for Understanding the Codebase\n\n- `oap-server/server-core/src/main/java/.../CoreModule.java` - Core module definition\n- `oap-server/server-library/library-module/src/main/java/.../ModuleDefine.java` - Module system base\n- `oap-server/oal-grammar/src/main/antlr4/.../OALParser.g4` - OAL grammar definition\n- `oap-server/server-starter/` - Application entry point\n- `docs/en/concepts-and-designs/` - Architecture documentation\n\n## Common Development Tasks\n\n### Adding a New Receiver Plugin\n1. Create module in `server-receiver-plugin/`\n2. Implement `ModuleDefine` and `ModuleProvider`\n3. Register via SPI in `META-INF/services/`\n4. Add configuration to `application.yml`\n\n### Adding a New Storage Plugin\n1. Create module in `server-storage-plugin/`\n2. Implement storage DAOs for each data type\n3. Follow existing plugin patterns (e.g., BanyanDB, elasticsearch)\n\n### Modifying OAL Metrics\n1. Edit `.oal` files in `oap-server/server-starter/src/main/resources/oal/`\n2. Regenerate by building the project\n3. Update storage schema if needed\n\n### MAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `otel-rules/` - OpenTelemetry metrics (Prometheus, etc.)\n- `meter-analyzer-config/` - SkyWalking native meter protocol\n\n### LAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `lal/` - Log processing rules\n- `log-mal-rules/` - Metrics extracted from logs\n\n## Documentation (in `docs/en/`, structure defined in `docs/menu.yml`)\n\n- `concepts-and-designs/` - Architecture and core concepts (OAL, MAL, LAL, profiling)\n- `setup/` - Installation and configuration guides\n- `api/` - Telemetry and query protocol documentation\n- `guides/` - Contributing guides, build instructions, testing\n- `changes/changes.md` - Changelog (update when making changes). Keep each bullet's **prose on one line**\n  — don't hard-wrap a sentence across source lines (nested sub-bullets `  - …` are fine; they're real list\n  items). The changelog becomes the GitHub release body, which GitHub renders as GFM where a single\n  newline continuing a bullet becomes a `<br>`, so prose-wrapped bullets show jagged mid-sentence breaks on\n  the release page (verified on v10.4.0; the docs website reflows and hides the problem). See\n  `guides/How-to-release.md`. Append a new entry at the **end** of its section, never at the top —\n  the section reads in the order the changes landed.\n- `swip/` - SkyWalking Improvement Proposals\n\n### SWIP vs. operator docs\n\nThe two kinds of document answer different questions and have different lifetimes:\n\n- **`docs/en/swip/SWIP-N.md` records the design discussion up to first acceptance.** It is a\n  historical record of what was proposed and agreed, not a description of current behaviour.\n- **The operator / end-user docs describe how the system behaves today.** They are the only\n  place a reader should be sent for current truth.\n\nWhen the implementation of a SWIP lands, sync the SWIP once to what was actually built — every\nproposed change written in implemented/past tense, with **no `TODO` / \"future work\" /\n\"open dependency\" / \"lands later\" / \"empty until X\" language for anything in the SWIP's\nscope**. **Once that SWIP ships as part of a release, it is frozen and must not be edited again.**\n\nFrozen means frozen even when the SWIP's text becomes factually stale. A later refactor that\nrenames a package, moves a class, deletes a constant or changes a class-naming scheme **does not**\nlicense updating the SWIP to match — not to keep code references accurate, not for internal\nconsistency, and not because the same PR already touched it. Ask instead: *would an end user read\nthis to learn how the system works today?* If yes it is an operator doc and must be updated; if it\nis a SWIP the answer is no, and the fix belongs in the operator doc plus `changes.md`.\n\nAfter that first sync, further metrics, dashboards, and incremental enhancements go into the\n**operator doc**, not the SWIP. Only genuinely small, optional follow-ups may stay under a SWIP's\n\"future work\" — never a big change.\n\n- BanyanDB self-observability: the SWIP is `docs/en/swip/SWIP-15.md`; the living operator\n  catalog is `docs/en/banyandb/dashboards-banyandb.md` (menu: \"BanyanDB self observability\n  dashboard\"). New `meter_banyandb_*` rules → update the operator doc; leave the SWIP alone.\n\n### Doc cross-references\n\nReference another section with a **Markdown anchor link** `[section title](#heading-slug)`,\nnever the `§` symbol or a bare \"section N\". Verify every anchor resolves to a real heading\n(GitHub slug rules: lowercase, spaces → `-`, punctuation dropped) — a broken `[](#…)`\nsilently renders as plain text on the website.\n\n## Handling Existing Tests\n\nIf existing test code (UT, IT, or E2E) fails to pass, try to fix it. If any test cases need to be modified, or test code needs to be deleted, the change must be double-checked by a human before it is applied — surface the proposed modification or deletion and the reasoning, and wait for explicit approval rather than changing or removing the test yourself.\n\n## Submitting Pull Requests\n\nUse the `/gh-pull-request` skill for committing and pushing to a PR branch. It runs pre-flight checks (compile, checkstyle, license headers) before every push, and creates the PR if one doesn't exist yet.\n\n## GitHub Actions Allow List\n\nApache enforces an allow list for third-party GitHub Actions. All third-party actions must be pinned to an approved SHA from:\nhttps://github.com/apache/infrastructure-actions/blob/main/approved_patterns.yml\n\nIf a PR is blocked by \"action is not allowed\" errors, check the approved list and update `.github/workflows/` files to use the approved SHA pin instead of a version tag.\n\nActions owned by `actions/*` (GitHub), `github/*`, and `apache/*` are always allowed (enterprise-owned).\n\n## Tips for AI Assistants\n\n1. **Always check submodules**: Protocol changes may require submodule updates\n2. **Generate sources first**: Run `mvnw compile` before analyzing generated code\n3. **Install package**: Use `mvnw flatten:flatten install` to build the precompiler and export generated classes before running tests. ref to [compile skill doc](.claude/skills/compile/SKILL.md)\n3. **Full rebuild on cross-module changes**: If you changed more than two modules or pulled/rebased code from git remote, run `mvnw clean install` (or `mvnw clean package`) on the **whole project** rather than picking individual modules with `-pl`. Incremental `-pl ... -am` builds can leave stale jars in `.m2` or `oap-libs/` when jar sizes don't change but content does, causing hard-to-debug runtime issues.\n3. **Respect checkstyle**: No System.out, no @author, no Chinese characters\n4. **Follow module patterns**: Use existing modules as templates\n5. **Check multiple storage implementations**: Logic may vary by storage type\n6. **OAL generates code**: Don't manually edit generated metrics classes\n7. **Use Lombok**: Prefer annotations over boilerplate code\n8. **Test both unit and integration**: Different test patterns for different scopes\n9. **Documentation is rendered via markdown**: When reviewing docs, consider how they will be rendered by a markdown engine\n10. **Relative paths in docs are valid**: Relative file paths (e.g., `../../../oap-server/...`) in documentation work both in the repo and on the documentation website, supported by website build tooling\n11. **Module service registration**: When adding a service to `CoreModule.services()`, update ALL `CoreModuleProvider` implementations — not just the main one. Search with `grep -rn \"extends CoreModuleProvider\" oap-server/ --include=\"*.java\"`. The `MockCoreModuleProvider` in `server-tools/profile-exporter/` also needs it, or the profile exporter e2e test will fail at startup.\n12. **Multiple OAP packagings**: The OAP server is not only the main `server-starter`. The `server-tools/` directory contains standalone tools (e.g., profile exporter) that boot with mock module providers and a subset of modules. Changes to core module contracts (services, required modules) must be reflected in these tools too.\n13. **`moduleManager.find(X.NAME)` requires `X.NAME` in `requiredModules()`**: every call to `moduleManager.find(SomeModule.NAME)` (direct or through a helper) must have `SomeModule.NAME` in the provider's `requiredModules()` array. Missing declarations cause runtime exceptions the first time the code path fires — not at module boot. Wrapping the call in `try { ... } catch (Throwable)` is NOT a substitute; declare the module and keep the try/catch only for defensive handling of transient provider outages. When auditing a branch, grep for `moduleManager.find(` across the touched module and verify each target name appears in `requiredModules()`. Example modules that frequently catch teams out: `AlarmModule` (used by alarm-kernel reset), `LogAnalyzerModule` (used by LAL factory lookup).\n14. **Don't look up `ClusterModule` services directly**: the `ClusterModule` (ZooKeeper / K8s / Nacos coordination) exposes `ClusterRegister` / `ClusterNodesQuery` / `ClusterCoordinator`. Most receiver / analyzer modules don't declare `ClusterModule` in `requiredModules()`, so calling `moduleManager.find(ClusterModule.NAME)` will throw at runtime. Instead, go through `CoreModule`'s `RemoteClientManager` service — it's already populated by the cluster module and exposes the peer list every OAP needs. If a module genuinely needs cluster-coordinator primitives, declare `ClusterModule.NAME` in `requiredModules()` explicitly.\n15. **No `ThreadLocal` side-channels to hijack downstream behaviour**: routing a caller's intent through a `ThreadLocal` that downstream code reads (e.g., `if (PeerMode.isActive()) skipSomething()`) is almost always the wrong answer — it creates invisible coupling between far-apart code paths, leaks across async hand-offs (executors, gRPC threads, Armeria event loops), and makes the behaviour impossible to understand from a method signature. The correct fix is almost always to **extend the interface** — add a parameter, a new method, a new mode enum that appears in the signature. Rare exceptions: propagating OpenTelemetry context where the whole industry has standardised on `ThreadLocal`, or security principals enforced by a framework. In all other cases, prefer an explicit API extension, even if it costs more lines.\n16. **BanyanDB schema-visibility: fence on `mod_revision`, do NOT poll metadata**: schema no longer lives in etcd — etcd was removed. It is now stored as property documents in BanyanDB's own `_schema` store and propagated to each data node's in-memory cache **asynchronously** (a `SchemaUpdateService.WatchSchemas` gRPC stream + a 30s reconcile sync, see `banyand/metadata/schema/property/client.go`). DDL `Create` / `Update` / `Delete` broadcast only to the schema-server (`ROLE_META`) nodes and **return immediately with a `mod_revision`** — now a client-stamped `time.Now().UnixNano()` timestamp, NOT an etcd index (which is why it still changes on every content-changing DDL; it is `0` on a delete that didn't record a tombstone) — **without waiting for data nodes**. A data write whose schema hasn't propagated yet is dropped at the data-node executor (`cannot find measure definition`, logged + skipped, `banyand/measure/write_standalone.go`), so the fence is still required — arguably more than under the old etcd-watch model. After firing DDL, fence on `BanyanDBClient.getSchemaWatcher().awaitRevisionApplied(maxRev, timeout)` before unparking dispatch / firing data writes — this blocks (liaison-side, all alive nodes, bounded by the timeout) until every data node's `notifiedModRevision` watermark reaches `maxRev`, which the registry's read-after-write does not guarantee. For deletes that returned `mod_revision == 0`, fall back to `awaitSchemaDeleted(SchemaKey, timeout)`. The previous \"poll `findMeasure` until you can read your own write\" idiom predates the `SchemaBarrierService` proto and has been replaced — do not reintroduce it. The `SchemaWatcher` class is in OAP's **in-tree** `library-banyandb-client` (`org.apache.skywalking.library.banyandb.v1.client.SchemaWatcher`); the standalone `skywalking-banyandb-java-client` repo does NOT have these symbols, so don't conclude the API is gone from grepping it. JDBC and ES are synchronous-DDL on the coordinator so they don't need a fence.\n\n## Analysis and Design Principles\n\n**Never guess or speculate.** All analysis must be grounded in source code, documentation, or verified behavior.\n\n### Before making claims\n- **Read the source code** — don't assume how a feature works based on naming or convention. Check the actual implementation.\n- **Read the documentation** — check `docs/en/`, CLAUDE.md files in submodules, and README files before proposing designs.\n- **Check configuration and flags** — verify what flags/env vars exist, their default values, and how they are parsed (e.g., BanyanDB uses viper with `BYDB_` prefix to auto-bind flags to env vars).\n- **Check dependent projects** — SkyWalking depends on BanyanDB, infra-e2e, Helm charts, etc. Read their source code and docs before assuming capabilities (e.g., check Helm chart `values.yaml` for supported fields, check infra-e2e for supported config options). For `skywalking-*` projects, ask the developer if they have the source code locally — searching a local clone is much faster than fetching files via GitHub API.\n\n### Before proposing changes\n- **Verify locally first** — run the code, start the container, execute the test before pushing to CI. Don't use CI as a trial-and-error environment.\n- **Validate file paths and directory structures** — check where data actually goes (e.g., BanyanDB `--stream-root-path /tmp` creates `/tmp/stream/`, `--access-log-root-path /tmp` creates `/tmp/accesslog/`). Don't assume directory names.\n- **Validate YAML syntax** — after editing YAML files (especially with sed/awk), validate with a YAML parser before committing. Corrupted YAML causes silent failures in CI.\n- **Check the actual Docker image** — verify what's available in the container (binaries, shell, directories) before writing commands that depend on them.\n\n### When uncertain\n- **Say \"I don't know\" and investigate** — reading the code is always better than guessing. Use grep, find, and read tools to locate the answer.\n- **Ask the developer first** — if you can't find the source code, don't know how to run something, or the code doesn't make the answer clear, ask the developer where to find it rather than speculate.\n- **Test with real data** — when investigating runtime behavior (e.g., what model names an API returns, what directory structure BanyanDB creates), set up a local test and observe the actual output.\n\n### Docker images\n- **Apache SkyWalking projects** — images are on `ghcr.io/apache/` (e.g., `ghcr.io/apache/skywalking-banyandb:${COMMIT_SHA}`). Tags are full commit SHAs, not short SHAs or version tags.\n- **Official and 3rd-party images** — on Docker Hub (e.g., `ollama/ollama`, `otel/opentelemetry-collector`, `envoyproxy/gateway`).\n- **Always verify the image exists** — `docker pull` before writing CI or e2e configs. Image tags depend on CI publish workflows completing successfully.\n"},"files":{".claude/skills/gh-pull-request/SKILL.md":"---\nname: gh-pull-request\ndescription: Verify, commit, and push changes on a PR branch. Runs pre-flight checks (compile, checkstyle, license headers) before every push. Also creates the PR if one doesn't exist yet.\n---\n\n# PR Branch Workflow\n\nRun pre-flight checks, commit, push, and optionally create a PR.\n\n## Pre-flight checks\n\nRun these checks before every commit+push and fix any failures:\n\n### 1. Compile and checkstyle\n\n```bash\n# Checkstyle\n./mvnw -B -q clean checkstyle:check\n\n# Full build (compile + javadoc)\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### 2. License header check\n\n```bash\nlicense-eye header check\n```\n\nIf invalid files are found, fix with `license-eye header fix` and re-check.\n\n### 3. Unnecessary fully-qualified class names\n\nThe project checkstyle forbids inline FQCNs — every type reference in code should resolve\nthrough an `import`, not a fully-qualified name. Checkstyle does not always catch this (it\nmisses cases like inline `java.util.HashMap`, `java.util.concurrent.TimeUnit`, or\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` used as a local\nvariable type, generic parameter, or `new` target). Audit the files the branch touched\nbefore pushing:\n\nUse the `Grep` tool (ripgrep) rather than BSD `grep` on macOS — the scan below relies on a\nnegative lookahead that BSD `grep` doesn't support and GNU `grep -P` does:\n\n```\npattern: ^(?!\\s*(import |package |\\s*\\*)).*\\b(java\\.util\\.|java\\.io\\.|java\\.nio\\.|java\\.util\\.concurrent\\.|javassist\\.|org\\.apache\\.skywalking\\.)[A-Z][A-Za-z0-9_]*\nglob:    *.java\noutput_mode: content\n-n: true\n```\n\nScope the scan to files the branch touched, not the whole tree — pre-existing FQDNs on\nunrelated files generate noise. Use `git diff --name-only master...HEAD -- '*.java'` to get\nthe changed list, then run the ripgrep pattern against each.\n\nAcceptable exceptions (same as the `CLAUDE.md` rule):\n - Two classes with the same simple name would collide if both imported.\n - A Javadoc `{@link}` where the short name would be ambiguous to the reader.\n - Inside a string literal (e.g., a class name passed to `Class.forName`).\n\nFix every other hit — add an `import` and switch to the short name. This includes\n`new java.util.HashMap<>()`, `java.util.Set<String>` parameter types, and\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` as a local\nvariable type. Field declarations, method signatures, local variables, and generic\ntype arguments should all use the imported short name.\n\nRe-run checkstyle after the fix — a sloppy `sed`/`replace_all` can corrupt the `import`\nline itself (e.g., turning `import java.util.concurrent.locks.ReentrantLock;` into\n`import ReentrantLock;`), which causes a cryptic checkstyle `Range [0, -1) out of\nbounds for length N` error, not a normal violation line. If you see that error, inspect\nthe imports block first.\n\n## Commit and push\n\nAfter checks pass, commit and push:\n\n```bash\ngit add <files>\ngit commit -m \"<message>\"\ngit push -u origin <branch-name>\n```\n\n### Branch strategy\n- **Never work directly on master branch**\n- If on master, create a new branch first: `git checkout -b feature/<name>` or `git checkout -b fix/<name>`\n\n## Create PR (if not yet created)\n\nCheck whether a PR already exists for the current branch:\n\n```bash\ngh pr view --json number 2>/dev/null\n```\n\nIf no PR exists, create one:\n\n### PR title\nSummarize the changes concisely. Examples:\n- `Fix BanyanDB query timeout issue`\n- `Add support for OpenTelemetry metrics`\n\n### PR description\n\nRead `.github/PULL_REQUEST_TEMPLATE` and use its **exact format with checkboxes**. Do NOT use a custom summary format.\n\nKey template sections — uncomment the relevant one:\n\n**For Bug Fixes:**\n```\n### Fix <bug description or issue link>\n- [ ] Add a unit test to verify that the fix works.\n- [ ] Explain briefly why the bug exists and how to fix it.\n```\n\n**For New Features:**\n```\n### <Feature description>\n- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.\n- [ ] Update the documentation to include this new feature.\n- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.\n- [ ] If it's UI related, attach the screenshots below.\n```\n\n**For Performance Improvements:**\n```\n### Improve the performance of <class or module or ...>\n- [ ] Add a benchmark for the improvement.\n- [ ] The benchmark result.\n- [ ] Links/URLs to the theory proof or discussion articles/blogs.\n```\n\n**Always include:**\n```\n- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.\n- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).\n```\n\n### Create command\n\n```bash\ngh pr create --title \"<title>\" --body \"$(cat <<'EOF'\n<PR body from template>\nEOF\n)\"\n```\n\n### Post-creation\n- Add `copilot` as a reviewer: `gh pr edit <number> --add-reviewer copilot`\n- Do NOT add AI assistant as co-author. Code responsibility is on the committer's hands.\n- Return the PR URL when done.\n\n## After the PR is merged\n\nOnce the PR is merged, sync the default branch and clean up the feature branch:\n\n```bash\n# 1. Prune stale remote refs. GitHub auto-deletes the PR's branch on merge, so\n#    the remote feature branch is usually already gone; --prune removes the\n#    dangling local tracking ref.\ngit fetch origin --prune\n\n# 2. Switch back to the default branch and fast-forward it to include the merge.\ngit checkout master\ngit pull --ff-only origin master\n\n# 3. Confirm the change actually landed in master before deleting anything —\n#    `git log --oneline -1` should show the merge/squash commit with the PR\n#    number, or grep for a symbol the PR introduced.\ngit log --oneline -1\n\n# 4. Delete the local feature branch. SkyWalking SQUASH-merges PRs, so the\n#    feature branch's commit is NOT an ancestor of master (master gets a new\n#    squash commit instead). `git branch -d` therefore reports \"not fully\n#    merged\" — that is expected, not an error. After confirming the content is\n#    in master (step 3), force-delete:\ngit branch -d <branch> 2>/dev/null || git branch -D <branch>\n```\n\nNotes:\n- `git branch -d` failing with \"not fully merged\" on a squash-merged PR is normal — the squash commit has a different SHA than the feature commit. Verify via step 3, then `-D`.\n- If the remote branch was not auto-deleted (some repo settings), remove it explicitly: `git push origin --delete <branch>`.\n- Do NOT skip step 3. Force-deleting a local branch whose work didn't actually merge loses it.\n",".claude/skills/generate-classes/SKILL.md":"---\nname: generate-classes\ndescription: Generate bytecode classes from DSL scripts (MAL, OAL, LAL, Hierarchy). Runs the compiler and dumps .class files for inspection.\nargument-hint: \"<mal|oal|lal|hierarchy|all>\"\n---\n\n# Generate DSL Classes\n\nRun the v2 compiler (ANTLR4 + Javassist) to generate bytecode classes from DSL scripts and dump `.class` files to disk for inspection.\n\n## Commands by argument\n\n### `mal` — MAL expression classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=MALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/mal/**/*.generated-classes/`\n\n### `oal` — OAL metrics/dispatcher/builder classes\n\n```bash\n./mvnw test -pl oap-server/oal-rt \\\n  -Dtest=RuntimeOALGenerationTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/oal-rt/target/test-classes/metrics/`, `metrics/builder/`, `dispatcher/`\n\n### `lal` — LAL filter/extractor classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=LALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/lal/**/*.generated-classes/`\n\n### `hierarchy` — Hierarchy rule classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=HierarchyRuleExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/hierarchy-rule/*.generated-classes/`\n\n### `all` or no argument — generate all DSLs\n\nRun all four commands above sequentially.\n\n## After generation\n\nPrint the output location for the requested DSL so the user knows where to find the generated `.class` files. Use `javap` to decompile:\n\n```bash\njavap -c -p <path-to-class-file>\n```\n\n## Cleaning generated classes\n\n```bash\n./mvnw clean -pl oap-server/analyzer/dsl-scripts-test\n```\n",".claude/skills/test/SKILL.md":"---\nname: test\ndescription: Run unit tests, integration tests, or slow integration tests matching CI. Use to validate changes before submitting a PR.\nargument-hint: \"[unit|integration|slow|module-name]\"\n---\n\n# Tests\n\nRun tests matching CI configuration.\n\n## Commands by argument\n\n### `unit` or no argument — unit tests\n\n```bash\n./mvnw clean test -q -B -D\"checkstyle.skip\"\n```\n\nCI runs this on:\n- JDK 11: ubuntu, macOS, Windows\n- JDK 17, 21, 25: ubuntu only\n\n### `integration` — integration tests (excludes slow)\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow\n```\n\nCI runs this on JDK 11, 17, 21, 25 (ubuntu only).\n\n### `slow` — slow integration tests\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true \\\n  -Dit.test=org.apache.skywalking.library.elasticsearch.ElasticSearchIT \\\n  -Dfailsafe.failIfNoSpecifiedTests=false\n```\n\nCI runs on JDK 11 (ubuntu only). Currently only ElasticSearch/OpenSearch IT is in the slow matrix.\n\n### Module name — single module tests\n\n```bash\n# Unit tests for a specific module\n./mvnw clean test -pl oap-server/analyzer/<module-name> -D\"checkstyle.skip\"\n\n# Integration tests for a specific module\n./mvnw -B clean integration-test -pl oap-server/analyzer/<module-name> -Dcheckstyle.skip -DskipUTs=true\n```\n\n## Test frameworks\n\n- JUnit 5 (`org.junit.jupiter`)\n- Mockito for mocking\n- AssertJ for assertions\n- PowerMock for reflection utilities\n\n## Test naming conventions\n\n| Type | Pattern | Maven phase |\n|------|---------|-------------|\n| Unit tests | `*Test.java` | `test` |\n| Integration tests | `IT*.java` or `*IT.java` | `integration-test` |\n\n## Slow test tagging\n\nTests tagged with `@Tag(\"slow\")` are excluded from the normal integration-test job and run separately in the slow-integration-test job. Use this tag for tests that spin up external services (Elasticsearch, etc.) and take significant time.\n\n## CI retry behavior\n\nAll three CI jobs retry on failure (run the same command twice with `||`). This handles flaky tests but masks intermittent issues — if a test fails locally, investigate rather than relying on retries.\n\n## CI reference\n\nCI workflow: `.github/workflows/skywalking.yaml`\n\n| Job | JDK | OS | Timeout |\n|-----|-----|----|---------|\n| `unit-test` | 11 (3 OS) + 17, 21, 25 (ubuntu) | ubuntu, macOS, Windows | 30 min |\n| `integration-test` | 11, 17, 21, 25 | ubuntu | 60 min |\n| `slow-integration-test` | 11 | ubuntu | 60 min |\n",".claude/skills/license/SKILL.md":"---\nname: license\ndescription: Check and fix Apache 2.0 license headers and dependency licenses using skywalking-eyes. Use before submitting a PR.\nargument-hint: \"[check|fix|deps]\"\n---\n\n# License Checks\n\nCheck and fix license compliance using [skywalking-eyes](https://github.com/apache/skywalking-eyes). Two CI jobs use this tool:\n\n1. **license-header** — verifies all source files have Apache 2.0 headers\n2. **dependency-license** — verifies the LICENSE file matches current dependencies\n\n## Steps\n\n### Header check (default, or `check` argument)\n\n```bash\nlicense-eye header check\n```\n\n- **All valid**: Output shows `valid: N, invalid: 0` — nothing to do.\n- **Invalid files found**: Fix with `license-eye header fix`, then re-check.\n\n### Header fix (`fix` argument)\n\n```bash\nlicense-eye header fix\nlicense-eye header check\n```\n\n### Dependency license check (`deps` argument)\n\nThis regenerates the LICENSE file from dependency metadata and checks for drift:\n\n```bash\nlicense-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl || exit 1\nif [ ! -z \"$(git diff -U0 ./dist-material/release-docs/LICENSE)\" ]; then\n  echo \"LICENSE file is not updated correctly\"\n  git diff -U0 ./dist-material/release-docs/LICENSE\nfi\n```\n\nIf the LICENSE file changed, review the diff and commit it. **Important**: CI runs on Linux — some dependencies have platform-specific variants. If you're on macOS/Windows, the LICENSE diff may be a platform artifact. Verify before committing.\n\n## Rules\n\nConfiguration is in `.licenserc.yaml`:\n- Java, XML, YAML/YML files require Apache 2.0 headers\n- JSON and Markdown files are excluded (JSON doesn't support comments)\n- Generated files and certain vendor paths are excluded\n- SPI service files (`META-INF/services/`) require headers (use `#` comment style)\n\n## Installation\n\n```bash\n# Same version as CI (pinned commit)\ngo install github.com/apache/skywalking-eyes/cmd/license-eye@5b7ee1731d036b5aac68f8bd3fc9e6f98ada082e\n\n# Or via Homebrew (macOS)\nbrew install license-eye\n```\n",".claude/skills/package/SKILL.md":"---\nname: package\ndescription: Rebuild the SkyWalking distribution and OAP Docker image after source changes. Use before running e2e tests so the image reflects your code changes. Avoids the \"image looks updated but runtime has stale jars\" trap.\nargument-hint: \"[oap|all|dist-only]\"\n---\n\n# Package OAP Distribution & Docker Image\n\nRebuilding the Docker image after a source-code change has a **two-step dependency**: rebuild the dist tarball, then rebuild the image. Skipping the first step silently produces an image with stale jars — the Docker build still \"succeeds,\" the image has a fresh timestamp, but the embedded jar is the one from your previous build.\n\n## Why this matters\n\n`make docker.oap` in the root `Makefile` is defined as:\n\n```\ndocker.% push.docker.%: $(CONTEXT)/$(DIST) $(SW_ROOT)/docker/%/*\n    $(DOCKER_RULE)\n```\n\nIt depends on `$(CONTEXT)/$(DIST)` (the dist tarball) **as a file prerequisite**. If that tarball already exists, make does not regenerate it — it just copies whatever is on disk into `dist/docker_build/oap/` and runs `docker buildx build`. There is **no** dependency that rebuilds the tarball from source.\n\nOnly `make docker` triggers the full chain (`init → build.all → docker.all`). So:\n\n| Command | Rebuilds source? | Rebuilds tarball? | Rebuilds image? |\n|---------|:--:|:--:|:--:|\n| `./mvnw -pl <module> package` | only that module | no | no |\n| `./mvnw -pl apm-dist -am package -Pbackend,dist` | all backend deps + dist | yes | no |\n| `make docker.oap` | **no** | **no** (uses whatever's on disk) | yes |\n| `make docker` | yes (`build.all`) | yes | yes (`docker.all`) |\n\n## `flatten:flatten` — always run it before `package`/`install`\n\nSkyWalking's poms use `${revision}` as a placeholder version (e.g., `10.5.0-SNAPSHOT`). The `flatten-maven-plugin` resolves `${revision}` into concrete versions and writes a `.flattened-pom.xml`. Without it:\n\n- Installed artifacts carry the literal string `${revision}` in their coordinates and cannot be resolved as dependencies.\n- Downstream modules (including `apm-dist`) see an inconsistent dependency graph.\n- Symptoms are subtle: `-pl <module> -am` may succeed in isolation but fail or pull in stale transitive artifacts when the same module is consumed by another build invocation in the same session.\n\n**Always run `flatten:flatten` in the same goal chain as `package` or `install`.** The CI `dist-tar` job and the `compile` skill both do this. Example:\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -DskipTests\n```\n\nNot optional — treat it as part of `package`.\n\n## Pick the right command\n\n### Changed OAP source → want a new image\n\n```bash\n# Recommended: full chain\nmake docker\n```\n\nor, faster and equivalent for backend-only changes (skips UI):\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true\nmake docker.oap\n```\n\n**Do not** just run `make docker.oap` after a code edit — the image will look rebuilt but your change will not be in the jars.\n**Do not** skip `flatten:flatten` — see the section above.\n\n### Changed only the dist packaging (e.g., `apm-dist/`, log4j2.xml)\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests\nmake docker.oap\n```\n\n### Changed only Dockerfile / entrypoint (`docker/oap/*`)\n\nThe tarball is untouched, so `make docker.oap` alone is correct:\n\n```bash\nmake docker.oap\n```\n\n## Verify the fix reached the image\n\nDocker's \"image created\" timestamp updates even when the content is stale (because `--no-cache` is used). **Don't trust the timestamp.** Verify the jar contents directly:\n\n```bash\n# Copy the jar out of the container\ndocker cp <container-name>:/skywalking/oap-libs/<module>-<version>.jar /tmp/verify.jar\n\n# Extract the specific class\ncd /tmp && jar -xf verify.jar <path/to/YourClass.class>\n\n# Grep for a string your fix introduced (unique literal, method name, etc.)\ngrep -oa \"myFixSignature\" <path/to/YourClass.class>\n```\n\nIf grep finds nothing, the image does not contain your fix — you forgot to rebuild the dist. Re-run with the correct chain.\n\n## Common pitfalls\n\n- **`make docker.oap` after `./mvnw package`**: The module jar is fresh in `oap-server/.../target/`, but the dist tarball at `dist/apache-skywalking-apm-bin.tar.gz` is untouched. Image uses the old jar.\n- **Cancelling a `make docker` mid-flight**: leaves the dist half-rebuilt. Re-run `make docker` or delete `dist/` first.\n- **Stale buildx cache**: not usually the issue (the Makefile passes `--no-cache`), but if you suspect it, run `docker buildx prune`.\n- **Trusting image timestamps**: `docker images` shows when the image was *tagged*, not when its content actually changed. A no-op rebuild still updates the timestamp.\n- **Interactive buildx hangs**: if `docker buildx build` sits silent for minutes with no stdout, kill it and retry. The buildx container driver occasionally stalls; a restart is faster than waiting.\n\n## The golden rule\n\nAfter any Java source edit that affects code shipped in `oap-libs/`, run one of:\n\n1. `make docker` (full, safest)\n2. `./mvnw -pl apm-dist -am -o package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true && make docker.oap` (fast)\n\nThen **verify the fix is in the image's jar** before running e2e. Don't assume.\n",".claude/skills/run-e2e/SKILL.md":"---\nname: run-e2e\ndescription: Run SkyWalking E2E tests locally\ndisable-model-invocation: true\nargument-hint: \"[test-case-path]\"\n---\n\n# Run SkyWalking E2E Test\n\nRun an E2E test case using `skywalking-infra-e2e`. The user provides a test case path (e.g., `simple/jdk`, `storage/banyandb`, `alarm`).\n\n## Prerequisites\n\nAll tools require **Go** installed. Check `.github/workflows/` for the exact `e2e` commit used in CI.\n\n### e2e CLI\n\nBuilt from [apache/skywalking-infra-e2e](https://github.com/apache/skywalking-infra-e2e), pinned by commit in CI:\n\n```bash\n# Install the pinned commit\ngo install github.com/apache/skywalking-infra-e2e/cmd/e2e@<commit-id>\n\n# Or clone and build locally (useful when debugging the e2e tool itself)\ngit clone https://github.com/apache/skywalking-infra-e2e.git\ncd skywalking-infra-e2e\ngit checkout <commit-id>\nmake build\n# binary is in bin/e2e — add to PATH or copy to $GOPATH/bin\n```\n\n### swctl, yq, and other tools\n\nE2E test cases run pre-install steps (see `setup.steps` in each `e2e.yaml`) that install tools into `/tmp/skywalking-infra-e2e/bin`. When running locally, you need these tools on your PATH.\n\n**swctl** — SkyWalking CLI, used in verify cases to query OAP's GraphQL API. Pinned at `SW_CTL_COMMIT` in `test/e2e-v2/script/env`:\n\n```bash\n# Option 1: Use the install script (same as CI)\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: Build from source\ngo install github.com/apache/skywalking-cli/cmd/swctl@<SW_CTL_COMMIT>\n```\n\n**yq** — YAML processor, used in verify cases:\n\n```bash\n# Option 1: Use the install script\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: brew install yq (macOS)\n```\n\n**Other tools** (only needed for specific test cases):\n\n| Tool | Install script | Used by |\n|------|---------------|---------|\n| `kubectl` | `install.sh kubectl` | Kubernetes-based tests |\n| `helm` | `install.sh helm` | Helm chart tests |\n| `istioctl` | `install.sh istioctl` | Istio/service mesh tests |\n| `etcdctl` | `install.sh etcdctl` | etcd cluster tests |\n\nAll install scripts are at `test/e2e-v2/script/prepare/setup-e2e-shell/`.\n\n## Steps\n\n### 1. Determine the test case\n\nResolve the user's argument to a full path under `test/e2e-v2/cases/`. If ambiguous, list matching directories and ask.\n\n```bash\nls test/e2e-v2/cases/<argument>/e2e.yaml\n```\n\n### 2. Check if rebuild is needed\n\nCompare source file timestamps against the last build:\n\n```bash\n# OAP server changes since last build\nfind oap-server apm-protocol -type f \\( \\\n  -name \"*.java\" -o -name \"*.yaml\" -o -name \"*.yml\" -o \\\n  -name \"*.json\" -o -name \"*.xml\" -o -name \"*.properties\" -o \\\n  -name \"*.proto\" \\\n\\) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null | head -5\n\n# Test service changes since last build\nfind test/e2e-v2/java-test-service -type f \\( \\\n  -name \"*.java\" -o -name \"*.xml\" -o -name \"*.yaml\" -o -name \"*.yml\" \\\n\\) -newer test/e2e-v2/java-test-service/e2e-service-provider/target/*.jar 2>/dev/null | head -5\n```\n\nIf files are found, warn the user and suggest rebuilding before running.\n\n### 3. Rebuild if needed (only with user confirmation)\n\n```bash\n# Rebuild OAP\n./mvnw clean flatten:flatten package -Pall -Dmaven.test.skip && make docker\n\n# Rebuild test services\n./mvnw -f test/e2e-v2/java-test-service/pom.xml clean flatten:flatten package\n```\n\n### 4. Run the E2E test\n\nSet required environment variables and run:\n\n```bash\nexport SW_AGENT_JDK_VERSION=8\ne2e run -c test/e2e-v2/cases/<case-path>/e2e.yaml\n```\n\n### 5. If the test fails\n\nDo NOT run cleanup immediately. Instead:\n\n1. Check container logs:\n   ```bash\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs oap\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs provider\n   ```\n\n2. Run verify separately (can retry after investigation):\n   ```bash\n   e2e verify -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n3. Only cleanup when done debugging:\n   ```bash\n   e2e cleanup -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n### 6. Manually fire each verify query (fast triage)\n\nThe `e2e verify` retry loop runs in sequence and stops at the first failing case, so a single bad query hides every case after it. When a verify fails, **run each verify case directly against the still-running OAP** before editing anything — you'll see the real error (bad flag, missing data, wrong expected), not the progress spinner. This is also the right way to author new verify cases: craft the query against live OAP, confirm the actual YAML, then write the expected file.\n\n```bash\n# Find the host-side port that infra-e2e bound to OAP's container port 12800.\n# (Each run picks a new random port; the trigger log prints it too.)\ndocker ps --filter \"name=skywalking_e2e-oap\" --format \"{{.Ports}}\" \\\n  | grep -oE \"[0-9]+->12800\" | head -1\n# => e.g. 56381->12800\n\nURL=http://localhost:56381/graphql\nSWCTL=/tmp/skywalking-infra-e2e/bin/swctl\n\n# Copy the query from e2e.yaml verbatim, then substitute ${oap_host} → localhost\n# and ${oap_12800} → the port you just found:\n$SWCTL --display yaml --base-url=$URL service ly IOS\n$SWCTL --display yaml --base-url=$URL logs list --service-name=MyiOSApp\n$SWCTL --display yaml --base-url=$URL metrics exec --expression=service_cpm --service-name=MyiOSApp\n```\n\nWhen a `swctl` subcommand rejects a flag (`Incorrect Usage: flag provided but not defined: -layer`), the e2e config is using syntax the pinned `swctl` commit doesn't support. Find the right syntax with `swctl <cmd> --help` and update the e2e config. Common cases encountered:\n\n| Broken flag/form | Working form |\n|---|---|\n| `service ls --layer IOS` | `service ly IOS` |\n| `metrics exec ... --is-normal=true` | drop `--is-normal` (default behavior) |\n\nFor queries that *don't* use `swctl` (raw `curl` against `/loki/...`, Zipkin, PromQL), hit the matching exposed port:\n\n```bash\ncurl \"http://localhost:$(docker ps --filter name=skywalking_e2e-oap --format '{{.Ports}}' | grep -oE '[0-9]+->3100' | head -1 | cut -d'-' -f1)/loki/api/v1/labels\"\n```\n\n### 7. UI template changes require a fresh DB\n\n`UITemplateInitializer.initTemplate()` (in `oap-server/server-core`) calls `uiTemplateManagementService.addIfNotExist(setting)` — keyed by the `id` field in each `ui-initialized-templates/**/*.json`. Same ID → skipped. So edits to an *existing* template JSON (adding widgets, relabeling, changing expressions) will **not** be applied on an already-initialized OAP, even after a container restart, because the old copy still lives in storage.\n\nTo pick up dashboard JSON changes:\n\n```bash\n# Remove both containers — BanyanDB stores state inside the container FS in the\n# e2e compose (no named volume), so removing the container wipes state cleanly.\ndocker rm -f skywalking_e2e-oap-1 skywalking_e2e-banyandb-1\n\n# For compose setups that use a named volume, also:\n# docker volume rm <volume-name>\n\n# Then re-run — OAP sees empty storage, loads the new template JSON.\ne2e run -c test/e2e-v2/cases/<case>/e2e.yaml\n```\n\nSymptom to watch for: you edit the JSON, rebuild, redeploy — dashboard in the UI still shows the pre-edit layout. That's not a caching bug; that's `addIfNotExist` doing exactly what its name says.\n\n### 8. Author the expected YAML from live output\n\nFor a new verify case, the workflow is:\n\n1. Fire the query manually (see step 6) and capture the YAML.\n2. Pick which fields are *meaningful domain values* (must match exactly) vs *dynamic runtime values* (`notEmpty` / `gt` / `ge`). See `test/e2e-v2/CLAUDE.md` for the decision guide.\n3. Write the expected file. If the response is a list, wrap the items in `{{- contains . }} ... {{- end }}` so ordering and extra actual items don't fail the match.\n4. Re-run `e2e verify` alone (the containers are still up from the previous run); iterate on the expected file without rebuilding.\n\n### 9. Expected-file authoring traps\n\nThese burn CI cycles and pass locally. Each was learned the hard way.\n\n**Unquoted `content: {{ notEmpty .content }}` with `:` inside the value.** Sim-generated or real log content routinely includes colons (`POST https://api.example.com/cart failed: 500`, `HTTP/1.1 500: Internal Server Error`). Without quoting, the template renders to invalid YAML (snakeyaml parses `failed:` as a nested key) and the *whole log entry* marshals to `nil`. Symptom: diff shows `- nil` at *every* position in the expected logs list vs real maps in actual. Fix: wrap in single quotes — `content: '{{ notEmpty .content }}'`. Single-quoted YAML preserves `:` in the scalar; only fails on embedded `'`. Double quotes also work unless the content has `\"`.\n\n**Nested `contains` with multiple per-element pattern assertions against a varied stream.** The template renders the block body once per actual element; when the outer block body has *multiple* inner `contains` patterns asserting specific tag key/value pairs, and only *some* actuals satisfy all the inner patterns, `go-cmp` with `contains` can end up comparing `[rendered_for_A0, nil, nil, ...]` vs `[A0, A1, A2, ...]` and fail despite `contains` being permissive on extras. Specifically: outer `contains .logs` with a single log pattern + inner `contains .tags` asserting two distinct key/value pairs. On a simulator emitting heterogeneous errors (js + promise + ajax + pageNotFound), only a subset satisfy the inner assertion. Passes locally with 1–2 logs, fails in CI with 6+.\n- Keep the outer `contains` body lenient: field-shape checks (`notEmpty`, `gt`), one discriminator tag that *every* element in the stream carries.\n- Cover per-category assertions via separate labeled-metric verify cases, not inside the nested template.\n- Rule of thumb: \"at least one log exists with the right layer routing\" inside the logs expected; per-category coverage via `meter_*_count{label=X}` verify cases.\n\n**Hand-crafted OTLP curl payloads drift from real SDK output.** When the upstream SDK ships a published simulator image (mini-program-monitor's `sim-wechat` / `sim-alipay`, browser-client-js sim, etc.), prefer driving the e2e with that image in `MODE=timed` with a bounded `DURATION_MS` over hand-rolling the OTLP JSON. Hand-crafted payloads miss real-world shape issues: delta-vs-cumulative temporality, label-cardinality surprises, stacktrace formatting variance, attribute key names that changed between SDK versions. Pin to a released tag (`v0.4.0`), not `:latest` or HEAD SHA — reproducibility.\n\n**`timeUnixNano: \"0\"` in an OTLP metric datapoint.** The receiver propagates this into MAL's bucket computation and the metric lands in the 1970 time bucket — `swctl metrics exec` over the \"last N minutes\" window won't find it. Either use `$(date +%s)000000000` at setup time or omit the field if the receiver accepts \"now\" as default.\n\n**Setup-step curl loop with `|| sleep` pattern.** The shell line `for ... do curl && break || sleep 5; done` exits 0 when every attempt connection-refused because the final `sleep 5` returns 0. OAP takes ~50 s to start in CI, so all attempts fail before OAP is ready, and the setup step silently succeeds with zero traffic ingested. Fix: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e` at step top.\n\n**`swctl` flag rejected.** If a verify case uses a flag the pinned `swctl` commit doesn't support (`service ls --layer` vs `service ly`), the whole case fails 20× before CI gives up. Fire each verify query by hand once before pushing (step 6 above).\n\n**Published image cache miss in CI.** `docker compose pull` sometimes hits rate limits or unreachable registries; the test spins until timeout with \"dependency failed to start\". Look at the CI log for `Error response from daemon: pull access denied` or `manifest unknown`. If you see that, pin a different image tag that's definitely published (check `docker manifest inspect <tag>` locally), not a floating one.\n\n## Common test cases\n\n| Shorthand | Path |\n|-----------|------|\n| `simple/jdk` | `test/e2e-v2/cases/simple/jdk/` |\n| `storage/banyandb` | `test/e2e-v2/cases/storage/banyandb/` |\n| `storage/elasticsearch` | `test/e2e-v2/cases/storage/elasticsearch/` |\n| `alarm` | `test/e2e-v2/cases/alarm/` |\n| `log` | `test/e2e-v2/cases/log/` |\n| `profiling/trace` | `test/e2e-v2/cases/profiling/trace/` |\n",".claude/skills/new-monitoring-feature/SKILL.md":"---\nname: new-monitoring-feature\ndescription: Add a new monitoring target / layer to SkyWalking OAP. Orients you to the OAL / MAL / LAL / SpanListener / SegmentListener extension points, the UI template + submodule touchpoints, the docs set that must move together, and the cross-cutting traps that don't live in any one skill.\n---\n\n# Adding a New Monitoring Feature / Layer\n\nSkyWalking monitoring for a new target (a new `Layer` — mobile OS, gateway, DB, language runtime, …) is composed from the same set of reusable extension points. This skill is the index: which extension point fits which signal, where its contract lives, which docs must be updated together, and which harness-wide traps the extension-specific docs don't cover.\n\nThere is one skill per narrow concern. This one is the wiring map.\n\n---\n\n## Working posture\n\n- **Default scope for a new monitoring feature is narrow.** For most new monitoring targets, traces are **re-used as-is** — you do not write or modify a trace analyzer. Metrics and log extensions via OTLP (MAL + LAL) are the common shape. OTLP / Zipkin span-based metrics analysis (`SpanListener`) already exists and only needs a new implementation if your feature introduces genuinely new semantic-convention attributes to extract (e.g. `gen_ai.*`, `mcp.*` for the GenAI layer). Do **not** touch `CommonAnalysisListener` / `RPCAnalysisListener` / `SegmentAnalysisListener` unless the OAP core protocol gains a new `SpanLayer` enum value — details in §2.\n- **If you find yourself editing more than this skill suggests, stop and confirm.** Scope creep on trace analysis, span-layer mapping, receiver handlers, or protobuf/proto contracts is a strong signal the framing is wrong. Ask the developer before committing that direction.\n- **Verify every new feature locally before pushing to CI.** Compile + checkstyle + license-eye are minimums, but they only tell you the code builds — they do **not** tell you the new MAL rule compiles its closures at startup, the LAL layer:auto dispatch works, the OTLP payload shape matches the MAL labels, the UI template renders against real data, or the layer shows up in `swctl service ly <LAYER>`. Run the new e2e case end-to-end locally against a live OAP (see [run-e2e skill](../run-e2e/SKILL.md)) and fire every verify step with `swctl` by hand at least once. Pushing to CI without a local green run wastes shared CI cycles and slow-loops review.\n\n---\n\n## 0. Register the `Layer` — the feature's entry point\n\nA `Layer` is how OAP slices services / instances / endpoints by data source. The UI, storage partitioning, menu navigation, and OAL aggregation all key off it.\n\n`Layer` is a registry-backed value type (no longer a closed enum). Built-in layers are declared as `public static final Layer` constants in `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java`; external layers are registered through the `Layer.register(name, ordinal, normal)` API at boot. **Pick the registration path that matches the scope of your feature:**\n\n| Your feature ships as | Registration path |\n|---|---|\n| Part of the OAP distribution (in-tree, the common case for new SkyWalking-supported targets) | Add a `public static final Layer` constant to `Layer.java` with the next sequential ordinal in `0–49`. Examples: `IOS = register(\"IOS\", 47, true)`, `APISIX = register(\"APISIX\", 21, true)`, `VIRTUAL_DATABASE = register(\"VIRTUAL_DATABASE\", 14, false)` for inferred/non-real services. |\n| An out-of-tree MAL or LAL rule file | Add a top-level `layerDefinitions:` block to the rule file. The DSL loader funnels each entry through `Layer.register` before compiling the rule. One file ships the layer + the rules that produce its telemetry. |\n| An out-of-tree plugin module (jar) | Implement `org.apache.skywalking.oap.server.core.analysis.LayerExtension` and register via `META-INF/services/`. Discovered by `LayerExtensionLoader` during `CoreModuleProvider.prepare()`. |\n| Operator-deployed config (no code, no DSL) | Add an entry to `oap-server/server-starter/src/main/resources/layer-extensions.yml` (or override on the OAP node's classpath). |\n\n**Ordinal conventions:** `0–49` is in active use by built-ins. `50–999` is reserved by convention for future built-in layers. External layers are recommended (not required) to start at `>= 1000` to avoid colliding with future built-ins on OAP upgrade. Collisions in either direction are detected at boot via the ordinal-uniqueness check, which fails OAP startup loudly.\n\n**Storage encoding is the ordinal int**, persisted in BanyanDB / Elasticsearch / JDBC. Every OAP node that reads or writes a given layer must agree on its `(name, ordinal)` mapping — deploy `layer-extensions.yml` and any `layerDefinitions:` rule files identically across all nodes. The registry is sealed at the start of `Core.notifyAfterCompleted()`; later registration attempts throw.\n\n**UI template folders are auto-discovered by file scan, not by `Layer.values()`.** `UITemplateInitializer` walks `ui-initialized-templates/**/*.json` recursively (depth 2) and trusts each template's own `configuration.layer` field. Drop a folder of dashboard JSONs on disk and the initializer picks them up on the next boot — folder name is purely organizational.\n\n**Component ID lookup in Java code**: IDs declared in `component-libraries.yml` are loaded at runtime into `ComponentLibraryCatalogService`'s `componentName2Id` map — they are **not** exposed as Java enum constants. To look up by name in listener code, inject the catalog service and resolve once at construction:\n```java\nIComponentLibraryCatalogService catalog = moduleManager.find(CoreModule.NAME)\n    .provider().getService(IComponentLibraryCatalogService.class);\nint myComponentId = catalog.getComponentId(\"My-Component-Name\");\n```\nCache as an `int` field; runtime comparisons are then plain `componentId == myComponentId`. **Trap:** there is a `ComponentsDefine` class under `skywalking-trace-receiver-plugin/src/test/java/.../mock/ComponentsDefine.java` — it is a test-only mock holding five hand-picked constants (Tomcat, Dubbo, RocketMQ, MongoDB). Do not import or extend it from production code.\n\nEmit the layer from every source object your feature produces. Built-in layers have a static-field accessor; external layers are looked up by name through the registry:\n\n```java\n// Built-in layer (constant)\nservice.setLayer(Layer.IOS);\n\n// External layer (registered via yaml / SPI / layerDefinitions:)\nservice.setLayer(Layer.nameOf(\"IOT_FLEET\"));\n```\n\nDownstream (the core OAL, `service ly <LAYER>` swctl query, topology filters, UI root dashboard's layer selector) all work off this single enum value.\n\n---\n\n## 1. Pick the right extension point\n\n| Signal you want to process | Extension point | Register via |\n|---|---|---|\n| **SkyWalking native trace segments** (from language agents) | `AnalysisListener` (TraceSegmentListener / EntryAnalysisListener / ExitAnalysisListener / LocalAnalysisListener / FirstAnalysisListener) + factory in `agent-analyzer` | `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory` |\n| **OTLP / Zipkin trace spans** (OTel SDKs, Istio, Envoy, iOS/Android, …) | `SpanListener` in `core.trace` | `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener` |\n| **OTel metrics** (Prometheus scrape, Envoy stats, Istio, …) | MAL rules — `otel-rules/<namespace>/*.yaml` | Append the rule path to `enabledOtelMetricsRules` in `application.yml` |\n| **Logs** (OTLP, SkyWalking native, Fluent Bit, Envoy ALS) | LAL rules — `lal/<rule>.yaml` | Append to `SW_LOG_LAL_FILES` (env) or `lalFiles` (YAML) under `log-analyzer` |\n| **Pre-computed meter / custom meter** (custom agent protocol, meter receiver) | MAL `meter-analyzer-config/*.yaml` | Append to `meter-analyzer-config` entries |\n| **Aggregation from Source entities** (service, instance, endpoint, relation) | OAL rules — `oal/*.oal` | Already loaded by `OAL grammar`; just add lines |\n\nAll five DSLs (OAL, MAL, LAL, Hierarchy + SpanListener SPI) compile via ANTLR4 + Javassist at startup. Source-of-truth details are in:\n- [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md)\n- [`docs/en/concepts-and-designs/mal.md`](../../../docs/en/concepts-and-designs/mal.md)\n- [`docs/en/concepts-and-designs/lal.md`](../../../docs/en/concepts-and-designs/lal.md)\n- [`docs/en/concepts-and-designs/scope-definitions.md`](../../../docs/en/concepts-and-designs/scope-definitions.md)\n- [`oap-server/analyzer/meter-analyzer/CLAUDE.md`](../../../oap-server/analyzer/meter-analyzer/CLAUDE.md) — MAL compiler internals\n- [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — LAL compiler internals\n\n---\n\n## 2. Trace ingestion — two parallel pipelines\n\nSkyWalking OAP ingests traces through two distinct entry points. Pick based on source.\n\n> **Before writing any trace extension — ask \"do I actually need one?\"**\n>\n> For **most new layers you do not touch trace analysis at all.** Segments (native or\n> OTLP) flow through the existing pipeline, `RPCAnalysisListener.parseExit` produces\n> the outbound `ServiceRelation` edges, the backend-side agent produces the inbound\n> metrics via its own `parseEntry`, and the new-layer Service / Instance / Endpoint\n> entities come from **MAL + LAL**, not from trace analysis. This is how browser\n> (`Layer.BROWSER`), iOS (`Layer.IOS`), mobile mini-programs\n> (`Layer.WECHAT_MINI_PROGRAM` / `Layer.ALIPAY_MINI_PROGRAM`), and most\n> scraped-metric layers work.\n>\n> Extend trace analysis **only** when the feature genuinely needs per-span\n> interpretation that the existing pipeline cannot express:\n>\n> - **A new `SpanLayer`** — OAP core protobuf has a new `SpanLayer` enum value that\n>   callers of `identifyServiceLayer` / `identifyRemoteServiceLayer` must dispatch on.\n>   That means touching `CommonAnalysisListener` / `RPCAnalysisListener`.\n> - **A genuinely new signal class not covered by existing listeners** — e.g. the GenAI\n>   work added `GenAISpanListener` for LLM / MCP instrumentation because OTLP spans\n>   carry semantic-convention attributes (`gen_ai.*`, `mcp.*`) that no prior listener\n>   extracted. That is rare.\n>\n> **Anti-pattern we've been bitten by:** using a componentId mapping in\n> `CommonAnalysisListener` to assign a client-side layer to exit-only segments. This\n> does not produce inbound metrics anyway (no entry spans → no\n> `service_cpm` / `endpoint_cpm` from OAL), and it is orthogonal to the Service entity\n> layer (which MAL / LAL sets). Mirror the browser pattern instead: ship `Layer` +\n> `component-libraries.yml` + MAL + LAL + dashboards, and leave the listener chain\n> alone.\n\n### 2.1 SkyWalking native segments — `AnalysisListener`\n\n**Source protocol**: `apm-protocol/apm-network/src/main/proto/language-agent/Tracing.proto` (`TraceSegmentObject`).\n\n**Receiver**: `oap-server/server-receiver-plugin/skywalking-trace-receiver-plugin`. After wire decoding, segments are handed to `SegmentParserServiceImpl` which fans out to registered `AnalysisListenerFactory` implementations.\n\n**Extension**: implement one or more of\n- `FirstAnalysisListener` — per-segment, before per-span phase\n- `EntryAnalysisListener` — per-entry-span\n- `ExitAnalysisListener` — per-exit-span\n- `LocalAnalysisListener` — per-local-span\n- `TraceSegmentListener` — whole-segment, after per-span phase\n\nRegister the **factory** in `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory`. Existing examples: `RPCAnalysisListener`, `MultiScopesAnalysisListener`, `SegmentAnalysisListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`.\n\nUse when: the data arrives on the SW native wire (language agents — Java, Python, Go, Node.js, .NET, PHP, Lua, etc.).\n\n### 2.2 OTLP / Zipkin spans — `SpanListener`\n\n**Source protocol**: OTLP proto (`opentelemetry/proto/trace/v1/trace.proto`) or Zipkin v2 JSON.\n\n**Receiver**: `oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/.../otlp/OpenTelemetryTraceHandler.java` (OTLP) and `oap-server/server-receiver-plugin/zipkin-receiver-plugin/.../ITZipkinTraceCollector` (Zipkin). After decoding, spans are iterated and passed to every registered `SpanListener` through `SpanListenerManager`.\n\n**Extension**: implement `org.apache.skywalking.oap.server.core.trace.SpanListener`:\n\n```java\npublic interface SpanListener {\n    String[] requiredModules();\n    void init(ModuleManager moduleManager);\n    SpanListenerResult onOTLPSpan(OTLPSpanReader span, String serviceName, Map<String,String> resourceAttributes,\n                                  String scopeName, String scopeVersion);\n}\n```\n\nName the service by `serviceName`, not the resource's `service.name`: it is the service the span is analyzed and\nstored under, which `receiver-otel.otlpServiceGroupKeys` may have grouped (`shop::cart` for `service.name=cart`).\n\nRegister in `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener`. Reference implementations:\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSHTTPSpanListener.java` — OTel Swift URLSession → OAL (Service / ServiceInstance / Endpoint)\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSMetricKitSpanListener.java` — OTel Swift MetricKit → MAL samples via `OpenTelemetryMetricRequestProcessor.toMeter`\n- `oap-server/analyzer/gen-ai-analyzer/src/main/java/.../GenAISpanListener.java` — LLM / MCP instrumentation\n\nReturn `SpanListenerResult.builder().shouldPersist(false).build()` to prevent a span being stored as a trace (useful for e.g. 24-hour aggregate spans that are not meaningful as traces).\n\n**When both native and OTLP should feed the same metric**, write your extension against the downstream scope (Service / ServiceInstance / Endpoint + OAL) and emit those entities from **both** a SegmentListener and a SpanListener. The OAL layer is protocol-agnostic.\n\n---\n\n## 3. Metrics — MAL / OAL rules\n\n### 3.1 OAL (for entities already produced by core sources)\n\n`oap-server/server-starter/src/main/resources/oal/core.oal` already defines the standard `service_cpm`, `service_resp_time`, `service_sla`, `endpoint_cpm`, etc. — keyed off the `Service`, `ServiceInstance`, `Endpoint` source scopes. If your new layer emits those entities with the right `Layer`, you get the core metrics for free.\n\n**Who emits those entities matters.** `service_cpm` / `service_resp_time` / `service_sla` / `endpoint_cpm` / `endpoint_resp_time` come from `RPCAnalysisListener.parseEntry` → `callingIn.toService()` / `toEndpoint()`. That only fires on **inbound entry spans**. Client-side / edge layers (browser, iOS, mini-programs) only emit exit spans, so these OAL metrics do **not** populate under those layers — their Service entities are produced by MAL / LAL, and their dashboards use MAL metrics (`meter_<layer>_*`) for load / latency, not `service_cpm` / `service_resp_time`. Don't wire dashboards to OAL-derived metrics that the layer never produces.\n\nAdditional OAL rules go in the same file. OAL grammar: `oap-server/oal-grammar/src/main/antlr4/OALParser.g4`. Syntax and examples: [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md).\n\n### 3.2 MAL (for meter / metric samples)\n\n**File location**: `oap-server/server-starter/src/main/resources/otel-rules/<namespace>/*.yaml` (OTel scrape) or `meter-analyzer-config/*.yaml` (SW native meter).\n\n**Registration** (MANDATORY): append the rule path to the default in `oap-server/server-starter/src/main/resources/application.yml`:\n```yaml\nreceiver-otel:\n  default:\n    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:\"...,<your-namespace>/*\"}\n```\nIf you skip this, the rule isn't loaded in the default starter and downstream `converters.forEach(...)` is a silent no-op — samples just disappear.\n\n**Also update** `test/e2e-v2/cases/storage/expected/config-dump.yml`. This e2e diffs `/debugging/config/dump` output against a static file; every default-value change in `application.yml` must mirror here or the storage e2e fails.\n\n**Naming**: keep attribute scope in the metric name. If a metric reads from `foo.bar.foreground.exit_count`, call it `foo_foreground_exit_count`, not `foo_exit_count`. When two attributes (foreground vs background, ingress vs egress) share a theme, keep them as separate distinguishable metrics — don't flatten.\n\n### 3.3 Histogram pitfalls (both OAL and MAL)\n\n- **Per-bucket counts, not cumulative.** `histogram_percentile` accumulates internally. Feeding cumulative counts doubles the accumulator and shifts percentiles up. Each observation contributes `1` to exactly one `le` bucket.\n- **`le` label scale.** `SampleFamily.histogram()` rescales `le` by `defaultHistogramBucketUnit.toMillis(1)` (default `SECONDS` ⇒ ×1000). If your raw labels are already in ms, use `SampleFamilyBuilder.newBuilder(samples).defaultHistogramBucketUnit(TimeUnit.MILLISECONDS).build()`.\n- **Overflow bucket.** MAL parses `le=\"Infinity\"` as `Double.POSITIVE_INFINITY` → stored as `Long.MAX_VALUE`. If a percentile lands there, the UI renders ≈ 9.2 × 10¹⁸. Prefer a finite sentinel (e.g., 30 s for hang/launch) so outliers remain visible without crashing the chart.\n- **`NamingControl`** — every `service` / `serviceInstance` / `endpoint` label that ends up as an entity key must go through `namingControl.formatServiceName()` / `formatInstanceName()` / `formatEndpointName()`. Raw resource attributes (long git-SHA build metadata, path-style endpoints) will otherwise exceed storage limits.\n\n---\n\n## 4. Logs — LAL\n\n**File location**: `oap-server/server-starter/src/main/resources/lal/<feature>.yaml`.\n\n**Registration**: append to `SW_LOG_LAL_FILES` env var, or `log-analyzer.default.lalFiles` in `application.yml`.\n\n**Grammar**: `oap-server/analyzer/log-analyzer/src/main/antlr4/LALParser.g4`. The compiler docs are thorough: [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — covers:\n- `filter { … }` block with `parsed.*`, `tag()`, `sourceAttribute()`, `log.*`, `def` locals, `extractor {}`, `metrics {}`, `sink {}`\n- `layer: auto` mode (script decides layer from source attrs)\n- `inputType` / `outputType` SPI (`LALSourceTypeProvider`) to bind typed proto to `parsed.*`\n- Test harness at `oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/oap-cases/` — ship YAML + `.data.yaml` pair; `LALScriptExecutionTest` picks them up automatically.\n\n**`layer: auto` gotcha**: the extractor's `layer \"X\"` assignment must propagate to `LogMetadata.layer` (not just the output builder) — `FilterSpec.doSink` reads metadata to decide drop/route. The current codegen (`LALBlockCodegen.generateFieldToOutput`) handles this correctly; if you extend codegen, preserve that invariant.\n\n---\n\n## 5. UI — templates, submodule, and menu\n\n### 5.1 Dashboard templates\n\nJSON files under `oap-server/server-starter/src/main/resources/ui-initialized-templates/<feature>/`. Conventional fileset:\n\n- `<feature>-root.json` — `ServiceList` widget, layered service index.\n- `<feature>-service.json` — per-service dashboard, usually a `Tab` with Overview / Instance / Endpoint / Log sub-tabs.\n- `<feature>-instance.json` — per-instance dashboard.\n- `<feature>-endpoint.json` — per-endpoint dashboard.\n- Menu link in `oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml`.\n\n> **Collaborate with the developer on dashboard design.** Dashboard layout and metric selection are product decisions, not derivable from the MAL rules alone. Before authoring the JSON, check in with the developer on: which metric goes on which panel, which percentiles / aggregations to show, what's on the root list vs. per-service vs. per-instance vs. per-endpoint pages, what stays in the Overview tab vs. a sub-tab, widget sizes / ordering, and whether Trace / Log tabs belong at all (depends on whether the feature emits traces and logs). Then **manually set up the feature against a live OAP** — emit a few real OTLP payloads, open each dashboard in the UI, confirm every widget renders non-empty, and walk the drill-down from root → service → instance → endpoint. Only after that eyes-on pass is the dashboard ready to ship.\n\nDetails that always bite:\n\n- **Tab chrome eats ~4 rows.** Set outer `Tab.h = content_max_y + 4` or you get an inner scrollbar on top of the page scrollbar.\n- **Widget heights consistent.** Use `h=15` per row for `Line` / `Bar` widgets to match mesh / general peer templates. Don't mix h=12 / h=15 / h=24 across a feature's dashboards.\n- **List widgets (ServiceList / InstanceList / EndpointList)** render two pieces per cell:\n  - `expressions` → the aggregated summary number (right side). Required. Wrap raw metrics in `avg(...)` / `sum(...)`. Omitting it leaves the column empty.\n  - `subExpressions` → the sparkline trend (left side). Optional, falls back to `expressions` if absent.\n  - `metricConfig` pairs 1-to-1 with `expressions` (not `subExpressions`). Use for `label` (column header) and `unit`.\n  - Compound columns via MQE addition: `avg(metric_a + metric_b)` in `expressions`, `metric_a + metric_b` in `subExpressions`.\n  - Normal widgets with multiple expressions on one chart — always give a human label per expression in `metricConfig`, otherwise legend renders raw metric names.\n\n### 5.2 Hot-reload (edits to shipped templates take effect on restart)\n\n`UITemplateInitializer` normally uses `addIfNotExist` (keyed on JSON `id`), so on a plain restart your edits to shipped templates are ignored — the seeded copy in storage wins. For dev / extension work, set the environment variable and restart:\n\n```bash\nSW_UI_TEMPLATE_FORCE_RELOAD=true\n# then restart OAP — shipped templates overwrite the seeded copies via addOrReplace.\n# Hard-refresh the browser (Cmd+Shift+R) because the booster UI caches config in memory.\n```\n\nThis is a dev / extension knob only — read directly from the OS environment, not wired through `application.yml`. Unset or `false` preserves the default production behavior where operators' UI edits persist across restarts.\n\n### 5.3 New top-level menu group (e.g. \"Mobile\", \"Gateway\")\n\nNew **child entries** only need template JSON + `menu.yaml`. A new **top-level group** needs submodule work in `skywalking-ui` (git submodule at `skywalking-ui/`):\n\n1. `skywalking-ui/src/assets/icons/<name>.svg` — menu icon.\n2. `skywalking-ui/src/locales/lang/menus/{en,es,zh}.ts` — i18n labels (`<name>`, `<name>_desc`, `<name>_<child>`, `<name>_<child>_desc`). Missing i18n falls back to raw keys on screen.\n3. Commit those upstream in `apache/skywalking-booster-ui`, bump the submodule pointer here.\n4. Rebuild with `-Pui,dist` (not `-Pbackend,dist`) and `make docker.ui`, otherwise the new assets aren't in the image. See [package skill](../package/SKILL.md).\n\n---\n\n## 6. Docs set — everything that moves together\n\n| File | What to add |\n|---|---|\n| `docs/en/setup/backend/<feature>-monitoring.md` | User-facing setup guide. Required SDK / collector, required env vars, sample OTLP payload, metrics / log tags list. |\n| `docs/menu.yml` | Add the guide to the docs navigation tree. |\n| `docs/en/security/README.md` | **Client-side monitoring**: mobile / browser / mini-program endpoints accept data from the public internet without auth by default. Add a rate-limit + abuse-protection note. Agent-side monitoring over private VPC usually doesn't need this. |\n| `docs/en/changes/changes.md` | Per-release changelog. Add under `#### OAP Server` for features/bug-fixes, `#### UI` if the submodule bumps, `#### Documentation` if new guides. |\n| `docs/en/swip/SWIP-<N>.md` | Design doc for non-trivial features; keep reconciled with final code after review. |\n| `docs/en/swip/readme.md` | Move SWIP from \"Proposed\" to \"Accepted\" once merged. |\n\n---\n\n## 7. E2E test\n\nFolder: `test/e2e-v2/cases/<feature>/`. See [run-e2e skill](../run-e2e/SKILL.md) for harness basics (setup / trigger / verify phases, template functions, expected-file authoring via `contains`).\n\nFeature-specific extras:\n\n1. **Docker-compose**: extend `../../script/docker-compose/base-compose.yml` oap + storage. Set `SW_OTEL_RECEIVER_ENABLED_HANDLERS`, `SW_LOG_LAL_FILES`, and any feature-specific env. Don't list default rules here — rely on `application.yml` so the config-dump test stays consistent.\n2. **Register in CI**: add an entry under the e2e matrix in `.github/workflows/skywalking.yaml`.\n3. **OTLP data hygiene**:\n   - `spanId` is exactly 16 hex chars; `traceId` is exactly 32. Protobuf-JSON decodes them as base64; other lengths return HTTP 400.\n   - Setup-step curl loops must fail loudly: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e`. The pattern `for …; do curl && break || sleep 5; done` silently succeeds when every attempt connection-refused (the final `sleep` returns 0) because OAP takes ~50 s to start in CI.\n4. **Verify each case by hand before pushing** and use a published simulator image where the upstream SDK has one. See [`run-e2e` skill §6–§9](../run-e2e/SKILL.md) for verify-query triage, expected-file authoring, and the list of expected-file traps (YAML quoting on colons, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl loops that swallow failure).\n5. **Config dump**: when you modify any `application.yml` default (rule list, handler list, ...), also update `test/e2e-v2/cases/storage/expected/config-dump.yml`.\n\n---\n\n## 8. Pre-submit checklist\n\nRun in order; each has a dedicated skill:\n\n1. [`compile`](../compile/SKILL.md) — `./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall -Dmaven.test.skip -Dcheckstyle.skip -Dgpg.skip`. Javadoc `[ERROR] ... error:` must be fixed; `[ERROR] ... warning:` is noise.\n2. Tests: `./mvnw -pl oap-server/analyzer/log-analyzer -am test -Dtest=LALScriptExecutionTest` for LAL changes, equivalent for MAL, OAL unit tests where the rule lives.\n3. [`license`](../license/SKILL.md) — `license-eye header check` (Apache 2.0 headers). If the UI submodule bumped npm deps, also `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit the `LICENSE` diff.\n4. [`package`](../package/SKILL.md) — full `make docker` (or dist-rebuild + `make docker.oap` / `make docker.ui`). Don't trust `docker.oap` alone to pick up code changes; the dist tarball is a file-level prereq.\n5. [`run-e2e`](../run-e2e/SKILL.md) — run the new case end-to-end locally before pushing.\n6. [`gh-pull-request`](../gh-pull-request/SKILL.md) — commit and open PR with the template, attach UI screenshots.\n\n---\n\n## 9. Cross-cutting traps collected from real extensions\n\n| Trap | Symptom | Fix |\n|---|---|---|\n| Layer folder named with hyphens | `ui-initialized-templates/my-layer/*.json` on disk but dashboard empty | Folder must be `Layer.name().toLowerCase()` (underscores). `wechat_mini_program/`, not `wechat-mini-program/` |\n| Extending `CommonAnalysisListener` for a client-side layer | Added componentId → layer mapping \"to get topology for free\"; then `service_cpm` etc. don't populate anyway | Client-side (exit-only) layers don't need trace-analysis changes. Ship `Layer` + `component-libraries.yml` + MAL + LAL. See §2 header for when extending trace analysis is actually justified |\n| Dashboards wired to OAL-derived metrics on a client-side layer | Charts stay empty — `service_cpm` / `service_resp_time` / `endpoint_cpm` never populate | Those come from inbound `parseEntry`, which mini-program / browser / iOS don't emit. Use MAL metrics (`meter_<layer>_*`) instead |\n| LAL `#` comment in DSL body | OAP boot fails with `extraneous input 'metrics'` at the `#` line | LAL grammar accepts `//` and `/* */` only. No `#` comments |\n| LAL bare def-var value access — `tag 'x': myvar` | OAP boot fails with `Cannot resolve expression '[myvar]'` | `LALValueCodegen` only treats def-vars as valid when there's a method chain (`myvar?.foo`). Inline `sourceAttribute(...)` or `tag(...)` in the three value-access sites instead of storing to a def, or add a no-op chain like `myvar.toString()`. Conditions (`if (myvar == ...)`) go through a different codegen path and DO handle bare def refs |\n| MAL rule file with multi-doc YAML (`---` separator) | `MALFilterExecutionTest` fails with `expected a single document`; OAP would also fail at startup via `Rules.loadRules` (calls `Yaml.loadAs`, single-doc) | Split into one file per top-level `expSuffix` / `filter` / `metricPrefix` combo. Reference each file by name in `application.yml` (`lalFiles`, `malFiles`, `enabledOtelMetricsRules`) |\n| Histogram bucket bounds in ms with default `.histogram()` call | Percentile values come out 1000× inflated (`P50 = 100000` for a 100 ms request) | MAL's default `defaultHistogramBucketUnit = SECONDS` multiplies `le` by 1000 when converting to stored ms. For SDKs that emit ms bounds directly (mini-program, most in-house JS SDKs), use `.histogram(\"le\", TimeUnit.MILLISECONDS)`. `TimeUnit` is registered in `MALCodegenHelper.ENUM_FQCN` so rule YAML can reference it |\n| `.rate()` / `.increase()` on a DELTA counter / histogram `_count` | Compute delta-of-delta — nonsense values | OTLP DELTA means each flush is already the per-flush count. MAL's per-minute aggregator sums directly to requests-per-minute. For CUMULATIVE (Prometheus default), use `.rate(\"PT1M\")` for per-sec or `.increase(\"PT1M\")` for per-min; they are **not** equivalent (differ by 60×) |\n| MAL bucket-family name with `_histogram` suffix | Rule compiles, metric stays empty | `PrometheusMetricConverter` emits the bucket family as `<metric_name>` (no suffix), with `<metric_name>_count` and `<metric_name>_sum` as the companion families. Reference the bucket family by its bare name in the rule |\n| LAL `tag(\"key\") == null` check on a missing tag | Check never triggers — LAL's `tag()` returns empty string `\"\"` for absent tags, not `null`, so conditional abort is a no-op | Guard on both: `tag(\"key\") == null || tag(\"key\") == \"\"`. Same for `sourceAttribute(\"key\")` |\n| MAL rule file with `expSuffix: service(...)` mixing service-scoped and endpoint-scoped rules | Endpoint rules' metrics appear under the service entity (no endpoint_traffic rows created); `swctl endpoint ls` returns empty. `MetricConvert.formatExp:132` appends `(exp).expSuffix` to every rule, so `.endpoint(...)` in the rule's expression is followed by `.service(...)` and the service scope wins. | Follow the APISIX pattern — leave `expSuffix:` empty and explicitly chain `.service(...)` / `.endpoint(...)` on each rule. If every rule in a file is same-scope (e.g. all instance-scoped in `*-instance.yaml`), `expSuffix` is still fine |\n\nE2E-side authoring traps (log-content YAML quoting, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl swallowing failure) live in the [`run-e2e` skill §9](../run-e2e/SKILL.md).\n| Missing layer-root template | Menu item lands on empty \"no dashboard\" view | `Layer.vue:41-44` requires a template with `isRoot: true`; ship a `<layer>-root.json` (precedent: `ios/ios-root.json`) |\n| New layer not selectable in UI / swctl | `service ly <LAYER>` returns empty | Add the enum in `Layer.java` with a fresh id; see §0 |\n| Stale dist in image | `make docker.oap` succeeds but behavior is old | See [`package` skill](../package/SKILL.md); rebuild dist first, then image |\n| Shipped template edits don't render after restart | Edited JSON on disk, but OAP shows old copy | Set `SW_UI_TEMPLATE_FORCE_RELOAD=true` (environment variable, not `application.yml`) and restart; OAP writes `addOrReplace`. Unset in production so operator UI edits aren't clobbered. |\n| MAL service-scoped rule fragmented by instance | \"Overall app\" metric has N series per version | Keep `service_instance_id` out of service-scoped `.sum`/`.avg`. Put instance-scoped metrics in a separate `<feature>-instance.yaml` with `expSuffix: instance(...)` — see `otel-rules/ios/` pattern. |\n| Config dump drift | Storage e2e fails after `application.yml` edit | Mirror the change in `test/e2e-v2/cases/storage/expected/config-dump.yml` |\n| LAL `layer:auto` WARN drop | Every log silently dropped | Codegen must set `metadata.layer` alongside output builder's layer |\n| Histogram 1000× off | p50 = 2 000 000 ms | `defaultHistogramBucketUnit(MILLISECONDS)` on the `SampleFamily` |\n| Histogram shifted up | p50 = 5 000 on 500–2500 ms data | Emit per-bucket, not cumulative |\n| Empty list column | Dashboard instance table has no numbers | Add `expressions` (not just `subExpressions`) |\n| Double scroll | Inner tab scroll + page scroll | `Tab.h = content_max + 4` chrome buffer |\n| Raw metric name in legend | Multiple expressions on one widget | Add parallel `metricConfig[].label` entries |\n| `spanKind` mismatch | Span skipped silently | `OTLPSpanReaderImpl.spanKind()` returns enum `.name()` like `SPAN_KIND_CLIENT`, not `CLIENT` |\n| Negative latency | `*_sla` inflated | Clamp `rawLatency < 0 ? 0 : (int) min(Integer.MAX_VALUE, rawLatency)` |\n| SLA too optimistic | Client-side DNS / timeout / connection-refused counted as success | Require a real response code for success: `status = code > 0 && code < 400`. Do not treat missing (0) as success. |\n| Silent drop with no rules | `converters == null` NPE elsewhere | Init to empty list at declaration |\n| `+Inf` bucket surfaces as `9.2e18` | UI shows garbage when a sample lands in overflow | Use finite sentinel bucket (e.g. 30 s) |\n| OTel submodule bump misses UI image | New icons / i18n not visible | Rebuild with `-Pui,dist` + `make docker.ui` |\n| LICENSE drift after submodule bump | CI `Dependency licenses` fails | `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit |\n| Setup-step curl loop swallows failure | e2e passes metrics but fails log verify | `curl -f --retry-connrefused --retry-all-errors` + `set -e` |\n| Verify case flag mismatch | CI fails early on `swctl` flag unknown to pinned commit | Fire every verify command locally first; bump `SW_CTL_COMMIT` if needed |\n\n---\n\n## 10. Reference extensions (read their code)\n\nBrowse these when you need a template to copy from. Grouped by the kind of data source, because the extension points differ significantly:\n\n### SW native trace (language agents) — the \"General\" layer and friends\n\n`Layer.GENERAL` is the default layer for services reporting via the SkyWalking native trace protocol (Java / Python / Go / Node.js / .NET / PHP / Lua / Rust agents). It's the richest reference because the native pipeline is where the **full topology** lives:\n\n- **Service → service dependency** (topology): derived by `MultiScopesAnalysisListener` and `NetworkAddressAliasMappingListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`. Entry/exit spans produce `ServiceRelation`, `ServiceInstanceRelation` sources.\n- **Endpoint → endpoint dependency**: `RPCAnalysisListener` emits `EndpointRelation` sources from cross-process refs and exit spans. This is unique to the native protocol — OTLP spans don't preserve upstream entry-span metadata the same way, so OTLP-only layers generally lose endpoint-level topology.\n- OAL aggregation: `oap-server/server-starter/src/main/resources/oal/core.oal` — all `service_relation_*`, `service_instance_relation_*`, `endpoint_relation_*` metrics are already defined there; a new native-tracing layer just needs to produce the right sources with the right `Layer`.\n\nRead order: `AnalysisListenerFactory` SPI registration → `SegmentParserServiceImpl` fan-out → individual listeners → OAL rules.\n\n### Service-mesh (Istio / Envoy ALS) — topology from proxy-side signals\n\nNo agent in the service; Envoy emits HTTP/TCP access logs (ALS) and metrics. Service-to-service topology is reconstructed from ALS request/response pairs.\n\n- `oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/` — ALS receiver + `EnvoyAccessLogBuilder` persistence (LAL output type).\n- `otel-rules/istio-*`, `otel-rules/envoy/*` — MAL rules for proxy-emitted OTel metrics.\n- `lal/envoy-als.yaml` — LAL rule that extracts topology-generating fields from ALS.\n- Layer: `Layer.MESH`, plus `Layer.MESH_CP` (control plane) and `Layer.MESH_DP` (data plane) for split-plane topology.\n\nTopology parity with native traces: ALS gives you service-level relations; endpoint-level relations are approximate (proxy sees inbound + outbound, not the full call-chain that a native agent observes).\n\n### OTLP-only layers (OTel SDKs, iOS/Android, AI gateway, …)\n\nOTLP spans feed a `SpanListener`. Topology is **partial by construction** — an OTel Swift SDK on a mobile device only knows about its own outbound requests; there's no upstream caller. So these layers typically produce only `Service` / `ServiceInstance` / `Endpoint` sources (no `*Relation`) and the Topology view shows isolated islands, by design.\n\n- **iOS** (`Layer.IOS`, OTLP traces + MetricKit + LAL diagnostics): `oap-server/analyzer/ios-analyzer/` + `docs/en/swip/SWIP-11.md`.\n- **Envoy AI Gateway** (`Layer.ENVOY_AI_GATEWAY`, LLM / MCP, OTLP metrics + Zipkin sampled traces + LAL access-log sampling): `oap-server/analyzer/gen-ai-analyzer/`, `otel-rules/envoy-ai-gateway/*`, `lal/envoy-ai-gateway.yaml`.\n\n### OTel-metrics-only layers (scraped exporters)\n\nNo spans, no logs — just MAL rules converting OTel metric scrapes.\n\n- **MySQL / PostgreSQL**: `otel-rules/mysql/*`, `otel-rules/postgresql/*`.\n- **Kubernetes**: `otel-rules/k8s/*` — paired with `virtual-k8s-service` (`Layer.K8S_SERVICE`) sources for synthetic services.\n- **AWS**: `otel-rules/aws-eks/*`, `otel-rules/aws-s3/*`, `otel-rules/aws-dynamodb/*`.\n\n### SW native meter (custom protocol)\n\nNo OTel; the agent / plugin sends pre-built meter samples on the SW `MeterProtocol`.\n\n- **Nginx Lua** (`Layer.NGINX`): `meter-analyzer-config/nginx-service.yaml`, with the Lua agent in `apache/skywalking-nginx-lua`.\n- **APISIX** (`Layer.APISIX`): `meter-analyzer-config/apisix-service.yaml` + LAL trace extraction.\n\n### Which to copy from\n\n| You're adding | Closest reference |\n|---|---|\n| Language agent (new JVM / runtime) | The General / Java layer flow; extend `AnalysisListener` SPI. |\n| Service mesh / proxy | `mesh-*` + `envoy-*` rules; LAL on ALS. |\n| Mobile / browser / mini-program SDK (OTLP) | `ios-analyzer` (OTLP client-side). Expect no topology relations. |\n| Database / middleware (metrics only) | `mysql`, `postgresql` — pure MAL rules. |\n| Cloud service (no agent, just metrics) | `aws-eks` / `aws-s3`. |\n| Custom protocol / home-grown telemetry | `apisix` / `nginx` — SW native meter. |\n",".claude/skills/ci-e2e-debug/SKILL.md":"---\nname: ci-e2e-debug\ndescription: Download and inspect CI e2e test logs from GitHub Actions artifacts. Use when investigating e2e test failures in CI.\nargument-hint: \"<RUN_ID or GitHub Actions URL>\"\n---\n\n# CI E2E Debug Skill\n\nDownload test log artifacts from a GitHub Actions CI run and inspect them for errors.\n\n## Steps\n\n1. **Find the artifact**: Use `gh api` to list artifacts for the given CI run:\n   ```bash\n   gh api repos/apache/skywalking/actions/runs/<RUN_ID>/artifacts --jq '.artifacts[] | {id: .id, name: .name}'\n   ```\n\n2. **Download and extract**: Download the artifact zip and extract it:\n   ```bash\n   cd /tmp && rm -rf e2e-debug-logs && mkdir e2e-debug-logs && cd e2e-debug-logs\n   gh api repos/apache/skywalking/actions/artifacts/<ARTIFACT_ID>/zip > artifact.zip\n   unzip -o artifact.zip\n   ```\n\n3. **Inspect OAP logs**: Look for errors in the OAP server logs:\n   ```bash\n   # Find OAP log files\n   find /tmp/e2e-debug-logs -name \"skywalking-oap-*.log\" -o -name \"oap.log\"\n   # Check for errors\n   grep -E \"ERROR|Exception|FATAL|CannotCompileException\" <log_file> | head -30\n   ```\n\n4. **Inspect other component logs**: Check BanyanDB, UI, and other pod logs as needed.\n\n5. **Report findings**: Summarize the root cause error from the logs.\n\n## Notes\n- CI artifacts are automatically uploaded by the e2e test framework to `$SW_INFRA_E2E_LOG_DIR`\n- Log files are organized by namespace/pod name\n- OAP init pods may have different errors than the main OAP pod — check all of them\n- Common errors: MAL/LAL/OAL compilation failures, storage connection issues, module initialization errors\n- **Profile exporter tests** (e.g., Trace Profiling ES): the exporter runs as a separate process with `MockCoreModuleProvider`. If a new service is added to `CoreModule.services()` but not registered in the mock, the exporter fails at startup with `requiredCheck()` error — but the OAP logs will show no errors (OAP is fine, the exporter subprocess is what fails). Check the test step that runs `profile_exporter.sh`.\n- **When OAP logs are clean but tests fail**: look at which specific test step failed. Some e2e tests invoke standalone tools (profile exporter, data generator) that boot with mock providers and may fail independently of OAP.\n",".claude/skills/compile/SKILL.md":"---\nname: compile\ndescription: Build SkyWalking OAP server, run javadoc checks, and verify checkstyle. Use to validate changes before submitting a PR.\nargument-hint: \"[all|backend|javadoc|checkstyle|module-name]\"\n---\n\n# Compile & Verify\n\nBuild the project and run static checks matching CI.\n\n## Prerequisites\n\n- JDK 11, 17, or 21 (LTS versions)\n- Maven 3.6+ (use `./mvnw` wrapper)\n\n## Maven profiles\n\n- `backend` (default): Builds OAP server modules\n- `ui` (default): Builds web application\n- `dist` (default): Creates distribution packages\n- `all`: Builds everything including submodule initialization\n\n## Commands by argument\n\n### `all` or no argument — full CI build\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### `backend` — backend only (faster)\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -Dmaven.test.skip\n```\n\n### `javadoc` — javadoc check only\n\nJavadoc requires delombok output, so `install` must run first:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nRunning `javadoc:javadoc` alone without `install` will miss errors because `${delombok.output.dir}` won't be populated.\n\n### `checkstyle` — checkstyle only\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n\n### Module name — single module build\n\n```bash\n./mvnw clean flatten:flatten package -pl oap-server/analyzer/<module-name> -Dmaven.test.skip\n```\n\n## Reading javadoc output\n\nMaven prefixes all javadoc output with `[ERROR]`, but the actual severity is in the message after the line number. Only lines containing `error:` fail the build; lines with `warning:` do not.\n\n```\n[ERROR] Foo.java:42: error: bad use of '>'        ← ACTUAL ERROR (must fix)\n[ERROR] Foo.java:50: warning: no @param for <T>   ← WARNING (does not fail build)\n```\n\n### Common javadoc errors\n\n| Error | Cause | Fix |\n|-------|-------|-----|\n| `bad use of '>'` | Bare `>` in javadoc HTML (e.g., `->` in `<pre>` blocks) | Use `{@code ->}` or `-&gt;` |\n| heading out of sequence | Heading level skips the expected hierarchy | See heading rules below |\n| reference not found | `{@link Foo#bar()}` with wrong signature | Match exact parameter types: `{@link Foo#bar(ArgType)}` |\n\n### Javadoc heading rules (JDK 13+)\n\nStrict heading validation was introduced in JDK 13. JDK 11 does **not** enforce it, but JDK 17/21/25 do. Write headings correctly for forward compatibility:\n\n| Javadoc location | Start heading at |\n|---|---|\n| Class, interface, enum, package, module | `<h2>` |\n| Constructor, method, field | `<h4>` |\n| Standalone HTML files (`doc-files/`) | `<h1>` |\n\nThe generated javadoc page uses `<h1>` for the class name and `<h3>` for member sections (Methods, Fields, etc.), so class-level subsections must use `<h2>` and method-level subsections must use `<h4>` to maintain proper nesting.\n\n## CI reference\n\nCI uses JDK 11 on Linux. The `dist-tar` job runs:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nThe `code-style` job runs:\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n","CLAUDE.md":"# CLAUDE.md - AI Assistant Guide for Apache SkyWalking\n\nThis file provides guidance for AI assistants working with the Apache SkyWalking codebase.\n\n## Project Overview\n\nApache SkyWalking is an open-source APM (Application Performance Monitoring) system designed for microservices, cloud-native, and container-based architectures. It provides distributed tracing, service mesh telemetry analysis, metrics aggregation, alerting, and observability capabilities.\n\n## Repository Structure\n\n```\nskywalking/\n├── oap-server/                    # OAP (Observability Analysis Platform) backend server\n│   ├── server-core/               # Core module with fundamental services\n│   ├── server-library/            # Shared libraries (module system, util, etc.)\n│   ├── server-receiver-plugin/    # Data receivers (gRPC, HTTP, Kafka, etc.)\n│   ├── server-storage-plugin/     # Storage implementations (BanyanDB, Elasticsearch, etc.)\n│   ├── server-cluster-plugin/     # Cluster coordination (Zookeeper, K8s, etc.)\n│   ├── server-query-plugin/       # Query interfaces (GraphQL)\n│   ├── server-alarm-plugin/       # Alerting system\n│   ├── server-fetcher-plugin/     # Data fetchers\n│   ├── server-configuration/      # Dynamic configuration providers\n│   ├── oal-grammar/               # OAL (Observability Analysis Language) grammar\n│   ├── oal-rt/                    # OAL runtime\n│   ├── mqe-grammar/               # MQE (Metrics Query Engine) grammar\n│   ├── mqe-rt/                    # MQE runtime\n│   ├── server-testing/             # Shared test utilities (DSL test framework)\n│   ├── analyzer/                  # Log and trace analyzers\n│   ├── ai-pipeline/               # AI/ML pipeline components\n│   ├── exporter/                  # Data export functionality\n│   └── server-tools/              # Standalone tools (profile exporter) with mock providers\n├── apm-protocol/                  # Protocol definitions (submodule)\n│   └── apm-network/               # gRPC/Protobuf network protocols\n├── skywalking-ui/                 # Web UI (submodule - skywalking-booster-ui)\n├── apm-webapp/                    # Web application packaging\n├── apm-dist/                      # Distribution packaging\n├── docs/                          # Documentation\n├── docker/                        # Docker build files\n├── test/                          # E2E and integration tests\n└── tools/                         # Development tools\n```\n\n## Architecture & Key Concepts\n\n### Module System\nSkyWalking uses a custom module/provider architecture based on Java SPI:\n\n- **ModuleDefine**: Declares a module and its required services\n- **ModuleProvider**: Implements a module with specific technology/approach\n- **Service**: Interface that modules expose to other modules\n\nKey pattern:\n```java\npublic class XxxModule extends ModuleDefine {\n    public Class[] services() {\n        return new Class[] { XxxService.class };\n    }\n}\n\npublic class XxxModuleProvider extends ModuleProvider {\n    public void prepare() { /* initialize */ }\n    public void start() { /* start services */ }\n}\n```\n\n### Core Concepts\n- **OAL (Observability Analysis Language)**: DSL for defining metrics aggregation rules\n- **MQE (Metrics Query Engine)**: DSL for querying metrics\n- **LAL (Log Analysis Language)**: DSL for log processing\n- **MAL (Meter Analysis Language)**: DSL for meter data processing\n- **Source/Scope**: Data model definitions for telemetry data\n- **Stream Processing**: Metrics, Records, and TopN processing pipelines\n\n### Data Flow\n1. Agents/Collectors send data via gRPC/HTTP/Kafka\n2. Receiver plugins parse and validate data\n3. Analysis engine processes data using OAL/LAL/MAL\n4. Storage plugins persist aggregated data\n5. Query plugins serve data to UI/API\n\n## Code Style & Conventions\n\n### Checkstyle Rules (enforced via `apm-checkstyle/checkStyle.xml`)\n\n**Prohibited patterns:**\n- No `System.out.println` - use proper logging (SLF4J)\n- No `@author` tags - ASF projects don't use author annotations\n- No Chinese characters in source files\n- No tab characters (use 4 spaces)\n- No star imports (`import xxx.*`)\n- No unused or redundant imports\n- No empty statements (standalone `;`)\n- No fully-qualified class names inline in code — always add an `import` statement and\n  use the short name. Acceptable exceptions: (a) two classes with the same simple name\n  would collide if both imported, (b) the class appears exactly once in a Javadoc\n  `{@link}` where the short name would be ambiguous to the reader, (c) the name is inside\n  GENERATED source — a codegen string literal or a `.ftl` template. Field declarations,\n  method signatures, local variables, and generic type arguments should always use the\n  imported short name — `private RemoteClientManager rcm;`, not `private\n  org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager rcm;`.\n  - Exception (c) is a Javassist constraint, not a preference: its compiler has no\n    `import` statement and resolves simple names only against `java.lang`\n    (`ClassPool.importedPackages`; widening it needs `importPackage`, which is pool-wide\n    mutable state and is never called here). So generated source must fully qualify\n    everything, `java.util.Map` included. Shortening an FQCN in a codegen string or a\n    template compiles as Java and then fails at runtime — audit real Java only.\n- No one-line delegate methods. A wrapper whose only body is a single forwarding call\n  to another class (`return Other.foo(a, b);`) adds a hop without value. Inline the\n  call at the use site, or call the underlying object directly (including via method\n  reference: `obj::foo` instead of `this::wrapperOfFoo`).\n\n**Required patterns:**\n- `@Override` annotation required for overridden methods\n- `equals()` and `hashCode()` must be overridden together\n- Javadoc `@param`, `@return`, `@throws` must have descriptions\n- Long constants must use uppercase `L` (e.g., `100L` not `100l`)\n- `default` case must come last in switch statements\n- One statement per line\n\n**Naming conventions:**\n- Constants/static variables: `UPPER_CASE_WITH_UNDERSCORES`\n- Type parameters: `UPPER_CASE` (e.g., `TYPE`, `KEY`, `VALUE`)\n- Package names: `org.apache.skywalking.*` or `test.apache.skywalking.*`\n- Type names: `PascalCase` or `UPPER_CASE_WITH_UNDERSCORES`\n- Local variables/parameters/members: `camelCase`\n- **Function-oriented naming, not abstract metaphor**: classes and methods are named for\n  what they do, not for an abstract concept. Prefer concrete verbs (`load`, `apply`,\n  `unregister`, `compile`, `verify`, `commit`, `rollback`) over metaphorical ones\n  (`seed`, `hydrate`, `bootstrap`, `prime`). Class names follow the same rule —\n  `StaticRuleLoader` (loads static rules), not `StaticBundleSeeder`; `DSLSyncTimer` (syncs\n  DB → state on a timer), not `TickRunner`. If you can't name a method without reaching\n  for a metaphor, the method is probably doing too much; split it.\n\n**File limits:**\n- Max file length: 3000 lines\n\n**Whitespace:**\n- Whitespace required after commas, semicolons, type casts\n- Whitespace required around operators\n- No multiple consecutive blank lines\n- Empty line separators between class members (fields can be grouped)\n\n### Code Style (via `codeStyle.xml` for IntelliJ IDEA)\n\n**Indentation:**\n- 4-space indentation\n- 4-space continuation indent\n\n**Imports:**\n- No star imports (threshold set to 999)\n- Import order: regular imports, blank line, static imports\n\n**Formatting:**\n- `while` in do-while on new line\n- Align multiline chained method calls\n- Align multiline parameters in calls\n- Array initializer braces on new lines\n- Wrap long method call chains\n\n**General:**\n- Use `final` for local variables and parameters\n- Use Lombok annotations (`@Getter`, `@Setter`, `@Builder`, `@Data`, `@Slf4j`, etc.)\n- Follow existing patterns in similar files\n\n**Comments:**\n- Don't explain what the code already says. Comment only when reading the code is\n  not enough — invariants that aren't visible in this method, cross-cutting\n  ordering, why a non-obvious branch exists, why a seemingly redundant call is\n  required. If the code is clear, no comment.\n- Connect logic that lives in different places. A check here that exists only\n  because of a guarantee made elsewhere deserves a one-line pointer to that\n  guarantee. A workaround for a known bug elsewhere deserves a link.\n- Trim restated javadoc. Method-level docs should describe the contract, not\n  re-narrate the body. Multi-paragraph rationale belongs in the design doc, not\n  the source.\n\n**Tests:**\n- Always use JUnit assertions (`assertTrue`, `assertEquals`, `assertThrows`, …).\n  Never write bare Java `assert` statements in tests. Although surefire defaults\n  `enableAssertions=true` so bare `assert` does fire in this project's tests,\n  JUnit assertions are the convention everywhere else in the codebase, survive\n  any test-runner configuration, and produce a structured failure that names\n  the actual value instead of a context-free `AssertionError`.\n\n### License Header\nJava, XML, and YAML/YML files must include the Apache 2.0 license header (see `HEADER` file).\nJSON and Markdown files are excluded (JSON doesn't support comments, see `.licenserc.yaml`).\n\n### JDK 11 Compatibility\n\nAll code must be compatible with JDK 11 (LTS). The project supports JDK 11, 17, and 21.\n\n**Prohibited Java features (post-JDK 11):**\n\n| Feature | JDK Version | Use Instead |\n|---------|-------------|-------------|\n| Switch expressions (`->`) | 14+ | Traditional `switch` with `case:` and `break` |\n| `Stream.toList()` | 16+ | `.collect(Collectors.toList())` |\n| Text blocks (`\"\"\"...\"\"\"`) | 15+ | String concatenation or `+` |\n| Records | 14+ | Regular classes with Lombok `@Data` |\n| Pattern matching for `instanceof` | 14+ | Traditional cast after `instanceof` |\n| Sealed classes/interfaces | 15+ | Regular classes/interfaces |\n\n**Allowed Java features (JDK 11 compatible):**\n- `List.of()`, `Set.of()`, `Map.of()` - Immutable collections (Java 9+)\n- `Optional` methods - `orElseThrow()`, `ifPresentOrElse()` (Java 9+)\n- Lambda expressions and method references (Java 8+)\n- Stream API (Java 8+)\n- Lombok annotations (`@Getter`, `@Builder`, `@Data`, `@Slf4j`)\n\n**Verification commands:**\n```bash\n# Check for switch expressions (should return no matches)\ngrep -r \"switch.*->\" src/ --include=\"*.java\"\n\n# Check for Stream.toList() (should return no matches)\ngrep -r \"\\.toList()\" src/ --include=\"*.java\"\n\n# Check for text blocks (should return no matches)\ngrep -r '\"\"\"' src/ --include=\"*.java\"\n```\n\n## Git Submodules\n\nThe project uses submodules for protocol definitions and UI:\n- `apm-protocol/apm-network/src/main/proto` - skywalking-data-collect-protocol\n- `oap-server/server-query-plugin/.../query-protocol` - skywalking-query-protocol\n- `skywalking-ui` - skywalking-booster-ui\n- `oap-server/server-library/library-banyandb-client/src/main/proto` - banyandb-client-proto\n\nAlways use `--recurse-submodules` when cloning or update submodules manually.\n\n## Key Files for Understanding the Codebase\n\n- `oap-server/server-core/src/main/java/.../CoreModule.java` - Core module definition\n- `oap-server/server-library/library-module/src/main/java/.../ModuleDefine.java` - Module system base\n- `oap-server/oal-grammar/src/main/antlr4/.../OALParser.g4` - OAL grammar definition\n- `oap-server/server-starter/` - Application entry point\n- `docs/en/concepts-and-designs/` - Architecture documentation\n\n## Common Development Tasks\n\n### Adding a New Receiver Plugin\n1. Create module in `server-receiver-plugin/`\n2. Implement `ModuleDefine` and `ModuleProvider`\n3. Register via SPI in `META-INF/services/`\n4. Add configuration to `application.yml`\n\n### Adding a New Storage Plugin\n1. Create module in `server-storage-plugin/`\n2. Implement storage DAOs for each data type\n3. Follow existing plugin patterns (e.g., BanyanDB, elasticsearch)\n\n### Modifying OAL Metrics\n1. Edit `.oal` files in `oap-server/server-starter/src/main/resources/oal/`\n2. Regenerate by building the project\n3. Update storage schema if needed\n\n### MAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `otel-rules/` - OpenTelemetry metrics (Prometheus, etc.)\n- `meter-analyzer-config/` - SkyWalking native meter protocol\n\n### LAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `lal/` - Log processing rules\n- `log-mal-rules/` - Metrics extracted from logs\n\n## Documentation (in `docs/en/`, structure defined in `docs/menu.yml`)\n\n- `concepts-and-designs/` - Architecture and core concepts (OAL, MAL, LAL, profiling)\n- `setup/` - Installation and configuration guides\n- `api/` - Telemetry and query protocol documentation\n- `guides/` - Contributing guides, build instructions, testing\n- `changes/changes.md` - Changelog (update when making changes). Keep each bullet's **prose on one line**\n  — don't hard-wrap a sentence across source lines (nested sub-bullets `  - …` are fine; they're real list\n  items). The changelog becomes the GitHub release body, which GitHub renders as GFM where a single\n  newline continuing a bullet becomes a `<br>`, so prose-wrapped bullets show jagged mid-sentence breaks on\n  the release page (verified on v10.4.0; the docs website reflows and hides the problem). See\n  `guides/How-to-release.md`. Append a new entry at the **end** of its section, never at the top —\n  the section reads in the order the changes landed.\n- `swip/` - SkyWalking Improvement Proposals\n\n### SWIP vs. operator docs\n\nThe two kinds of document answer different questions and have different lifetimes:\n\n- **`docs/en/swip/SWIP-N.md` records the design discussion up to first acceptance.** It is a\n  historical record of what was proposed and agreed, not a description of current behaviour.\n- **The operator / end-user docs describe how the system behaves today.** They are the only\n  place a reader should be sent for current truth.\n\nWhen the implementation of a SWIP lands, sync the SWIP once to what was actually built — every\nproposed change written in implemented/past tense, with **no `TODO` / \"future work\" /\n\"open dependency\" / \"lands later\" / \"empty until X\" language for anything in the SWIP's\nscope**. **Once that SWIP ships as part of a release, it is frozen and must not be edited again.**\n\nFrozen means frozen even when the SWIP's text becomes factually stale. A later refactor that\nrenames a package, moves a class, deletes a constant or changes a class-naming scheme **does not**\nlicense updating the SWIP to match — not to keep code references accurate, not for internal\nconsistency, and not because the same PR already touched it. Ask instead: *would an end user read\nthis to learn how the system works today?* If yes it is an operator doc and must be updated; if it\nis a SWIP the answer is no, and the fix belongs in the operator doc plus `changes.md`.\n\nAfter that first sync, further metrics, dashboards, and incremental enhancements go into the\n**operator doc**, not the SWIP. Only genuinely small, optional follow-ups may stay under a SWIP's\n\"future work\" — never a big change.\n\n- BanyanDB self-observability: the SWIP is `docs/en/swip/SWIP-15.md`; the living operator\n  catalog is `docs/en/banyandb/dashboards-banyandb.md` (menu: \"BanyanDB self observability\n  dashboard\"). New `meter_banyandb_*` rules → update the operator doc; leave the SWIP alone.\n\n### Doc cross-references\n\nReference another section with a **Markdown anchor link** `[section title](#heading-slug)`,\nnever the `§` symbol or a bare \"section N\". Verify every anchor resolves to a real heading\n(GitHub slug rules: lowercase, spaces → `-`, punctuation dropped) — a broken `[](#…)`\nsilently renders as plain text on the website.\n\n## Handling Existing Tests\n\nIf existing test code (UT, IT, or E2E) fails to pass, try to fix it. If any test cases need to be modified, or test code needs to be deleted, the change must be double-checked by a human before it is applied — surface the proposed modification or deletion and the reasoning, and wait for explicit approval rather than changing or removing the test yourself.\n\n## Submitting Pull Requests\n\nUse the `/gh-pull-request` skill for committing and pushing to a PR branch. It runs pre-flight checks (compile, checkstyle, license headers) before every push, and creates the PR if one doesn't exist yet.\n\n## GitHub Actions Allow List\n\nApache enforces an allow list for third-party GitHub Actions. All third-party actions must be pinned to an approved SHA from:\nhttps://github.com/apache/infrastructure-actions/blob/main/approved_patterns.yml\n\nIf a PR is blocked by \"action is not allowed\" errors, check the approved list and update `.github/workflows/` files to use the approved SHA pin instead of a version tag.\n\nActions owned by `actions/*` (GitHub), `github/*`, and `apache/*` are always allowed (enterprise-owned).\n\n## Tips for AI Assistants\n\n1. **Always check submodules**: Protocol changes may require submodule updates\n2. **Generate sources first**: Run `mvnw compile` before analyzing generated code\n3. **Install package**: Use `mvnw flatten:flatten install` to build the precompiler and export generated classes before running tests. ref to [compile skill doc](.claude/skills/compile/SKILL.md)\n3. **Full rebuild on cross-module changes**: If you changed more than two modules or pulled/rebased code from git remote, run `mvnw clean install` (or `mvnw clean package`) on the **whole project** rather than picking individual modules with `-pl`. Incremental `-pl ... -am` builds can leave stale jars in `.m2` or `oap-libs/` when jar sizes don't change but content does, causing hard-to-debug runtime issues.\n3. **Respect checkstyle**: No System.out, no @author, no Chinese characters\n4. **Follow module patterns**: Use existing modules as templates\n5. **Check multiple storage implementations**: Logic may vary by storage type\n6. **OAL generates code**: Don't manually edit generated metrics classes\n7. **Use Lombok**: Prefer annotations over boilerplate code\n8. **Test both unit and integration**: Different test patterns for different scopes\n9. **Documentation is rendered via markdown**: When reviewing docs, consider how they will be rendered by a markdown engine\n10. **Relative paths in docs are valid**: Relative file paths (e.g., `../../../oap-server/...`) in documentation work both in the repo and on the documentation website, supported by website build tooling\n11. **Module service registration**: When adding a service to `CoreModule.services()`, update ALL `CoreModuleProvider` implementations — not just the main one. Search with `grep -rn \"extends CoreModuleProvider\" oap-server/ --include=\"*.java\"`. The `MockCoreModuleProvider` in `server-tools/profile-exporter/` also needs it, or the profile exporter e2e test will fail at startup.\n12. **Multiple OAP packagings**: The OAP server is not only the main `server-starter`. The `server-tools/` directory contains standalone tools (e.g., profile exporter) that boot with mock module providers and a subset of modules. Changes to core module contracts (services, required modules) must be reflected in these tools too.\n13. **`moduleManager.find(X.NAME)` requires `X.NAME` in `requiredModules()`**: every call to `moduleManager.find(SomeModule.NAME)` (direct or through a helper) must have `SomeModule.NAME` in the provider's `requiredModules()` array. Missing declarations cause runtime exceptions the first time the code path fires — not at module boot. Wrapping the call in `try { ... } catch (Throwable)` is NOT a substitute; declare the module and keep the try/catch only for defensive handling of transient provider outages. When auditing a branch, grep for `moduleManager.find(` across the touched module and verify each target name appears in `requiredModules()`. Example modules that frequently catch teams out: `AlarmModule` (used by alarm-kernel reset), `LogAnalyzerModule` (used by LAL factory lookup).\n14. **Don't look up `ClusterModule` services directly**: the `ClusterModule` (ZooKeeper / K8s / Nacos coordination) exposes `ClusterRegister` / `ClusterNodesQuery` / `ClusterCoordinator`. Most receiver / analyzer modules don't declare `ClusterModule` in `requiredModules()`, so calling `moduleManager.find(ClusterModule.NAME)` will throw at runtime. Instead, go through `CoreModule`'s `RemoteClientManager` service — it's already populated by the cluster module and exposes the peer list every OAP needs. If a module genuinely needs cluster-coordinator primitives, declare `ClusterModule.NAME` in `requiredModules()` explicitly.\n15. **No `ThreadLocal` side-channels to hijack downstream behaviour**: routing a caller's intent through a `ThreadLocal` that downstream code reads (e.g., `if (PeerMode.isActive()) skipSomething()`) is almost always the wrong answer — it creates invisible coupling between far-apart code paths, leaks across async hand-offs (executors, gRPC threads, Armeria event loops), and makes the behaviour impossible to understand from a method signature. The correct fix is almost always to **extend the interface** — add a parameter, a new method, a new mode enum that appears in the signature. Rare exceptions: propagating OpenTelemetry context where the whole industry has standardised on `ThreadLocal`, or security principals enforced by a framework. In all other cases, prefer an explicit API extension, even if it costs more lines.\n16. **BanyanDB schema-visibility: fence on `mod_revision`, do NOT poll metadata**: schema no longer lives in etcd — etcd was removed. It is now stored as property documents in BanyanDB's own `_schema` store and propagated to each data node's in-memory cache **asynchronously** (a `SchemaUpdateService.WatchSchemas` gRPC stream + a 30s reconcile sync, see `banyand/metadata/schema/property/client.go`). DDL `Create` / `Update` / `Delete` broadcast only to the schema-server (`ROLE_META`) nodes and **return immediately with a `mod_revision`** — now a client-stamped `time.Now().UnixNano()` timestamp, NOT an etcd index (which is why it still changes on every content-changing DDL; it is `0` on a delete that didn't record a tombstone) — **without waiting for data nodes**. A data write whose schema hasn't propagated yet is dropped at the data-node executor (`cannot find measure definition`, logged + skipped, `banyand/measure/write_standalone.go`), so the fence is still required — arguably more than under the old etcd-watch model. After firing DDL, fence on `BanyanDBClient.getSchemaWatcher().awaitRevisionApplied(maxRev, timeout)` before unparking dispatch / firing data writes — this blocks (liaison-side, all alive nodes, bounded by the timeout) until every data node's `notifiedModRevision` watermark reaches `maxRev`, which the registry's read-after-write does not guarantee. For deletes that returned `mod_revision == 0`, fall back to `awaitSchemaDeleted(SchemaKey, timeout)`. The previous \"poll `findMeasure` until you can read your own write\" idiom predates the `SchemaBarrierService` proto and has been replaced — do not reintroduce it. The `SchemaWatcher` class is in OAP's **in-tree** `library-banyandb-client` (`org.apache.skywalking.library.banyandb.v1.client.SchemaWatcher`); the standalone `skywalking-banyandb-java-client` repo does NOT have these symbols, so don't conclude the API is gone from grepping it. JDBC and ES are synchronous-DDL on the coordinator so they don't need a fence.\n\n## Analysis and Design Principles\n\n**Never guess or speculate.** All analysis must be grounded in source code, documentation, or verified behavior.\n\n### Before making claims\n- **Read the source code** — don't assume how a feature works based on naming or convention. Check the actual implementation.\n- **Read the documentation** — check `docs/en/`, CLAUDE.md files in submodules, and README files before proposing designs.\n- **Check configuration and flags** — verify what flags/env vars exist, their default values, and how they are parsed (e.g., BanyanDB uses viper with `BYDB_` prefix to auto-bind flags to env vars).\n- **Check dependent projects** — SkyWalking depends on BanyanDB, infra-e2e, Helm charts, etc. Read their source code and docs before assuming capabilities (e.g., check Helm chart `values.yaml` for supported fields, check infra-e2e for supported config options). For `skywalking-*` projects, ask the developer if they have the source code locally — searching a local clone is much faster than fetching files via GitHub API.\n\n### Before proposing changes\n- **Verify locally first** — run the code, start the container, execute the test before pushing to CI. Don't use CI as a trial-and-error environment.\n- **Validate file paths and directory structures** — check where data actually goes (e.g., BanyanDB `--stream-root-path /tmp` creates `/tmp/stream/`, `--access-log-root-path /tmp` creates `/tmp/accesslog/`). Don't assume directory names.\n- **Validate YAML syntax** — after editing YAML files (especially with sed/awk), validate with a YAML parser before committing. Corrupted YAML causes silent failures in CI.\n- **Check the actual Docker image** — verify what's available in the container (binaries, shell, directories) before writing commands that depend on them.\n\n### When uncertain\n- **Say \"I don't know\" and investigate** — reading the code is always better than guessing. Use grep, find, and read tools to locate the answer.\n- **Ask the developer first** — if you can't find the source code, don't know how to run something, or the code doesn't make the answer clear, ask the developer where to find it rather than speculate.\n- **Test with real data** — when investigating runtime behavior (e.g., what model names an API returns, what directory structure BanyanDB creates), set up a local test and observe the actual output.\n\n### Docker images\n- **Apache SkyWalking projects** — images are on `ghcr.io/apache/` (e.g., `ghcr.io/apache/skywalking-banyandb:${COMMIT_SHA}`). Tags are full commit SHAs, not short SHAs or version tags.\n- **Official and 3rd-party images** — on Docker Hub (e.g., `ollama/ollama`, `otel/opentelemetry-collector`, `envoyproxy/gateway`).\n- **Always verify the image exists** — `docker pull` before writing CI or e2e configs. Image tags depend on CI publish workflows completing successfully.\n"},"items":[{"name":"SKILL.md","path":".claude/skills/ci-e2e-debug/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/ci-e2e-debug/SKILL.md","title":"ci-e2e-debug","category":"anthropic-skill","format":"markdown","content":"---\nname: ci-e2e-debug\ndescription: Download and inspect CI e2e test logs from GitHub Actions artifacts. Use when investigating e2e test failures in CI.\nargument-hint: \"<RUN_ID or GitHub Actions URL>\"\n---\n\n# CI E2E Debug Skill\n\nDownload test log artifacts from a GitHub Actions CI run and inspect them for errors.\n\n## Steps\n\n1. **Find the artifact**: Use `gh api` to list artifacts for the given CI run:\n   ```bash\n   gh api repos/apache/skywalking/actions/runs/<RUN_ID>/artifacts --jq '.artifacts[] | {id: .id, name: .name}'\n   ```\n\n2. **Download and extract**: Download the artifact zip and extract it:\n   ```bash\n   cd /tmp && rm -rf e2e-debug-logs && mkdir e2e-debug-logs && cd e2e-debug-logs\n   gh api repos/apache/skywalking/actions/artifacts/<ARTIFACT_ID>/zip > artifact.zip\n   unzip -o artifact.zip\n   ```\n\n3. **Inspect OAP logs**: Look for errors in the OAP server logs:\n   ```bash\n   # Find OAP log files\n   find /tmp/e2e-debug-logs -name \"skywalking-oap-*.log\" -o -name \"oap.log\"\n   # Check for errors\n   grep -E \"ERROR|Exception|FATAL|CannotCompileException\" <log_file> | head -30\n   ```\n\n4. **Inspect other component logs**: Check BanyanDB, UI, and other pod logs as needed.\n\n5. **Report findings**: Summarize the root cause error from the logs.\n\n## Notes\n- CI artifacts are automatically uploaded by the e2e test framework to `$SW_INFRA_E2E_LOG_DIR`\n- Log files are organized by namespace/pod name\n- OAP init pods may have different errors than the main OAP pod — check all of them\n- Common errors: MAL/LAL/OAL compilation failures, storage connection issues, module initialization errors\n- **Profile exporter tests** (e.g., Trace Profiling ES): the exporter runs as a separate process with `MockCoreModuleProvider`. If a new service is added to `CoreModule.services()` but not registered in the mock, the exporter fails at startup with `requiredCheck()` error — but the OAP logs will show no errors (OAP is fine, the exporter subprocess is what fails). Check the test step that runs `profile_exporter.sh`.\n- **When OAP logs are clean but tests fail**: look at which specific test step failed. Some e2e tests invoke standalone tools (profile exporter, data generator) that boot with mock providers and may fail independently of OAP.\n","frontmatter":{"name":"ci-e2e-debug","description":"Download and inspect CI e2e test logs from GitHub Actions artifacts. Use when investigating e2e test failures in CI.","argument-hint":"<RUN_ID or GitHub Actions URL>"},"isInternal":false,"tokens":576,"sizeBytes":2251},{"name":"SKILL.md","path":".claude/skills/compile/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/compile/SKILL.md","title":"compile","category":"anthropic-skill","format":"markdown","content":"---\nname: compile\ndescription: Build SkyWalking OAP server, run javadoc checks, and verify checkstyle. Use to validate changes before submitting a PR.\nargument-hint: \"[all|backend|javadoc|checkstyle|module-name]\"\n---\n\n# Compile & Verify\n\nBuild the project and run static checks matching CI.\n\n## Prerequisites\n\n- JDK 11, 17, or 21 (LTS versions)\n- Maven 3.6+ (use `./mvnw` wrapper)\n\n## Maven profiles\n\n- `backend` (default): Builds OAP server modules\n- `ui` (default): Builds web application\n- `dist` (default): Creates distribution packages\n- `all`: Builds everything including submodule initialization\n\n## Commands by argument\n\n### `all` or no argument — full CI build\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### `backend` — backend only (faster)\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -Dmaven.test.skip\n```\n\n### `javadoc` — javadoc check only\n\nJavadoc requires delombok output, so `install` must run first:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nRunning `javadoc:javadoc` alone without `install` will miss errors because `${delombok.output.dir}` won't be populated.\n\n### `checkstyle` — checkstyle only\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n\n### Module name — single module build\n\n```bash\n./mvnw clean flatten:flatten package -pl oap-server/analyzer/<module-name> -Dmaven.test.skip\n```\n\n## Reading javadoc output\n\nMaven prefixes all javadoc output with `[ERROR]`, but the actual severity is in the message after the line number. Only lines containing `error:` fail the build; lines with `warning:` do not.\n\n```\n[ERROR] Foo.java:42: error: bad use of '>'        ← ACTUAL ERROR (must fix)\n[ERROR] Foo.java:50: warning: no @param for <T>   ← WARNING (does not fail build)\n```\n\n### Common javadoc errors\n\n| Error | Cause | Fix |\n|-------|-------|-----|\n| `bad use of '>'` | Bare `>` in javadoc HTML (e.g., `->` in `<pre>` blocks) | Use `{@code ->}` or `-&gt;` |\n| heading out of sequence | Heading level skips the expected hierarchy | See heading rules below |\n| reference not found | `{@link Foo#bar()}` with wrong signature | Match exact parameter types: `{@link Foo#bar(ArgType)}` |\n\n### Javadoc heading rules (JDK 13+)\n\nStrict heading validation was introduced in JDK 13. JDK 11 does **not** enforce it, but JDK 17/21/25 do. Write headings correctly for forward compatibility:\n\n| Javadoc location | Start heading at |\n|---|---|\n| Class, interface, enum, package, module | `<h2>` |\n| Constructor, method, field | `<h4>` |\n| Standalone HTML files (`doc-files/`) | `<h1>` |\n\nThe generated javadoc page uses `<h1>` for the class name and `<h3>` for member sections (Methods, Fields, etc.), so class-level subsections must use `<h2>` and method-level subsections must use `<h4>` to maintain proper nesting.\n\n## CI reference\n\nCI uses JDK 11 on Linux. The `dist-tar` job runs:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nThe `code-style` job runs:\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n","frontmatter":{"name":"compile","description":"Build SkyWalking OAP server, run javadoc checks, and verify checkstyle. Use to validate changes before submitting a PR.","argument-hint":"[all|backend|javadoc|checkstyle|module-name]"},"isInternal":false,"tokens":907,"sizeBytes":3261},{"name":"SKILL.md","path":".claude/skills/generate-classes/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/generate-classes/SKILL.md","title":"generate-classes","category":"anthropic-skill","format":"markdown","content":"---\nname: generate-classes\ndescription: Generate bytecode classes from DSL scripts (MAL, OAL, LAL, Hierarchy). Runs the compiler and dumps .class files for inspection.\nargument-hint: \"<mal|oal|lal|hierarchy|all>\"\n---\n\n# Generate DSL Classes\n\nRun the v2 compiler (ANTLR4 + Javassist) to generate bytecode classes from DSL scripts and dump `.class` files to disk for inspection.\n\n## Commands by argument\n\n### `mal` — MAL expression classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=MALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/mal/**/*.generated-classes/`\n\n### `oal` — OAL metrics/dispatcher/builder classes\n\n```bash\n./mvnw test -pl oap-server/oal-rt \\\n  -Dtest=RuntimeOALGenerationTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/oal-rt/target/test-classes/metrics/`, `metrics/builder/`, `dispatcher/`\n\n### `lal` — LAL filter/extractor classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=LALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/lal/**/*.generated-classes/`\n\n### `hierarchy` — Hierarchy rule classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=HierarchyRuleExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/hierarchy-rule/*.generated-classes/`\n\n### `all` or no argument — generate all DSLs\n\nRun all four commands above sequentially.\n\n## After generation\n\nPrint the output location for the requested DSL so the user knows where to find the generated `.class` files. Use `javap` to decompile:\n\n```bash\njavap -c -p <path-to-class-file>\n```\n\n## Cleaning generated classes\n\n```bash\n./mvnw clean -pl oap-server/analyzer/dsl-scripts-test\n```\n","frontmatter":{"name":"generate-classes","description":"Generate bytecode classes from DSL scripts (MAL, OAL, LAL, Hierarchy). Runs the compiler and dumps .class files for inspection.","argument-hint":"<mal|oal|lal|hierarchy|all>"},"isInternal":false,"tokens":521,"sizeBytes":1976},{"name":"SKILL.md","path":".claude/skills/gh-pull-request/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/gh-pull-request/SKILL.md","title":"gh-pull-request","category":"anthropic-skill","format":"markdown","content":"---\nname: gh-pull-request\ndescription: Verify, commit, and push changes on a PR branch. Runs pre-flight checks (compile, checkstyle, license headers) before every push. Also creates the PR if one doesn't exist yet.\n---\n\n# PR Branch Workflow\n\nRun pre-flight checks, commit, push, and optionally create a PR.\n\n## Pre-flight checks\n\nRun these checks before every commit+push and fix any failures:\n\n### 1. Compile and checkstyle\n\n```bash\n# Checkstyle\n./mvnw -B -q clean checkstyle:check\n\n# Full build (compile + javadoc)\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### 2. License header check\n\n```bash\nlicense-eye header check\n```\n\nIf invalid files are found, fix with `license-eye header fix` and re-check.\n\n### 3. Unnecessary fully-qualified class names\n\nThe project checkstyle forbids inline FQCNs — every type reference in code should resolve\nthrough an `import`, not a fully-qualified name. Checkstyle does not always catch this (it\nmisses cases like inline `java.util.HashMap`, `java.util.concurrent.TimeUnit`, or\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` used as a local\nvariable type, generic parameter, or `new` target). Audit the files the branch touched\nbefore pushing:\n\nUse the `Grep` tool (ripgrep) rather than BSD `grep` on macOS — the scan below relies on a\nnegative lookahead that BSD `grep` doesn't support and GNU `grep -P` does:\n\n```\npattern: ^(?!\\s*(import |package |\\s*\\*)).*\\b(java\\.util\\.|java\\.io\\.|java\\.nio\\.|java\\.util\\.concurrent\\.|javassist\\.|org\\.apache\\.skywalking\\.)[A-Z][A-Za-z0-9_]*\nglob:    *.java\noutput_mode: content\n-n: true\n```\n\nScope the scan to files the branch touched, not the whole tree — pre-existing FQDNs on\nunrelated files generate noise. Use `git diff --name-only master...HEAD -- '*.java'` to get\nthe changed list, then run the ripgrep pattern against each.\n\nAcceptable exceptions (same as the `CLAUDE.md` rule):\n - Two classes with the same simple name would collide if both imported.\n - A Javadoc `{@link}` where the short name would be ambiguous to the reader.\n - Inside a string literal (e.g., a class name passed to `Class.forName`).\n\nFix every other hit — add an `import` and switch to the short name. This includes\n`new java.util.HashMap<>()`, `java.util.Set<String>` parameter types, and\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` as a local\nvariable type. Field declarations, method signatures, local variables, and generic\ntype arguments should all use the imported short name.\n\nRe-run checkstyle after the fix — a sloppy `sed`/`replace_all` can corrupt the `import`\nline itself (e.g., turning `import java.util.concurrent.locks.ReentrantLock;` into\n`import ReentrantLock;`), which causes a cryptic checkstyle `Range [0, -1) out of\nbounds for length N` error, not a normal violation line. If you see that error, inspect\nthe imports block first.\n\n## Commit and push\n\nAfter checks pass, commit and push:\n\n```bash\ngit add <files>\ngit commit -m \"<message>\"\ngit push -u origin <branch-name>\n```\n\n### Branch strategy\n- **Never work directly on master branch**\n- If on master, create a new branch first: `git checkout -b feature/<name>` or `git checkout -b fix/<name>`\n\n## Create PR (if not yet created)\n\nCheck whether a PR already exists for the current branch:\n\n```bash\ngh pr view --json number 2>/dev/null\n```\n\nIf no PR exists, create one:\n\n### PR title\nSummarize the changes concisely. Examples:\n- `Fix BanyanDB query timeout issue`\n- `Add support for OpenTelemetry metrics`\n\n### PR description\n\nRead `.github/PULL_REQUEST_TEMPLATE` and use its **exact format with checkboxes**. Do NOT use a custom summary format.\n\nKey template sections — uncomment the relevant one:\n\n**For Bug Fixes:**\n```\n### Fix <bug description or issue link>\n- [ ] Add a unit test to verify that the fix works.\n- [ ] Explain briefly why the bug exists and how to fix it.\n```\n\n**For New Features:**\n```\n### <Feature description>\n- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.\n- [ ] Update the documentation to include this new feature.\n- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.\n- [ ] If it's UI related, attach the screenshots below.\n```\n\n**For Performance Improvements:**\n```\n### Improve the performance of <class or module or ...>\n- [ ] Add a benchmark for the improvement.\n- [ ] The benchmark result.\n- [ ] Links/URLs to the theory proof or discussion articles/blogs.\n```\n\n**Always include:**\n```\n- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.\n- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).\n```\n\n### Create command\n\n```bash\ngh pr create --title \"<title>\" --body \"$(cat <<'EOF'\n<PR body from template>\nEOF\n)\"\n```\n\n### Post-creation\n- Add `copilot` as a reviewer: `gh pr edit <number> --add-reviewer copilot`\n- Do NOT add AI assistant as co-author. Code responsibility is on the committer's hands.\n- Return the PR URL when done.\n\n## After the PR is merged\n\nOnce the PR is merged, sync the default branch and clean up the feature branch:\n\n```bash\n# 1. Prune stale remote refs. GitHub auto-deletes the PR's branch on merge, so\n#    the remote feature branch is usually already gone; --prune removes the\n#    dangling local tracking ref.\ngit fetch origin --prune\n\n# 2. Switch back to the default branch and fast-forward it to include the merge.\ngit checkout master\ngit pull --ff-only origin master\n\n# 3. Confirm the change actually landed in master before deleting anything —\n#    `git log --oneline -1` should show the merge/squash commit with the PR\n#    number, or grep for a symbol the PR introduced.\ngit log --oneline -1\n\n# 4. Delete the local feature branch. SkyWalking SQUASH-merges PRs, so the\n#    feature branch's commit is NOT an ancestor of master (master gets a new\n#    squash commit instead). `git branch -d` therefore reports \"not fully\n#    merged\" — that is expected, not an error. After confirming the content is\n#    in master (step 3), force-delete:\ngit branch -d <branch> 2>/dev/null || git branch -D <branch>\n```\n\nNotes:\n- `git branch -d` failing with \"not fully merged\" on a squash-merged PR is normal — the squash commit has a different SHA than the feature commit. Verify via step 3, then `-D`.\n- If the remote branch was not auto-deleted (some repo settings), remove it explicitly: `git push origin --delete <branch>`.\n- Do NOT skip step 3. Force-deleting a local branch whose work didn't actually merge loses it.\n","frontmatter":{"name":"gh-pull-request","description":"Verify, commit, and push changes on a PR branch. Runs pre-flight checks (compile, checkstyle, license headers) before every push. Also creates the PR if one doesn't exist yet."},"isInternal":false,"tokens":1696,"sizeBytes":6622},{"name":"SKILL.md","path":".claude/skills/license/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/license/SKILL.md","title":"license","category":"anthropic-skill","format":"markdown","content":"---\nname: license\ndescription: Check and fix Apache 2.0 license headers and dependency licenses using skywalking-eyes. Use before submitting a PR.\nargument-hint: \"[check|fix|deps]\"\n---\n\n# License Checks\n\nCheck and fix license compliance using [skywalking-eyes](https://github.com/apache/skywalking-eyes). Two CI jobs use this tool:\n\n1. **license-header** — verifies all source files have Apache 2.0 headers\n2. **dependency-license** — verifies the LICENSE file matches current dependencies\n\n## Steps\n\n### Header check (default, or `check` argument)\n\n```bash\nlicense-eye header check\n```\n\n- **All valid**: Output shows `valid: N, invalid: 0` — nothing to do.\n- **Invalid files found**: Fix with `license-eye header fix`, then re-check.\n\n### Header fix (`fix` argument)\n\n```bash\nlicense-eye header fix\nlicense-eye header check\n```\n\n### Dependency license check (`deps` argument)\n\nThis regenerates the LICENSE file from dependency metadata and checks for drift:\n\n```bash\nlicense-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl || exit 1\nif [ ! -z \"$(git diff -U0 ./dist-material/release-docs/LICENSE)\" ]; then\n  echo \"LICENSE file is not updated correctly\"\n  git diff -U0 ./dist-material/release-docs/LICENSE\nfi\n```\n\nIf the LICENSE file changed, review the diff and commit it. **Important**: CI runs on Linux — some dependencies have platform-specific variants. If you're on macOS/Windows, the LICENSE diff may be a platform artifact. Verify before committing.\n\n## Rules\n\nConfiguration is in `.licenserc.yaml`:\n- Java, XML, YAML/YML files require Apache 2.0 headers\n- JSON and Markdown files are excluded (JSON doesn't support comments)\n- Generated files and certain vendor paths are excluded\n- SPI service files (`META-INF/services/`) require headers (use `#` comment style)\n\n## Installation\n\n```bash\n# Same version as CI (pinned commit)\ngo install github.com/apache/skywalking-eyes/cmd/license-eye@5b7ee1731d036b5aac68f8bd3fc9e6f98ada082e\n\n# Or via Homebrew (macOS)\nbrew install license-eye\n```\n","frontmatter":{"name":"license","description":"Check and fix Apache 2.0 license headers and dependency licenses using skywalking-eyes. Use before submitting a PR.","argument-hint":"[check|fix|deps]"},"isInternal":false,"tokens":491,"sizeBytes":2028},{"name":"SKILL.md","path":".claude/skills/new-monitoring-feature/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/new-monitoring-feature/SKILL.md","title":"new-monitoring-feature","category":"anthropic-skill","format":"markdown","content":"---\nname: new-monitoring-feature\ndescription: Add a new monitoring target / layer to SkyWalking OAP. Orients you to the OAL / MAL / LAL / SpanListener / SegmentListener extension points, the UI template + submodule touchpoints, the docs set that must move together, and the cross-cutting traps that don't live in any one skill.\n---\n\n# Adding a New Monitoring Feature / Layer\n\nSkyWalking monitoring for a new target (a new `Layer` — mobile OS, gateway, DB, language runtime, …) is composed from the same set of reusable extension points. This skill is the index: which extension point fits which signal, where its contract lives, which docs must be updated together, and which harness-wide traps the extension-specific docs don't cover.\n\nThere is one skill per narrow concern. This one is the wiring map.\n\n---\n\n## Working posture\n\n- **Default scope for a new monitoring feature is narrow.** For most new monitoring targets, traces are **re-used as-is** — you do not write or modify a trace analyzer. Metrics and log extensions via OTLP (MAL + LAL) are the common shape. OTLP / Zipkin span-based metrics analysis (`SpanListener`) already exists and only needs a new implementation if your feature introduces genuinely new semantic-convention attributes to extract (e.g. `gen_ai.*`, `mcp.*` for the GenAI layer). Do **not** touch `CommonAnalysisListener` / `RPCAnalysisListener` / `SegmentAnalysisListener` unless the OAP core protocol gains a new `SpanLayer` enum value — details in §2.\n- **If you find yourself editing more than this skill suggests, stop and confirm.** Scope creep on trace analysis, span-layer mapping, receiver handlers, or protobuf/proto contracts is a strong signal the framing is wrong. Ask the developer before committing that direction.\n- **Verify every new feature locally before pushing to CI.** Compile + checkstyle + license-eye are minimums, but they only tell you the code builds — they do **not** tell you the new MAL rule compiles its closures at startup, the LAL layer:auto dispatch works, the OTLP payload shape matches the MAL labels, the UI template renders against real data, or the layer shows up in `swctl service ly <LAYER>`. Run the new e2e case end-to-end locally against a live OAP (see [run-e2e skill](../run-e2e/SKILL.md)) and fire every verify step with `swctl` by hand at least once. Pushing to CI without a local green run wastes shared CI cycles and slow-loops review.\n\n---\n\n## 0. Register the `Layer` — the feature's entry point\n\nA `Layer` is how OAP slices services / instances / endpoints by data source. The UI, storage partitioning, menu navigation, and OAL aggregation all key off it.\n\n`Layer` is a registry-backed value type (no longer a closed enum). Built-in layers are declared as `public static final Layer` constants in `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java`; external layers are registered through the `Layer.register(name, ordinal, normal)` API at boot. **Pick the registration path that matches the scope of your feature:**\n\n| Your feature ships as | Registration path |\n|---|---|\n| Part of the OAP distribution (in-tree, the common case for new SkyWalking-supported targets) | Add a `public static final Layer` constant to `Layer.java` with the next sequential ordinal in `0–49`. Examples: `IOS = register(\"IOS\", 47, true)`, `APISIX = register(\"APISIX\", 21, true)`, `VIRTUAL_DATABASE = register(\"VIRTUAL_DATABASE\", 14, false)` for inferred/non-real services. |\n| An out-of-tree MAL or LAL rule file | Add a top-level `layerDefinitions:` block to the rule file. The DSL loader funnels each entry through `Layer.register` before compiling the rule. One file ships the layer + the rules that produce its telemetry. |\n| An out-of-tree plugin module (jar) | Implement `org.apache.skywalking.oap.server.core.analysis.LayerExtension` and register via `META-INF/services/`. Discovered by `LayerExtensionLoader` during `CoreModuleProvider.prepare()`. |\n| Operator-deployed config (no code, no DSL) | Add an entry to `oap-server/server-starter/src/main/resources/layer-extensions.yml` (or override on the OAP node's classpath). |\n\n**Ordinal conventions:** `0–49` is in active use by built-ins. `50–999` is reserved by convention for future built-in layers. External layers are recommended (not required) to start at `>= 1000` to avoid colliding with future built-ins on OAP upgrade. Collisions in either direction are detected at boot via the ordinal-uniqueness check, which fails OAP startup loudly.\n\n**Storage encoding is the ordinal int**, persisted in BanyanDB / Elasticsearch / JDBC. Every OAP node that reads or writes a given layer must agree on its `(name, ordinal)` mapping — deploy `layer-extensions.yml` and any `layerDefinitions:` rule files identically across all nodes. The registry is sealed at the start of `Core.notifyAfterCompleted()`; later registration attempts throw.\n\n**UI template folders are auto-discovered by file scan, not by `Layer.values()`.** `UITemplateInitializer` walks `ui-initialized-templates/**/*.json` recursively (depth 2) and trusts each template's own `configuration.layer` field. Drop a folder of dashboard JSONs on disk and the initializer picks them up on the next boot — folder name is purely organizational.\n\n**Component ID lookup in Java code**: IDs declared in `component-libraries.yml` are loaded at runtime into `ComponentLibraryCatalogService`'s `componentName2Id` map — they are **not** exposed as Java enum constants. To look up by name in listener code, inject the catalog service and resolve once at construction:\n```java\nIComponentLibraryCatalogService catalog = moduleManager.find(CoreModule.NAME)\n    .provider().getService(IComponentLibraryCatalogService.class);\nint myComponentId = catalog.getComponentId(\"My-Component-Name\");\n```\nCache as an `int` field; runtime comparisons are then plain `componentId == myComponentId`. **Trap:** there is a `ComponentsDefine` class under `skywalking-trace-receiver-plugin/src/test/java/.../mock/ComponentsDefine.java` — it is a test-only mock holding five hand-picked constants (Tomcat, Dubbo, RocketMQ, MongoDB). Do not import or extend it from production code.\n\nEmit the layer from every source object your feature produces. Built-in layers have a static-field accessor; external layers are looked up by name through the registry:\n\n```java\n// Built-in layer (constant)\nservice.setLayer(Layer.IOS);\n\n// External layer (registered via yaml / SPI / layerDefinitions:)\nservice.setLayer(Layer.nameOf(\"IOT_FLEET\"));\n```\n\nDownstream (the core OAL, `service ly <LAYER>` swctl query, topology filters, UI root dashboard's layer selector) all work off this single enum value.\n\n---\n\n## 1. Pick the right extension point\n\n| Signal you want to process | Extension point | Register via |\n|---|---|---|\n| **SkyWalking native trace segments** (from language agents) | `AnalysisListener` (TraceSegmentListener / EntryAnalysisListener / ExitAnalysisListener / LocalAnalysisListener / FirstAnalysisListener) + factory in `agent-analyzer` | `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory` |\n| **OTLP / Zipkin trace spans** (OTel SDKs, Istio, Envoy, iOS/Android, …) | `SpanListener` in `core.trace` | `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener` |\n| **OTel metrics** (Prometheus scrape, Envoy stats, Istio, …) | MAL rules — `otel-rules/<namespace>/*.yaml` | Append the rule path to `enabledOtelMetricsRules` in `application.yml` |\n| **Logs** (OTLP, SkyWalking native, Fluent Bit, Envoy ALS) | LAL rules — `lal/<rule>.yaml` | Append to `SW_LOG_LAL_FILES` (env) or `lalFiles` (YAML) under `log-analyzer` |\n| **Pre-computed meter / custom meter** (custom agent protocol, meter receiver) | MAL `meter-analyzer-config/*.yaml` | Append to `meter-analyzer-config` entries |\n| **Aggregation from Source entities** (service, instance, endpoint, relation) | OAL rules — `oal/*.oal` | Already loaded by `OAL grammar`; just add lines |\n\nAll five DSLs (OAL, MAL, LAL, Hierarchy + SpanListener SPI) compile via ANTLR4 + Javassist at startup. Source-of-truth details are in:\n- [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md)\n- [`docs/en/concepts-and-designs/mal.md`](../../../docs/en/concepts-and-designs/mal.md)\n- [`docs/en/concepts-and-designs/lal.md`](../../../docs/en/concepts-and-designs/lal.md)\n- [`docs/en/concepts-and-designs/scope-definitions.md`](../../../docs/en/concepts-and-designs/scope-definitions.md)\n- [`oap-server/analyzer/meter-analyzer/CLAUDE.md`](../../../oap-server/analyzer/meter-analyzer/CLAUDE.md) — MAL compiler internals\n- [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — LAL compiler internals\n\n---\n\n## 2. Trace ingestion — two parallel pipelines\n\nSkyWalking OAP ingests traces through two distinct entry points. Pick based on source.\n\n> **Before writing any trace extension — ask \"do I actually need one?\"**\n>\n> For **most new layers you do not touch trace analysis at all.** Segments (native or\n> OTLP) flow through the existing pipeline, `RPCAnalysisListener.parseExit` produces\n> the outbound `ServiceRelation` edges, the backend-side agent produces the inbound\n> metrics via its own `parseEntry`, and the new-layer Service / Instance / Endpoint\n> entities come from **MAL + LAL**, not from trace analysis. This is how browser\n> (`Layer.BROWSER`), iOS (`Layer.IOS`), mobile mini-programs\n> (`Layer.WECHAT_MINI_PROGRAM` / `Layer.ALIPAY_MINI_PROGRAM`), and most\n> scraped-metric layers work.\n>\n> Extend trace analysis **only** when the feature genuinely needs per-span\n> interpretation that the existing pipeline cannot express:\n>\n> - **A new `SpanLayer`** — OAP core protobuf has a new `SpanLayer` enum value that\n>   callers of `identifyServiceLayer` / `identifyRemoteServiceLayer` must dispatch on.\n>   That means touching `CommonAnalysisListener` / `RPCAnalysisListener`.\n> - **A genuinely new signal class not covered by existing listeners** — e.g. the GenAI\n>   work added `GenAISpanListener` for LLM / MCP instrumentation because OTLP spans\n>   carry semantic-convention attributes (`gen_ai.*`, `mcp.*`) that no prior listener\n>   extracted. That is rare.\n>\n> **Anti-pattern we've been bitten by:** using a componentId mapping in\n> `CommonAnalysisListener` to assign a client-side layer to exit-only segments. This\n> does not produce inbound metrics anyway (no entry spans → no\n> `service_cpm` / `endpoint_cpm` from OAL), and it is orthogonal to the Service entity\n> layer (which MAL / LAL sets). Mirror the browser pattern instead: ship `Layer` +\n> `component-libraries.yml` + MAL + LAL + dashboards, and leave the listener chain\n> alone.\n\n### 2.1 SkyWalking native segments — `AnalysisListener`\n\n**Source protocol**: `apm-protocol/apm-network/src/main/proto/language-agent/Tracing.proto` (`TraceSegmentObject`).\n\n**Receiver**: `oap-server/server-receiver-plugin/skywalking-trace-receiver-plugin`. After wire decoding, segments are handed to `SegmentParserServiceImpl` which fans out to registered `AnalysisListenerFactory` implementations.\n\n**Extension**: implement one or more of\n- `FirstAnalysisListener` — per-segment, before per-span phase\n- `EntryAnalysisListener` — per-entry-span\n- `ExitAnalysisListener` — per-exit-span\n- `LocalAnalysisListener` — per-local-span\n- `TraceSegmentListener` — whole-segment, after per-span phase\n\nRegister the **factory** in `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory`. Existing examples: `RPCAnalysisListener`, `MultiScopesAnalysisListener`, `SegmentAnalysisListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`.\n\nUse when: the data arrives on the SW native wire (language agents — Java, Python, Go, Node.js, .NET, PHP, Lua, etc.).\n\n### 2.2 OTLP / Zipkin spans — `SpanListener`\n\n**Source protocol**: OTLP proto (`opentelemetry/proto/trace/v1/trace.proto`) or Zipkin v2 JSON.\n\n**Receiver**: `oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/.../otlp/OpenTelemetryTraceHandler.java` (OTLP) and `oap-server/server-receiver-plugin/zipkin-receiver-plugin/.../ITZipkinTraceCollector` (Zipkin). After decoding, spans are iterated and passed to every registered `SpanListener` through `SpanListenerManager`.\n\n**Extension**: implement `org.apache.skywalking.oap.server.core.trace.SpanListener`:\n\n```java\npublic interface SpanListener {\n    String[] requiredModules();\n    void init(ModuleManager moduleManager);\n    SpanListenerResult onOTLPSpan(OTLPSpanReader span, String serviceName, Map<String,String> resourceAttributes,\n                                  String scopeName, String scopeVersion);\n}\n```\n\nName the service by `serviceName`, not the resource's `service.name`: it is the service the span is analyzed and\nstored under, which `receiver-otel.otlpServiceGroupKeys` may have grouped (`shop::cart` for `service.name=cart`).\n\nRegister in `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener`. Reference implementations:\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSHTTPSpanListener.java` — OTel Swift URLSession → OAL (Service / ServiceInstance / Endpoint)\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSMetricKitSpanListener.java` — OTel Swift MetricKit → MAL samples via `OpenTelemetryMetricRequestProcessor.toMeter`\n- `oap-server/analyzer/gen-ai-analyzer/src/main/java/.../GenAISpanListener.java` — LLM / MCP instrumentation\n\nReturn `SpanListenerResult.builder().shouldPersist(false).build()` to prevent a span being stored as a trace (useful for e.g. 24-hour aggregate spans that are not meaningful as traces).\n\n**When both native and OTLP should feed the same metric**, write your extension against the downstream scope (Service / ServiceInstance / Endpoint + OAL) and emit those entities from **both** a SegmentListener and a SpanListener. The OAL layer is protocol-agnostic.\n\n---\n\n## 3. Metrics — MAL / OAL rules\n\n### 3.1 OAL (for entities already produced by core sources)\n\n`oap-server/server-starter/src/main/resources/oal/core.oal` already defines the standard `service_cpm`, `service_resp_time`, `service_sla`, `endpoint_cpm`, etc. — keyed off the `Service`, `ServiceInstance`, `Endpoint` source scopes. If your new layer emits those entities with the right `Layer`, you get the core metrics for free.\n\n**Who emits those entities matters.** `service_cpm` / `service_resp_time` / `service_sla` / `endpoint_cpm` / `endpoint_resp_time` come from `RPCAnalysisListener.parseEntry` → `callingIn.toService()` / `toEndpoint()`. That only fires on **inbound entry spans**. Client-side / edge layers (browser, iOS, mini-programs) only emit exit spans, so these OAL metrics do **not** populate under those layers — their Service entities are produced by MAL / LAL, and their dashboards use MAL metrics (`meter_<layer>_*`) for load / latency, not `service_cpm` / `service_resp_time`. Don't wire dashboards to OAL-derived metrics that the layer never produces.\n\nAdditional OAL rules go in the same file. OAL grammar: `oap-server/oal-grammar/src/main/antlr4/OALParser.g4`. Syntax and examples: [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md).\n\n### 3.2 MAL (for meter / metric samples)\n\n**File location**: `oap-server/server-starter/src/main/resources/otel-rules/<namespace>/*.yaml` (OTel scrape) or `meter-analyzer-config/*.yaml` (SW native meter).\n\n**Registration** (MANDATORY): append the rule path to the default in `oap-server/server-starter/src/main/resources/application.yml`:\n```yaml\nreceiver-otel:\n  default:\n    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:\"...,<your-namespace>/*\"}\n```\nIf you skip this, the rule isn't loaded in the default starter and downstream `converters.forEach(...)` is a silent no-op — samples just disappear.\n\n**Also update** `test/e2e-v2/cases/storage/expected/config-dump.yml`. This e2e diffs `/debugging/config/dump` output against a static file; every default-value change in `application.yml` must mirror here or the storage e2e fails.\n\n**Naming**: keep attribute scope in the metric name. If a metric reads from `foo.bar.foreground.exit_count`, call it `foo_foreground_exit_count`, not `foo_exit_count`. When two attributes (foreground vs background, ingress vs egress) share a theme, keep them as separate distinguishable metrics — don't flatten.\n\n### 3.3 Histogram pitfalls (both OAL and MAL)\n\n- **Per-bucket counts, not cumulative.** `histogram_percentile` accumulates internally. Feeding cumulative counts doubles the accumulator and shifts percentiles up. Each observation contributes `1` to exactly one `le` bucket.\n- **`le` label scale.** `SampleFamily.histogram()` rescales `le` by `defaultHistogramBucketUnit.toMillis(1)` (default `SECONDS` ⇒ ×1000). If your raw labels are already in ms, use `SampleFamilyBuilder.newBuilder(samples).defaultHistogramBucketUnit(TimeUnit.MILLISECONDS).build()`.\n- **Overflow bucket.** MAL parses `le=\"Infinity\"` as `Double.POSITIVE_INFINITY` → stored as `Long.MAX_VALUE`. If a percentile lands there, the UI renders ≈ 9.2 × 10¹⁸. Prefer a finite sentinel (e.g., 30 s for hang/launch) so outliers remain visible without crashing the chart.\n- **`NamingControl`** — every `service` / `serviceInstance` / `endpoint` label that ends up as an entity key must go through `namingControl.formatServiceName()` / `formatInstanceName()` / `formatEndpointName()`. Raw resource attributes (long git-SHA build metadata, path-style endpoints) will otherwise exceed storage limits.\n\n---\n\n## 4. Logs — LAL\n\n**File location**: `oap-server/server-starter/src/main/resources/lal/<feature>.yaml`.\n\n**Registration**: append to `SW_LOG_LAL_FILES` env var, or `log-analyzer.default.lalFiles` in `application.yml`.\n\n**Grammar**: `oap-server/analyzer/log-analyzer/src/main/antlr4/LALParser.g4`. The compiler docs are thorough: [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — covers:\n- `filter { … }` block with `parsed.*`, `tag()`, `sourceAttribute()`, `log.*`, `def` locals, `extractor {}`, `metrics {}`, `sink {}`\n- `layer: auto` mode (script decides layer from source attrs)\n- `inputType` / `outputType` SPI (`LALSourceTypeProvider`) to bind typed proto to `parsed.*`\n- Test harness at `oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/oap-cases/` — ship YAML + `.data.yaml` pair; `LALScriptExecutionTest` picks them up automatically.\n\n**`layer: auto` gotcha**: the extractor's `layer \"X\"` assignment must propagate to `LogMetadata.layer` (not just the output builder) — `FilterSpec.doSink` reads metadata to decide drop/route. The current codegen (`LALBlockCodegen.generateFieldToOutput`) handles this correctly; if you extend codegen, preserve that invariant.\n\n---\n\n## 5. UI — templates, submodule, and menu\n\n### 5.1 Dashboard templates\n\nJSON files under `oap-server/server-starter/src/main/resources/ui-initialized-templates/<feature>/`. Conventional fileset:\n\n- `<feature>-root.json` — `ServiceList` widget, layered service index.\n- `<feature>-service.json` — per-service dashboard, usually a `Tab` with Overview / Instance / Endpoint / Log sub-tabs.\n- `<feature>-instance.json` — per-instance dashboard.\n- `<feature>-endpoint.json` — per-endpoint dashboard.\n- Menu link in `oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml`.\n\n> **Collaborate with the developer on dashboard design.** Dashboard layout and metric selection are product decisions, not derivable from the MAL rules alone. Before authoring the JSON, check in with the developer on: which metric goes on which panel, which percentiles / aggregations to show, what's on the root list vs. per-service vs. per-instance vs. per-endpoint pages, what stays in the Overview tab vs. a sub-tab, widget sizes / ordering, and whether Trace / Log tabs belong at all (depends on whether the feature emits traces and logs). Then **manually set up the feature against a live OAP** — emit a few real OTLP payloads, open each dashboard in the UI, confirm every widget renders non-empty, and walk the drill-down from root → service → instance → endpoint. Only after that eyes-on pass is the dashboard ready to ship.\n\nDetails that always bite:\n\n- **Tab chrome eats ~4 rows.** Set outer `Tab.h = content_max_y + 4` or you get an inner scrollbar on top of the page scrollbar.\n- **Widget heights consistent.** Use `h=15` per row for `Line` / `Bar` widgets to match mesh / general peer templates. Don't mix h=12 / h=15 / h=24 across a feature's dashboards.\n- **List widgets (ServiceList / InstanceList / EndpointList)** render two pieces per cell:\n  - `expressions` → the aggregated summary number (right side). Required. Wrap raw metrics in `avg(...)` / `sum(...)`. Omitting it leaves the column empty.\n  - `subExpressions` → the sparkline trend (left side). Optional, falls back to `expressions` if absent.\n  - `metricConfig` pairs 1-to-1 with `expressions` (not `subExpressions`). Use for `label` (column header) and `unit`.\n  - Compound columns via MQE addition: `avg(metric_a + metric_b)` in `expressions`, `metric_a + metric_b` in `subExpressions`.\n  - Normal widgets with multiple expressions on one chart — always give a human label per expression in `metricConfig`, otherwise legend renders raw metric names.\n\n### 5.2 Hot-reload (edits to shipped templates take effect on restart)\n\n`UITemplateInitializer` normally uses `addIfNotExist` (keyed on JSON `id`), so on a plain restart your edits to shipped templates are ignored — the seeded copy in storage wins. For dev / extension work, set the environment variable and restart:\n\n```bash\nSW_UI_TEMPLATE_FORCE_RELOAD=true\n# then restart OAP — shipped templates overwrite the seeded copies via addOrReplace.\n# Hard-refresh the browser (Cmd+Shift+R) because the booster UI caches config in memory.\n```\n\nThis is a dev / extension knob only — read directly from the OS environment, not wired through `application.yml`. Unset or `false` preserves the default production behavior where operators' UI edits persist across restarts.\n\n### 5.3 New top-level menu group (e.g. \"Mobile\", \"Gateway\")\n\nNew **child entries** only need template JSON + `menu.yaml`. A new **top-level group** needs submodule work in `skywalking-ui` (git submodule at `skywalking-ui/`):\n\n1. `skywalking-ui/src/assets/icons/<name>.svg` — menu icon.\n2. `skywalking-ui/src/locales/lang/menus/{en,es,zh}.ts` — i18n labels (`<name>`, `<name>_desc`, `<name>_<child>`, `<name>_<child>_desc`). Missing i18n falls back to raw keys on screen.\n3. Commit those upstream in `apache/skywalking-booster-ui`, bump the submodule pointer here.\n4. Rebuild with `-Pui,dist` (not `-Pbackend,dist`) and `make docker.ui`, otherwise the new assets aren't in the image. See [package skill](../package/SKILL.md).\n\n---\n\n## 6. Docs set — everything that moves together\n\n| File | What to add |\n|---|---|\n| `docs/en/setup/backend/<feature>-monitoring.md` | User-facing setup guide. Required SDK / collector, required env vars, sample OTLP payload, metrics / log tags list. |\n| `docs/menu.yml` | Add the guide to the docs navigation tree. |\n| `docs/en/security/README.md` | **Client-side monitoring**: mobile / browser / mini-program endpoints accept data from the public internet without auth by default. Add a rate-limit + abuse-protection note. Agent-side monitoring over private VPC usually doesn't need this. |\n| `docs/en/changes/changes.md` | Per-release changelog. Add under `#### OAP Server` for features/bug-fixes, `#### UI` if the submodule bumps, `#### Documentation` if new guides. |\n| `docs/en/swip/SWIP-<N>.md` | Design doc for non-trivial features; keep reconciled with final code after review. |\n| `docs/en/swip/readme.md` | Move SWIP from \"Proposed\" to \"Accepted\" once merged. |\n\n---\n\n## 7. E2E test\n\nFolder: `test/e2e-v2/cases/<feature>/`. See [run-e2e skill](../run-e2e/SKILL.md) for harness basics (setup / trigger / verify phases, template functions, expected-file authoring via `contains`).\n\nFeature-specific extras:\n\n1. **Docker-compose**: extend `../../script/docker-compose/base-compose.yml` oap + storage. Set `SW_OTEL_RECEIVER_ENABLED_HANDLERS`, `SW_LOG_LAL_FILES`, and any feature-specific env. Don't list default rules here — rely on `application.yml` so the config-dump test stays consistent.\n2. **Register in CI**: add an entry under the e2e matrix in `.github/workflows/skywalking.yaml`.\n3. **OTLP data hygiene**:\n   - `spanId` is exactly 16 hex chars; `traceId` is exactly 32. Protobuf-JSON decodes them as base64; other lengths return HTTP 400.\n   - Setup-step curl loops must fail loudly: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e`. The pattern `for …; do curl && break || sleep 5; done` silently succeeds when every attempt connection-refused (the final `sleep` returns 0) because OAP takes ~50 s to start in CI.\n4. **Verify each case by hand before pushing** and use a published simulator image where the upstream SDK has one. See [`run-e2e` skill §6–§9](../run-e2e/SKILL.md) for verify-query triage, expected-file authoring, and the list of expected-file traps (YAML quoting on colons, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl loops that swallow failure).\n5. **Config dump**: when you modify any `application.yml` default (rule list, handler list, ...), also update `test/e2e-v2/cases/storage/expected/config-dump.yml`.\n\n---\n\n## 8. Pre-submit checklist\n\nRun in order; each has a dedicated skill:\n\n1. [`compile`](../compile/SKILL.md) — `./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall -Dmaven.test.skip -Dcheckstyle.skip -Dgpg.skip`. Javadoc `[ERROR] ... error:` must be fixed; `[ERROR] ... warning:` is noise.\n2. Tests: `./mvnw -pl oap-server/analyzer/log-analyzer -am test -Dtest=LALScriptExecutionTest` for LAL changes, equivalent for MAL, OAL unit tests where the rule lives.\n3. [`license`](../license/SKILL.md) — `license-eye header check` (Apache 2.0 headers). If the UI submodule bumped npm deps, also `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit the `LICENSE` diff.\n4. [`package`](../package/SKILL.md) — full `make docker` (or dist-rebuild + `make docker.oap` / `make docker.ui`). Don't trust `docker.oap` alone to pick up code changes; the dist tarball is a file-level prereq.\n5. [`run-e2e`](../run-e2e/SKILL.md) — run the new case end-to-end locally before pushing.\n6. [`gh-pull-request`](../gh-pull-request/SKILL.md) — commit and open PR with the template, attach UI screenshots.\n\n---\n\n## 9. Cross-cutting traps collected from real extensions\n\n| Trap | Symptom | Fix |\n|---|---|---|\n| Layer folder named with hyphens | `ui-initialized-templates/my-layer/*.json` on disk but dashboard empty | Folder must be `Layer.name().toLowerCase()` (underscores). `wechat_mini_program/`, not `wechat-mini-program/` |\n| Extending `CommonAnalysisListener` for a client-side layer | Added componentId → layer mapping \"to get topology for free\"; then `service_cpm` etc. don't populate anyway | Client-side (exit-only) layers don't need trace-analysis changes. Ship `Layer` + `component-libraries.yml` + MAL + LAL. See §2 header for when extending trace analysis is actually justified |\n| Dashboards wired to OAL-derived metrics on a client-side layer | Charts stay empty — `service_cpm` / `service_resp_time` / `endpoint_cpm` never populate | Those come from inbound `parseEntry`, which mini-program / browser / iOS don't emit. Use MAL metrics (`meter_<layer>_*`) instead |\n| LAL `#` comment in DSL body | OAP boot fails with `extraneous input 'metrics'` at the `#` line | LAL grammar accepts `//` and `/* */` only. No `#` comments |\n| LAL bare def-var value access — `tag 'x': myvar` | OAP boot fails with `Cannot resolve expression '[myvar]'` | `LALValueCodegen` only treats def-vars as valid when there's a method chain (`myvar?.foo`). Inline `sourceAttribute(...)` or `tag(...)` in the three value-access sites instead of storing to a def, or add a no-op chain like `myvar.toString()`. Conditions (`if (myvar == ...)`) go through a different codegen path and DO handle bare def refs |\n| MAL rule file with multi-doc YAML (`---` separator) | `MALFilterExecutionTest` fails with `expected a single document`; OAP would also fail at startup via `Rules.loadRules` (calls `Yaml.loadAs`, single-doc) | Split into one file per top-level `expSuffix` / `filter` / `metricPrefix` combo. Reference each file by name in `application.yml` (`lalFiles`, `malFiles`, `enabledOtelMetricsRules`) |\n| Histogram bucket bounds in ms with default `.histogram()` call | Percentile values come out 1000× inflated (`P50 = 100000` for a 100 ms request) | MAL's default `defaultHistogramBucketUnit = SECONDS` multiplies `le` by 1000 when converting to stored ms. For SDKs that emit ms bounds directly (mini-program, most in-house JS SDKs), use `.histogram(\"le\", TimeUnit.MILLISECONDS)`. `TimeUnit` is registered in `MALCodegenHelper.ENUM_FQCN` so rule YAML can reference it |\n| `.rate()` / `.increase()` on a DELTA counter / histogram `_count` | Compute delta-of-delta — nonsense values | OTLP DELTA means each flush is already the per-flush count. MAL's per-minute aggregator sums directly to requests-per-minute. For CUMULATIVE (Prometheus default), use `.rate(\"PT1M\")` for per-sec or `.increase(\"PT1M\")` for per-min; they are **not** equivalent (differ by 60×) |\n| MAL bucket-family name with `_histogram` suffix | Rule compiles, metric stays empty | `PrometheusMetricConverter` emits the bucket family as `<metric_name>` (no suffix), with `<metric_name>_count` and `<metric_name>_sum` as the companion families. Reference the bucket family by its bare name in the rule |\n| LAL `tag(\"key\") == null` check on a missing tag | Check never triggers — LAL's `tag()` returns empty string `\"\"` for absent tags, not `null`, so conditional abort is a no-op | Guard on both: `tag(\"key\") == null || tag(\"key\") == \"\"`. Same for `sourceAttribute(\"key\")` |\n| MAL rule file with `expSuffix: service(...)` mixing service-scoped and endpoint-scoped rules | Endpoint rules' metrics appear under the service entity (no endpoint_traffic rows created); `swctl endpoint ls` returns empty. `MetricConvert.formatExp:132` appends `(exp).expSuffix` to every rule, so `.endpoint(...)` in the rule's expression is followed by `.service(...)` and the service scope wins. | Follow the APISIX pattern — leave `expSuffix:` empty and explicitly chain `.service(...)` / `.endpoint(...)` on each rule. If every rule in a file is same-scope (e.g. all instance-scoped in `*-instance.yaml`), `expSuffix` is still fine |\n\nE2E-side authoring traps (log-content YAML quoting, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl swallowing failure) live in the [`run-e2e` skill §9](../run-e2e/SKILL.md).\n| Missing layer-root template | Menu item lands on empty \"no dashboard\" view | `Layer.vue:41-44` requires a template with `isRoot: true`; ship a `<layer>-root.json` (precedent: `ios/ios-root.json`) |\n| New layer not selectable in UI / swctl | `service ly <LAYER>` returns empty | Add the enum in `Layer.java` with a fresh id; see §0 |\n| Stale dist in image | `make docker.oap` succeeds but behavior is old | See [`package` skill](../package/SKILL.md); rebuild dist first, then image |\n| Shipped template edits don't render after restart | Edited JSON on disk, but OAP shows old copy | Set `SW_UI_TEMPLATE_FORCE_RELOAD=true` (environment variable, not `application.yml`) and restart; OAP writes `addOrReplace`. Unset in production so operator UI edits aren't clobbered. |\n| MAL service-scoped rule fragmented by instance | \"Overall app\" metric has N series per version | Keep `service_instance_id` out of service-scoped `.sum`/`.avg`. Put instance-scoped metrics in a separate `<feature>-instance.yaml` with `expSuffix: instance(...)` — see `otel-rules/ios/` pattern. |\n| Config dump drift | Storage e2e fails after `application.yml` edit | Mirror the change in `test/e2e-v2/cases/storage/expected/config-dump.yml` |\n| LAL `layer:auto` WARN drop | Every log silently dropped | Codegen must set `metadata.layer` alongside output builder's layer |\n| Histogram 1000× off | p50 = 2 000 000 ms | `defaultHistogramBucketUnit(MILLISECONDS)` on the `SampleFamily` |\n| Histogram shifted up | p50 = 5 000 on 500–2500 ms data | Emit per-bucket, not cumulative |\n| Empty list column | Dashboard instance table has no numbers | Add `expressions` (not just `subExpressions`) |\n| Double scroll | Inner tab scroll + page scroll | `Tab.h = content_max + 4` chrome buffer |\n| Raw metric name in legend | Multiple expressions on one widget | Add parallel `metricConfig[].label` entries |\n| `spanKind` mismatch | Span skipped silently | `OTLPSpanReaderImpl.spanKind()` returns enum `.name()` like `SPAN_KIND_CLIENT`, not `CLIENT` |\n| Negative latency | `*_sla` inflated | Clamp `rawLatency < 0 ? 0 : (int) min(Integer.MAX_VALUE, rawLatency)` |\n| SLA too optimistic | Client-side DNS / timeout / connection-refused counted as success | Require a real response code for success: `status = code > 0 && code < 400`. Do not treat missing (0) as success. |\n| Silent drop with no rules | `converters == null` NPE elsewhere | Init to empty list at declaration |\n| `+Inf` bucket surfaces as `9.2e18` | UI shows garbage when a sample lands in overflow | Use finite sentinel bucket (e.g. 30 s) |\n| OTel submodule bump misses UI image | New icons / i18n not visible | Rebuild with `-Pui,dist` + `make docker.ui` |\n| LICENSE drift after submodule bump | CI `Dependency licenses` fails | `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit |\n| Setup-step curl loop swallows failure | e2e passes metrics but fails log verify | `curl -f --retry-connrefused --retry-all-errors` + `set -e` |\n| Verify case flag mismatch | CI fails early on `swctl` flag unknown to pinned commit | Fire every verify command locally first; bump `SW_CTL_COMMIT` if needed |\n\n---\n\n## 10. Reference extensions (read their code)\n\nBrowse these when you need a template to copy from. Grouped by the kind of data source, because the extension points differ significantly:\n\n### SW native trace (language agents) — the \"General\" layer and friends\n\n`Layer.GENERAL` is the default layer for services reporting via the SkyWalking native trace protocol (Java / Python / Go / Node.js / .NET / PHP / Lua / Rust agents). It's the richest reference because the native pipeline is where the **full topology** lives:\n\n- **Service → service dependency** (topology): derived by `MultiScopesAnalysisListener` and `NetworkAddressAliasMappingListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`. Entry/exit spans produce `ServiceRelation`, `ServiceInstanceRelation` sources.\n- **Endpoint → endpoint dependency**: `RPCAnalysisListener` emits `EndpointRelation` sources from cross-process refs and exit spans. This is unique to the native protocol — OTLP spans don't preserve upstream entry-span metadata the same way, so OTLP-only layers generally lose endpoint-level topology.\n- OAL aggregation: `oap-server/server-starter/src/main/resources/oal/core.oal` — all `service_relation_*`, `service_instance_relation_*`, `endpoint_relation_*` metrics are already defined there; a new native-tracing layer just needs to produce the right sources with the right `Layer`.\n\nRead order: `AnalysisListenerFactory` SPI registration → `SegmentParserServiceImpl` fan-out → individual listeners → OAL rules.\n\n### Service-mesh (Istio / Envoy ALS) — topology from proxy-side signals\n\nNo agent in the service; Envoy emits HTTP/TCP access logs (ALS) and metrics. Service-to-service topology is reconstructed from ALS request/response pairs.\n\n- `oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/` — ALS receiver + `EnvoyAccessLogBuilder` persistence (LAL output type).\n- `otel-rules/istio-*`, `otel-rules/envoy/*` — MAL rules for proxy-emitted OTel metrics.\n- `lal/envoy-als.yaml` — LAL rule that extracts topology-generating fields from ALS.\n- Layer: `Layer.MESH`, plus `Layer.MESH_CP` (control plane) and `Layer.MESH_DP` (data plane) for split-plane topology.\n\nTopology parity with native traces: ALS gives you service-level relations; endpoint-level relations are approximate (proxy sees inbound + outbound, not the full call-chain that a native agent observes).\n\n### OTLP-only layers (OTel SDKs, iOS/Android, AI gateway, …)\n\nOTLP spans feed a `SpanListener`. Topology is **partial by construction** — an OTel Swift SDK on a mobile device only knows about its own outbound requests; there's no upstream caller. So these layers typically produce only `Service` / `ServiceInstance` / `Endpoint` sources (no `*Relation`) and the Topology view shows isolated islands, by design.\n\n- **iOS** (`Layer.IOS`, OTLP traces + MetricKit + LAL diagnostics): `oap-server/analyzer/ios-analyzer/` + `docs/en/swip/SWIP-11.md`.\n- **Envoy AI Gateway** (`Layer.ENVOY_AI_GATEWAY`, LLM / MCP, OTLP metrics + Zipkin sampled traces + LAL access-log sampling): `oap-server/analyzer/gen-ai-analyzer/`, `otel-rules/envoy-ai-gateway/*`, `lal/envoy-ai-gateway.yaml`.\n\n### OTel-metrics-only layers (scraped exporters)\n\nNo spans, no logs — just MAL rules converting OTel metric scrapes.\n\n- **MySQL / PostgreSQL**: `otel-rules/mysql/*`, `otel-rules/postgresql/*`.\n- **Kubernetes**: `otel-rules/k8s/*` — paired with `virtual-k8s-service` (`Layer.K8S_SERVICE`) sources for synthetic services.\n- **AWS**: `otel-rules/aws-eks/*`, `otel-rules/aws-s3/*`, `otel-rules/aws-dynamodb/*`.\n\n### SW native meter (custom protocol)\n\nNo OTel; the agent / plugin sends pre-built meter samples on the SW `MeterProtocol`.\n\n- **Nginx Lua** (`Layer.NGINX`): `meter-analyzer-config/nginx-service.yaml`, with the Lua agent in `apache/skywalking-nginx-lua`.\n- **APISIX** (`Layer.APISIX`): `meter-analyzer-config/apisix-service.yaml` + LAL trace extraction.\n\n### Which to copy from\n\n| You're adding | Closest reference |\n|---|---|\n| Language agent (new JVM / runtime) | The General / Java layer flow; extend `AnalysisListener` SPI. |\n| Service mesh / proxy | `mesh-*` + `envoy-*` rules; LAL on ALS. |\n| Mobile / browser / mini-program SDK (OTLP) | `ios-analyzer` (OTLP client-side). Expect no topology relations. |\n| Database / middleware (metrics only) | `mysql`, `postgresql` — pure MAL rules. |\n| Cloud service (no agent, just metrics) | `aws-eks` / `aws-s3`. |\n| Custom protocol / home-grown telemetry | `apisix` / `nginx` — SW native meter. |\n","frontmatter":{"name":"new-monitoring-feature","description":"Add a new monitoring target / layer to SkyWalking OAP. Orients you to the OAL / MAL / LAL / SpanListener / SegmentListener extension points, the UI template + submodule touchpoints, the docs set that must move together, and the cross-cutting traps that don't live in any one skill."},"isInternal":false,"tokens":9600,"sizeBytes":38835},{"name":"SKILL.md","path":".claude/skills/package/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/package/SKILL.md","title":"package","category":"anthropic-skill","format":"markdown","content":"---\nname: package\ndescription: Rebuild the SkyWalking distribution and OAP Docker image after source changes. Use before running e2e tests so the image reflects your code changes. Avoids the \"image looks updated but runtime has stale jars\" trap.\nargument-hint: \"[oap|all|dist-only]\"\n---\n\n# Package OAP Distribution & Docker Image\n\nRebuilding the Docker image after a source-code change has a **two-step dependency**: rebuild the dist tarball, then rebuild the image. Skipping the first step silently produces an image with stale jars — the Docker build still \"succeeds,\" the image has a fresh timestamp, but the embedded jar is the one from your previous build.\n\n## Why this matters\n\n`make docker.oap` in the root `Makefile` is defined as:\n\n```\ndocker.% push.docker.%: $(CONTEXT)/$(DIST) $(SW_ROOT)/docker/%/*\n    $(DOCKER_RULE)\n```\n\nIt depends on `$(CONTEXT)/$(DIST)` (the dist tarball) **as a file prerequisite**. If that tarball already exists, make does not regenerate it — it just copies whatever is on disk into `dist/docker_build/oap/` and runs `docker buildx build`. There is **no** dependency that rebuilds the tarball from source.\n\nOnly `make docker` triggers the full chain (`init → build.all → docker.all`). So:\n\n| Command | Rebuilds source? | Rebuilds tarball? | Rebuilds image? |\n|---------|:--:|:--:|:--:|\n| `./mvnw -pl <module> package` | only that module | no | no |\n| `./mvnw -pl apm-dist -am package -Pbackend,dist` | all backend deps + dist | yes | no |\n| `make docker.oap` | **no** | **no** (uses whatever's on disk) | yes |\n| `make docker` | yes (`build.all`) | yes | yes (`docker.all`) |\n\n## `flatten:flatten` — always run it before `package`/`install`\n\nSkyWalking's poms use `${revision}` as a placeholder version (e.g., `10.5.0-SNAPSHOT`). The `flatten-maven-plugin` resolves `${revision}` into concrete versions and writes a `.flattened-pom.xml`. Without it:\n\n- Installed artifacts carry the literal string `${revision}` in their coordinates and cannot be resolved as dependencies.\n- Downstream modules (including `apm-dist`) see an inconsistent dependency graph.\n- Symptoms are subtle: `-pl <module> -am` may succeed in isolation but fail or pull in stale transitive artifacts when the same module is consumed by another build invocation in the same session.\n\n**Always run `flatten:flatten` in the same goal chain as `package` or `install`.** The CI `dist-tar` job and the `compile` skill both do this. Example:\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -DskipTests\n```\n\nNot optional — treat it as part of `package`.\n\n## Pick the right command\n\n### Changed OAP source → want a new image\n\n```bash\n# Recommended: full chain\nmake docker\n```\n\nor, faster and equivalent for backend-only changes (skips UI):\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true\nmake docker.oap\n```\n\n**Do not** just run `make docker.oap` after a code edit — the image will look rebuilt but your change will not be in the jars.\n**Do not** skip `flatten:flatten` — see the section above.\n\n### Changed only the dist packaging (e.g., `apm-dist/`, log4j2.xml)\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests\nmake docker.oap\n```\n\n### Changed only Dockerfile / entrypoint (`docker/oap/*`)\n\nThe tarball is untouched, so `make docker.oap` alone is correct:\n\n```bash\nmake docker.oap\n```\n\n## Verify the fix reached the image\n\nDocker's \"image created\" timestamp updates even when the content is stale (because `--no-cache` is used). **Don't trust the timestamp.** Verify the jar contents directly:\n\n```bash\n# Copy the jar out of the container\ndocker cp <container-name>:/skywalking/oap-libs/<module>-<version>.jar /tmp/verify.jar\n\n# Extract the specific class\ncd /tmp && jar -xf verify.jar <path/to/YourClass.class>\n\n# Grep for a string your fix introduced (unique literal, method name, etc.)\ngrep -oa \"myFixSignature\" <path/to/YourClass.class>\n```\n\nIf grep finds nothing, the image does not contain your fix — you forgot to rebuild the dist. Re-run with the correct chain.\n\n## Common pitfalls\n\n- **`make docker.oap` after `./mvnw package`**: The module jar is fresh in `oap-server/.../target/`, but the dist tarball at `dist/apache-skywalking-apm-bin.tar.gz` is untouched. Image uses the old jar.\n- **Cancelling a `make docker` mid-flight**: leaves the dist half-rebuilt. Re-run `make docker` or delete `dist/` first.\n- **Stale buildx cache**: not usually the issue (the Makefile passes `--no-cache`), but if you suspect it, run `docker buildx prune`.\n- **Trusting image timestamps**: `docker images` shows when the image was *tagged*, not when its content actually changed. A no-op rebuild still updates the timestamp.\n- **Interactive buildx hangs**: if `docker buildx build` sits silent for minutes with no stdout, kill it and retry. The buildx container driver occasionally stalls; a restart is faster than waiting.\n\n## The golden rule\n\nAfter any Java source edit that affects code shipped in `oap-libs/`, run one of:\n\n1. `make docker` (full, safest)\n2. `./mvnw -pl apm-dist -am -o package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true && make docker.oap` (fast)\n\nThen **verify the fix is in the image's jar** before running e2e. Don't assume.\n","frontmatter":{"name":"package","description":"Rebuild the SkyWalking distribution and OAP Docker image after source changes. Use before running e2e tests so the image reflects your code changes. Avoids the \"image looks updated but runtime has stale jars\" trap.","argument-hint":"[oap|all|dist-only]"},"isInternal":false,"tokens":1374,"sizeBytes":5316},{"name":"SKILL.md","path":".claude/skills/run-e2e/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/run-e2e/SKILL.md","title":"run-e2e","category":"anthropic-skill","format":"markdown","content":"---\nname: run-e2e\ndescription: Run SkyWalking E2E tests locally\ndisable-model-invocation: true\nargument-hint: \"[test-case-path]\"\n---\n\n# Run SkyWalking E2E Test\n\nRun an E2E test case using `skywalking-infra-e2e`. The user provides a test case path (e.g., `simple/jdk`, `storage/banyandb`, `alarm`).\n\n## Prerequisites\n\nAll tools require **Go** installed. Check `.github/workflows/` for the exact `e2e` commit used in CI.\n\n### e2e CLI\n\nBuilt from [apache/skywalking-infra-e2e](https://github.com/apache/skywalking-infra-e2e), pinned by commit in CI:\n\n```bash\n# Install the pinned commit\ngo install github.com/apache/skywalking-infra-e2e/cmd/e2e@<commit-id>\n\n# Or clone and build locally (useful when debugging the e2e tool itself)\ngit clone https://github.com/apache/skywalking-infra-e2e.git\ncd skywalking-infra-e2e\ngit checkout <commit-id>\nmake build\n# binary is in bin/e2e — add to PATH or copy to $GOPATH/bin\n```\n\n### swctl, yq, and other tools\n\nE2E test cases run pre-install steps (see `setup.steps` in each `e2e.yaml`) that install tools into `/tmp/skywalking-infra-e2e/bin`. When running locally, you need these tools on your PATH.\n\n**swctl** — SkyWalking CLI, used in verify cases to query OAP's GraphQL API. Pinned at `SW_CTL_COMMIT` in `test/e2e-v2/script/env`:\n\n```bash\n# Option 1: Use the install script (same as CI)\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: Build from source\ngo install github.com/apache/skywalking-cli/cmd/swctl@<SW_CTL_COMMIT>\n```\n\n**yq** — YAML processor, used in verify cases:\n\n```bash\n# Option 1: Use the install script\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: brew install yq (macOS)\n```\n\n**Other tools** (only needed for specific test cases):\n\n| Tool | Install script | Used by |\n|------|---------------|---------|\n| `kubectl` | `install.sh kubectl` | Kubernetes-based tests |\n| `helm` | `install.sh helm` | Helm chart tests |\n| `istioctl` | `install.sh istioctl` | Istio/service mesh tests |\n| `etcdctl` | `install.sh etcdctl` | etcd cluster tests |\n\nAll install scripts are at `test/e2e-v2/script/prepare/setup-e2e-shell/`.\n\n## Steps\n\n### 1. Determine the test case\n\nResolve the user's argument to a full path under `test/e2e-v2/cases/`. If ambiguous, list matching directories and ask.\n\n```bash\nls test/e2e-v2/cases/<argument>/e2e.yaml\n```\n\n### 2. Check if rebuild is needed\n\nCompare source file timestamps against the last build:\n\n```bash\n# OAP server changes since last build\nfind oap-server apm-protocol -type f \\( \\\n  -name \"*.java\" -o -name \"*.yaml\" -o -name \"*.yml\" -o \\\n  -name \"*.json\" -o -name \"*.xml\" -o -name \"*.properties\" -o \\\n  -name \"*.proto\" \\\n\\) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null | head -5\n\n# Test service changes since last build\nfind test/e2e-v2/java-test-service -type f \\( \\\n  -name \"*.java\" -o -name \"*.xml\" -o -name \"*.yaml\" -o -name \"*.yml\" \\\n\\) -newer test/e2e-v2/java-test-service/e2e-service-provider/target/*.jar 2>/dev/null | head -5\n```\n\nIf files are found, warn the user and suggest rebuilding before running.\n\n### 3. Rebuild if needed (only with user confirmation)\n\n```bash\n# Rebuild OAP\n./mvnw clean flatten:flatten package -Pall -Dmaven.test.skip && make docker\n\n# Rebuild test services\n./mvnw -f test/e2e-v2/java-test-service/pom.xml clean flatten:flatten package\n```\n\n### 4. Run the E2E test\n\nSet required environment variables and run:\n\n```bash\nexport SW_AGENT_JDK_VERSION=8\ne2e run -c test/e2e-v2/cases/<case-path>/e2e.yaml\n```\n\n### 5. If the test fails\n\nDo NOT run cleanup immediately. Instead:\n\n1. Check container logs:\n   ```bash\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs oap\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs provider\n   ```\n\n2. Run verify separately (can retry after investigation):\n   ```bash\n   e2e verify -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n3. Only cleanup when done debugging:\n   ```bash\n   e2e cleanup -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n### 6. Manually fire each verify query (fast triage)\n\nThe `e2e verify` retry loop runs in sequence and stops at the first failing case, so a single bad query hides every case after it. When a verify fails, **run each verify case directly against the still-running OAP** before editing anything — you'll see the real error (bad flag, missing data, wrong expected), not the progress spinner. This is also the right way to author new verify cases: craft the query against live OAP, confirm the actual YAML, then write the expected file.\n\n```bash\n# Find the host-side port that infra-e2e bound to OAP's container port 12800.\n# (Each run picks a new random port; the trigger log prints it too.)\ndocker ps --filter \"name=skywalking_e2e-oap\" --format \"{{.Ports}}\" \\\n  | grep -oE \"[0-9]+->12800\" | head -1\n# => e.g. 56381->12800\n\nURL=http://localhost:56381/graphql\nSWCTL=/tmp/skywalking-infra-e2e/bin/swctl\n\n# Copy the query from e2e.yaml verbatim, then substitute ${oap_host} → localhost\n# and ${oap_12800} → the port you just found:\n$SWCTL --display yaml --base-url=$URL service ly IOS\n$SWCTL --display yaml --base-url=$URL logs list --service-name=MyiOSApp\n$SWCTL --display yaml --base-url=$URL metrics exec --expression=service_cpm --service-name=MyiOSApp\n```\n\nWhen a `swctl` subcommand rejects a flag (`Incorrect Usage: flag provided but not defined: -layer`), the e2e config is using syntax the pinned `swctl` commit doesn't support. Find the right syntax with `swctl <cmd> --help` and update the e2e config. Common cases encountered:\n\n| Broken flag/form | Working form |\n|---|---|\n| `service ls --layer IOS` | `service ly IOS` |\n| `metrics exec ... --is-normal=true` | drop `--is-normal` (default behavior) |\n\nFor queries that *don't* use `swctl` (raw `curl` against `/loki/...`, Zipkin, PromQL), hit the matching exposed port:\n\n```bash\ncurl \"http://localhost:$(docker ps --filter name=skywalking_e2e-oap --format '{{.Ports}}' | grep -oE '[0-9]+->3100' | head -1 | cut -d'-' -f1)/loki/api/v1/labels\"\n```\n\n### 7. UI template changes require a fresh DB\n\n`UITemplateInitializer.initTemplate()` (in `oap-server/server-core`) calls `uiTemplateManagementService.addIfNotExist(setting)` — keyed by the `id` field in each `ui-initialized-templates/**/*.json`. Same ID → skipped. So edits to an *existing* template JSON (adding widgets, relabeling, changing expressions) will **not** be applied on an already-initialized OAP, even after a container restart, because the old copy still lives in storage.\n\nTo pick up dashboard JSON changes:\n\n```bash\n# Remove both containers — BanyanDB stores state inside the container FS in the\n# e2e compose (no named volume), so removing the container wipes state cleanly.\ndocker rm -f skywalking_e2e-oap-1 skywalking_e2e-banyandb-1\n\n# For compose setups that use a named volume, also:\n# docker volume rm <volume-name>\n\n# Then re-run — OAP sees empty storage, loads the new template JSON.\ne2e run -c test/e2e-v2/cases/<case>/e2e.yaml\n```\n\nSymptom to watch for: you edit the JSON, rebuild, redeploy — dashboard in the UI still shows the pre-edit layout. That's not a caching bug; that's `addIfNotExist` doing exactly what its name says.\n\n### 8. Author the expected YAML from live output\n\nFor a new verify case, the workflow is:\n\n1. Fire the query manually (see step 6) and capture the YAML.\n2. Pick which fields are *meaningful domain values* (must match exactly) vs *dynamic runtime values* (`notEmpty` / `gt` / `ge`). See `test/e2e-v2/CLAUDE.md` for the decision guide.\n3. Write the expected file. If the response is a list, wrap the items in `{{- contains . }} ... {{- end }}` so ordering and extra actual items don't fail the match.\n4. Re-run `e2e verify` alone (the containers are still up from the previous run); iterate on the expected file without rebuilding.\n\n### 9. Expected-file authoring traps\n\nThese burn CI cycles and pass locally. Each was learned the hard way.\n\n**Unquoted `content: {{ notEmpty .content }}` with `:` inside the value.** Sim-generated or real log content routinely includes colons (`POST https://api.example.com/cart failed: 500`, `HTTP/1.1 500: Internal Server Error`). Without quoting, the template renders to invalid YAML (snakeyaml parses `failed:` as a nested key) and the *whole log entry* marshals to `nil`. Symptom: diff shows `- nil` at *every* position in the expected logs list vs real maps in actual. Fix: wrap in single quotes — `content: '{{ notEmpty .content }}'`. Single-quoted YAML preserves `:` in the scalar; only fails on embedded `'`. Double quotes also work unless the content has `\"`.\n\n**Nested `contains` with multiple per-element pattern assertions against a varied stream.** The template renders the block body once per actual element; when the outer block body has *multiple* inner `contains` patterns asserting specific tag key/value pairs, and only *some* actuals satisfy all the inner patterns, `go-cmp` with `contains` can end up comparing `[rendered_for_A0, nil, nil, ...]` vs `[A0, A1, A2, ...]` and fail despite `contains` being permissive on extras. Specifically: outer `contains .logs` with a single log pattern + inner `contains .tags` asserting two distinct key/value pairs. On a simulator emitting heterogeneous errors (js + promise + ajax + pageNotFound), only a subset satisfy the inner assertion. Passes locally with 1–2 logs, fails in CI with 6+.\n- Keep the outer `contains` body lenient: field-shape checks (`notEmpty`, `gt`), one discriminator tag that *every* element in the stream carries.\n- Cover per-category assertions via separate labeled-metric verify cases, not inside the nested template.\n- Rule of thumb: \"at least one log exists with the right layer routing\" inside the logs expected; per-category coverage via `meter_*_count{label=X}` verify cases.\n\n**Hand-crafted OTLP curl payloads drift from real SDK output.** When the upstream SDK ships a published simulator image (mini-program-monitor's `sim-wechat` / `sim-alipay`, browser-client-js sim, etc.), prefer driving the e2e with that image in `MODE=timed` with a bounded `DURATION_MS` over hand-rolling the OTLP JSON. Hand-crafted payloads miss real-world shape issues: delta-vs-cumulative temporality, label-cardinality surprises, stacktrace formatting variance, attribute key names that changed between SDK versions. Pin to a released tag (`v0.4.0`), not `:latest` or HEAD SHA — reproducibility.\n\n**`timeUnixNano: \"0\"` in an OTLP metric datapoint.** The receiver propagates this into MAL's bucket computation and the metric lands in the 1970 time bucket — `swctl metrics exec` over the \"last N minutes\" window won't find it. Either use `$(date +%s)000000000` at setup time or omit the field if the receiver accepts \"now\" as default.\n\n**Setup-step curl loop with `|| sleep` pattern.** The shell line `for ... do curl && break || sleep 5; done` exits 0 when every attempt connection-refused because the final `sleep 5` returns 0. OAP takes ~50 s to start in CI, so all attempts fail before OAP is ready, and the setup step silently succeeds with zero traffic ingested. Fix: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e` at step top.\n\n**`swctl` flag rejected.** If a verify case uses a flag the pinned `swctl` commit doesn't support (`service ls --layer` vs `service ly`), the whole case fails 20× before CI gives up. Fire each verify query by hand once before pushing (step 6 above).\n\n**Published image cache miss in CI.** `docker compose pull` sometimes hits rate limits or unreachable registries; the test spins until timeout with \"dependency failed to start\". Look at the CI log for `Error response from daemon: pull access denied` or `manifest unknown`. If you see that, pin a different image tag that's definitely published (check `docker manifest inspect <tag>` locally), not a floating one.\n\n## Common test cases\n\n| Shorthand | Path |\n|-----------|------|\n| `simple/jdk` | `test/e2e-v2/cases/simple/jdk/` |\n| `storage/banyandb` | `test/e2e-v2/cases/storage/banyandb/` |\n| `storage/elasticsearch` | `test/e2e-v2/cases/storage/elasticsearch/` |\n| `alarm` | `test/e2e-v2/cases/alarm/` |\n| `log` | `test/e2e-v2/cases/log/` |\n| `profiling/trace` | `test/e2e-v2/cases/profiling/trace/` |\n","frontmatter":{"name":"run-e2e","description":"Run SkyWalking E2E tests locally","disable-model-invocation":true,"argument-hint":"[test-case-path]","disableModelInvocation":true},"isInternal":false,"tokens":3365,"sizeBytes":12362},{"name":"SKILL.md","path":".claude/skills/test/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/.claude/skills/test/SKILL.md","title":"test","category":"anthropic-skill","format":"markdown","content":"---\nname: test\ndescription: Run unit tests, integration tests, or slow integration tests matching CI. Use to validate changes before submitting a PR.\nargument-hint: \"[unit|integration|slow|module-name]\"\n---\n\n# Tests\n\nRun tests matching CI configuration.\n\n## Commands by argument\n\n### `unit` or no argument — unit tests\n\n```bash\n./mvnw clean test -q -B -D\"checkstyle.skip\"\n```\n\nCI runs this on:\n- JDK 11: ubuntu, macOS, Windows\n- JDK 17, 21, 25: ubuntu only\n\n### `integration` — integration tests (excludes slow)\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow\n```\n\nCI runs this on JDK 11, 17, 21, 25 (ubuntu only).\n\n### `slow` — slow integration tests\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true \\\n  -Dit.test=org.apache.skywalking.library.elasticsearch.ElasticSearchIT \\\n  -Dfailsafe.failIfNoSpecifiedTests=false\n```\n\nCI runs on JDK 11 (ubuntu only). Currently only ElasticSearch/OpenSearch IT is in the slow matrix.\n\n### Module name — single module tests\n\n```bash\n# Unit tests for a specific module\n./mvnw clean test -pl oap-server/analyzer/<module-name> -D\"checkstyle.skip\"\n\n# Integration tests for a specific module\n./mvnw -B clean integration-test -pl oap-server/analyzer/<module-name> -Dcheckstyle.skip -DskipUTs=true\n```\n\n## Test frameworks\n\n- JUnit 5 (`org.junit.jupiter`)\n- Mockito for mocking\n- AssertJ for assertions\n- PowerMock for reflection utilities\n\n## Test naming conventions\n\n| Type | Pattern | Maven phase |\n|------|---------|-------------|\n| Unit tests | `*Test.java` | `test` |\n| Integration tests | `IT*.java` or `*IT.java` | `integration-test` |\n\n## Slow test tagging\n\nTests tagged with `@Tag(\"slow\")` are excluded from the normal integration-test job and run separately in the slow-integration-test job. Use this tag for tests that spin up external services (Elasticsearch, etc.) and take significant time.\n\n## CI retry behavior\n\nAll three CI jobs retry on failure (run the same command twice with `||`). This handles flaky tests but masks intermittent issues — if a test fails locally, investigate rather than relying on retries.\n\n## CI reference\n\nCI workflow: `.github/workflows/skywalking.yaml`\n\n| Job | JDK | OS | Timeout |\n|-----|-----|----|---------|\n| `unit-test` | 11 (3 OS) + 17, 21, 25 (ubuntu) | ubuntu, macOS, Windows | 30 min |\n| `integration-test` | 11, 17, 21, 25 | ubuntu | 60 min |\n| `slow-integration-test` | 11 | ubuntu | 60 min |\n","frontmatter":{"name":"test","description":"Run unit tests, integration tests, or slow integration tests matching CI. Use to validate changes before submitting a PR.","argument-hint":"[unit|integration|slow|module-name]"},"isInternal":false,"tokens":653,"sizeBytes":2464},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/apache/skywalking/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md - AI Assistant Guide for Apache SkyWalking\n\nThis file provides guidance for AI assistants working with the Apache SkyWalking codebase.\n\n## Project Overview\n\nApache SkyWalking is an open-source APM (Application Performance Monitoring) system designed for microservices, cloud-native, and container-based architectures. It provides distributed tracing, service mesh telemetry analysis, metrics aggregation, alerting, and observability capabilities.\n\n## Repository Structure\n\n```\nskywalking/\n├── oap-server/                    # OAP (Observability Analysis Platform) backend server\n│   ├── server-core/               # Core module with fundamental services\n│   ├── server-library/            # Shared libraries (module system, util, etc.)\n│   ├── server-receiver-plugin/    # Data receivers (gRPC, HTTP, Kafka, etc.)\n│   ├── server-storage-plugin/     # Storage implementations (BanyanDB, Elasticsearch, etc.)\n│   ├── server-cluster-plugin/     # Cluster coordination (Zookeeper, K8s, etc.)\n│   ├── server-query-plugin/       # Query interfaces (GraphQL)\n│   ├── server-alarm-plugin/       # Alerting system\n│   ├── server-fetcher-plugin/     # Data fetchers\n│   ├── server-configuration/      # Dynamic configuration providers\n│   ├── oal-grammar/               # OAL (Observability Analysis Language) grammar\n│   ├── oal-rt/                    # OAL runtime\n│   ├── mqe-grammar/               # MQE (Metrics Query Engine) grammar\n│   ├── mqe-rt/                    # MQE runtime\n│   ├── server-testing/             # Shared test utilities (DSL test framework)\n│   ├── analyzer/                  # Log and trace analyzers\n│   ├── ai-pipeline/               # AI/ML pipeline components\n│   ├── exporter/                  # Data export functionality\n│   └── server-tools/              # Standalone tools (profile exporter) with mock providers\n├── apm-protocol/                  # Protocol definitions (submodule)\n│   └── apm-network/               # gRPC/Protobuf network protocols\n├── skywalking-ui/                 # Web UI (submodule - skywalking-booster-ui)\n├── apm-webapp/                    # Web application packaging\n├── apm-dist/                      # Distribution packaging\n├── docs/                          # Documentation\n├── docker/                        # Docker build files\n├── test/                          # E2E and integration tests\n└── tools/                         # Development tools\n```\n\n## Architecture & Key Concepts\n\n### Module System\nSkyWalking uses a custom module/provider architecture based on Java SPI:\n\n- **ModuleDefine**: Declares a module and its required services\n- **ModuleProvider**: Implements a module with specific technology/approach\n- **Service**: Interface that modules expose to other modules\n\nKey pattern:\n```java\npublic class XxxModule extends ModuleDefine {\n    public Class[] services() {\n        return new Class[] { XxxService.class };\n    }\n}\n\npublic class XxxModuleProvider extends ModuleProvider {\n    public void prepare() { /* initialize */ }\n    public void start() { /* start services */ }\n}\n```\n\n### Core Concepts\n- **OAL (Observability Analysis Language)**: DSL for defining metrics aggregation rules\n- **MQE (Metrics Query Engine)**: DSL for querying metrics\n- **LAL (Log Analysis Language)**: DSL for log processing\n- **MAL (Meter Analysis Language)**: DSL for meter data processing\n- **Source/Scope**: Data model definitions for telemetry data\n- **Stream Processing**: Metrics, Records, and TopN processing pipelines\n\n### Data Flow\n1. Agents/Collectors send data via gRPC/HTTP/Kafka\n2. Receiver plugins parse and validate data\n3. Analysis engine processes data using OAL/LAL/MAL\n4. Storage plugins persist aggregated data\n5. Query plugins serve data to UI/API\n\n## Code Style & Conventions\n\n### Checkstyle Rules (enforced via `apm-checkstyle/checkStyle.xml`)\n\n**Prohibited patterns:**\n- No `System.out.println` - use proper logging (SLF4J)\n- No `@author` tags - ASF projects don't use author annotations\n- No Chinese characters in source files\n- No tab characters (use 4 spaces)\n- No star imports (`import xxx.*`)\n- No unused or redundant imports\n- No empty statements (standalone `;`)\n- No fully-qualified class names inline in code — always add an `import` statement and\n  use the short name. Acceptable exceptions: (a) two classes with the same simple name\n  would collide if both imported, (b) the class appears exactly once in a Javadoc\n  `{@link}` where the short name would be ambiguous to the reader, (c) the name is inside\n  GENERATED source — a codegen string literal or a `.ftl` template. Field declarations,\n  method signatures, local variables, and generic type arguments should always use the\n  imported short name — `private RemoteClientManager rcm;`, not `private\n  org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager rcm;`.\n  - Exception (c) is a Javassist constraint, not a preference: its compiler has no\n    `import` statement and resolves simple names only against `java.lang`\n    (`ClassPool.importedPackages`; widening it needs `importPackage`, which is pool-wide\n    mutable state and is never called here). So generated source must fully qualify\n    everything, `java.util.Map` included. Shortening an FQCN in a codegen string or a\n    template compiles as Java and then fails at runtime — audit real Java only.\n- No one-line delegate methods. A wrapper whose only body is a single forwarding call\n  to another class (`return Other.foo(a, b);`) adds a hop without value. Inline the\n  call at the use site, or call the underlying object directly (including via method\n  reference: `obj::foo` instead of `this::wrapperOfFoo`).\n\n**Required patterns:**\n- `@Override` annotation required for overridden methods\n- `equals()` and `hashCode()` must be overridden together\n- Javadoc `@param`, `@return`, `@throws` must have descriptions\n- Long constants must use uppercase `L` (e.g., `100L` not `100l`)\n- `default` case must come last in switch statements\n- One statement per line\n\n**Naming conventions:**\n- Constants/static variables: `UPPER_CASE_WITH_UNDERSCORES`\n- Type parameters: `UPPER_CASE` (e.g., `TYPE`, `KEY`, `VALUE`)\n- Package names: `org.apache.skywalking.*` or `test.apache.skywalking.*`\n- Type names: `PascalCase` or `UPPER_CASE_WITH_UNDERSCORES`\n- Local variables/parameters/members: `camelCase`\n- **Function-oriented naming, not abstract metaphor**: classes and methods are named for\n  what they do, not for an abstract concept. Prefer concrete verbs (`load`, `apply`,\n  `unregister`, `compile`, `verify`, `commit`, `rollback`) over metaphorical ones\n  (`seed`, `hydrate`, `bootstrap`, `prime`). Class names follow the same rule —\n  `StaticRuleLoader` (loads static rules), not `StaticBundleSeeder`; `DSLSyncTimer` (syncs\n  DB → state on a timer), not `TickRunner`. If you can't name a method without reaching\n  for a metaphor, the method is probably doing too much; split it.\n\n**File limits:**\n- Max file length: 3000 lines\n\n**Whitespace:**\n- Whitespace required after commas, semicolons, type casts\n- Whitespace required around operators\n- No multiple consecutive blank lines\n- Empty line separators between class members (fields can be grouped)\n\n### Code Style (via `codeStyle.xml` for IntelliJ IDEA)\n\n**Indentation:**\n- 4-space indentation\n- 4-space continuation indent\n\n**Imports:**\n- No star imports (threshold set to 999)\n- Import order: regular imports, blank line, static imports\n\n**Formatting:**\n- `while` in do-while on new line\n- Align multiline chained method calls\n- Align multiline parameters in calls\n- Array initializer braces on new lines\n- Wrap long method call chains\n\n**General:**\n- Use `final` for local variables and parameters\n- Use Lombok annotations (`@Getter`, `@Setter`, `@Builder`, `@Data`, `@Slf4j`, etc.)\n- Follow existing patterns in similar files\n\n**Comments:**\n- Don't explain what the code already says. Comment only when reading the code is\n  not enough — invariants that aren't visible in this method, cross-cutting\n  ordering, why a non-obvious branch exists, why a seemingly redundant call is\n  required. If the code is clear, no comment.\n- Connect logic that lives in different places. A check here that exists only\n  because of a guarantee made elsewhere deserves a one-line pointer to that\n  guarantee. A workaround for a known bug elsewhere deserves a link.\n- Trim restated javadoc. Method-level docs should describe the contract, not\n  re-narrate the body. Multi-paragraph rationale belongs in the design doc, not\n  the source.\n\n**Tests:**\n- Always use JUnit assertions (`assertTrue`, `assertEquals`, `assertThrows`, …).\n  Never write bare Java `assert` statements in tests. Although surefire defaults\n  `enableAssertions=true` so bare `assert` does fire in this project's tests,\n  JUnit assertions are the convention everywhere else in the codebase, survive\n  any test-runner configuration, and produce a structured failure that names\n  the actual value instead of a context-free `AssertionError`.\n\n### License Header\nJava, XML, and YAML/YML files must include the Apache 2.0 license header (see `HEADER` file).\nJSON and Markdown files are excluded (JSON doesn't support comments, see `.licenserc.yaml`).\n\n### JDK 11 Compatibility\n\nAll code must be compatible with JDK 11 (LTS). The project supports JDK 11, 17, and 21.\n\n**Prohibited Java features (post-JDK 11):**\n\n| Feature | JDK Version | Use Instead |\n|---------|-------------|-------------|\n| Switch expressions (`->`) | 14+ | Traditional `switch` with `case:` and `break` |\n| `Stream.toList()` | 16+ | `.collect(Collectors.toList())` |\n| Text blocks (`\"\"\"...\"\"\"`) | 15+ | String concatenation or `+` |\n| Records | 14+ | Regular classes with Lombok `@Data` |\n| Pattern matching for `instanceof` | 14+ | Traditional cast after `instanceof` |\n| Sealed classes/interfaces | 15+ | Regular classes/interfaces |\n\n**Allowed Java features (JDK 11 compatible):**\n- `List.of()`, `Set.of()`, `Map.of()` - Immutable collections (Java 9+)\n- `Optional` methods - `orElseThrow()`, `ifPresentOrElse()` (Java 9+)\n- Lambda expressions and method references (Java 8+)\n- Stream API (Java 8+)\n- Lombok annotations (`@Getter`, `@Builder`, `@Data`, `@Slf4j`)\n\n**Verification commands:**\n```bash\n# Check for switch expressions (should return no matches)\ngrep -r \"switch.*->\" src/ --include=\"*.java\"\n\n# Check for Stream.toList() (should return no matches)\ngrep -r \"\\.toList()\" src/ --include=\"*.java\"\n\n# Check for text blocks (should return no matches)\ngrep -r '\"\"\"' src/ --include=\"*.java\"\n```\n\n## Git Submodules\n\nThe project uses submodules for protocol definitions and UI:\n- `apm-protocol/apm-network/src/main/proto` - skywalking-data-collect-protocol\n- `oap-server/server-query-plugin/.../query-protocol` - skywalking-query-protocol\n- `skywalking-ui` - skywalking-booster-ui\n- `oap-server/server-library/library-banyandb-client/src/main/proto` - banyandb-client-proto\n\nAlways use `--recurse-submodules` when cloning or update submodules manually.\n\n## Key Files for Understanding the Codebase\n\n- `oap-server/server-core/src/main/java/.../CoreModule.java` - Core module definition\n- `oap-server/server-library/library-module/src/main/java/.../ModuleDefine.java` - Module system base\n- `oap-server/oal-grammar/src/main/antlr4/.../OALParser.g4` - OAL grammar definition\n- `oap-server/server-starter/` - Application entry point\n- `docs/en/concepts-and-designs/` - Architecture documentation\n\n## Common Development Tasks\n\n### Adding a New Receiver Plugin\n1. Create module in `server-receiver-plugin/`\n2. Implement `ModuleDefine` and `ModuleProvider`\n3. Register via SPI in `META-INF/services/`\n4. Add configuration to `application.yml`\n\n### Adding a New Storage Plugin\n1. Create module in `server-storage-plugin/`\n2. Implement storage DAOs for each data type\n3. Follow existing plugin patterns (e.g., BanyanDB, elasticsearch)\n\n### Modifying OAL Metrics\n1. Edit `.oal` files in `oap-server/server-starter/src/main/resources/oal/`\n2. Regenerate by building the project\n3. Update storage schema if needed\n\n### MAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `otel-rules/` - OpenTelemetry metrics (Prometheus, etc.)\n- `meter-analyzer-config/` - SkyWalking native meter protocol\n\n### LAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `lal/` - Log processing rules\n- `log-mal-rules/` - Metrics extracted from logs\n\n## Documentation (in `docs/en/`, structure defined in `docs/menu.yml`)\n\n- `concepts-and-designs/` - Architecture and core concepts (OAL, MAL, LAL, profiling)\n- `setup/` - Installation and configuration guides\n- `api/` - Telemetry and query protocol documentation\n- `guides/` - Contributing guides, build instructions, testing\n- `changes/changes.md` - Changelog (update when making changes). Keep each bullet's **prose on one line**\n  — don't hard-wrap a sentence across source lines (nested sub-bullets `  - …` are fine; they're real list\n  items). The changelog becomes the GitHub release body, which GitHub renders as GFM where a single\n  newline continuing a bullet becomes a `<br>`, so prose-wrapped bullets show jagged mid-sentence breaks on\n  the release page (verified on v10.4.0; the docs website reflows and hides the problem). See\n  `guides/How-to-release.md`. Append a new entry at the **end** of its section, never at the top —\n  the section reads in the order the changes landed.\n- `swip/` - SkyWalking Improvement Proposals\n\n### SWIP vs. operator docs\n\nThe two kinds of document answer different questions and have different lifetimes:\n\n- **`docs/en/swip/SWIP-N.md` records the design discussion up to first acceptance.** It is a\n  historical record of what was proposed and agreed, not a description of current behaviour.\n- **The operator / end-user docs describe how the system behaves today.** They are the only\n  place a reader should be sent for current truth.\n\nWhen the implementation of a SWIP lands, sync the SWIP once to what was actually built — every\nproposed change written in implemented/past tense, with **no `TODO` / \"future work\" /\n\"open dependency\" / \"lands later\" / \"empty until X\" language for anything in the SWIP's\nscope**. **Once that SWIP ships as part of a release, it is frozen and must not be edited again.**\n\nFrozen means frozen even when the SWIP's text becomes factually stale. A later refactor that\nrenames a package, moves a class, deletes a constant or changes a class-naming scheme **does not**\nlicense updating the SWIP to match — not to keep code references accurate, not for internal\nconsistency, and not because the same PR already touched it. Ask instead: *would an end user read\nthis to learn how the system works today?* If yes it is an operator doc and must be updated; if it\nis a SWIP the answer is no, and the fix belongs in the operator doc plus `changes.md`.\n\nAfter that first sync, further metrics, dashboards, and incremental enhancements go into the\n**operator doc**, not the SWIP. Only genuinely small, optional follow-ups may stay under a SWIP's\n\"future work\" — never a big change.\n\n- BanyanDB self-observability: the SWIP is `docs/en/swip/SWIP-15.md`; the living operator\n  catalog is `docs/en/banyandb/dashboards-banyandb.md` (menu: \"BanyanDB self observability\n  dashboard\"). New `meter_banyandb_*` rules → update the operator doc; leave the SWIP alone.\n\n### Doc cross-references\n\nReference another section with a **Markdown anchor link** `[section title](#heading-slug)`,\nnever the `§` symbol or a bare \"section N\". Verify every anchor resolves to a real heading\n(GitHub slug rules: lowercase, spaces → `-`, punctuation dropped) — a broken `[](#…)`\nsilently renders as plain text on the website.\n\n## Handling Existing Tests\n\nIf existing test code (UT, IT, or E2E) fails to pass, try to fix it. If any test cases need to be modified, or test code needs to be deleted, the change must be double-checked by a human before it is applied — surface the proposed modification or deletion and the reasoning, and wait for explicit approval rather than changing or removing the test yourself.\n\n## Submitting Pull Requests\n\nUse the `/gh-pull-request` skill for committing and pushing to a PR branch. It runs pre-flight checks (compile, checkstyle, license headers) before every push, and creates the PR if one doesn't exist yet.\n\n## GitHub Actions Allow List\n\nApache enforces an allow list for third-party GitHub Actions. All third-party actions must be pinned to an approved SHA from:\nhttps://github.com/apache/infrastructure-actions/blob/main/approved_patterns.yml\n\nIf a PR is blocked by \"action is not allowed\" errors, check the approved list and update `.github/workflows/` files to use the approved SHA pin instead of a version tag.\n\nActions owned by `actions/*` (GitHub), `github/*`, and `apache/*` are always allowed (enterprise-owned).\n\n## Tips for AI Assistants\n\n1. **Always check submodules**: Protocol changes may require submodule updates\n2. **Generate sources first**: Run `mvnw compile` before analyzing generated code\n3. **Install package**: Use `mvnw flatten:flatten install` to build the precompiler and export generated classes before running tests. ref to [compile skill doc](.claude/skills/compile/SKILL.md)\n3. **Full rebuild on cross-module changes**: If you changed more than two modules or pulled/rebased code from git remote, run `mvnw clean install` (or `mvnw clean package`) on the **whole project** rather than picking individual modules with `-pl`. Incremental `-pl ... -am` builds can leave stale jars in `.m2` or `oap-libs/` when jar sizes don't change but content does, causing hard-to-debug runtime issues.\n3. **Respect checkstyle**: No System.out, no @author, no Chinese characters\n4. **Follow module patterns**: Use existing modules as templates\n5. **Check multiple storage implementations**: Logic may vary by storage type\n6. **OAL generates code**: Don't manually edit generated metrics classes\n7. **Use Lombok**: Prefer annotations over boilerplate code\n8. **Test both unit and integration**: Different test patterns for different scopes\n9. **Documentation is rendered via markdown**: When reviewing docs, consider how they will be rendered by a markdown engine\n10. **Relative paths in docs are valid**: Relative file paths (e.g., `../../../oap-server/...`) in documentation work both in the repo and on the documentation website, supported by website build tooling\n11. **Module service registration**: When adding a service to `CoreModule.services()`, update ALL `CoreModuleProvider` implementations — not just the main one. Search with `grep -rn \"extends CoreModuleProvider\" oap-server/ --include=\"*.java\"`. The `MockCoreModuleProvider` in `server-tools/profile-exporter/` also needs it, or the profile exporter e2e test will fail at startup.\n12. **Multiple OAP packagings**: The OAP server is not only the main `server-starter`. The `server-tools/` directory contains standalone tools (e.g., profile exporter) that boot with mock module providers and a subset of modules. Changes to core module contracts (services, required modules) must be reflected in these tools too.\n13. **`moduleManager.find(X.NAME)` requires `X.NAME` in `requiredModules()`**: every call to `moduleManager.find(SomeModule.NAME)` (direct or through a helper) must have `SomeModule.NAME` in the provider's `requiredModules()` array. Missing declarations cause runtime exceptions the first time the code path fires — not at module boot. Wrapping the call in `try { ... } catch (Throwable)` is NOT a substitute; declare the module and keep the try/catch only for defensive handling of transient provider outages. When auditing a branch, grep for `moduleManager.find(` across the touched module and verify each target name appears in `requiredModules()`. Example modules that frequently catch teams out: `AlarmModule` (used by alarm-kernel reset), `LogAnalyzerModule` (used by LAL factory lookup).\n14. **Don't look up `ClusterModule` services directly**: the `ClusterModule` (ZooKeeper / K8s / Nacos coordination) exposes `ClusterRegister` / `ClusterNodesQuery` / `ClusterCoordinator`. Most receiver / analyzer modules don't declare `ClusterModule` in `requiredModules()`, so calling `moduleManager.find(ClusterModule.NAME)` will throw at runtime. Instead, go through `CoreModule`'s `RemoteClientManager` service — it's already populated by the cluster module and exposes the peer list every OAP needs. If a module genuinely needs cluster-coordinator primitives, declare `ClusterModule.NAME` in `requiredModules()` explicitly.\n15. **No `ThreadLocal` side-channels to hijack downstream behaviour**: routing a caller's intent through a `ThreadLocal` that downstream code reads (e.g., `if (PeerMode.isActive()) skipSomething()`) is almost always the wrong answer — it creates invisible coupling between far-apart code paths, leaks across async hand-offs (executors, gRPC threads, Armeria event loops), and makes the behaviour impossible to understand from a method signature. The correct fix is almost always to **extend the interface** — add a parameter, a new method, a new mode enum that appears in the signature. Rare exceptions: propagating OpenTelemetry context where the whole industry has standardised on `ThreadLocal`, or security principals enforced by a framework. In all other cases, prefer an explicit API extension, even if it costs more lines.\n16. **BanyanDB schema-visibility: fence on `mod_revision`, do NOT poll metadata**: schema no longer lives in etcd — etcd was removed. It is now stored as property documents in BanyanDB's own `_schema` store and propagated to each data node's in-memory cache **asynchronously** (a `SchemaUpdateService.WatchSchemas` gRPC stream + a 30s reconcile sync, see `banyand/metadata/schema/property/client.go`). DDL `Create` / `Update` / `Delete` broadcast only to the schema-server (`ROLE_META`) nodes and **return immediately with a `mod_revision`** — now a client-stamped `time.Now().UnixNano()` timestamp, NOT an etcd index (which is why it still changes on every content-changing DDL; it is `0` on a delete that didn't record a tombstone) — **without waiting for data nodes**. A data write whose schema hasn't propagated yet is dropped at the data-node executor (`cannot find measure definition`, logged + skipped, `banyand/measure/write_standalone.go`), so the fence is still required — arguably more than under the old etcd-watch model. After firing DDL, fence on `BanyanDBClient.getSchemaWatcher().awaitRevisionApplied(maxRev, timeout)` before unparking dispatch / firing data writes — this blocks (liaison-side, all alive nodes, bounded by the timeout) until every data node's `notifiedModRevision` watermark reaches `maxRev`, which the registry's read-after-write does not guarantee. For deletes that returned `mod_revision == 0`, fall back to `awaitSchemaDeleted(SchemaKey, timeout)`. The previous \"poll `findMeasure` until you can read your own write\" idiom predates the `SchemaBarrierService` proto and has been replaced — do not reintroduce it. The `SchemaWatcher` class is in OAP's **in-tree** `library-banyandb-client` (`org.apache.skywalking.library.banyandb.v1.client.SchemaWatcher`); the standalone `skywalking-banyandb-java-client` repo does NOT have these symbols, so don't conclude the API is gone from grepping it. JDBC and ES are synchronous-DDL on the coordinator so they don't need a fence.\n\n## Analysis and Design Principles\n\n**Never guess or speculate.** All analysis must be grounded in source code, documentation, or verified behavior.\n\n### Before making claims\n- **Read the source code** — don't assume how a feature works based on naming or convention. Check the actual implementation.\n- **Read the documentation** — check `docs/en/`, CLAUDE.md files in submodules, and README files before proposing designs.\n- **Check configuration and flags** — verify what flags/env vars exist, their default values, and how they are parsed (e.g., BanyanDB uses viper with `BYDB_` prefix to auto-bind flags to env vars).\n- **Check dependent projects** — SkyWalking depends on BanyanDB, infra-e2e, Helm charts, etc. Read their source code and docs before assuming capabilities (e.g., check Helm chart `values.yaml` for supported fields, check infra-e2e for supported config options). For `skywalking-*` projects, ask the developer if they have the source code locally — searching a local clone is much faster than fetching files via GitHub API.\n\n### Before proposing changes\n- **Verify locally first** — run the code, start the container, execute the test before pushing to CI. Don't use CI as a trial-and-error environment.\n- **Validate file paths and directory structures** — check where data actually goes (e.g., BanyanDB `--stream-root-path /tmp` creates `/tmp/stream/`, `--access-log-root-path /tmp` creates `/tmp/accesslog/`). Don't assume directory names.\n- **Validate YAML syntax** — after editing YAML files (especially with sed/awk), validate with a YAML parser before committing. Corrupted YAML causes silent failures in CI.\n- **Check the actual Docker image** — verify what's available in the container (binaries, shell, directories) before writing commands that depend on them.\n\n### When uncertain\n- **Say \"I don't know\" and investigate** — reading the code is always better than guessing. Use grep, find, and read tools to locate the answer.\n- **Ask the developer first** — if you can't find the source code, don't know how to run something, or the code doesn't make the answer clear, ask the developer where to find it rather than speculate.\n- **Test with real data** — when investigating runtime behavior (e.g., what model names an API returns, what directory structure BanyanDB creates), set up a local test and observe the actual output.\n\n### Docker images\n- **Apache SkyWalking projects** — images are on `ghcr.io/apache/` (e.g., `ghcr.io/apache/skywalking-banyandb:${COMMIT_SHA}`). Tags are full commit SHAs, not short SHAs or version tags.\n- **Official and 3rd-party images** — on Docker Hub (e.g., `ollama/ollama`, `otel/opentelemetry-collector`, `envoyproxy/gateway`).\n- **Always verify the image exists** — `docker pull` before writing CI or e2e configs. Image tags depend on CI publish workflows completing successfully.\n","isInternal":false,"tokens":6158,"sizeBytes":26416}],"systemPromptSnippet":"<yakaai_skills repo=\"apache/skywalking\">\n<!-- File: .claude/skills/ci-e2e-debug/SKILL.md (Tokens: ~576 | Category: anthropic-skill) -->\n---\nname: ci-e2e-debug\ndescription: Download and inspect CI e2e test logs from GitHub Actions artifacts. Use when investigating e2e test failures in CI.\nargument-hint: \"<RUN_ID or GitHub Actions URL>\"\n---\n\n# CI E2E Debug Skill\n\nDownload test log artifacts from a GitHub Actions CI run and inspect them for errors.\n\n## Steps\n\n1. **Find the artifact**: Use `gh api` to list artifacts for the given CI run:\n   ```bash\n   gh api repos/apache/skywalking/actions/runs/<RUN_ID>/artifacts --jq '.artifacts[] | {id: .id, name: .name}'\n   ```\n\n2. **Download and extract**: Download the artifact zip and extract it:\n   ```bash\n   cd /tmp && rm -rf e2e-debug-logs && mkdir e2e-debug-logs && cd e2e-debug-logs\n   gh api repos/apache/skywalking/actions/artifacts/<ARTIFACT_ID>/zip > artifact.zip\n   unzip -o artifact.zip\n   ```\n\n3. **Inspect OAP logs**: Look for errors in the OAP server logs:\n   ```bash\n   # Find OAP log files\n   find /tmp/e2e-debug-logs -name \"skywalking-oap-*.log\" -o -name \"oap.log\"\n   # Check for errors\n   grep -E \"ERROR|Exception|FATAL|CannotCompileException\" <log_file> | head -30\n   ```\n\n4. **Inspect other component logs**: Check BanyanDB, UI, and other pod logs as needed.\n\n5. **Report findings**: Summarize the root cause error from the logs.\n\n## Notes\n- CI artifacts are automatically uploaded by the e2e test framework to `$SW_INFRA_E2E_LOG_DIR`\n- Log files are organized by namespace/pod name\n- OAP init pods may have different errors than the main OAP pod — check all of them\n- Common errors: MAL/LAL/OAL compilation failures, storage connection issues, module initialization errors\n- **Profile exporter tests** (e.g., Trace Profiling ES): the exporter runs as a separate process with `MockCoreModuleProvider`. If a new service is added to `CoreModule.services()` but not registered in the mock, the exporter fails at startup with `requiredCheck()` error — but the OAP logs will show no errors (OAP is fine, the exporter subprocess is what fails). Check the test step that runs `profile_exporter.sh`.\n- **When OAP logs are clean but tests fail**: look at which specific test step failed. Some e2e tests invoke standalone tools (profile exporter, data generator) that boot with mock providers and may fail independently of OAP.\n\n<!-- File: .claude/skills/compile/SKILL.md (Tokens: ~907 | Category: anthropic-skill) -->\n---\nname: compile\ndescription: Build SkyWalking OAP server, run javadoc checks, and verify checkstyle. Use to validate changes before submitting a PR.\nargument-hint: \"[all|backend|javadoc|checkstyle|module-name]\"\n---\n\n# Compile & Verify\n\nBuild the project and run static checks matching CI.\n\n## Prerequisites\n\n- JDK 11, 17, or 21 (LTS versions)\n- Maven 3.6+ (use `./mvnw` wrapper)\n\n## Maven profiles\n\n- `backend` (default): Builds OAP server modules\n- `ui` (default): Builds web application\n- `dist` (default): Creates distribution packages\n- `all`: Builds everything including submodule initialization\n\n## Commands by argument\n\n### `all` or no argument — full CI build\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### `backend` — backend only (faster)\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -Dmaven.test.skip\n```\n\n### `javadoc` — javadoc check only\n\nJavadoc requires delombok output, so `install` must run first:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nRunning `javadoc:javadoc` alone without `install` will miss errors because `${delombok.output.dir}` won't be populated.\n\n### `checkstyle` — checkstyle only\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n\n### Module name — single module build\n\n```bash\n./mvnw clean flatten:flatten package -pl oap-server/analyzer/<module-name> -Dmaven.test.skip\n```\n\n## Reading javadoc output\n\nMaven prefixes all javadoc output with `[ERROR]`, but the actual severity is in the message after the line number. Only lines containing `error:` fail the build; lines with `warning:` do not.\n\n```\n[ERROR] Foo.java:42: error: bad use of '>'        ← ACTUAL ERROR (must fix)\n[ERROR] Foo.java:50: warning: no @param for <T>   ← WARNING (does not fail build)\n```\n\n### Common javadoc errors\n\n| Error | Cause | Fix |\n|-------|-------|-----|\n| `bad use of '>'` | Bare `>` in javadoc HTML (e.g., `->` in `<pre>` blocks) | Use `{@code ->}` or `-&gt;` |\n| heading out of sequence | Heading level skips the expected hierarchy | See heading rules below |\n| reference not found | `{@link Foo#bar()}` with wrong signature | Match exact parameter types: `{@link Foo#bar(ArgType)}` |\n\n### Javadoc heading rules (JDK 13+)\n\nStrict heading validation was introduced in JDK 13. JDK 11 does **not** enforce it, but JDK 17/21/25 do. Write headings correctly for forward compatibility:\n\n| Javadoc location | Start heading at |\n|---|---|\n| Class, interface, enum, package, module | `<h2>` |\n| Constructor, method, field | `<h4>` |\n| Standalone HTML files (`doc-files/`) | `<h1>` |\n\nThe generated javadoc page uses `<h1>` for the class name and `<h3>` for member sections (Methods, Fields, etc.), so class-level subsections must use `<h2>` and method-level subsections must use `<h4>` to maintain proper nesting.\n\n## CI reference\n\nCI uses JDK 11 on Linux. The `dist-tar` job runs:\n\n```bash\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\nThe `code-style` job runs:\n\n```bash\n./mvnw -B -q clean flatten:flatten checkstyle:check\n```\n\n<!-- File: .claude/skills/generate-classes/SKILL.md (Tokens: ~521 | Category: anthropic-skill) -->\n---\nname: generate-classes\ndescription: Generate bytecode classes from DSL scripts (MAL, OAL, LAL, Hierarchy). Runs the compiler and dumps .class files for inspection.\nargument-hint: \"<mal|oal|lal|hierarchy|all>\"\n---\n\n# Generate DSL Classes\n\nRun the v2 compiler (ANTLR4 + Javassist) to generate bytecode classes from DSL scripts and dump `.class` files to disk for inspection.\n\n## Commands by argument\n\n### `mal` — MAL expression classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=MALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/mal/**/*.generated-classes/`\n\n### `oal` — OAL metrics/dispatcher/builder classes\n\n```bash\n./mvnw test -pl oap-server/oal-rt \\\n  -Dtest=RuntimeOALGenerationTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/oal-rt/target/test-classes/metrics/`, `metrics/builder/`, `dispatcher/`\n\n### `lal` — LAL filter/extractor classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=LALExpressionExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/lal/**/*.generated-classes/`\n\n### `hierarchy` — Hierarchy rule classes\n\n```bash\n./mvnw test -pl oap-server/analyzer/dsl-scripts-test \\\n  -Dtest=HierarchyRuleExecutionTest -DfailIfNoTests=false -Dcheckstyle.skip\n```\n\nOutput location: `oap-server/analyzer/dsl-scripts-test/src/test/resources/scripts/hierarchy-rule/*.generated-classes/`\n\n### `all` or no argument — generate all DSLs\n\nRun all four commands above sequentially.\n\n## After generation\n\nPrint the output location for the requested DSL so the user knows where to find the generated `.class` files. Use `javap` to decompile:\n\n```bash\njavap -c -p <path-to-class-file>\n```\n\n## Cleaning generated classes\n\n```bash\n./mvnw clean -pl oap-server/analyzer/dsl-scripts-test\n```\n\n<!-- File: .claude/skills/gh-pull-request/SKILL.md (Tokens: ~1696 | Category: anthropic-skill) -->\n---\nname: gh-pull-request\ndescription: Verify, commit, and push changes on a PR branch. Runs pre-flight checks (compile, checkstyle, license headers) before every push. Also creates the PR if one doesn't exist yet.\n---\n\n# PR Branch Workflow\n\nRun pre-flight checks, commit, push, and optionally create a PR.\n\n## Pre-flight checks\n\nRun these checks before every commit+push and fix any failures:\n\n### 1. Compile and checkstyle\n\n```bash\n# Checkstyle\n./mvnw -B -q clean checkstyle:check\n\n# Full build (compile + javadoc)\n./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall \\\n  -Dmaven.test.skip \\\n  -Dcheckstyle.skip \\\n  -Dgpg.skip\n```\n\n### 2. License header check\n\n```bash\nlicense-eye header check\n```\n\nIf invalid files are found, fix with `license-eye header fix` and re-check.\n\n### 3. Unnecessary fully-qualified class names\n\nThe project checkstyle forbids inline FQCNs — every type reference in code should resolve\nthrough an `import`, not a fully-qualified name. Checkstyle does not always catch this (it\nmisses cases like inline `java.util.HashMap`, `java.util.concurrent.TimeUnit`, or\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` used as a local\nvariable type, generic parameter, or `new` target). Audit the files the branch touched\nbefore pushing:\n\nUse the `Grep` tool (ripgrep) rather than BSD `grep` on macOS — the scan below relies on a\nnegative lookahead that BSD `grep` doesn't support and GNU `grep -P` does:\n\n```\npattern: ^(?!\\s*(import |package |\\s*\\*)).*\\b(java\\.util\\.|java\\.io\\.|java\\.nio\\.|java\\.util\\.concurrent\\.|javassist\\.|org\\.apache\\.skywalking\\.)[A-Z][A-Za-z0-9_]*\nglob:    *.java\noutput_mode: content\n-n: true\n```\n\nScope the scan to files the branch touched, not the whole tree — pre-existing FQDNs on\nunrelated files generate noise. Use `git diff --name-only master...HEAD -- '*.java'` to get\nthe changed list, then run the ripgrep pattern against each.\n\nAcceptable exceptions (same as the `CLAUDE.md` rule):\n - Two classes with the same simple name would collide if both imported.\n - A Javadoc `{@link}` where the short name would be ambiguous to the reader.\n - Inside a string literal (e.g., a class name passed to `Class.forName`).\n\nFix every other hit — add an `import` and switch to the short name. This includes\n`new java.util.HashMap<>()`, `java.util.Set<String>` parameter types, and\n`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` as a local\nvariable type. Field declarations, method signatures, local variables, and generic\ntype arguments should all use the imported short name.\n\nRe-run checkstyle after the fix — a sloppy `sed`/`replace_all` can corrupt the `import`\nline itself (e.g., turning `import java.util.concurrent.locks.ReentrantLock;` into\n`import ReentrantLock;`), which causes a cryptic checkstyle `Range [0, -1) out of\nbounds for length N` error, not a normal violation line. If you see that error, inspect\nthe imports block first.\n\n## Commit and push\n\nAfter checks pass, commit and push:\n\n```bash\ngit add <files>\ngit commit -m \"<message>\"\ngit push -u origin <branch-name>\n```\n\n### Branch strategy\n- **Never work directly on master branch**\n- If on master, create a new branch first: `git checkout -b feature/<name>` or `git checkout -b fix/<name>`\n\n## Create PR (if not yet created)\n\nCheck whether a PR already exists for the current branch:\n\n```bash\ngh pr view --json number 2>/dev/null\n```\n\nIf no PR exists, create one:\n\n### PR title\nSummarize the changes concisely. Examples:\n- `Fix BanyanDB query timeout issue`\n- `Add support for OpenTelemetry metrics`\n\n### PR description\n\nRead `.github/PULL_REQUEST_TEMPLATE` and use its **exact format with checkboxes**. Do NOT use a custom summary format.\n\nKey template sections — uncomment the relevant one:\n\n**For Bug Fixes:**\n```\n### Fix <bug description or issue link>\n- [ ] Add a unit test to verify that the fix works.\n- [ ] Explain briefly why the bug exists and how to fix it.\n```\n\n**For New Features:**\n```\n### <Feature description>\n- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.\n- [ ] Update the documentation to include this new feature.\n- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.\n- [ ] If it's UI related, attach the screenshots below.\n```\n\n**For Performance Improvements:**\n```\n### Improve the performance of <class or module or ...>\n- [ ] Add a benchmark for the improvement.\n- [ ] The benchmark result.\n- [ ] Links/URLs to the theory proof or discussion articles/blogs.\n```\n\n**Always include:**\n```\n- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.\n- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).\n```\n\n### Create command\n\n```bash\ngh pr create --title \"<title>\" --body \"$(cat <<'EOF'\n<PR body from template>\nEOF\n)\"\n```\n\n### Post-creation\n- Add `copilot` as a reviewer: `gh pr edit <number> --add-reviewer copilot`\n- Do NOT add AI assistant as co-author. Code responsibility is on the committer's hands.\n- Return the PR URL when done.\n\n## After the PR is merged\n\nOnce the PR is merged, sync the default branch and clean up the feature branch:\n\n```bash\n# 1. Prune stale remote refs. GitHub auto-deletes the PR's branch on merge, so\n#    the remote feature branch is usually already gone; --prune removes the\n#    dangling local tracking ref.\ngit fetch origin --prune\n\n# 2. Switch back to the default branch and fast-forward it to include the merge.\ngit checkout master\ngit pull --ff-only origin master\n\n# 3. Confirm the change actually landed in master before deleting anything —\n#    `git log --oneline -1` should show the merge/squash commit with the PR\n#    number, or grep for a symbol the PR introduced.\ngit log --oneline -1\n\n# 4. Delete the local feature branch. SkyWalking SQUASH-merges PRs, so the\n#    feature branch's commit is NOT an ancestor of master (master gets a new\n#    squash commit instead). `git branch -d` therefore reports \"not fully\n#    merged\" — that is expected, not an error. After confirming the content is\n#    in master (step 3), force-delete:\ngit branch -d <branch> 2>/dev/null || git branch -D <branch>\n```\n\nNotes:\n- `git branch -d` failing with \"not fully merged\" on a squash-merged PR is normal — the squash commit has a different SHA than the feature commit. Verify via step 3, then `-D`.\n- If the remote branch was not auto-deleted (some repo settings), remove it explicitly: `git push origin --delete <branch>`.\n- Do NOT skip step 3. Force-deleting a local branch whose work didn't actually merge loses it.\n\n<!-- File: .claude/skills/license/SKILL.md (Tokens: ~491 | Category: anthropic-skill) -->\n---\nname: license\ndescription: Check and fix Apache 2.0 license headers and dependency licenses using skywalking-eyes. Use before submitting a PR.\nargument-hint: \"[check|fix|deps]\"\n---\n\n# License Checks\n\nCheck and fix license compliance using [skywalking-eyes](https://github.com/apache/skywalking-eyes). Two CI jobs use this tool:\n\n1. **license-header** — verifies all source files have Apache 2.0 headers\n2. **dependency-license** — verifies the LICENSE file matches current dependencies\n\n## Steps\n\n### Header check (default, or `check` argument)\n\n```bash\nlicense-eye header check\n```\n\n- **All valid**: Output shows `valid: N, invalid: 0` — nothing to do.\n- **Invalid files found**: Fix with `license-eye header fix`, then re-check.\n\n### Header fix (`fix` argument)\n\n```bash\nlicense-eye header fix\nlicense-eye header check\n```\n\n### Dependency license check (`deps` argument)\n\nThis regenerates the LICENSE file from dependency metadata and checks for drift:\n\n```bash\nlicense-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl || exit 1\nif [ ! -z \"$(git diff -U0 ./dist-material/release-docs/LICENSE)\" ]; then\n  echo \"LICENSE file is not updated correctly\"\n  git diff -U0 ./dist-material/release-docs/LICENSE\nfi\n```\n\nIf the LICENSE file changed, review the diff and commit it. **Important**: CI runs on Linux — some dependencies have platform-specific variants. If you're on macOS/Windows, the LICENSE diff may be a platform artifact. Verify before committing.\n\n## Rules\n\nConfiguration is in `.licenserc.yaml`:\n- Java, XML, YAML/YML files require Apache 2.0 headers\n- JSON and Markdown files are excluded (JSON doesn't support comments)\n- Generated files and certain vendor paths are excluded\n- SPI service files (`META-INF/services/`) require headers (use `#` comment style)\n\n## Installation\n\n```bash\n# Same version as CI (pinned commit)\ngo install github.com/apache/skywalking-eyes/cmd/license-eye@5b7ee1731d036b5aac68f8bd3fc9e6f98ada082e\n\n# Or via Homebrew (macOS)\nbrew install license-eye\n```\n\n<!-- File: .claude/skills/new-monitoring-feature/SKILL.md (Tokens: ~9600 | Category: anthropic-skill) -->\n---\nname: new-monitoring-feature\ndescription: Add a new monitoring target / layer to SkyWalking OAP. Orients you to the OAL / MAL / LAL / SpanListener / SegmentListener extension points, the UI template + submodule touchpoints, the docs set that must move together, and the cross-cutting traps that don't live in any one skill.\n---\n\n# Adding a New Monitoring Feature / Layer\n\nSkyWalking monitoring for a new target (a new `Layer` — mobile OS, gateway, DB, language runtime, …) is composed from the same set of reusable extension points. This skill is the index: which extension point fits which signal, where its contract lives, which docs must be updated together, and which harness-wide traps the extension-specific docs don't cover.\n\nThere is one skill per narrow concern. This one is the wiring map.\n\n---\n\n## Working posture\n\n- **Default scope for a new monitoring feature is narrow.** For most new monitoring targets, traces are **re-used as-is** — you do not write or modify a trace analyzer. Metrics and log extensions via OTLP (MAL + LAL) are the common shape. OTLP / Zipkin span-based metrics analysis (`SpanListener`) already exists and only needs a new implementation if your feature introduces genuinely new semantic-convention attributes to extract (e.g. `gen_ai.*`, `mcp.*` for the GenAI layer). Do **not** touch `CommonAnalysisListener` / `RPCAnalysisListener` / `SegmentAnalysisListener` unless the OAP core protocol gains a new `SpanLayer` enum value — details in §2.\n- **If you find yourself editing more than this skill suggests, stop and confirm.** Scope creep on trace analysis, span-layer mapping, receiver handlers, or protobuf/proto contracts is a strong signal the framing is wrong. Ask the developer before committing that direction.\n- **Verify every new feature locally before pushing to CI.** Compile + checkstyle + license-eye are minimums, but they only tell you the code builds — they do **not** tell you the new MAL rule compiles its closures at startup, the LAL layer:auto dispatch works, the OTLP payload shape matches the MAL labels, the UI template renders against real data, or the layer shows up in `swctl service ly <LAYER>`. Run the new e2e case end-to-end locally against a live OAP (see [run-e2e skill](../run-e2e/SKILL.md)) and fire every verify step with `swctl` by hand at least once. Pushing to CI without a local green run wastes shared CI cycles and slow-loops review.\n\n---\n\n## 0. Register the `Layer` — the feature's entry point\n\nA `Layer` is how OAP slices services / instances / endpoints by data source. The UI, storage partitioning, menu navigation, and OAL aggregation all key off it.\n\n`Layer` is a registry-backed value type (no longer a closed enum). Built-in layers are declared as `public static final Layer` constants in `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java`; external layers are registered through the `Layer.register(name, ordinal, normal)` API at boot. **Pick the registration path that matches the scope of your feature:**\n\n| Your feature ships as | Registration path |\n|---|---|\n| Part of the OAP distribution (in-tree, the common case for new SkyWalking-supported targets) | Add a `public static final Layer` constant to `Layer.java` with the next sequential ordinal in `0–49`. Examples: `IOS = register(\"IOS\", 47, true)`, `APISIX = register(\"APISIX\", 21, true)`, `VIRTUAL_DATABASE = register(\"VIRTUAL_DATABASE\", 14, false)` for inferred/non-real services. |\n| An out-of-tree MAL or LAL rule file | Add a top-level `layerDefinitions:` block to the rule file. The DSL loader funnels each entry through `Layer.register` before compiling the rule. One file ships the layer + the rules that produce its telemetry. |\n| An out-of-tree plugin module (jar) | Implement `org.apache.skywalking.oap.server.core.analysis.LayerExtension` and register via `META-INF/services/`. Discovered by `LayerExtensionLoader` during `CoreModuleProvider.prepare()`. |\n| Operator-deployed config (no code, no DSL) | Add an entry to `oap-server/server-starter/src/main/resources/layer-extensions.yml` (or override on the OAP node's classpath). |\n\n**Ordinal conventions:** `0–49` is in active use by built-ins. `50–999` is reserved by convention for future built-in layers. External layers are recommended (not required) to start at `>= 1000` to avoid colliding with future built-ins on OAP upgrade. Collisions in either direction are detected at boot via the ordinal-uniqueness check, which fails OAP startup loudly.\n\n**Storage encoding is the ordinal int**, persisted in BanyanDB / Elasticsearch / JDBC. Every OAP node that reads or writes a given layer must agree on its `(name, ordinal)` mapping — deploy `layer-extensions.yml` and any `layerDefinitions:` rule files identically across all nodes. The registry is sealed at the start of `Core.notifyAfterCompleted()`; later registration attempts throw.\n\n**UI template folders are auto-discovered by file scan, not by `Layer.values()`.** `UITemplateInitializer` walks `ui-initialized-templates/**/*.json` recursively (depth 2) and trusts each template's own `configuration.layer` field. Drop a folder of dashboard JSONs on disk and the initializer picks them up on the next boot — folder name is purely organizational.\n\n**Component ID lookup in Java code**: IDs declared in `component-libraries.yml` are loaded at runtime into `ComponentLibraryCatalogService`'s `componentName2Id` map — they are **not** exposed as Java enum constants. To look up by name in listener code, inject the catalog service and resolve once at construction:\n```java\nIComponentLibraryCatalogService catalog = moduleManager.find(CoreModule.NAME)\n    .provider().getService(IComponentLibraryCatalogService.class);\nint myComponentId = catalog.getComponentId(\"My-Component-Name\");\n```\nCache as an `int` field; runtime comparisons are then plain `componentId == myComponentId`. **Trap:** there is a `ComponentsDefine` class under `skywalking-trace-receiver-plugin/src/test/java/.../mock/ComponentsDefine.java` — it is a test-only mock holding five hand-picked constants (Tomcat, Dubbo, RocketMQ, MongoDB). Do not import or extend it from production code.\n\nEmit the layer from every source object your feature produces. Built-in layers have a static-field accessor; external layers are looked up by name through the registry:\n\n```java\n// Built-in layer (constant)\nservice.setLayer(Layer.IOS);\n\n// External layer (registered via yaml / SPI / layerDefinitions:)\nservice.setLayer(Layer.nameOf(\"IOT_FLEET\"));\n```\n\nDownstream (the core OAL, `service ly <LAYER>` swctl query, topology filters, UI root dashboard's layer selector) all work off this single enum value.\n\n---\n\n## 1. Pick the right extension point\n\n| Signal you want to process | Extension point | Register via |\n|---|---|---|\n| **SkyWalking native trace segments** (from language agents) | `AnalysisListener` (TraceSegmentListener / EntryAnalysisListener / ExitAnalysisListener / LocalAnalysisListener / FirstAnalysisListener) + factory in `agent-analyzer` | `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory` |\n| **OTLP / Zipkin trace spans** (OTel SDKs, Istio, Envoy, iOS/Android, …) | `SpanListener` in `core.trace` | `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener` |\n| **OTel metrics** (Prometheus scrape, Envoy stats, Istio, …) | MAL rules — `otel-rules/<namespace>/*.yaml` | Append the rule path to `enabledOtelMetricsRules` in `application.yml` |\n| **Logs** (OTLP, SkyWalking native, Fluent Bit, Envoy ALS) | LAL rules — `lal/<rule>.yaml` | Append to `SW_LOG_LAL_FILES` (env) or `lalFiles` (YAML) under `log-analyzer` |\n| **Pre-computed meter / custom meter** (custom agent protocol, meter receiver) | MAL `meter-analyzer-config/*.yaml` | Append to `meter-analyzer-config` entries |\n| **Aggregation from Source entities** (service, instance, endpoint, relation) | OAL rules — `oal/*.oal` | Already loaded by `OAL grammar`; just add lines |\n\nAll five DSLs (OAL, MAL, LAL, Hierarchy + SpanListener SPI) compile via ANTLR4 + Javassist at startup. Source-of-truth details are in:\n- [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md)\n- [`docs/en/concepts-and-designs/mal.md`](../../../docs/en/concepts-and-designs/mal.md)\n- [`docs/en/concepts-and-designs/lal.md`](../../../docs/en/concepts-and-designs/lal.md)\n- [`docs/en/concepts-and-designs/scope-definitions.md`](../../../docs/en/concepts-and-designs/scope-definitions.md)\n- [`oap-server/analyzer/meter-analyzer/CLAUDE.md`](../../../oap-server/analyzer/meter-analyzer/CLAUDE.md) — MAL compiler internals\n- [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — LAL compiler internals\n\n---\n\n## 2. Trace ingestion — two parallel pipelines\n\nSkyWalking OAP ingests traces through two distinct entry points. Pick based on source.\n\n> **Before writing any trace extension — ask \"do I actually need one?\"**\n>\n> For **most new layers you do not touch trace analysis at all.** Segments (native or\n> OTLP) flow through the existing pipeline, `RPCAnalysisListener.parseExit` produces\n> the outbound `ServiceRelation` edges, the backend-side agent produces the inbound\n> metrics via its own `parseEntry`, and the new-layer Service / Instance / Endpoint\n> entities come from **MAL + LAL**, not from trace analysis. This is how browser\n> (`Layer.BROWSER`), iOS (`Layer.IOS`), mobile mini-programs\n> (`Layer.WECHAT_MINI_PROGRAM` / `Layer.ALIPAY_MINI_PROGRAM`), and most\n> scraped-metric layers work.\n>\n> Extend trace analysis **only** when the feature genuinely needs per-span\n> interpretation that the existing pipeline cannot express:\n>\n> - **A new `SpanLayer`** — OAP core protobuf has a new `SpanLayer` enum value that\n>   callers of `identifyServiceLayer` / `identifyRemoteServiceLayer` must dispatch on.\n>   That means touching `CommonAnalysisListener` / `RPCAnalysisListener`.\n> - **A genuinely new signal class not covered by existing listeners** — e.g. the GenAI\n>   work added `GenAISpanListener` for LLM / MCP instrumentation because OTLP spans\n>   carry semantic-convention attributes (`gen_ai.*`, `mcp.*`) that no prior listener\n>   extracted. That is rare.\n>\n> **Anti-pattern we've been bitten by:** using a componentId mapping in\n> `CommonAnalysisListener` to assign a client-side layer to exit-only segments. This\n> does not produce inbound metrics anyway (no entry spans → no\n> `service_cpm` / `endpoint_cpm` from OAL), and it is orthogonal to the Service entity\n> layer (which MAL / LAL sets). Mirror the browser pattern instead: ship `Layer` +\n> `component-libraries.yml` + MAL + LAL + dashboards, and leave the listener chain\n> alone.\n\n### 2.1 SkyWalking native segments — `AnalysisListener`\n\n**Source protocol**: `apm-protocol/apm-network/src/main/proto/language-agent/Tracing.proto` (`TraceSegmentObject`).\n\n**Receiver**: `oap-server/server-receiver-plugin/skywalking-trace-receiver-plugin`. After wire decoding, segments are handed to `SegmentParserServiceImpl` which fans out to registered `AnalysisListenerFactory` implementations.\n\n**Extension**: implement one or more of\n- `FirstAnalysisListener` — per-segment, before per-span phase\n- `EntryAnalysisListener` — per-entry-span\n- `ExitAnalysisListener` — per-exit-span\n- `LocalAnalysisListener` — per-local-span\n- `TraceSegmentListener` — whole-segment, after per-span phase\n\nRegister the **factory** in `META-INF/services/org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.AnalysisListenerFactory`. Existing examples: `RPCAnalysisListener`, `MultiScopesAnalysisListener`, `SegmentAnalysisListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`.\n\nUse when: the data arrives on the SW native wire (language agents — Java, Python, Go, Node.js, .NET, PHP, Lua, etc.).\n\n### 2.2 OTLP / Zipkin spans — `SpanListener`\n\n**Source protocol**: OTLP proto (`opentelemetry/proto/trace/v1/trace.proto`) or Zipkin v2 JSON.\n\n**Receiver**: `oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/.../otlp/OpenTelemetryTraceHandler.java` (OTLP) and `oap-server/server-receiver-plugin/zipkin-receiver-plugin/.../ITZipkinTraceCollector` (Zipkin). After decoding, spans are iterated and passed to every registered `SpanListener` through `SpanListenerManager`.\n\n**Extension**: implement `org.apache.skywalking.oap.server.core.trace.SpanListener`:\n\n```java\npublic interface SpanListener {\n    String[] requiredModules();\n    void init(ModuleManager moduleManager);\n    SpanListenerResult onOTLPSpan(OTLPSpanReader span, String serviceName, Map<String,String> resourceAttributes,\n                                  String scopeName, String scopeVersion);\n}\n```\n\nName the service by `serviceName`, not the resource's `service.name`: it is the service the span is analyzed and\nstored under, which `receiver-otel.otlpServiceGroupKeys` may have grouped (`shop::cart` for `service.name=cart`).\n\nRegister in `META-INF/services/org.apache.skywalking.oap.server.core.trace.SpanListener`. Reference implementations:\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSHTTPSpanListener.java` — OTel Swift URLSession → OAL (Service / ServiceInstance / Endpoint)\n- `oap-server/analyzer/ios-analyzer/src/main/java/.../IOSMetricKitSpanListener.java` — OTel Swift MetricKit → MAL samples via `OpenTelemetryMetricRequestProcessor.toMeter`\n- `oap-server/analyzer/gen-ai-analyzer/src/main/java/.../GenAISpanListener.java` — LLM / MCP instrumentation\n\nReturn `SpanListenerResult.builder().shouldPersist(false).build()` to prevent a span being stored as a trace (useful for e.g. 24-hour aggregate spans that are not meaningful as traces).\n\n**When both native and OTLP should feed the same metric**, write your extension against the downstream scope (Service / ServiceInstance / Endpoint + OAL) and emit those entities from **both** a SegmentListener and a SpanListener. The OAL layer is protocol-agnostic.\n\n---\n\n## 3. Metrics — MAL / OAL rules\n\n### 3.1 OAL (for entities already produced by core sources)\n\n`oap-server/server-starter/src/main/resources/oal/core.oal` already defines the standard `service_cpm`, `service_resp_time`, `service_sla`, `endpoint_cpm`, etc. — keyed off the `Service`, `ServiceInstance`, `Endpoint` source scopes. If your new layer emits those entities with the right `Layer`, you get the core metrics for free.\n\n**Who emits those entities matters.** `service_cpm` / `service_resp_time` / `service_sla` / `endpoint_cpm` / `endpoint_resp_time` come from `RPCAnalysisListener.parseEntry` → `callingIn.toService()` / `toEndpoint()`. That only fires on **inbound entry spans**. Client-side / edge layers (browser, iOS, mini-programs) only emit exit spans, so these OAL metrics do **not** populate under those layers — their Service entities are produced by MAL / LAL, and their dashboards use MAL metrics (`meter_<layer>_*`) for load / latency, not `service_cpm` / `service_resp_time`. Don't wire dashboards to OAL-derived metrics that the layer never produces.\n\nAdditional OAL rules go in the same file. OAL grammar: `oap-server/oal-grammar/src/main/antlr4/OALParser.g4`. Syntax and examples: [`docs/en/concepts-and-designs/oal.md`](../../../docs/en/concepts-and-designs/oal.md).\n\n### 3.2 MAL (for meter / metric samples)\n\n**File location**: `oap-server/server-starter/src/main/resources/otel-rules/<namespace>/*.yaml` (OTel scrape) or `meter-analyzer-config/*.yaml` (SW native meter).\n\n**Registration** (MANDATORY): append the rule path to the default in `oap-server/server-starter/src/main/resources/application.yml`:\n```yaml\nreceiver-otel:\n  default:\n    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:\"...,<your-namespace>/*\"}\n```\nIf you skip this, the rule isn't loaded in the default starter and downstream `converters.forEach(...)` is a silent no-op — samples just disappear.\n\n**Also update** `test/e2e-v2/cases/storage/expected/config-dump.yml`. This e2e diffs `/debugging/config/dump` output against a static file; every default-value change in `application.yml` must mirror here or the storage e2e fails.\n\n**Naming**: keep attribute scope in the metric name. If a metric reads from `foo.bar.foreground.exit_count`, call it `foo_foreground_exit_count`, not `foo_exit_count`. When two attributes (foreground vs background, ingress vs egress) share a theme, keep them as separate distinguishable metrics — don't flatten.\n\n### 3.3 Histogram pitfalls (both OAL and MAL)\n\n- **Per-bucket counts, not cumulative.** `histogram_percentile` accumulates internally. Feeding cumulative counts doubles the accumulator and shifts percentiles up. Each observation contributes `1` to exactly one `le` bucket.\n- **`le` label scale.** `SampleFamily.histogram()` rescales `le` by `defaultHistogramBucketUnit.toMillis(1)` (default `SECONDS` ⇒ ×1000). If your raw labels are already in ms, use `SampleFamilyBuilder.newBuilder(samples).defaultHistogramBucketUnit(TimeUnit.MILLISECONDS).build()`.\n- **Overflow bucket.** MAL parses `le=\"Infinity\"` as `Double.POSITIVE_INFINITY` → stored as `Long.MAX_VALUE`. If a percentile lands there, the UI renders ≈ 9.2 × 10¹⁸. Prefer a finite sentinel (e.g., 30 s for hang/launch) so outliers remain visible without crashing the chart.\n- **`NamingControl`** — every `service` / `serviceInstance` / `endpoint` label that ends up as an entity key must go through `namingControl.formatServiceName()` / `formatInstanceName()` / `formatEndpointName()`. Raw resource attributes (long git-SHA build metadata, path-style endpoints) will otherwise exceed storage limits.\n\n---\n\n## 4. Logs — LAL\n\n**File location**: `oap-server/server-starter/src/main/resources/lal/<feature>.yaml`.\n\n**Registration**: append to `SW_LOG_LAL_FILES` env var, or `log-analyzer.default.lalFiles` in `application.yml`.\n\n**Grammar**: `oap-server/analyzer/log-analyzer/src/main/antlr4/LALParser.g4`. The compiler docs are thorough: [`oap-server/analyzer/log-analyzer/CLAUDE.md`](../../../oap-server/analyzer/log-analyzer/CLAUDE.md) — covers:\n- `filter { … }` block with `parsed.*`, `tag()`, `sourceAttribute()`, `log.*`, `def` locals, `extractor {}`, `metrics {}`, `sink {}`\n- `layer: auto` mode (script decides layer from source attrs)\n- `inputType` / `outputType` SPI (`LALSourceTypeProvider`) to bind typed proto to `parsed.*`\n- Test harness at `oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/oap-cases/` — ship YAML + `.data.yaml` pair; `LALScriptExecutionTest` picks them up automatically.\n\n**`layer: auto` gotcha**: the extractor's `layer \"X\"` assignment must propagate to `LogMetadata.layer` (not just the output builder) — `FilterSpec.doSink` reads metadata to decide drop/route. The current codegen (`LALBlockCodegen.generateFieldToOutput`) handles this correctly; if you extend codegen, preserve that invariant.\n\n---\n\n## 5. UI — templates, submodule, and menu\n\n### 5.1 Dashboard templates\n\nJSON files under `oap-server/server-starter/src/main/resources/ui-initialized-templates/<feature>/`. Conventional fileset:\n\n- `<feature>-root.json` — `ServiceList` widget, layered service index.\n- `<feature>-service.json` — per-service dashboard, usually a `Tab` with Overview / Instance / Endpoint / Log sub-tabs.\n- `<feature>-instance.json` — per-instance dashboard.\n- `<feature>-endpoint.json` — per-endpoint dashboard.\n- Menu link in `oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml`.\n\n> **Collaborate with the developer on dashboard design.** Dashboard layout and metric selection are product decisions, not derivable from the MAL rules alone. Before authoring the JSON, check in with the developer on: which metric goes on which panel, which percentiles / aggregations to show, what's on the root list vs. per-service vs. per-instance vs. per-endpoint pages, what stays in the Overview tab vs. a sub-tab, widget sizes / ordering, and whether Trace / Log tabs belong at all (depends on whether the feature emits traces and logs). Then **manually set up the feature against a live OAP** — emit a few real OTLP payloads, open each dashboard in the UI, confirm every widget renders non-empty, and walk the drill-down from root → service → instance → endpoint. Only after that eyes-on pass is the dashboard ready to ship.\n\nDetails that always bite:\n\n- **Tab chrome eats ~4 rows.** Set outer `Tab.h = content_max_y + 4` or you get an inner scrollbar on top of the page scrollbar.\n- **Widget heights consistent.** Use `h=15` per row for `Line` / `Bar` widgets to match mesh / general peer templates. Don't mix h=12 / h=15 / h=24 across a feature's dashboards.\n- **List widgets (ServiceList / InstanceList / EndpointList)** render two pieces per cell:\n  - `expressions` → the aggregated summary number (right side). Required. Wrap raw metrics in `avg(...)` / `sum(...)`. Omitting it leaves the column empty.\n  - `subExpressions` → the sparkline trend (left side). Optional, falls back to `expressions` if absent.\n  - `metricConfig` pairs 1-to-1 with `expressions` (not `subExpressions`). Use for `label` (column header) and `unit`.\n  - Compound columns via MQE addition: `avg(metric_a + metric_b)` in `expressions`, `metric_a + metric_b` in `subExpressions`.\n  - Normal widgets with multiple expressions on one chart — always give a human label per expression in `metricConfig`, otherwise legend renders raw metric names.\n\n### 5.2 Hot-reload (edits to shipped templates take effect on restart)\n\n`UITemplateInitializer` normally uses `addIfNotExist` (keyed on JSON `id`), so on a plain restart your edits to shipped templates are ignored — the seeded copy in storage wins. For dev / extension work, set the environment variable and restart:\n\n```bash\nSW_UI_TEMPLATE_FORCE_RELOAD=true\n# then restart OAP — shipped templates overwrite the seeded copies via addOrReplace.\n# Hard-refresh the browser (Cmd+Shift+R) because the booster UI caches config in memory.\n```\n\nThis is a dev / extension knob only — read directly from the OS environment, not wired through `application.yml`. Unset or `false` preserves the default production behavior where operators' UI edits persist across restarts.\n\n### 5.3 New top-level menu group (e.g. \"Mobile\", \"Gateway\")\n\nNew **child entries** only need template JSON + `menu.yaml`. A new **top-level group** needs submodule work in `skywalking-ui` (git submodule at `skywalking-ui/`):\n\n1. `skywalking-ui/src/assets/icons/<name>.svg` — menu icon.\n2. `skywalking-ui/src/locales/lang/menus/{en,es,zh}.ts` — i18n labels (`<name>`, `<name>_desc`, `<name>_<child>`, `<name>_<child>_desc`). Missing i18n falls back to raw keys on screen.\n3. Commit those upstream in `apache/skywalking-booster-ui`, bump the submodule pointer here.\n4. Rebuild with `-Pui,dist` (not `-Pbackend,dist`) and `make docker.ui`, otherwise the new assets aren't in the image. See [package skill](../package/SKILL.md).\n\n---\n\n## 6. Docs set — everything that moves together\n\n| File | What to add |\n|---|---|\n| `docs/en/setup/backend/<feature>-monitoring.md` | User-facing setup guide. Required SDK / collector, required env vars, sample OTLP payload, metrics / log tags list. |\n| `docs/menu.yml` | Add the guide to the docs navigation tree. |\n| `docs/en/security/README.md` | **Client-side monitoring**: mobile / browser / mini-program endpoints accept data from the public internet without auth by default. Add a rate-limit + abuse-protection note. Agent-side monitoring over private VPC usually doesn't need this. |\n| `docs/en/changes/changes.md` | Per-release changelog. Add under `#### OAP Server` for features/bug-fixes, `#### UI` if the submodule bumps, `#### Documentation` if new guides. |\n| `docs/en/swip/SWIP-<N>.md` | Design doc for non-trivial features; keep reconciled with final code after review. |\n| `docs/en/swip/readme.md` | Move SWIP from \"Proposed\" to \"Accepted\" once merged. |\n\n---\n\n## 7. E2E test\n\nFolder: `test/e2e-v2/cases/<feature>/`. See [run-e2e skill](../run-e2e/SKILL.md) for harness basics (setup / trigger / verify phases, template functions, expected-file authoring via `contains`).\n\nFeature-specific extras:\n\n1. **Docker-compose**: extend `../../script/docker-compose/base-compose.yml` oap + storage. Set `SW_OTEL_RECEIVER_ENABLED_HANDLERS`, `SW_LOG_LAL_FILES`, and any feature-specific env. Don't list default rules here — rely on `application.yml` so the config-dump test stays consistent.\n2. **Register in CI**: add an entry under the e2e matrix in `.github/workflows/skywalking.yaml`.\n3. **OTLP data hygiene**:\n   - `spanId` is exactly 16 hex chars; `traceId` is exactly 32. Protobuf-JSON decodes them as base64; other lengths return HTTP 400.\n   - Setup-step curl loops must fail loudly: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e`. The pattern `for …; do curl && break || sleep 5; done` silently succeeds when every attempt connection-refused (the final `sleep` returns 0) because OAP takes ~50 s to start in CI.\n4. **Verify each case by hand before pushing** and use a published simulator image where the upstream SDK has one. See [`run-e2e` skill §6–§9](../run-e2e/SKILL.md) for verify-query triage, expected-file authoring, and the list of expected-file traps (YAML quoting on colons, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl loops that swallow failure).\n5. **Config dump**: when you modify any `application.yml` default (rule list, handler list, ...), also update `test/e2e-v2/cases/storage/expected/config-dump.yml`.\n\n---\n\n## 8. Pre-submit checklist\n\nRun in order; each has a dedicated skill:\n\n1. [`compile`](../compile/SKILL.md) — `./mvnw clean flatten:flatten install javadoc:javadoc -B -q -Pall -Dmaven.test.skip -Dcheckstyle.skip -Dgpg.skip`. Javadoc `[ERROR] ... error:` must be fixed; `[ERROR] ... warning:` is noise.\n2. Tests: `./mvnw -pl oap-server/analyzer/log-analyzer -am test -Dtest=LALScriptExecutionTest` for LAL changes, equivalent for MAL, OAL unit tests where the rule lives.\n3. [`license`](../license/SKILL.md) — `license-eye header check` (Apache 2.0 headers). If the UI submodule bumped npm deps, also `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit the `LICENSE` diff.\n4. [`package`](../package/SKILL.md) — full `make docker` (or dist-rebuild + `make docker.oap` / `make docker.ui`). Don't trust `docker.oap` alone to pick up code changes; the dist tarball is a file-level prereq.\n5. [`run-e2e`](../run-e2e/SKILL.md) — run the new case end-to-end locally before pushing.\n6. [`gh-pull-request`](../gh-pull-request/SKILL.md) — commit and open PR with the template, attach UI screenshots.\n\n---\n\n## 9. Cross-cutting traps collected from real extensions\n\n| Trap | Symptom | Fix |\n|---|---|---|\n| Layer folder named with hyphens | `ui-initialized-templates/my-layer/*.json` on disk but dashboard empty | Folder must be `Layer.name().toLowerCase()` (underscores). `wechat_mini_program/`, not `wechat-mini-program/` |\n| Extending `CommonAnalysisListener` for a client-side layer | Added componentId → layer mapping \"to get topology for free\"; then `service_cpm` etc. don't populate anyway | Client-side (exit-only) layers don't need trace-analysis changes. Ship `Layer` + `component-libraries.yml` + MAL + LAL. See §2 header for when extending trace analysis is actually justified |\n| Dashboards wired to OAL-derived metrics on a client-side layer | Charts stay empty — `service_cpm` / `service_resp_time` / `endpoint_cpm` never populate | Those come from inbound `parseEntry`, which mini-program / browser / iOS don't emit. Use MAL metrics (`meter_<layer>_*`) instead |\n| LAL `#` comment in DSL body | OAP boot fails with `extraneous input 'metrics'` at the `#` line | LAL grammar accepts `//` and `/* */` only. No `#` comments |\n| LAL bare def-var value access — `tag 'x': myvar` | OAP boot fails with `Cannot resolve expression '[myvar]'` | `LALValueCodegen` only treats def-vars as valid when there's a method chain (`myvar?.foo`). Inline `sourceAttribute(...)` or `tag(...)` in the three value-access sites instead of storing to a def, or add a no-op chain like `myvar.toString()`. Conditions (`if (myvar == ...)`) go through a different codegen path and DO handle bare def refs |\n| MAL rule file with multi-doc YAML (`---` separator) | `MALFilterExecutionTest` fails with `expected a single document`; OAP would also fail at startup via `Rules.loadRules` (calls `Yaml.loadAs`, single-doc) | Split into one file per top-level `expSuffix` / `filter` / `metricPrefix` combo. Reference each file by name in `application.yml` (`lalFiles`, `malFiles`, `enabledOtelMetricsRules`) |\n| Histogram bucket bounds in ms with default `.histogram()` call | Percentile values come out 1000× inflated (`P50 = 100000` for a 100 ms request) | MAL's default `defaultHistogramBucketUnit = SECONDS` multiplies `le` by 1000 when converting to stored ms. For SDKs that emit ms bounds directly (mini-program, most in-house JS SDKs), use `.histogram(\"le\", TimeUnit.MILLISECONDS)`. `TimeUnit` is registered in `MALCodegenHelper.ENUM_FQCN` so rule YAML can reference it |\n| `.rate()` / `.increase()` on a DELTA counter / histogram `_count` | Compute delta-of-delta — nonsense values | OTLP DELTA means each flush is already the per-flush count. MAL's per-minute aggregator sums directly to requests-per-minute. For CUMULATIVE (Prometheus default), use `.rate(\"PT1M\")` for per-sec or `.increase(\"PT1M\")` for per-min; they are **not** equivalent (differ by 60×) |\n| MAL bucket-family name with `_histogram` suffix | Rule compiles, metric stays empty | `PrometheusMetricConverter` emits the bucket family as `<metric_name>` (no suffix), with `<metric_name>_count` and `<metric_name>_sum` as the companion families. Reference the bucket family by its bare name in the rule |\n| LAL `tag(\"key\") == null` check on a missing tag | Check never triggers — LAL's `tag()` returns empty string `\"\"` for absent tags, not `null`, so conditional abort is a no-op | Guard on both: `tag(\"key\") == null || tag(\"key\") == \"\"`. Same for `sourceAttribute(\"key\")` |\n| MAL rule file with `expSuffix: service(...)` mixing service-scoped and endpoint-scoped rules | Endpoint rules' metrics appear under the service entity (no endpoint_traffic rows created); `swctl endpoint ls` returns empty. `MetricConvert.formatExp:132` appends `(exp).expSuffix` to every rule, so `.endpoint(...)` in the rule's expression is followed by `.service(...)` and the service scope wins. | Follow the APISIX pattern — leave `expSuffix:` empty and explicitly chain `.service(...)` / `.endpoint(...)` on each rule. If every rule in a file is same-scope (e.g. all instance-scoped in `*-instance.yaml`), `expSuffix` is still fine |\n\nE2E-side authoring traps (log-content YAML quoting, nested `contains` fragility, `timeUnixNano: \"0\"`, setup-step curl swallowing failure) live in the [`run-e2e` skill §9](../run-e2e/SKILL.md).\n| Missing layer-root template | Menu item lands on empty \"no dashboard\" view | `Layer.vue:41-44` requires a template with `isRoot: true`; ship a `<layer>-root.json` (precedent: `ios/ios-root.json`) |\n| New layer not selectable in UI / swctl | `service ly <LAYER>` returns empty | Add the enum in `Layer.java` with a fresh id; see §0 |\n| Stale dist in image | `make docker.oap` succeeds but behavior is old | See [`package` skill](../package/SKILL.md); rebuild dist first, then image |\n| Shipped template edits don't render after restart | Edited JSON on disk, but OAP shows old copy | Set `SW_UI_TEMPLATE_FORCE_RELOAD=true` (environment variable, not `application.yml`) and restart; OAP writes `addOrReplace`. Unset in production so operator UI edits aren't clobbered. |\n| MAL service-scoped rule fragmented by instance | \"Overall app\" metric has N series per version | Keep `service_instance_id` out of service-scoped `.sum`/`.avg`. Put instance-scoped metrics in a separate `<feature>-instance.yaml` with `expSuffix: instance(...)` — see `otel-rules/ios/` pattern. |\n| Config dump drift | Storage e2e fails after `application.yml` edit | Mirror the change in `test/e2e-v2/cases/storage/expected/config-dump.yml` |\n| LAL `layer:auto` WARN drop | Every log silently dropped | Codegen must set `metadata.layer` alongside output builder's layer |\n| Histogram 1000× off | p50 = 2 000 000 ms | `defaultHistogramBucketUnit(MILLISECONDS)` on the `SampleFamily` |\n| Histogram shifted up | p50 = 5 000 on 500–2500 ms data | Emit per-bucket, not cumulative |\n| Empty list column | Dashboard instance table has no numbers | Add `expressions` (not just `subExpressions`) |\n| Double scroll | Inner tab scroll + page scroll | `Tab.h = content_max + 4` chrome buffer |\n| Raw metric name in legend | Multiple expressions on one widget | Add parallel `metricConfig[].label` entries |\n| `spanKind` mismatch | Span skipped silently | `OTLPSpanReaderImpl.spanKind()` returns enum `.name()` like `SPAN_KIND_CLIENT`, not `CLIENT` |\n| Negative latency | `*_sla` inflated | Clamp `rawLatency < 0 ? 0 : (int) min(Integer.MAX_VALUE, rawLatency)` |\n| SLA too optimistic | Client-side DNS / timeout / connection-refused counted as success | Require a real response code for success: `status = code > 0 && code < 400`. Do not treat missing (0) as success. |\n| Silent drop with no rules | `converters == null` NPE elsewhere | Init to empty list at declaration |\n| `+Inf` bucket surfaces as `9.2e18` | UI shows garbage when a sample lands in overflow | Use finite sentinel bucket (e.g. 30 s) |\n| OTel submodule bump misses UI image | New icons / i18n not visible | Rebuild with `-Pui,dist` + `make docker.ui` |\n| LICENSE drift after submodule bump | CI `Dependency licenses` fails | `license-eye dependency resolve --summary ./dist-material/release-docs/LICENSE.tpl` and commit |\n| Setup-step curl loop swallows failure | e2e passes metrics but fails log verify | `curl -f --retry-connrefused --retry-all-errors` + `set -e` |\n| Verify case flag mismatch | CI fails early on `swctl` flag unknown to pinned commit | Fire every verify command locally first; bump `SW_CTL_COMMIT` if needed |\n\n---\n\n## 10. Reference extensions (read their code)\n\nBrowse these when you need a template to copy from. Grouped by the kind of data source, because the extension points differ significantly:\n\n### SW native trace (language agents) — the \"General\" layer and friends\n\n`Layer.GENERAL` is the default layer for services reporting via the SkyWalking native trace protocol (Java / Python / Go / Node.js / .NET / PHP / Lua / Rust agents). It's the richest reference because the native pipeline is where the **full topology** lives:\n\n- **Service → service dependency** (topology): derived by `MultiScopesAnalysisListener` and `NetworkAddressAliasMappingListener` in `oap-server/analyzer/agent-analyzer/src/main/java/.../trace/parser/listener/`. Entry/exit spans produce `ServiceRelation`, `ServiceInstanceRelation` sources.\n- **Endpoint → endpoint dependency**: `RPCAnalysisListener` emits `EndpointRelation` sources from cross-process refs and exit spans. This is unique to the native protocol — OTLP spans don't preserve upstream entry-span metadata the same way, so OTLP-only layers generally lose endpoint-level topology.\n- OAL aggregation: `oap-server/server-starter/src/main/resources/oal/core.oal` — all `service_relation_*`, `service_instance_relation_*`, `endpoint_relation_*` metrics are already defined there; a new native-tracing layer just needs to produce the right sources with the right `Layer`.\n\nRead order: `AnalysisListenerFactory` SPI registration → `SegmentParserServiceImpl` fan-out → individual listeners → OAL rules.\n\n### Service-mesh (Istio / Envoy ALS) — topology from proxy-side signals\n\nNo agent in the service; Envoy emits HTTP/TCP access logs (ALS) and metrics. Service-to-service topology is reconstructed from ALS request/response pairs.\n\n- `oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/` — ALS receiver + `EnvoyAccessLogBuilder` persistence (LAL output type).\n- `otel-rules/istio-*`, `otel-rules/envoy/*` — MAL rules for proxy-emitted OTel metrics.\n- `lal/envoy-als.yaml` — LAL rule that extracts topology-generating fields from ALS.\n- Layer: `Layer.MESH`, plus `Layer.MESH_CP` (control plane) and `Layer.MESH_DP` (data plane) for split-plane topology.\n\nTopology parity with native traces: ALS gives you service-level relations; endpoint-level relations are approximate (proxy sees inbound + outbound, not the full call-chain that a native agent observes).\n\n### OTLP-only layers (OTel SDKs, iOS/Android, AI gateway, …)\n\nOTLP spans feed a `SpanListener`. Topology is **partial by construction** — an OTel Swift SDK on a mobile device only knows about its own outbound requests; there's no upstream caller. So these layers typically produce only `Service` / `ServiceInstance` / `Endpoint` sources (no `*Relation`) and the Topology view shows isolated islands, by design.\n\n- **iOS** (`Layer.IOS`, OTLP traces + MetricKit + LAL diagnostics): `oap-server/analyzer/ios-analyzer/` + `docs/en/swip/SWIP-11.md`.\n- **Envoy AI Gateway** (`Layer.ENVOY_AI_GATEWAY`, LLM / MCP, OTLP metrics + Zipkin sampled traces + LAL access-log sampling): `oap-server/analyzer/gen-ai-analyzer/`, `otel-rules/envoy-ai-gateway/*`, `lal/envoy-ai-gateway.yaml`.\n\n### OTel-metrics-only layers (scraped exporters)\n\nNo spans, no logs — just MAL rules converting OTel metric scrapes.\n\n- **MySQL / PostgreSQL**: `otel-rules/mysql/*`, `otel-rules/postgresql/*`.\n- **Kubernetes**: `otel-rules/k8s/*` — paired with `virtual-k8s-service` (`Layer.K8S_SERVICE`) sources for synthetic services.\n- **AWS**: `otel-rules/aws-eks/*`, `otel-rules/aws-s3/*`, `otel-rules/aws-dynamodb/*`.\n\n### SW native meter (custom protocol)\n\nNo OTel; the agent / plugin sends pre-built meter samples on the SW `MeterProtocol`.\n\n- **Nginx Lua** (`Layer.NGINX`): `meter-analyzer-config/nginx-service.yaml`, with the Lua agent in `apache/skywalking-nginx-lua`.\n- **APISIX** (`Layer.APISIX`): `meter-analyzer-config/apisix-service.yaml` + LAL trace extraction.\n\n### Which to copy from\n\n| You're adding | Closest reference |\n|---|---|\n| Language agent (new JVM / runtime) | The General / Java layer flow; extend `AnalysisListener` SPI. |\n| Service mesh / proxy | `mesh-*` + `envoy-*` rules; LAL on ALS. |\n| Mobile / browser / mini-program SDK (OTLP) | `ios-analyzer` (OTLP client-side). Expect no topology relations. |\n| Database / middleware (metrics only) | `mysql`, `postgresql` — pure MAL rules. |\n| Cloud service (no agent, just metrics) | `aws-eks` / `aws-s3`. |\n| Custom protocol / home-grown telemetry | `apisix` / `nginx` — SW native meter. |\n\n<!-- File: .claude/skills/package/SKILL.md (Tokens: ~1374 | Category: anthropic-skill) -->\n---\nname: package\ndescription: Rebuild the SkyWalking distribution and OAP Docker image after source changes. Use before running e2e tests so the image reflects your code changes. Avoids the \"image looks updated but runtime has stale jars\" trap.\nargument-hint: \"[oap|all|dist-only]\"\n---\n\n# Package OAP Distribution & Docker Image\n\nRebuilding the Docker image after a source-code change has a **two-step dependency**: rebuild the dist tarball, then rebuild the image. Skipping the first step silently produces an image with stale jars — the Docker build still \"succeeds,\" the image has a fresh timestamp, but the embedded jar is the one from your previous build.\n\n## Why this matters\n\n`make docker.oap` in the root `Makefile` is defined as:\n\n```\ndocker.% push.docker.%: $(CONTEXT)/$(DIST) $(SW_ROOT)/docker/%/*\n    $(DOCKER_RULE)\n```\n\nIt depends on `$(CONTEXT)/$(DIST)` (the dist tarball) **as a file prerequisite**. If that tarball already exists, make does not regenerate it — it just copies whatever is on disk into `dist/docker_build/oap/` and runs `docker buildx build`. There is **no** dependency that rebuilds the tarball from source.\n\nOnly `make docker` triggers the full chain (`init → build.all → docker.all`). So:\n\n| Command | Rebuilds source? | Rebuilds tarball? | Rebuilds image? |\n|---------|:--:|:--:|:--:|\n| `./mvnw -pl <module> package` | only that module | no | no |\n| `./mvnw -pl apm-dist -am package -Pbackend,dist` | all backend deps + dist | yes | no |\n| `make docker.oap` | **no** | **no** (uses whatever's on disk) | yes |\n| `make docker` | yes (`build.all`) | yes | yes (`docker.all`) |\n\n## `flatten:flatten` — always run it before `package`/`install`\n\nSkyWalking's poms use `${revision}` as a placeholder version (e.g., `10.5.0-SNAPSHOT`). The `flatten-maven-plugin` resolves `${revision}` into concrete versions and writes a `.flattened-pom.xml`. Without it:\n\n- Installed artifacts carry the literal string `${revision}` in their coordinates and cannot be resolved as dependencies.\n- Downstream modules (including `apm-dist`) see an inconsistent dependency graph.\n- Symptoms are subtle: `-pl <module> -am` may succeed in isolation but fail or pull in stale transitive artifacts when the same module is consumed by another build invocation in the same session.\n\n**Always run `flatten:flatten` in the same goal chain as `package` or `install`.** The CI `dist-tar` job and the `compile` skill both do this. Example:\n\n```bash\n./mvnw clean flatten:flatten package -Pbackend,dist -DskipTests\n```\n\nNot optional — treat it as part of `package`.\n\n## Pick the right command\n\n### Changed OAP source → want a new image\n\n```bash\n# Recommended: full chain\nmake docker\n```\n\nor, faster and equivalent for backend-only changes (skips UI):\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true\nmake docker.oap\n```\n\n**Do not** just run `make docker.oap` after a code edit — the image will look rebuilt but your change will not be in the jars.\n**Do not** skip `flatten:flatten` — see the section above.\n\n### Changed only the dist packaging (e.g., `apm-dist/`, log4j2.xml)\n\n```bash\n./mvnw -pl apm-dist -am -o clean flatten:flatten package -Pbackend,dist -DskipTests\nmake docker.oap\n```\n\n### Changed only Dockerfile / entrypoint (`docker/oap/*`)\n\nThe tarball is untouched, so `make docker.oap` alone is correct:\n\n```bash\nmake docker.oap\n```\n\n## Verify the fix reached the image\n\nDocker's \"image created\" timestamp updates even when the content is stale (because `--no-cache` is used). **Don't trust the timestamp.** Verify the jar contents directly:\n\n```bash\n# Copy the jar out of the container\ndocker cp <container-name>:/skywalking/oap-libs/<module>-<version>.jar /tmp/verify.jar\n\n# Extract the specific class\ncd /tmp && jar -xf verify.jar <path/to/YourClass.class>\n\n# Grep for a string your fix introduced (unique literal, method name, etc.)\ngrep -oa \"myFixSignature\" <path/to/YourClass.class>\n```\n\nIf grep finds nothing, the image does not contain your fix — you forgot to rebuild the dist. Re-run with the correct chain.\n\n## Common pitfalls\n\n- **`make docker.oap` after `./mvnw package`**: The module jar is fresh in `oap-server/.../target/`, but the dist tarball at `dist/apache-skywalking-apm-bin.tar.gz` is untouched. Image uses the old jar.\n- **Cancelling a `make docker` mid-flight**: leaves the dist half-rebuilt. Re-run `make docker` or delete `dist/` first.\n- **Stale buildx cache**: not usually the issue (the Makefile passes `--no-cache`), but if you suspect it, run `docker buildx prune`.\n- **Trusting image timestamps**: `docker images` shows when the image was *tagged*, not when its content actually changed. A no-op rebuild still updates the timestamp.\n- **Interactive buildx hangs**: if `docker buildx build` sits silent for minutes with no stdout, kill it and retry. The buildx container driver occasionally stalls; a restart is faster than waiting.\n\n## The golden rule\n\nAfter any Java source edit that affects code shipped in `oap-libs/`, run one of:\n\n1. `make docker` (full, safest)\n2. `./mvnw -pl apm-dist -am -o package -Pbackend,dist -DskipTests -Dcheckstyle.skip=true && make docker.oap` (fast)\n\nThen **verify the fix is in the image's jar** before running e2e. Don't assume.\n\n<!-- File: .claude/skills/run-e2e/SKILL.md (Tokens: ~3365 | Category: anthropic-skill) -->\n---\nname: run-e2e\ndescription: Run SkyWalking E2E tests locally\ndisable-model-invocation: true\nargument-hint: \"[test-case-path]\"\n---\n\n# Run SkyWalking E2E Test\n\nRun an E2E test case using `skywalking-infra-e2e`. The user provides a test case path (e.g., `simple/jdk`, `storage/banyandb`, `alarm`).\n\n## Prerequisites\n\nAll tools require **Go** installed. Check `.github/workflows/` for the exact `e2e` commit used in CI.\n\n### e2e CLI\n\nBuilt from [apache/skywalking-infra-e2e](https://github.com/apache/skywalking-infra-e2e), pinned by commit in CI:\n\n```bash\n# Install the pinned commit\ngo install github.com/apache/skywalking-infra-e2e/cmd/e2e@<commit-id>\n\n# Or clone and build locally (useful when debugging the e2e tool itself)\ngit clone https://github.com/apache/skywalking-infra-e2e.git\ncd skywalking-infra-e2e\ngit checkout <commit-id>\nmake build\n# binary is in bin/e2e — add to PATH or copy to $GOPATH/bin\n```\n\n### swctl, yq, and other tools\n\nE2E test cases run pre-install steps (see `setup.steps` in each `e2e.yaml`) that install tools into `/tmp/skywalking-infra-e2e/bin`. When running locally, you need these tools on your PATH.\n\n**swctl** — SkyWalking CLI, used in verify cases to query OAP's GraphQL API. Pinned at `SW_CTL_COMMIT` in `test/e2e-v2/script/env`:\n\n```bash\n# Option 1: Use the install script (same as CI)\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: Build from source\ngo install github.com/apache/skywalking-cli/cmd/swctl@<SW_CTL_COMMIT>\n```\n\n**yq** — YAML processor, used in verify cases:\n\n```bash\n# Option 1: Use the install script\nbash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq\nexport PATH=/tmp/skywalking-infra-e2e/bin:$PATH\n\n# Option 2: brew install yq (macOS)\n```\n\n**Other tools** (only needed for specific test cases):\n\n| Tool | Install script | Used by |\n|------|---------------|---------|\n| `kubectl` | `install.sh kubectl` | Kubernetes-based tests |\n| `helm` | `install.sh helm` | Helm chart tests |\n| `istioctl` | `install.sh istioctl` | Istio/service mesh tests |\n| `etcdctl` | `install.sh etcdctl` | etcd cluster tests |\n\nAll install scripts are at `test/e2e-v2/script/prepare/setup-e2e-shell/`.\n\n## Steps\n\n### 1. Determine the test case\n\nResolve the user's argument to a full path under `test/e2e-v2/cases/`. If ambiguous, list matching directories and ask.\n\n```bash\nls test/e2e-v2/cases/<argument>/e2e.yaml\n```\n\n### 2. Check if rebuild is needed\n\nCompare source file timestamps against the last build:\n\n```bash\n# OAP server changes since last build\nfind oap-server apm-protocol -type f \\( \\\n  -name \"*.java\" -o -name \"*.yaml\" -o -name \"*.yml\" -o \\\n  -name \"*.json\" -o -name \"*.xml\" -o -name \"*.properties\" -o \\\n  -name \"*.proto\" \\\n\\) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null | head -5\n\n# Test service changes since last build\nfind test/e2e-v2/java-test-service -type f \\( \\\n  -name \"*.java\" -o -name \"*.xml\" -o -name \"*.yaml\" -o -name \"*.yml\" \\\n\\) -newer test/e2e-v2/java-test-service/e2e-service-provider/target/*.jar 2>/dev/null | head -5\n```\n\nIf files are found, warn the user and suggest rebuilding before running.\n\n### 3. Rebuild if needed (only with user confirmation)\n\n```bash\n# Rebuild OAP\n./mvnw clean flatten:flatten package -Pall -Dmaven.test.skip && make docker\n\n# Rebuild test services\n./mvnw -f test/e2e-v2/java-test-service/pom.xml clean flatten:flatten package\n```\n\n### 4. Run the E2E test\n\nSet required environment variables and run:\n\n```bash\nexport SW_AGENT_JDK_VERSION=8\ne2e run -c test/e2e-v2/cases/<case-path>/e2e.yaml\n```\n\n### 5. If the test fails\n\nDo NOT run cleanup immediately. Instead:\n\n1. Check container logs:\n   ```bash\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs oap\n   docker compose -f test/e2e-v2/cases/<case-path>/docker-compose.yml logs provider\n   ```\n\n2. Run verify separately (can retry after investigation):\n   ```bash\n   e2e verify -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n3. Only cleanup when done debugging:\n   ```bash\n   e2e cleanup -c test/e2e-v2/cases/<case-path>/e2e.yaml\n   ```\n\n### 6. Manually fire each verify query (fast triage)\n\nThe `e2e verify` retry loop runs in sequence and stops at the first failing case, so a single bad query hides every case after it. When a verify fails, **run each verify case directly against the still-running OAP** before editing anything — you'll see the real error (bad flag, missing data, wrong expected), not the progress spinner. This is also the right way to author new verify cases: craft the query against live OAP, confirm the actual YAML, then write the expected file.\n\n```bash\n# Find the host-side port that infra-e2e bound to OAP's container port 12800.\n# (Each run picks a new random port; the trigger log prints it too.)\ndocker ps --filter \"name=skywalking_e2e-oap\" --format \"{{.Ports}}\" \\\n  | grep -oE \"[0-9]+->12800\" | head -1\n# => e.g. 56381->12800\n\nURL=http://localhost:56381/graphql\nSWCTL=/tmp/skywalking-infra-e2e/bin/swctl\n\n# Copy the query from e2e.yaml verbatim, then substitute ${oap_host} → localhost\n# and ${oap_12800} → the port you just found:\n$SWCTL --display yaml --base-url=$URL service ly IOS\n$SWCTL --display yaml --base-url=$URL logs list --service-name=MyiOSApp\n$SWCTL --display yaml --base-url=$URL metrics exec --expression=service_cpm --service-name=MyiOSApp\n```\n\nWhen a `swctl` subcommand rejects a flag (`Incorrect Usage: flag provided but not defined: -layer`), the e2e config is using syntax the pinned `swctl` commit doesn't support. Find the right syntax with `swctl <cmd> --help` and update the e2e config. Common cases encountered:\n\n| Broken flag/form | Working form |\n|---|---|\n| `service ls --layer IOS` | `service ly IOS` |\n| `metrics exec ... --is-normal=true` | drop `--is-normal` (default behavior) |\n\nFor queries that *don't* use `swctl` (raw `curl` against `/loki/...`, Zipkin, PromQL), hit the matching exposed port:\n\n```bash\ncurl \"http://localhost:$(docker ps --filter name=skywalking_e2e-oap --format '{{.Ports}}' | grep -oE '[0-9]+->3100' | head -1 | cut -d'-' -f1)/loki/api/v1/labels\"\n```\n\n### 7. UI template changes require a fresh DB\n\n`UITemplateInitializer.initTemplate()` (in `oap-server/server-core`) calls `uiTemplateManagementService.addIfNotExist(setting)` — keyed by the `id` field in each `ui-initialized-templates/**/*.json`. Same ID → skipped. So edits to an *existing* template JSON (adding widgets, relabeling, changing expressions) will **not** be applied on an already-initialized OAP, even after a container restart, because the old copy still lives in storage.\n\nTo pick up dashboard JSON changes:\n\n```bash\n# Remove both containers — BanyanDB stores state inside the container FS in the\n# e2e compose (no named volume), so removing the container wipes state cleanly.\ndocker rm -f skywalking_e2e-oap-1 skywalking_e2e-banyandb-1\n\n# For compose setups that use a named volume, also:\n# docker volume rm <volume-name>\n\n# Then re-run — OAP sees empty storage, loads the new template JSON.\ne2e run -c test/e2e-v2/cases/<case>/e2e.yaml\n```\n\nSymptom to watch for: you edit the JSON, rebuild, redeploy — dashboard in the UI still shows the pre-edit layout. That's not a caching bug; that's `addIfNotExist` doing exactly what its name says.\n\n### 8. Author the expected YAML from live output\n\nFor a new verify case, the workflow is:\n\n1. Fire the query manually (see step 6) and capture the YAML.\n2. Pick which fields are *meaningful domain values* (must match exactly) vs *dynamic runtime values* (`notEmpty` / `gt` / `ge`). See `test/e2e-v2/CLAUDE.md` for the decision guide.\n3. Write the expected file. If the response is a list, wrap the items in `{{- contains . }} ... {{- end }}` so ordering and extra actual items don't fail the match.\n4. Re-run `e2e verify` alone (the containers are still up from the previous run); iterate on the expected file without rebuilding.\n\n### 9. Expected-file authoring traps\n\nThese burn CI cycles and pass locally. Each was learned the hard way.\n\n**Unquoted `content: {{ notEmpty .content }}` with `:` inside the value.** Sim-generated or real log content routinely includes colons (`POST https://api.example.com/cart failed: 500`, `HTTP/1.1 500: Internal Server Error`). Without quoting, the template renders to invalid YAML (snakeyaml parses `failed:` as a nested key) and the *whole log entry* marshals to `nil`. Symptom: diff shows `- nil` at *every* position in the expected logs list vs real maps in actual. Fix: wrap in single quotes — `content: '{{ notEmpty .content }}'`. Single-quoted YAML preserves `:` in the scalar; only fails on embedded `'`. Double quotes also work unless the content has `\"`.\n\n**Nested `contains` with multiple per-element pattern assertions against a varied stream.** The template renders the block body once per actual element; when the outer block body has *multiple* inner `contains` patterns asserting specific tag key/value pairs, and only *some* actuals satisfy all the inner patterns, `go-cmp` with `contains` can end up comparing `[rendered_for_A0, nil, nil, ...]` vs `[A0, A1, A2, ...]` and fail despite `contains` being permissive on extras. Specifically: outer `contains .logs` with a single log pattern + inner `contains .tags` asserting two distinct key/value pairs. On a simulator emitting heterogeneous errors (js + promise + ajax + pageNotFound), only a subset satisfy the inner assertion. Passes locally with 1–2 logs, fails in CI with 6+.\n- Keep the outer `contains` body lenient: field-shape checks (`notEmpty`, `gt`), one discriminator tag that *every* element in the stream carries.\n- Cover per-category assertions via separate labeled-metric verify cases, not inside the nested template.\n- Rule of thumb: \"at least one log exists with the right layer routing\" inside the logs expected; per-category coverage via `meter_*_count{label=X}` verify cases.\n\n**Hand-crafted OTLP curl payloads drift from real SDK output.** When the upstream SDK ships a published simulator image (mini-program-monitor's `sim-wechat` / `sim-alipay`, browser-client-js sim, etc.), prefer driving the e2e with that image in `MODE=timed` with a bounded `DURATION_MS` over hand-rolling the OTLP JSON. Hand-crafted payloads miss real-world shape issues: delta-vs-cumulative temporality, label-cardinality surprises, stacktrace formatting variance, attribute key names that changed between SDK versions. Pin to a released tag (`v0.4.0`), not `:latest` or HEAD SHA — reproducibility.\n\n**`timeUnixNano: \"0\"` in an OTLP metric datapoint.** The receiver propagates this into MAL's bucket computation and the metric lands in the 1970 time bucket — `swctl metrics exec` over the \"last N minutes\" window won't find it. Either use `$(date +%s)000000000` at setup time or omit the field if the receiver accepts \"now\" as default.\n\n**Setup-step curl loop with `|| sleep` pattern.** The shell line `for ... do curl && break || sleep 5; done` exits 0 when every attempt connection-refused because the final `sleep 5` returns 0. OAP takes ~50 s to start in CI, so all attempts fail before OAP is ready, and the setup step silently succeeds with zero traffic ingested. Fix: `curl -sS -f --retry 30 --retry-delay 5 --retry-connrefused --retry-all-errors --max-time 10 ...` + `set -e` at step top.\n\n**`swctl` flag rejected.** If a verify case uses a flag the pinned `swctl` commit doesn't support (`service ls --layer` vs `service ly`), the whole case fails 20× before CI gives up. Fire each verify query by hand once before pushing (step 6 above).\n\n**Published image cache miss in CI.** `docker compose pull` sometimes hits rate limits or unreachable registries; the test spins until timeout with \"dependency failed to start\". Look at the CI log for `Error response from daemon: pull access denied` or `manifest unknown`. If you see that, pin a different image tag that's definitely published (check `docker manifest inspect <tag>` locally), not a floating one.\n\n## Common test cases\n\n| Shorthand | Path |\n|-----------|------|\n| `simple/jdk` | `test/e2e-v2/cases/simple/jdk/` |\n| `storage/banyandb` | `test/e2e-v2/cases/storage/banyandb/` |\n| `storage/elasticsearch` | `test/e2e-v2/cases/storage/elasticsearch/` |\n| `alarm` | `test/e2e-v2/cases/alarm/` |\n| `log` | `test/e2e-v2/cases/log/` |\n| `profiling/trace` | `test/e2e-v2/cases/profiling/trace/` |\n\n<!-- File: .claude/skills/test/SKILL.md (Tokens: ~653 | Category: anthropic-skill) -->\n---\nname: test\ndescription: Run unit tests, integration tests, or slow integration tests matching CI. Use to validate changes before submitting a PR.\nargument-hint: \"[unit|integration|slow|module-name]\"\n---\n\n# Tests\n\nRun tests matching CI configuration.\n\n## Commands by argument\n\n### `unit` or no argument — unit tests\n\n```bash\n./mvnw clean test -q -B -D\"checkstyle.skip\"\n```\n\nCI runs this on:\n- JDK 11: ubuntu, macOS, Windows\n- JDK 17, 21, 25: ubuntu only\n\n### `integration` — integration tests (excludes slow)\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow\n```\n\nCI runs this on JDK 11, 17, 21, 25 (ubuntu only).\n\n### `slow` — slow integration tests\n\n```bash\n./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true \\\n  -Dit.test=org.apache.skywalking.library.elasticsearch.ElasticSearchIT \\\n  -Dfailsafe.failIfNoSpecifiedTests=false\n```\n\nCI runs on JDK 11 (ubuntu only). Currently only ElasticSearch/OpenSearch IT is in the slow matrix.\n\n### Module name — single module tests\n\n```bash\n# Unit tests for a specific module\n./mvnw clean test -pl oap-server/analyzer/<module-name> -D\"checkstyle.skip\"\n\n# Integration tests for a specific module\n./mvnw -B clean integration-test -pl oap-server/analyzer/<module-name> -Dcheckstyle.skip -DskipUTs=true\n```\n\n## Test frameworks\n\n- JUnit 5 (`org.junit.jupiter`)\n- Mockito for mocking\n- AssertJ for assertions\n- PowerMock for reflection utilities\n\n## Test naming conventions\n\n| Type | Pattern | Maven phase |\n|------|---------|-------------|\n| Unit tests | `*Test.java` | `test` |\n| Integration tests | `IT*.java` or `*IT.java` | `integration-test` |\n\n## Slow test tagging\n\nTests tagged with `@Tag(\"slow\")` are excluded from the normal integration-test job and run separately in the slow-integration-test job. Use this tag for tests that spin up external services (Elasticsearch, etc.) and take significant time.\n\n## CI retry behavior\n\nAll three CI jobs retry on failure (run the same command twice with `||`). This handles flaky tests but masks intermittent issues — if a test fails locally, investigate rather than relying on retries.\n\n## CI reference\n\nCI workflow: `.github/workflows/skywalking.yaml`\n\n| Job | JDK | OS | Timeout |\n|-----|-----|----|---------|\n| `unit-test` | 11 (3 OS) + 17, 21, 25 (ubuntu) | ubuntu, macOS, Windows | 30 min |\n| `integration-test` | 11, 17, 21, 25 | ubuntu | 60 min |\n| `slow-integration-test` | 11 | ubuntu | 60 min |\n\n<!-- File: CLAUDE.md (Tokens: ~6158 | Category: claude-rule) -->\n# CLAUDE.md - AI Assistant Guide for Apache SkyWalking\n\nThis file provides guidance for AI assistants working with the Apache SkyWalking codebase.\n\n## Project Overview\n\nApache SkyWalking is an open-source APM (Application Performance Monitoring) system designed for microservices, cloud-native, and container-based architectures. It provides distributed tracing, service mesh telemetry analysis, metrics aggregation, alerting, and observability capabilities.\n\n## Repository Structure\n\n```\nskywalking/\n├── oap-server/                    # OAP (Observability Analysis Platform) backend server\n│   ├── server-core/               # Core module with fundamental services\n│   ├── server-library/            # Shared libraries (module system, util, etc.)\n│   ├── server-receiver-plugin/    # Data receivers (gRPC, HTTP, Kafka, etc.)\n│   ├── server-storage-plugin/     # Storage implementations (BanyanDB, Elasticsearch, etc.)\n│   ├── server-cluster-plugin/     # Cluster coordination (Zookeeper, K8s, etc.)\n│   ├── server-query-plugin/       # Query interfaces (GraphQL)\n│   ├── server-alarm-plugin/       # Alerting system\n│   ├── server-fetcher-plugin/     # Data fetchers\n│   ├── server-configuration/      # Dynamic configuration providers\n│   ├── oal-grammar/               # OAL (Observability Analysis Language) grammar\n│   ├── oal-rt/                    # OAL runtime\n│   ├── mqe-grammar/               # MQE (Metrics Query Engine) grammar\n│   ├── mqe-rt/                    # MQE runtime\n│   ├── server-testing/             # Shared test utilities (DSL test framework)\n│   ├── analyzer/                  # Log and trace analyzers\n│   ├── ai-pipeline/               # AI/ML pipeline components\n│   ├── exporter/                  # Data export functionality\n│   └── server-tools/              # Standalone tools (profile exporter) with mock providers\n├── apm-protocol/                  # Protocol definitions (submodule)\n│   └── apm-network/               # gRPC/Protobuf network protocols\n├── skywalking-ui/                 # Web UI (submodule - skywalking-booster-ui)\n├── apm-webapp/                    # Web application packaging\n├── apm-dist/                      # Distribution packaging\n├── docs/                          # Documentation\n├── docker/                        # Docker build files\n├── test/                          # E2E and integration tests\n└── tools/                         # Development tools\n```\n\n## Architecture & Key Concepts\n\n### Module System\nSkyWalking uses a custom module/provider architecture based on Java SPI:\n\n- **ModuleDefine**: Declares a module and its required services\n- **ModuleProvider**: Implements a module with specific technology/approach\n- **Service**: Interface that modules expose to other modules\n\nKey pattern:\n```java\npublic class XxxModule extends ModuleDefine {\n    public Class[] services() {\n        return new Class[] { XxxService.class };\n    }\n}\n\npublic class XxxModuleProvider extends ModuleProvider {\n    public void prepare() { /* initialize */ }\n    public void start() { /* start services */ }\n}\n```\n\n### Core Concepts\n- **OAL (Observability Analysis Language)**: DSL for defining metrics aggregation rules\n- **MQE (Metrics Query Engine)**: DSL for querying metrics\n- **LAL (Log Analysis Language)**: DSL for log processing\n- **MAL (Meter Analysis Language)**: DSL for meter data processing\n- **Source/Scope**: Data model definitions for telemetry data\n- **Stream Processing**: Metrics, Records, and TopN processing pipelines\n\n### Data Flow\n1. Agents/Collectors send data via gRPC/HTTP/Kafka\n2. Receiver plugins parse and validate data\n3. Analysis engine processes data using OAL/LAL/MAL\n4. Storage plugins persist aggregated data\n5. Query plugins serve data to UI/API\n\n## Code Style & Conventions\n\n### Checkstyle Rules (enforced via `apm-checkstyle/checkStyle.xml`)\n\n**Prohibited patterns:**\n- No `System.out.println` - use proper logging (SLF4J)\n- No `@author` tags - ASF projects don't use author annotations\n- No Chinese characters in source files\n- No tab characters (use 4 spaces)\n- No star imports (`import xxx.*`)\n- No unused or redundant imports\n- No empty statements (standalone `;`)\n- No fully-qualified class names inline in code — always add an `import` statement and\n  use the short name. Acceptable exceptions: (a) two classes with the same simple name\n  would collide if both imported, (b) the class appears exactly once in a Javadoc\n  `{@link}` where the short name would be ambiguous to the reader, (c) the name is inside\n  GENERATED source — a codegen string literal or a `.ftl` template. Field declarations,\n  method signatures, local variables, and generic type arguments should always use the\n  imported short name — `private RemoteClientManager rcm;`, not `private\n  org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager rcm;`.\n  - Exception (c) is a Javassist constraint, not a preference: its compiler has no\n    `import` statement and resolves simple names only against `java.lang`\n    (`ClassPool.importedPackages`; widening it needs `importPackage`, which is pool-wide\n    mutable state and is never called here). So generated source must fully qualify\n    everything, `java.util.Map` included. Shortening an FQCN in a codegen string or a\n    template compiles as Java and then fails at runtime — audit real Java only.\n- No one-line delegate methods. A wrapper whose only body is a single forwarding call\n  to another class (`return Other.foo(a, b);`) adds a hop without value. Inline the\n  call at the use site, or call the underlying object directly (including via method\n  reference: `obj::foo` instead of `this::wrapperOfFoo`).\n\n**Required patterns:**\n- `@Override` annotation required for overridden methods\n- `equals()` and `hashCode()` must be overridden together\n- Javadoc `@param`, `@return`, `@throws` must have descriptions\n- Long constants must use uppercase `L` (e.g., `100L` not `100l`)\n- `default` case must come last in switch statements\n- One statement per line\n\n**Naming conventions:**\n- Constants/static variables: `UPPER_CASE_WITH_UNDERSCORES`\n- Type parameters: `UPPER_CASE` (e.g., `TYPE`, `KEY`, `VALUE`)\n- Package names: `org.apache.skywalking.*` or `test.apache.skywalking.*`\n- Type names: `PascalCase` or `UPPER_CASE_WITH_UNDERSCORES`\n- Local variables/parameters/members: `camelCase`\n- **Function-oriented naming, not abstract metaphor**: classes and methods are named for\n  what they do, not for an abstract concept. Prefer concrete verbs (`load`, `apply`,\n  `unregister`, `compile`, `verify`, `commit`, `rollback`) over metaphorical ones\n  (`seed`, `hydrate`, `bootstrap`, `prime`). Class names follow the same rule —\n  `StaticRuleLoader` (loads static rules), not `StaticBundleSeeder`; `DSLSyncTimer` (syncs\n  DB → state on a timer), not `TickRunner`. If you can't name a method without reaching\n  for a metaphor, the method is probably doing too much; split it.\n\n**File limits:**\n- Max file length: 3000 lines\n\n**Whitespace:**\n- Whitespace required after commas, semicolons, type casts\n- Whitespace required around operators\n- No multiple consecutive blank lines\n- Empty line separators between class members (fields can be grouped)\n\n### Code Style (via `codeStyle.xml` for IntelliJ IDEA)\n\n**Indentation:**\n- 4-space indentation\n- 4-space continuation indent\n\n**Imports:**\n- No star imports (threshold set to 999)\n- Import order: regular imports, blank line, static imports\n\n**Formatting:**\n- `while` in do-while on new line\n- Align multiline chained method calls\n- Align multiline parameters in calls\n- Array initializer braces on new lines\n- Wrap long method call chains\n\n**General:**\n- Use `final` for local variables and parameters\n- Use Lombok annotations (`@Getter`, `@Setter`, `@Builder`, `@Data`, `@Slf4j`, etc.)\n- Follow existing patterns in similar files\n\n**Comments:**\n- Don't explain what the code already says. Comment only when reading the code is\n  not enough — invariants that aren't visible in this method, cross-cutting\n  ordering, why a non-obvious branch exists, why a seemingly redundant call is\n  required. If the code is clear, no comment.\n- Connect logic that lives in different places. A check here that exists only\n  because of a guarantee made elsewhere deserves a one-line pointer to that\n  guarantee. A workaround for a known bug elsewhere deserves a link.\n- Trim restated javadoc. Method-level docs should describe the contract, not\n  re-narrate the body. Multi-paragraph rationale belongs in the design doc, not\n  the source.\n\n**Tests:**\n- Always use JUnit assertions (`assertTrue`, `assertEquals`, `assertThrows`, …).\n  Never write bare Java `assert` statements in tests. Although surefire defaults\n  `enableAssertions=true` so bare `assert` does fire in this project's tests,\n  JUnit assertions are the convention everywhere else in the codebase, survive\n  any test-runner configuration, and produce a structured failure that names\n  the actual value instead of a context-free `AssertionError`.\n\n### License Header\nJava, XML, and YAML/YML files must include the Apache 2.0 license header (see `HEADER` file).\nJSON and Markdown files are excluded (JSON doesn't support comments, see `.licenserc.yaml`).\n\n### JDK 11 Compatibility\n\nAll code must be compatible with JDK 11 (LTS). The project supports JDK 11, 17, and 21.\n\n**Prohibited Java features (post-JDK 11):**\n\n| Feature | JDK Version | Use Instead |\n|---------|-------------|-------------|\n| Switch expressions (`->`) | 14+ | Traditional `switch` with `case:` and `break` |\n| `Stream.toList()` | 16+ | `.collect(Collectors.toList())` |\n| Text blocks (`\"\"\"...\"\"\"`) | 15+ | String concatenation or `+` |\n| Records | 14+ | Regular classes with Lombok `@Data` |\n| Pattern matching for `instanceof` | 14+ | Traditional cast after `instanceof` |\n| Sealed classes/interfaces | 15+ | Regular classes/interfaces |\n\n**Allowed Java features (JDK 11 compatible):**\n- `List.of()`, `Set.of()`, `Map.of()` - Immutable collections (Java 9+)\n- `Optional` methods - `orElseThrow()`, `ifPresentOrElse()` (Java 9+)\n- Lambda expressions and method references (Java 8+)\n- Stream API (Java 8+)\n- Lombok annotations (`@Getter`, `@Builder`, `@Data`, `@Slf4j`)\n\n**Verification commands:**\n```bash\n# Check for switch expressions (should return no matches)\ngrep -r \"switch.*->\" src/ --include=\"*.java\"\n\n# Check for Stream.toList() (should return no matches)\ngrep -r \"\\.toList()\" src/ --include=\"*.java\"\n\n# Check for text blocks (should return no matches)\ngrep -r '\"\"\"' src/ --include=\"*.java\"\n```\n\n## Git Submodules\n\nThe project uses submodules for protocol definitions and UI:\n- `apm-protocol/apm-network/src/main/proto` - skywalking-data-collect-protocol\n- `oap-server/server-query-plugin/.../query-protocol` - skywalking-query-protocol\n- `skywalking-ui` - skywalking-booster-ui\n- `oap-server/server-library/library-banyandb-client/src/main/proto` - banyandb-client-proto\n\nAlways use `--recurse-submodules` when cloning or update submodules manually.\n\n## Key Files for Understanding the Codebase\n\n- `oap-server/server-core/src/main/java/.../CoreModule.java` - Core module definition\n- `oap-server/server-library/library-module/src/main/java/.../ModuleDefine.java` - Module system base\n- `oap-server/oal-grammar/src/main/antlr4/.../OALParser.g4` - OAL grammar definition\n- `oap-server/server-starter/` - Application entry point\n- `docs/en/concepts-and-designs/` - Architecture documentation\n\n## Common Development Tasks\n\n### Adding a New Receiver Plugin\n1. Create module in `server-receiver-plugin/`\n2. Implement `ModuleDefine` and `ModuleProvider`\n3. Register via SPI in `META-INF/services/`\n4. Add configuration to `application.yml`\n\n### Adding a New Storage Plugin\n1. Create module in `server-storage-plugin/`\n2. Implement storage DAOs for each data type\n3. Follow existing plugin patterns (e.g., BanyanDB, elasticsearch)\n\n### Modifying OAL Metrics\n1. Edit `.oal` files in `oap-server/server-starter/src/main/resources/oal/`\n2. Regenerate by building the project\n3. Update storage schema if needed\n\n### MAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `otel-rules/` - OpenTelemetry metrics (Prometheus, etc.)\n- `meter-analyzer-config/` - SkyWalking native meter protocol\n\n### LAL Scripts (in `oap-server/server-starter/src/main/resources/`)\n- `lal/` - Log processing rules\n- `log-mal-rules/` - Metrics extracted from logs\n\n## Documentation (in `docs/en/`, structure defined in `docs/menu.yml`)\n\n- `concepts-and-designs/` - Architecture and core concepts (OAL, MAL, LAL, profiling)\n- `setup/` - Installation and configuration guides\n- `api/` - Telemetry and query protocol documentation\n- `guides/` - Contributing guides, build instructions, testing\n- `changes/changes.md` - Changelog (update when making changes). Keep each bullet's **prose on one line**\n  — don't hard-wrap a sentence across source lines (nested sub-bullets `  - …` are fine; they're real list\n  items). The changelog becomes the GitHub release body, which GitHub renders as GFM where a single\n  newline continuing a bullet becomes a `<br>`, so prose-wrapped bullets show jagged mid-sentence breaks on\n  the release page (verified on v10.4.0; the docs website reflows and hides the problem). See\n  `guides/How-to-release.md`. Append a new entry at the **end** of its section, never at the top —\n  the section reads in the order the changes landed.\n- `swip/` - SkyWalking Improvement Proposals\n\n### SWIP vs. operator docs\n\nThe two kinds of document answer different questions and have different lifetimes:\n\n- **`docs/en/swip/SWIP-N.md` records the design discussion up to first acceptance.** It is a\n  historical record of what was proposed and agreed, not a description of current behaviour.\n- **The operator / end-user docs describe how the system behaves today.** They are the only\n  place a reader should be sent for current truth.\n\nWhen the implementation of a SWIP lands, sync the SWIP once to what was actually built — every\nproposed change written in implemented/past tense, with **no `TODO` / \"future work\" /\n\"open dependency\" / \"lands later\" / \"empty until X\" language for anything in the SWIP's\nscope**. **Once that SWIP ships as part of a release, it is frozen and must not be edited again.**\n\nFrozen means frozen even when the SWIP's text becomes factually stale. A later refactor that\nrenames a package, moves a class, deletes a constant or changes a class-naming scheme **does not**\nlicense updating the SWIP to match — not to keep code references accurate, not for internal\nconsistency, and not because the same PR already touched it. Ask instead: *would an end user read\nthis to learn how the system works today?* If yes it is an operator doc and must be updated; if it\nis a SWIP the answer is no, and the fix belongs in the operator doc plus `changes.md`.\n\nAfter that first sync, further metrics, dashboards, and incremental enhancements go into the\n**operator doc**, not the SWIP. Only genuinely small, optional follow-ups may stay under a SWIP's\n\"future work\" — never a big change.\n\n- BanyanDB self-observability: the SWIP is `docs/en/swip/SWIP-15.md`; the living operator\n  catalog is `docs/en/banyandb/dashboards-banyandb.md` (menu: \"BanyanDB self observability\n  dashboard\"). New `meter_banyandb_*` rules → update the operator doc; leave the SWIP alone.\n\n### Doc cross-references\n\nReference another section with a **Markdown anchor link** `[section title](#heading-slug)`,\nnever the `§` symbol or a bare \"section N\". Verify every anchor resolves to a real heading\n(GitHub slug rules: lowercase, spaces → `-`, punctuation dropped) — a broken `[](#…)`\nsilently renders as plain text on the website.\n\n## Handling Existing Tests\n\nIf existing test code (UT, IT, or E2E) fails to pass, try to fix it. If any test cases need to be modified, or test code needs to be deleted, the change must be double-checked by a human before it is applied — surface the proposed modification or deletion and the reasoning, and wait for explicit approval rather than changing or removing the test yourself.\n\n## Submitting Pull Requests\n\nUse the `/gh-pull-request` skill for committing and pushing to a PR branch. It runs pre-flight checks (compile, checkstyle, license headers) before every push, and creates the PR if one doesn't exist yet.\n\n## GitHub Actions Allow List\n\nApache enforces an allow list for third-party GitHub Actions. All third-party actions must be pinned to an approved SHA from:\nhttps://github.com/apache/infrastructure-actions/blob/main/approved_patterns.yml\n\nIf a PR is blocked by \"action is not allowed\" errors, check the approved list and update `.github/workflows/` files to use the approved SHA pin instead of a version tag.\n\nActions owned by `actions/*` (GitHub), `github/*`, and `apache/*` are always allowed (enterprise-owned).\n\n## Tips for AI Assistants\n\n1. **Always check submodules**: Protocol changes may require submodule updates\n2. **Generate sources first**: Run `mvnw compile` before analyzing generated code\n3. **Install package**: Use `mvnw flatten:flatten install` to build the precompiler and export generated classes before running tests. ref to [compile skill doc](.claude/skills/compile/SKILL.md)\n3. **Full rebuild on cross-module changes**: If you changed more than two modules or pulled/rebased code from git remote, run `mvnw clean install` (or `mvnw clean package`) on the **whole project** rather than picking individual modules with `-pl`. Incremental `-pl ... -am` builds can leave stale jars in `.m2` or `oap-libs/` when jar sizes don't change but content does, causing hard-to-debug runtime issues.\n3. **Respect checkstyle**: No System.out, no @author, no Chinese characters\n4. **Follow module patterns**: Use existing modules as templates\n5. **Check multiple storage implementations**: Logic may vary by storage type\n6. **OAL generates code**: Don't manually edit generated metrics classes\n7. **Use Lombok**: Prefer annotations over boilerplate code\n8. **Test both unit and integration**: Different test patterns for different scopes\n9. **Documentation is rendered via markdown**: When reviewing docs, consider how they will be rendered by a markdown engine\n10. **Relative paths in docs are valid**: Relative file paths (e.g., `../../../oap-server/...`) in documentation work both in the repo and on the documentation website, supported by website build tooling\n11. **Module service registration**: When adding a service to `CoreModule.services()`, update ALL `CoreModuleProvider` implementations — not just the main one. Search with `grep -rn \"extends CoreModuleProvider\" oap-server/ --include=\"*.java\"`. The `MockCoreModuleProvider` in `server-tools/profile-exporter/` also needs it, or the profile exporter e2e test will fail at startup.\n12. **Multiple OAP packagings**: The OAP server is not only the main `server-starter`. The `server-tools/` directory contains standalone tools (e.g., profile exporter) that boot with mock module providers and a subset of modules. Changes to core module contracts (services, required modules) must be reflected in these tools too.\n13. **`moduleManager.find(X.NAME)` requires `X.NAME` in `requiredModules()`**: every call to `moduleManager.find(SomeModule.NAME)` (direct or through a helper) must have `SomeModule.NAME` in the provider's `requiredModules()` array. Missing declarations cause runtime exceptions the first time the code path fires — not at module boot. Wrapping the call in `try { ... } catch (Throwable)` is NOT a substitute; declare the module and keep the try/catch only for defensive handling of transient provider outages. When auditing a branch, grep for `moduleManager.find(` across the touched module and verify each target name appears in `requiredModules()`. Example modules that frequently catch teams out: `AlarmModule` (used by alarm-kernel reset), `LogAnalyzerModule` (used by LAL factory lookup).\n14. **Don't look up `ClusterModule` services directly**: the `ClusterModule` (ZooKeeper / K8s / Nacos coordination) exposes `ClusterRegister` / `ClusterNodesQuery` / `ClusterCoordinator`. Most receiver / analyzer modules don't declare `ClusterModule` in `requiredModules()`, so calling `moduleManager.find(ClusterModule.NAME)` will throw at runtime. Instead, go through `CoreModule`'s `RemoteClientManager` service — it's already populated by the cluster module and exposes the peer list every OAP needs. If a module genuinely needs cluster-coordinator primitives, declare `ClusterModule.NAME` in `requiredModules()` explicitly.\n15. **No `ThreadLocal` side-channels to hijack downstream behaviour**: routing a caller's intent through a `ThreadLocal` that downstream code reads (e.g., `if (PeerMode.isActive()) skipSomething()`) is almost always the wrong answer — it creates invisible coupling between far-apart code paths, leaks across async hand-offs (executors, gRPC threads, Armeria event loops), and makes the behaviour impossible to understand from a method signature. The correct fix is almost always to **extend the interface** — add a parameter, a new method, a new mode enum that appears in the signature. Rare exceptions: propagating OpenTelemetry context where the whole industry has standardised on `ThreadLocal`, or security principals enforced by a framework. In all other cases, prefer an explicit API extension, even if it costs more lines.\n16. **BanyanDB schema-visibility: fence on `mod_revision`, do NOT poll metadata**: schema no longer lives in etcd — etcd was removed. It is now stored as property documents in BanyanDB's own `_schema` store and propagated to each data node's in-memory cache **asynchronously** (a `SchemaUpdateService.WatchSchemas` gRPC stream + a 30s reconcile sync, see `banyand/metadata/schema/property/client.go`). DDL `Create` / `Update` / `Delete` broadcast only to the schema-server (`ROLE_META`) nodes and **return immediately with a `mod_revision`** — now a client-stamped `time.Now().UnixNano()` timestamp, NOT an etcd index (which is why it still changes on every content-changing DDL; it is `0` on a delete that didn't record a tombstone) — **without waiting for data nodes**. A data write whose schema hasn't propagated yet is dropped at the data-node executor (`cannot find measure definition`, logged + skipped, `banyand/measure/write_standalone.go`), so the fence is still required — arguably more than under the old etcd-watch model. After firing DDL, fence on `BanyanDBClient.getSchemaWatcher().awaitRevisionApplied(maxRev, timeout)` before unparking dispatch / firing data writes — this blocks (liaison-side, all alive nodes, bounded by the timeout) until every data node's `notifiedModRevision` watermark reaches `maxRev`, which the registry's read-after-write does not guarantee. For deletes that returned `mod_revision == 0`, fall back to `awaitSchemaDeleted(SchemaKey, timeout)`. The previous \"poll `findMeasure` until you can read your own write\" idiom predates the `SchemaBarrierService` proto and has been replaced — do not reintroduce it. The `SchemaWatcher` class is in OAP's **in-tree** `library-banyandb-client` (`org.apache.skywalking.library.banyandb.v1.client.SchemaWatcher`); the standalone `skywalking-banyandb-java-client` repo does NOT have these symbols, so don't conclude the API is gone from grepping it. JDBC and ES are synchronous-DDL on the coordinator so they don't need a fence.\n\n## Analysis and Design Principles\n\n**Never guess or speculate.** All analysis must be grounded in source code, documentation, or verified behavior.\n\n### Before making claims\n- **Read the source code** — don't assume how a feature works based on naming or convention. Check the actual implementation.\n- **Read the documentation** — check `docs/en/`, CLAUDE.md files in submodules, and README files before proposing designs.\n- **Check configuration and flags** — verify what flags/env vars exist, their default values, and how they are parsed (e.g., BanyanDB uses viper with `BYDB_` prefix to auto-bind flags to env vars).\n- **Check dependent projects** — SkyWalking depends on BanyanDB, infra-e2e, Helm charts, etc. Read their source code and docs before assuming capabilities (e.g., check Helm chart `values.yaml` for supported fields, check infra-e2e for supported config options). For `skywalking-*` projects, ask the developer if they have the source code locally — searching a local clone is much faster than fetching files via GitHub API.\n\n### Before proposing changes\n- **Verify locally first** — run the code, start the container, execute the test before pushing to CI. Don't use CI as a trial-and-error environment.\n- **Validate file paths and directory structures** — check where data actually goes (e.g., BanyanDB `--stream-root-path /tmp` creates `/tmp/stream/`, `--access-log-root-path /tmp` creates `/tmp/accesslog/`). Don't assume directory names.\n- **Validate YAML syntax** — after editing YAML files (especially with sed/awk), validate with a YAML parser before committing. Corrupted YAML causes silent failures in CI.\n- **Check the actual Docker image** — verify what's available in the container (binaries, shell, directories) before writing commands that depend on them.\n\n### When uncertain\n- **Say \"I don't know\" and investigate** — reading the code is always better than guessing. Use grep, find, and read tools to locate the answer.\n- **Ask the developer first** — if you can't find the source code, don't know how to run something, or the code doesn't make the answer clear, ask the developer where to find it rather than speculate.\n- **Test with real data** — when investigating runtime behavior (e.g., what model names an API returns, what directory structure BanyanDB creates), set up a local test and observe the actual output.\n\n### Docker images\n- **Apache SkyWalking projects** — images are on `ghcr.io/apache/` (e.g., `ghcr.io/apache/skywalking-banyandb:${COMMIT_SHA}`). Tags are full commit SHAs, not short SHAs or version tags.\n- **Official and 3rd-party images** — on Docker Hub (e.g., `ollama/ollama`, `otel/opentelemetry-collector`, `envoyproxy/gateway`).\n- **Always verify the image exists** — `docker pull` before writing CI or e2e configs. Image tags depend on CI publish workflows completing successfully.\n</yakaai_skills>"}