### Ai Ready # AI-Ready Integrations jscpd integrates into AI-powered development workflows through three complementary mechanisms: the AI reporter, agent skills, and an MCP server. ## AI Reporter The `ai` reporter produces compact, token-efficient output designed to be piped directly into an LLM prompt or agentic pipeline. It uses common-path-prefix compression and omits code fragments and colors — just the clone locations and a summary. ### TypeScript (v4) ```bash jscpd --reporters ai /path/to/source ``` ### Rust (v5) ```bash cpd --reporters ai /path/to/source ``` ### Example Output ``` src/utils/ auth.ts:10-25 ~ helpers.ts:40-55 src/utils/auth.ts 30-45 ~ 80-95 src/ utils/auth.ts:10-25 ~ api/routes.ts:5-20 --- 23 clones · 4.2% duplication ``` ### Token Efficiency Benchmarked on the `fixtures/` directory (212 clones, 347 files): | Reporter | Output size | Estimated tokens | |----------|-------------|------------------| | `console` (default) | ~21,800 chars | ~5,400 | | `ai` | ~4,500 chars | ~1,100 | ~79% fewer tokens than the default console reporter. ### Codebase Summary (v5) Add `--summary` for a compact refactoring-hotspot overview — top files and folders by tokens, lines, size, and a complexity estimate. In the `ai` reporter each entry is one line with all metrics inline, so an agent gets the full picture for a handful of tokens: ``` Summary by tokens (321 files, 129 folders): files (tokens/lines/size/cx/dup%): src/core/files.ts 2052/363/11.4K/80/0.0% ... folders (files/tokens/lines/size): src/core 8/5264/843/26.5K ... ``` ```bash cpd --reporters ai --summary --no-tips /path/to/source ``` See [rust.md](rust.md#summary) for the metric definitions and `--summary-top` / `--summary-by` options. ## Agent Skills jscpd ships two AI agent skills that teach coding assistants how to use jscpd and refactor detected duplications. ### jscpd — Tool Reference Skill Covers all CLI options, the AI reporter output format, and configuration file syntax. ```bash npx skills add kucherenko/jscpd --skill jscpd ``` ### dry-refactoring — Refactoring Workflow Skill A guided process for reading clone output, choosing the right extraction strategy, applying the refactor, and verifying the clone is eliminated. ```bash npx skills add kucherenko/jscpd --skill dry-refactoring ``` After installation, ask your agent to "find and fix code duplication" and it will invoke jscpd with the right options and act on the results. ## MCP Server [jscpd-server](../apps/jscpd-server) implements the [Model Context Protocol (MCP)](https://modelcontextprotocol.io), exposing jscpd's detection capabilities as tools that AI assistants can call directly from the editor. Start the server once against your codebase, then let your AI assistant check any snippet for duplication on demand — no CLI invocation needed. ### Installation ```bash npm install jscpd-server ``` ### Usage Start the server: ```bash jscpd-server /path/to/project ``` Options: - `--port` — Port number (default: 3000) - `--host` — Host to bind (default: 127.0.0.1) - `--allowed-origin` — Extra `Origin` hostname accepted by the MCP and REST endpoints (repeatable) - `--allowed-host` — `Host` hostname the MCP and REST endpoints answer on (repeatable) - `--store leveldb` — Use LevelDB persistent storage - Plus all standard jscpd detection options ### MCP Configuration Add to your MCP client config (e.g. Claude Desktop): ```json { "mcpServers": { "jscpd": { "type": "streamable-http", "url": "http://localhost:3000/mcp" } } } ``` The endpoint serves protocol revision `2026-07-28`: requests are direct and stateless, carrying their protocol version and client capabilities in the per-request `_meta` envelope, so there is no `initialize` handshake and no `Mcp-Session-Id`. Clients discover the server with `server/discover`. 2025-era clients keep working through the SDK's stateless legacy fallback. `/mcp`, `POST /api/check`, `POST /api/recheck`, and `GET /api/stats` validate the `Origin` and `Host` headers, as the transport specification requires. Loopback origins and hosts are allowed by default; add `--allowed-origin` for a browser client served under another name, and `--allowed-host` to pin extra hostnames a reachable deployment answers on. A concrete `--host` is always included in the Host allowlist. ### REST API | Method | Path | Description | |--------|------|-------------| | `POST` | `/api/check` | Check a code snippet for duplications. Body: `{"code": "...", "format": "javascript"}` | | `POST` | `/api/recheck` | Trigger a re-scan of the directory | | `GET` | `/api/stats` | Get overall project duplication statistics | | `GET` | `/api/health` | Health check — returns `{ status, workingDirectory, lastScanTime }` | | `GET` | `/` | API info with endpoint listing | ### MCP Tools Available MCP tools exposed via the `/mcp` endpoint: - `check_duplication` — Check a code snippet for duplications (inputs: `code`, `format`) - `get_statistics` — Get project stats (no inputs) - `check_current_directory` — Re-scan the working directory (no inputs) Snippet checking uses an ephemeral in-memory store per request for isolation — no cross-request contamination, automatic cleanup, concurrent-request safe. --- ### Api # Programming API Both jscpd v4 (TypeScript) and v5 (Rust) provide programmatic APIs for integration into your own tools. ## TypeScript (v4) ### `jscpd` Function The `jscpd` function accepts an `argv`-style array and returns a `Promise`: ```typescript import { IClone } from '@jscpd/core'; import { jscpd } from 'jscpd'; const clones: IClone[] = await jscpd([]); ``` Pass options as CLI-like arguments: ```typescript const clones: IClone[] = await jscpd([ '', '', __dirname + '/../fixtures', '-m', 'weak', '--silent', ]); ``` ### `detectClones` Function A higher-level API with an options object: ```typescript import { detectClones } from 'jscpd'; const clones = await detectClones({ path: ['./src'], silent: true, format: ['javascript', 'typescript'], minLines: 5, minTokens: 50, mode: 'mild', }); ``` ### Custom Store Use `detectClones` with a custom store for incremental detection: ```typescript import { detectClones } from 'jscpd'; import { IMapFrame, MemoryStore } from '@jscpd/core'; const store = new MemoryStore(); await detectClones({ path: ['./src'], }, store); // Re-use the store for incremental detection await detectClones({ path: ['./src'], silent: true, }, store); ``` For large repositories, use the LevelDB store: ```typescript import { detectClones } from 'jscpd'; import { IMapFrame } from '@jscpd/core'; import { LevelDBStore } from '@jscpd/leveldb-store'; const store = new LevelDBStore('/path/to/leveldb/dir'); await detectClones({ path: ['./src'], }, store); ``` ### Building Custom Tools Compose the lower-level packages for deep customization: - **`@jscpd/core`** — Core detection algorithm (Rabin-Karp), event emitter interface. Single dependency on `eventemitter3`. - **`@jscpd/tokenizer`** — Source code tokenization (224+ formats via reprism). - **`@jscpd/finder`** — File walking, clone detection orchestration, built-in reporters, subscribers, validators. - **`@jscpd/leveldb-store`** — LevelDB persistent store for large repositories. - **`@jscpd/redis-store`** — Redis store for distributed/CI environments. See [Packages](./packages.md) for details on each package. ## Rust (v5) The Rust engine is available as two npm packages — `jscpd@5` (installs the `jscpd` command) and `cpd` (installs the `cpd` command). On crates.io it is published as `jscpd`, which installs both `jscpd` and `cpd` binaries. For integration in Rust applications, use the `cpd-finder` crate: ```rust use cpd_finder::orchestrate::{RunConfig, run}; let config = RunConfig { paths: vec!["./src".into()], min_tokens: 50, ..Default::default() }; let result = run(&config).unwrap(); println!("Found {} clones", result.clones.len()); println!("Analyzed {} files", result.statistics.total.sources); ``` ### Crate Architecture | Crate | Description | |-------|-------------| | `cpd-core` | Core data models and hashing (Rabin-Karp rolling hash) | | `cpd-tokenizer` | Source code tokenization (223+ formats, uses `oxc_parser`) | | `cpd-finder` | File walking, orchestration, git blame (`rayon` + `ignore` + `globset`) | | `cpd-reporter` | Output format rendering (13 reporters) | There is no Node.js API for v5 — use v4's TypeScript API for Node.js integration, or v5's Rust API for Rust integration. --- ### Ci And Hooks # CI & Pre-Commit Hooks jscpd can enforce duplication thresholds in CI pipelines and as a local pre-commit hook — catching copy/pasted code before it reaches the main branch. ## GitHub Action The [jscpd-copy-paste-detector](https://github.com/marketplace/actions/jscpd-copy-paste-detector) GitHub Action runs jscpd in your CI workflow. It installs the Rust engine, runs detection, uploads SARIF to GitHub Code Scanning, and optionally uploads the report as an artifact. ### Basic Usage ```yaml name: Duplication Check on: [push, pull_request] jobs: jscpd: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: kucherenko/jscpd@master ``` This scans the entire repository with default settings and uploads SARIF results to GitHub Code Scanning. ### Fail on Threshold Set `threshold` to fail the build when duplication exceeds a percentage: ```yaml - uses: kucherenko/jscpd@master with: threshold: 5 ``` The workflow fails if more than 5% of the code is duplicated. ### Action Inputs | Input | Description | Default | |-------|-------------|---------| | `path` | Paths to scan (space-separated) | `.` | | `config` | Path to `.jscpd.json` config file | — | | `min-tokens` | Minimum tokens for a clone | `50` | | `min-lines` | Minimum lines for a clone | `5` | | `max-lines` | Maximum lines per block | — | | `mode` | Detection mode: `mild`, `weak`, `strict` | `mild` | | `format` | Comma-separated formats to check | — | | `ignore` | Comma-separated glob patterns to ignore | — | | `ignore-pattern` | Comma-separated regex patterns to skip | — | | `reporters` | Comma-separated reporters | `console` | | `output` | Output directory for file reporters | `report` | | `threshold` | Max duplication % before exit 1 | — | | `blame` | Enrich clones with git blame data | `false` | | `exit-code` | Exit with code when duplicates found (`true` or integer) | — | | `pattern` | Glob pattern for file search | — | | `max-size` | Skip files larger than SIZE | — | | `skip-local` | Skip clones in same directory | `false` | | `ignore-case` | Ignore case of symbols (experimental) | `false` | | `follow-symlinks` | Follow symbolic links | `false` | | `no-gitignore` | Don't respect .gitignore files | `false` | | `absolute` | Use absolute paths in reports | `false` | | `formats-exts` | Custom format-to-extension mappings | — | | `formats-names` | Custom format-to-filename mappings | — | | `version` | jscpd version to install | `latest` | | `install-prefix` | Installation directory for the binary | — | | `skip-install` | Skip installation (binary already present) | `false` | | `extra-args` | Additional arguments passed to jscpd | — | | `upload-report` | Upload report directory as artifact | `false` | | `upload-sarif` | Upload SARIF to GitHub Code Scanning | `true` | ### Action Outputs | Output | Description | |--------|-------------| | `duplication-percentage` | Percentage of duplicated code found | | `clones-found` | Number of clone pairs found | | `duplicated-lines` | Number of duplicated lines | | `total-lines` | Total lines scanned | | `files-count` | Number of source files scanned | | `report-path` | Path to the output directory | | `sarif-path` | Path to the SARIF report file | | `exit-code` | Exit code from jscpd | ### Examples #### Scan specific directories with threshold ```yaml - uses: kucherenko/jscpd@master with: path: src/lib src/utils threshold: 3 ignore: "**/*.test.*,**/*.spec.*" ``` #### Use a config file ```yaml - uses: kucherenko/jscpd@master with: config: .jscpd.json upload-report: true ``` #### Multi-reporter with artifact upload ```yaml - uses: kucherenko/jscpd@master with: reporters: console,json,html,sarif output: jscpd-report upload-report: true ``` #### Pin a specific version ```yaml - uses: kucherenko/jscpd@master with: version: "5.0.9" ``` #### Skip install (binary already in image) ```yaml - uses: kucherenko/jscpd@master with: skip-install: true ``` #### Use outputs in subsequent steps ```yaml - uses: kucherenko/jscpd@master id: jscpd - name: Check results if: steps.jscpd.outputs.duplication-percentage > 5 run: | echo "Duplication is ${{ steps.jscpd.outputs.duplication-percentage }}%" echo "Found ${{ steps.jscpd.outputs.clones-found }} clones" ``` ## Pre-Commit Hook Run jscpd before every commit to prevent duplicated code from entering the repository. ### Using pre-commit framework The [pre-commit](https://pre-commit.com) framework manages git hooks for you. After configuring the hook, it runs automatically on every `git commit`. **1. Install pre-commit** (one time, any of these): ```bash # pip pip install pre-commit # brew brew install pre-commit # npm (wrapper around the Python tool) npm install pre-commit ``` **2. Add the hook config** to `.pre-commit-config.yaml` in your repo: **Option A: `language: node`** — pre-commit installs jscpd automatically: ```yaml repos: - repo: local hooks: - id: jscpd name: jscpd - copy/paste detector entry: jscpd language: node additional_dependencies: ['jscpd@5'] args: [--threshold, "5", --reporters, console,silent] pass_filenames: false always_run: true ``` **Option B: `language: system`** — jscpd must be pre-installed globally: ```yaml repos: - repo: local hooks: - id: jscpd name: jscpd - copy/paste detector entry: jscpd language: system args: [--threshold, "5", --reporters, console,silent] pass_filenames: false always_run: true ``` If using Option B, install jscpd globally first: `npm install jscpd@5` or `cargo install jscpd`. **3. Install the hook into git:** ```bash pre-commit install ``` That's it — jscpd now runs on every `git commit`. If duplication exceeds the threshold, the commit is blocked. To run manually without committing: ```bash pre-commit run jscpd --all-files ``` ### Using Husky ```bash npm install -D husky npx husky init ``` Add the hook: ```bash echo 'npx jscpd@5 --threshold 5 --reporters console,silent .' > .husky/pre-commit ``` ### Manual git hook No extra tools required — just a shell script in `.git/hooks/`. 1. Create `.git/hooks/pre-commit`: ```bash #!/bin/sh jscpd --threshold 5 --reporters console,silent . ``` 2. Make it executable: ```bash chmod +x .git/hooks/pre-commit ``` Hooks in `.git/hooks/` are not version-controlled. To share the hook with your team, store it in the repo and symlink or copy it: **Option A: Symlink from a versioned script** Store the hook logic in the repo (e.g. `scripts/pre-commit`), then symlink: ```bash ln -s ../../scripts/pre-commit .git/hooks/pre-commit ``` Each developer runs the symlink command once after cloning. **Option B: `core.hooksPath` (Git 2.9+)** Point Git at a versioned hooks directory: ```bash git config core.hooksPath .githooks ``` Create `.githooks/pre-commit`: ```bash #!/bin/sh jscpd --threshold 5 --reporters console,silent . ``` ```bash chmod +x .githooks/pre-commit ``` Commit `.githooks/` to the repo. New contributors run the `git config` command once after cloning. Add it to your onboarding docs or a `scripts/setup.sh`: ```bash #!/bin/sh git config core.hooksPath .githooks ``` **Option C: npm `prepare` script** Add to `package.json`: ```json { "scripts": { "prepare": "git config core.hooksPath .githooks" } } ``` `npm install` (and `npm ci`) automatically run `prepare`, so the hooks path is set with no manual steps. **Option D: Makefile** ```makefile .PHONY: hooks hooks: git config core.hooksPath .githooks ``` Contributors run `make hooks` after cloning. ### Tips - Use `--reporters console,silent` to show clone details without writing report files on every commit - Use `--threshold` to set a failure threshold — the hook exits with code 1 if exceeded - Use `--ignore` to exclude generated files, test fixtures, or vendor directories - For large repos, use the Rust engine (`jscpd@5` / `cpd`) — it runs 24-37x faster, keeping commit latency low - Consider `--format` to limit detection to specific languages during the hook, with a full scan in CI --- ### Packages # Packages The jscpd monorepo contains two apps and several supporting packages. ## Apps ### jscpd **Path:** `apps/jscpd` **npm:** [`jscpd`](https://www.npmjs.com/package/jscpd) **Version:** 4.2.5 Main package for jscpd — CLI and Node.js API for copy/paste detection. See [TypeScript docs](./typescript.md). ### jscpd-server **Path:** `apps/jscpd-server` **npm:** [`jscpd-server`](https://www.npmjs.com/package/jscpd-server) **Version:** 4.2.5 Standalone server application providing REST API and MCP server for on-demand code duplication detection. See [AI-Ready docs](./ai-ready.md) for details. ## Packages (TypeScript / Node.js) ### @jscpd/core **Path:** `packages/core` **npm:** [`@jscpd/core`](https://www.npmjs.com/package/@jscpd/core) **Version:** 4.2.5 Core detection algorithm. Implements Rabin-Karp rolling hash for finding duplicate code blocks. Single dependency on `eventemitter3`. Provides `IClone`, `IMapFrame`, `MemoryStore`, and event interfaces. ### @jscpd/finder **Path:** `packages/finder` **npm:** [`@jscpd/finder`](https://www.npmjs.com/package/@jscpd/finder) **Version:** 4.2.5 Detector of duplications in files. Walks filesystem, runs clone detection, provides built-in reporters, subscribers, validators, and hooks. ### @jscpd/tokenizer **Path:** `packages/tokenizer` **npm:** [`@jscpd/tokenizer`](https://www.npmjs.com/package/@jscpd/tokenizer) **Version:** 4.2.5 Tokenizer — converts source code into tokens for duplicate detection. Supports 224 languages/formats via reprism-based grammar engine with lazy loading. Cross-format tokenization for Vue SFC, Svelte, Astro, and Markdown. ### @jscpd/html-reporter **Path:** `packages/html-reporter` **npm:** [`@jscpd/html-reporter`](https://www.npmjs.com/package/@jscpd/html-reporter) **Version:** 4.2.5 HTML reporter — generates interactive HTML report with per-format statistics, duplication graph, and syntax-highlighted clone diffs. ### @jscpd/badge-reporter **Path:** `packages/badge-reporter` **npm:** [`@jscpd/badge-reporter`](https://www.npmjs.com/package/@jscpd/badge-reporter) **Version:** 4.2.5 Badge reporter — generates SVG badges showing copy/paste level. ### jscpd-sarif-reporter **Path:** `packages/sarif-reporter` **npm:** [`jscpd-sarif-reporter`](https://www.npmjs.com/package/jscpd-sarif-reporter) **Version:** 4.2.5 SARIF reporter — generates Static Analysis Results Interchange Format output for GitHub Code Scanning. Emits warning-level results per clone, plus error if threshold exceeded. ### @jscpd/leveldb-store **Path:** `packages/leveldb-store` **npm:** [`@jscpd/leveldb-store`](https://www.npmjs.com/package/@jscpd/leveldb-store) **Version:** 4.2.5 LevelDB store — persistent disk-backed token store for large repositories. Slower than default in-memory store but can handle very large codebases. ### @jscpd/redis-store **Path:** `packages/redis-store` **npm:** [`@jscpd/redis-store`](https://www.npmjs.com/package/@jscpd/redis-store) **Version:** 4.2.5 Redis store — offloads in-memory hash map to Redis. Useful for large codebases or distributed/CI environments. ## Crates (Rust / v5) ### cpd (binary) **Path:** `rust/crates/cpd` **npm:** [`jscpd@5`](https://www.npmjs.com/package/jscpd) (installs the `jscpd` command) | [`cpd`](https://www.npmjs.com/package/cpd) (installs the `cpd` command) **crates.io:** [`jscpd`](https://crates.io/crates/jscpd) (installs both `jscpd` and `cpd` binaries) **Version:** 5.0.4 (npm) / 0.1.4 (crates.io) CLI binary, entry point. Published as `jscpd@5` on npm (self-contained binary, installs the `jscpd` command, no Node.js runtime) and `cpd` on npm (installs the `cpd` command). See [Rust docs](./rust.md). ### cpd-core **Path:** `rust/crates/cpd-core` **Version:** 0.1.3 Core data models and Rabin-Karp rolling hash implementation. ### cpd-tokenizer **Path:** `rust/crates/cpd-tokenizer` **Version:** 0.1.3 Source code tokenizer (223+ formats). Uses `oxc_parser` for Go, TypeScript/JSX tokenization. ### cpd-finder **Path:** `rust/crates/cpd-finder` **Version:** 0.1.4 File walking, orchestration, and git blame. Uses `rayon` for parallelism, `ignore` + `globset` for file matching. ### cpd-reporter **Path:** `rust/crates/cpd-reporter` **Version:** 0.1.4 Output format rendering for 13 reporters. --- ### Performance Comparison # jscpd Performance Comparison: v4 (TypeScript) vs v5 (Rust) **Date:** 2026-06-08 **Runs per configuration:** 10 (fixtures, svelte), 3 (CopilotKit) **Machine:** macOS (Apple Silicon) ## Versions | Tool | Version | Runtime | |------|---------|---------| | jscpd v4 | 4.2.5 | Node.js | | jscpd v5 (cpd) | 5.0.4 | Native binary (Rust) | ## Benchmark Targets | Target | Files | Size | Description | |--------|-------|------|-------------| | fixtures | 548 | 1.5 MB | Multi-language test fixtures (126+ formats) | | svelte | 8,963 | 38 MB | Svelte framework source code | | CopilotKit | 17,092 | 159 MB | Large real-world TypeScript/React project | ## Execution Time Results ### fixtures (548 files, 1.5 MB) | Metric | jscpd v4 | jscpd v5 | Speedup | |--------|----------|----------|---------| | Mean real time | 1.030s | 0.030s | **34.3x** | | Std dev | 0.042s | 0.000s | | | Min | 1.000s | 0.030s | | | Max | 1.130s | 0.030s | | | Mean user time | 1.174s | 0.085s | | | Mean sys time | 0.074s | 0.050s | | ### svelte (8,963 files, 38 MB) | Metric | jscpd v4 | jscpd v5 | Speedup | |--------|----------|----------|---------| | Mean real time | 15.803s | 0.428s | **36.9x** | | Std dev | 1.010s | 0.021s | | | Min | 14.740s | 0.390s | | | Max | 17.790s | 0.450s | | | Mean user time | 16.075s | 0.553s | | | Mean sys time | 0.738s | 1.110s | | ### CopilotKit (17,092 files, 159 MB) | Metric | jscpd v4 | jscpd v5 | Speedup | |--------|----------|----------|---------| | Mean real time | 82.890s | 3.440s | **24.1x** | | Std dev | 4.086s | 0.699s | | | Min | 79.560s | 2.900s | | | Max | 87.450s | 4.230s | | | Mean user time | 100.020s | 7.323s | | | Mean sys time | 18.263s | 3.100s | | ## Detection Results Comparison ### fixtures | Metric | jscpd v4 | jscpd v5 | |--------|----------|----------| | Files analyzed | 364 | 347 | | Clones found | 211 | 212 | | Duplicated lines | 9,969 (47.08%) | 9,133 (37.12%) | | Duplicated tokens | 73,416 (47.64%) | 56,491 (43.30%) | ### svelte | Metric | jscpd v4 | jscpd v5 | |--------|----------|----------| | Files analyzed | 11,672 | 4,322 | | Clones found | 903 | 1,055 | | Duplicated lines | 18,246 (7.34%) | 21,821 (8.78%) | ### CopilotKit | Metric | jscpd v4 | jscpd v5 | |--------|----------|----------| | Files analyzed | 13,944 | 12,386 | | Clones found | 12,272 | 22,487 | ## Raw Timing Data ### jscpd v4 — fixtures | Run | Real (s) | User (s) | Sys (s) | |-----|----------|----------|---------| | 1 | 1.13 | 1.17 | 0.10 | | 2 | 1.00 | 1.15 | 0.07 | | 3 | 1.01 | 1.16 | 0.07 | | 4 | 1.01 | 1.17 | 0.07 | | 5 | 1.00 | 1.16 | 0.07 | | 6 | 1.01 | 1.17 | 0.07 | | 7 | 1.01 | 1.17 | 0.07 | | 8 | 1.02 | 1.19 | 0.07 | | 9 | 1.03 | 1.20 | 0.07 | | 10 | 1.08 | 1.20 | 0.08 | ### jscpd v5 — fixtures | Run | Real (s) | User (s) | Sys (s) | |-----|----------|----------|---------| | 1 | 0.03 | 0.08 | 0.05 | | 2 | 0.03 | 0.09 | 0.05 | | 3 | 0.03 | 0.08 | 0.05 | | 4 | 0.03 | 0.08 | 0.05 | | 5 | 0.03 | 0.08 | 0.05 | | 6 | 0.03 | 0.09 | 0.06 | | 7 | 0.03 | 0.09 | 0.04 | | 8 | 0.03 | 0.09 | 0.05 | | 9 | 0.03 | 0.08 | 0.05 | | 10 | 0.03 | 0.09 | 0.05 | ### jscpd v4 — svelte | Run | Real (s) | User (s) | Sys (s) | |-----|----------|----------|---------| | 1 | 15.98 | 16.06 | 0.70 | | 2 | 15.06 | 15.55 | 0.59 | | 3 | 14.74 | 15.35 | 0.56 | | 4 | 17.37 | 16.03 | 1.04 | | 5 | 17.79 | 17.54 | 1.38 | | 6 | 15.86 | 16.00 | 0.72 | | 7 | 15.11 | 15.67 | 0.63 | | 8 | 15.28 | 15.89 | 0.60 | | 9 | 15.47 | 15.99 | 0.59 | | 10 | 15.37 | 16.67 | 0.57 | ### jscpd v5 — svelte | Run | Real (s) | User (s) | Sys (s) | |-----|----------|----------|---------| | 1 | 0.39 | 0.55 | 0.96 | | 2 | 0.43 | 0.55 | 1.23 | | 3 | 0.40 | 0.55 | 0.95 | | 4 | 0.41 | 0.54 | 1.18 | | 5 | 0.45 | 0.56 | 1.11 | | 6 | 0.45 | 0.56 | 1.20 | | 7 | 0.44 | 0.56 | 1.19 | | 8 | 0.44 | 0.55 | 1.17 | | 9 | 0.43 | 0.56 | 0.97 | | 10 | 0.44 | 0.55 | 1.14 | ### jscpd v4 — CopilotKit | Run | Real (s) | User (s) | Sys (s) | |-----|----------|----------|---------| | 1 | 87.45 | 99.76 | 20.15 | | 2 | 79.56 | 97.08 | 16.02 | | 3 | 81.66 | 103.22 | 18.62 | ### jscpd v5 — CopilotKit | Run | Real (s) | User (s) | Sys (s) | |-----|----------|----------|---------| | 1 | 2.90 | 7.41 | 3.13 | | 2 | 4.23 | 7.16 | 3.18 | | 3 | 3.19 | 7.40 | 2.99 | ## Analysis ### v5 is dramatically faster across all targets After correcting the benchmark methodology, v5 is consistently **24–37x faster** than v4: | Target | v4 (TypeScript) | v5 (Rust) | Speedup | |--------|----------------|-----------|---------| | fixtures (548 files, 1.5 MB) | 1.03s | 0.03s | **34.3x** | | svelte (9K files, 38 MB) | 15.80s | 0.43s | **36.9x** | | CopilotKit (17K files, 159 MB) | 82.89s | 3.44s | **24.1x** | ### Key observations 1. **Startup overhead**: v5's native binary has near-zero startup cost. v4's Node.js runtime adds ~1s even for tiny fixtures. 2. **Scaling**: v5 scales well from small to large codebases. CopilotKit (159 MB) takes only 3.4s. v4 takes 83s on the same target. 3. **CPU utilization**: v5's higher user time relative to real time (e.g., CopilotKit: 7.3s user vs 3.4s real) shows effective multi-threading. v4 is single-threaded (user ≈ real). 4. **Consistency**: v5 has tighter variance across all runs. On CopilotKit, v5's std dev is 0.7s (20% of mean) vs v4's 4.1s (5% of mean, but absolute variation is much larger). 5. **File scanning differences**: v4 with `--no-gitignore` analyzes more files than v5 on svelte (11,672 vs 4,322) because v5's gitignore handling differs. This means v5 is even more efficient per-file analyzed than the raw speedup numbers suggest. 6. **Detection accuracy**: v5 finds more clones on large codebases (1,055 vs 903 on svelte, 22,487 vs 12,272 on CopilotKit), likely due to different token counting and the `maxSize` default behavior. --- ### Rust # jscpd v5 (Rust Engine) The Rust engine is a ground-up rewrite of jscpd. It is a drop-in replacement for the Node.js CLI — same algorithm, same reporters, same `.jscpd.json` config — but 24-37x faster. The Rust engine is distributed as two npm packages: | Package | Installs commands | Notes | |---------|-------------------|-------| | [`jscpd@5`](https://www.npmjs.com/package/jscpd) | `jscpd` | Same command name as v4; drop-in CLI replacement | | [`cpd`](https://www.npmjs.com/package/cpd) | `cpd` | Lighter package, shorter command only | | [`jscpd` (crates.io)](https://crates.io/crates/jscpd) | `jscpd` **and** `cpd` | Rust-native install; both binaries | All three install the identical Rust binary and accept the same CLI options. Only the crates.io install exposes both command names from a single package. ## Performance Benchmarks on macOS (Apple Silicon), 10 runs per target (3 for CopilotKit). v4 ran with `--no-gitignore -i "node_modules"` to ensure comparable file scanning. See [performance-comparison.md](performance-comparison.md) for full methodology. | Codebase | Files | Size | `jscpd` v4 (Node.js) | `cpd`/`jscpd` v5 (Rust) | Speedup | |----------|-------|------|----------------------|-------------------------|---------| | Multi-format fixtures | 548 | 1.5 MB | 1.03s | 0.03s | **34.3x** | | Svelte source | 9K | 38 MB | 15.80s | 0.43s | **36.9x** | | CopilotKit | 17K | 159 MB | 82.89s | 3.44s | **24.1x** | ## Installation ```bash # npm — installs the jscpd command (same binary as v4 command name) npm install jscpd@5 jscpd /path/to/code # npm — installs only the cpd command (lighter) npm install cpd cpd /path/to/code # crates.io — Rust-native install (exposes both jscpd and cpd commands) cargo install jscpd jscpd /path/to/code cpd /path/to/code # Nix — run without installing nix run github:kucherenko/jscpd -- /path/to/code # Nix — install permanently nix profile install github:kucherenko/jscpd # Homebrew (macOS/Linux) brew install jscpd ``` The npm packages ship prebuilt binaries for 6 platforms: macOS arm64/x64, Linux arm64/x64 (glibc/musl), Windows x64. No Node.js runtime is required — the binary is self-contained. ## CLI Usage The `jscpd` command is available after installing `jscpd@5`; the `cpd` command is available after installing either `cpd` (npm) or `jscpd` (crates.io). Both commands accept the same options and are identical: ```bash jscpd [OPTIONS] [PATH]... cpd [OPTIONS] [PATH]... ``` ### Options | Option | Short | Description | Default | |--------|-------|-------------|---------| | `--min-tokens` | `-k` | Minimum tokens in a clone | 50 | | `--min-lines` | `-l` | Minimum lines in a clone | 5 | | `--max-lines` | `-x` | Maximum source file lines | — | | `--max-size` | `-z` | Skip files larger than SIZE (e.g. `1kb`, `1mb`, `100kb`) | no limit | | `--mode` | `-m` | Detection mode: `mild`, `weak`, `strict` | `mild` | | `--workers` | | Number of worker threads for parallel tokenization/detection | auto (all CPU cores) | | `--no-colors` | | Disable ANSI color output | off | | `--absolute` | `-a` | Use absolute paths in reports | off | | `--ignore-case` | | Ignore case of symbols in code (experimental) | off | | `--formats-exts` | | Custom format-to-extension mapping (e.g. `javascript:es,es6;dart:dt`) | — | | `--formats-names` | | Custom format-to-filename mapping | — | | `--cross-formats` | | Detect clones across formats: `;`-separated groups of `,`-separated formats (e.g. `javascript,typescript`). Preset `js-ts` = `javascript,jsx,typescript,tsx` | — | | `--list` | | List all supported formats and exit | — | | `--skip-local` | | Skip clones where both fragments are in the same directory | off | | `--sarif-error-tokens` | | Report SARIF results as `error` for clones with at least this many tokens (smaller clones stay `warning`). When overall duplication exceeds `--threshold`, all SARIF results become `error` regardless of size. | — (all `warning`) | | `--min-duplicated-lines` | | Minimum percentage of duplication to report (0-100) | 0 | | `--summary` | | Print a codebase summary: top files and folders by tokens, lines, size, and a complexity estimate. See [Summary](#summary) | off | | `--summary-top` | | Number of entries in each summary top list | 10 | | `--summary-by` | | Summary sort metric: `tokens`, `lines`, `size`, `complexity` | `tokens` | | `--silent` | `-s` | Suppress console output | off | | `--no-tips` | | Suppress tips and promotional messages | off | | `--version` | `-V` | Print version | — | | `--help` | `-h` | Print help | — | ### Reporters 13 built-in reporters: | Reporter | Output | |----------|--------| | `console` | Clone list + statistics table (default) | | `console-full` | Clone list with source snippets; with `--blame` shows side-by-side author comparison | | `json` | `report/jscpd-report.json` | | `xml` | `report/jscpd-report.xml` | | `csv` | `report/jscpd-report.csv` | | `html` | `report/jscpd-report.html` | | `markdown` | `report/jscpd-report.md` | | `badge` | `report/jscpd-badge.svg` + `report/jscpd-lines-badge.svg` | | `sarif` | `report/jscpd-report.sarif` (GitHub Code Scanning) | | `ai` | Token-efficient output for LLM pipelines | | `xcode` | Xcode-compatible warnings | | `threshold` | Exit 1 if duplication percentage exceeds `--threshold` | | `silent` | No console output | Output file names differ from v4: v5 uses `jscpd-report.*` prefix (e.g. `jscpd-report.json`, `jscpd-report.sarif`) while v4 uses `jscpd-report.json`, `html/` directory, etc. ### Summary `--summary` appends a codebase summary to the run output — the statistics jscpd already collects while scanning, aggregated to answer "where should I refactor first": ``` Summary (by tokens; 321 files, 129 folders analyzed) Top files: TOKENS LINES SIZE CX DUP% PATH 2052 363 11.4K 80 0.0 files.ts ... Top folders: FILES TOKENS LINES SIZE CX PATH 8 5264 843 26.5K 15 src/core ... ``` - **Top files** lists the top `--summary-top` files ranked by the `--summary-by` metric. Every row carries all metrics, so re-ranking by another lens is a `--summary-by size` (or `lines`, `complexity`) away. - **Top folders** aggregates files into their direct parent directory (each file counted exactly once; no cumulative ancestor totals). - **CX** is a language-agnostic cyclomatic-complexity estimate computed from the token stream: 1 + the number of decision-point tokens (`if`, `elif`/`elsif`/`elseif`, `unless`, `for`, `foreach`, `while`, `until`, `case`, `cond`, `when`, `catch`, `rescue`, `except`, `and`, `or`, `andalso`, `orelse`, `&&`, `||`, `?`, `??`). Matching is case-insensitive, so uppercase-keyword languages (SQL, PL/SQL, Fortran, COBOL, BASIC) count too. For folders it is the per-file mean. Languages that branch without such keywords (Smalltalk `ifTrue:` messages, Prolog clauses) stay at 1 — treat CX as a ranking signal, not an exact metric. - **DUP%** is the share of the file's lines covered by detected clone fragments (both fragments of a clone count toward their files; display is capped at 100%). The summary is fully opt-in and computed after detection from data already in memory, so runs without `--summary` are unaffected. It integrates with: - `console` / `console-full` — the block shown above - `ai` — a compact, LLM-token-efficient variant (one line per file/folder) - `json` — an additive `summary` key in `jscpd-report.json` (absent when the flag is off, so the schema is unchanged for existing consumers) Config file equivalents: `"summary": true`, `"summaryTop": 10`, `"summaryBy": "tokens"`. ```bash # Refactoring hotspots: biggest files by tokens plus duplication share cpd ./src --summary # Agent-friendly: compact clone list + compact summary cpd ./src --summary --reporters ai --no-tips # Focus on the most complex files, top 5 lists, machine-readable cpd ./src --summary --summary-by complexity --summary-top 5 --reporters json ``` ### Blame Output With `--blame --reporters console-full`, clones are displayed with a side-by-side author comparison: ``` 176 │ Andrii Kucherenko │ <= │ 196 │ Josh Soref │ ## TODO 177 │ Andrii Kucherenko │ <= │ 197 │ Josh Soref │ 180 │ Andrii Kucherenko │ == │ 200 │ Andrii Kucherenko │ ## License ``` `==` means both lines were written by the same author; `<=` means different authors (potential copy). ### Examples ```bash # Drop-in replacement for jscpd v4 jscpd /path/to/source # or cpd /path/to/source # Same flags as v4 cpd /path/to/source --min-tokens 30 --min-lines 3 --reporters console,json,html # Git blame with side-by-side author comparison cpd /path/to/source --blame --reporters console-full # List supported formats cpd --list # Use multiple reporters with custom output cpd ./src -r console,json,sarif -o ./reports # Skip clones within the same directory cpd --skip-local /path/to/source ``` ### Config File v5 reads the same `.jscpd.json` config file format as v4: ```json { "path": ["./src"], "reporters": ["console", "json"], "minLines": 5, "minTokens": 50, "threshold": 0, "format": ["javascript", "typescript"], "ignore": ["**/node_modules/**"], "gitignore": true, "mode": "mild" } ``` ## Format Support v5 supports **223 formats** (verified via `--list`). Use `cpd --list` to see the full list. ### Cross-Format Detection Vue SFC (`.vue`), Svelte (`.svelte`), Astro (`.astro`), and Markdown (`.md`) files are tokenized per-block/per-section, enabling duplicate detection across file types — same as v4. ### Cross-Format Groups (`--cross-formats`) By default every format is compared in its own isolated pool, so a TypeScript file never matches a near-identical JavaScript file. `--cross-formats` declares format equivalence groups that share one comparison pool — useful for finding leftover `.js` copies during a TypeScript migration: ```bash cpd --cross-formats "javascript,typescript" ./src cpd --cross-formats js-ts ./src # preset: javascript,jsx,typescript,tsx cpd --cross-formats "js-ts;css,scss" ./src # multiple groups ``` When a group mixes TypeScript (`typescript`/`tsx`) with JavaScript (`javascript`/`jsx`), TypeScript files are compared with erasable type syntax stripped from the detection token stream — type annotations, generics, `interface`/`type` declarations, `as`/`satisfies`, `?`/`!` markers, access modifiers, `implements` clauses, type-only imports/exports, overload signatures, and `declare` statements. Reported clone positions always reference the original sources. Config file equivalents (all three shapes are accepted): ```json { "crossFormats": "javascript,typescript;css,scss" } { "crossFormats": ["javascript,typescript", "css,scss"] } { "crossFormats": [["javascript", "typescript"], ["css", "scss"]] } ``` Notes: - TypeScript syntax with runtime semantics is not erased and will not cross-match: `enum`, non-declare `namespace`, parameter properties (`constructor(private x)`), `import x = require()`, `export =`. - A cross-format clone is attributed to one member format in the per-format statistics. - Overlapping groups are merged; groups with fewer than two formats are ignored. ## Differences from jscpd v4 (Node.js) | Feature | jscpd v4 (Node.js) | cpd v5 (Rust) | |---------|--------------------|-----------------| | `--blame` | Calls `git` CLI for each file | Same output (`==`/`<=` markers), calls `git blame --porcelain` per file | | `--store` (LevelDB/Redis) | Persistent store for large repos | Not supported. Use jscpd v4.x for external stores. | | `--formats-exts` | Custom format-to-extension mapping | Same flag name, same behavior | | `--formats-names` | Custom format-to-filename mapping | Same flag name, same behavior | | Programming API | `jscpd()` Promise API, `detectClones()` | Rust API via `cpd-finder` crate; no Node.js API | | Config file | `.jscpd.json` with camelCase keys | Same — `.jscpd.json` with camelCase keys | | Cross-format detection | Vue SFC, Svelte, Astro, Markdown | Same — per-block tokenization | | Token counts | Varies by tokenizer | May differ by 1-2% due to Rust tokenizer; clone detection matches | | `--reporters` | All v4 reporters | All v4 reporters except `full` (use `console-full`) | | `--no-gitignore` | Default respects `.gitignore` | Same behavior, same flag name | | `--workers` | Not available | Available — control parallelism for file tokenization/detection | | Output filenames | `jscpd-report.json`, `html/` directory | `jscpd-report.json`, `jscpd-report.html`, `jscpd-report.sarif`, `jscpd-report.csv`, `jscpd-report.md`, `jscpd-badge.svg`, `jscpd-lines-badge.svg` | ## Rust API For integration in Rust applications: ```rust use cpd_finder::orchestrate::{RunConfig, run}; let config = RunConfig { paths: vec!["./src".into()], min_tokens: 50, ..Default::default() }; let result = run(&config).unwrap(); println!("Found {} clones", result.clones.len()); println!("Analyzed {} files", result.statistics.total.sources); ``` ## Architecture ``` cpd (CLI binary) ├── cpd-core — Detection algorithm (Rabin-Karp rolling hash) ├── cpd-tokenizer — Language tokenization (223 formats) ├── cpd-finder — File walking, orchestration, git blame └── cpd-reporter — Output formatting (13 reporters) ``` --- ### Typescript # jscpd v4 (TypeScript / Node.js) Copy/paste detector for programming source code. The TypeScript engine runs on Node.js and is published as [`jscpd`](https://www.npmjs.com/package/jscpd) on npm. ## Installation ```bash # npm npm install jscpd # npx (no install required) npx jscpd /path/to/code ``` ## CLI Usage ```bash jscpd [options] ``` ### Options | Option | Short | Description | Default | |--------|-------|-------------|---------| | `--min-lines` | `-l` | Minimum lines in a clone | 5 | | `--min-tokens` | `-k` | Minimum tokens in a clone | 50 | | `--max-lines` | `-x` | Maximum source file lines | 1000 | | `--max-size` | `-z` | Maximum source file size (e.g. `1kb`, `1mb`) | `100kb` | | `--threshold` | `-t` | Duplication percentage threshold (exit with error if exceeded) | — | | `--config` | `-c` | Path to config file | `.jscpd.json` in path | | `--ignore` | `-i` | Glob patterns to exclude | — | | `--ignore-pattern` | | Regex patterns to ignore code blocks | — | | `--reporters` | `-r` | Reporters (comma-separated) | `time,console` | | `--output` | `-o` | Output directory for file reporters | `./report/` | | `--mode` | `-m` | Detection mode: `strict`, `mild`, `weak` | `mild` | | `--format` | `-f` | Formats to check (comma-separated) | all detected | | `--pattern` | `-p` | Glob pattern for file search | — | | `--blame` | `-b` | Enrich clones with git blame author data | off | | `--silent` | `-s` | Suppress console output | off | | `--store` | | Custom store (e.g. `leveldb` for large repos) | memory | | `--store-path` | | Directory for store cache (parallel runs) | — | | `--absolute` | `-a` | Use absolute paths in reports | off | | `--noSymlinks` | `-n` | Don't follow symlinks | off | | `--ignoreCase` | | Ignore case of symbols (experimental) | off | | `--gitignore` | | Respect `.gitignore` files (default: enabled) | on | | `--no-gitignore` | | Don't respect `.gitignore` files | — | | `--colors` | | Force ANSI colors even when stdout is not a TTY | — | | `--no-colors` | | Disable ANSI colors in console output | — | | `--formats-exts` | | Custom format-to-extension mapping (e.g. `javascript:es,es6;dart:dt`) | — | | `--formats-names` | | Custom format-to-filename mapping (e.g. `makefile:Makefile;docker:Dockerfile`) | — | | `--skipLocal` | | Skip clones within the same directory | off | | `--skipComments` | | Alias for `--mode weak` (ignore comments) | off | | `--noTips` | | Suppress tips and promotional messages | off | | `--exitCode` | | Exit code when clones detected | — | | `--debug` | `-d` | Show debug info, don't run detection | off | | `--verbose` | `-v` | Show full info during detection | off | | `--list` | | List all supported formats and exit | — | | `--version` | `-V` | Print version | — | | `--help` | `-h` | Print help | — | ### Color output Console colors are auto-detected. When neither `--colors` nor `--no-colors` is passed, jscpd emits ANSI colors only when stdout is a TTY, so piped or captured output (CI logs, coding agents, `jscpd . > out.txt`) stays free of escape sequences. Resolution order: `--colors` / `--no-colors` (or `colors: true|false` in config) → `FORCE_COLOR` (force on, unless `0` or `false`) → `NO_COLOR` (force off) → TTY detection. `FORCE_COLOR` is checked before `NO_COLOR` so that opting back in still works in images that export `NO_COLOR` globally, matching Node core, chalk, and the rest of the ecosystem. ### Reporters | Reporter | Output | |----------|--------| | `console` | Clone list with per-format statistics table | | `consoleFull` | Full source snippets for each clone | | `json` | `report/jscpd-report.json` | | `xml` | `report/jscpd-report.xml` | | `csv` | `report/jscpd-report.csv` | | `html` | Interactive HTML report (via `@jscpd/html-reporter`) | | `markdown` | `report/jscpd-report.md` | | `badge` | SVG badges (via `@jscpd/badge-reporter`) | | `sarif` | SARIF output for GitHub Code Scanning (via `jscpd-sarif-reporter`) | | `ai` | Token-efficient output for LLM pipelines | | `xcode` | Xcode-compatible warnings | | `threshold` | Exit 1 if duplication exceeds `--threshold` | | `silent` | No console output | You can also install third-party reporters as npm packages (e.g. `jscpd-full-reporter`). ### Config File Create `.jscpd.json` in the target directory: ```json { "path": ["./src"], "reporters": ["console", "json"], "minLines": 5, "minTokens": 50, "maxLines": 1000, "maxSize": "100kb", "threshold": 0, "format": ["javascript", "typescript"], "ignore": ["**/node_modules/**"], "gitignore": true, "mode": "mild", "absolute": false, "skipLocal": false, "skipComments": false } ``` ### Detection Modes | Mode | Behavior | |------|----------| | `strict` | All tokens must match (including whitespace, newlines) | | `mild` | Ignore empty and newline tokens | | `weak` | Ignore comments, empty tokens, and newlines (`--skipComments` is an alias) | ### Examples ```bash # Scan current directory jscpd . # Scan specific paths with options jscpd --min-lines 10 --min-tokens 100 --reporters console,json,html ./src # Scan only TypeScript files jscpd --format typescript --pattern "**/*.ts" ./src # Ignore directories jscpd --ignore "**/dist/**,**/node_modules/**" . # Skip clones within the same folder jscpd --skipLocal . # Use LevelDB store for large repos jscpd --store leveldb /path/to/large/repo # Configure LevelDB cache directory for parallel runs jscpd --store leveldb --store-path /tmp/jscpd-cache /path/to/repo ``` ## Programming API ### `jscpd` Promise API ```typescript import { IClone } from '@jscpd/core'; import { jscpd } from 'jscpd'; const clones: IClone[] = await jscpd([]); ``` ### `jscpd` with argv ```typescript import { IClone } from '@jscpd/core'; import { jscpd } from 'jscpd'; const clones: IClone[] = await jscpd(['', '', './fixtures', '-m', 'weak', '--silent']); ``` ### `detectClones` API ```typescript import { detectClones } from 'jscpd'; const clones = await detectClones({ path: ['./src'], silent: true, format: ['javascript', 'typescript'], minLines: 5, minTokens: 50, mode: 'mild', }); ``` ### `detectClones` with custom store ```typescript import { detectClones } from 'jscpd'; import { IMapFrame, MemoryStore } from '@jscpd/core'; const store = new MemoryStore(); await detectClones({ path: ['./src'], }, store); // Re-use the store for incremental detection await detectClones({ path: ['./src'], silent: true, }, store); ``` ### Building custom tools For deep customization, compose the lower-level packages: - `@jscpd/core` — Core detection algorithm, event emitter interface - `@jscpd/tokenizer` — Source code tokenization (224+ formats via reprism) - `@jscpd/finder` — File walking, clone detection, built-in reporters - `@jscpd/leveldb-store` — LevelDB persistent store for large repos - `@jscpd/redis-store` — Redis store for distributed/CI environments ## Format Support v4 supports **224 formats** (verified via `--list`). Use `jscpd --list` to see the full list. ### Cross-Format Detection Vue SFC (`.vue`), Svelte (`.svelte`), Astro (`.astro`), and Markdown (`.md`) files are tokenized per-block/per-section, enabling duplicate detection across file types (e.g., a `