{"owner":"kepano","repo":"defuddle","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md"],"skills":{"CLAUDE.md":"# Defuddle\n\nExtracts main content from web pages as clean HTML.\n\n## Project structure\n\n- `src/defuddle.ts` — Core parsing pipeline\n- `src/standardize.ts` — HTML normalization (headings, code blocks, footnotes)\n- `src/scoring.ts` — Content scoring to remove non-content blocks\n- `src/constants.ts` — Exact/partial selectors for clutter removal\n- `src/elements/` — Element-specific rules (code, footnotes, math)\n- `src/extractors/` — Site-specific extractors\n- `src/utils/dom.ts` — DOM utilities (`parseHTML`, `serializeHTML`)\n- `src/index.ts` / `src/index.full.ts` — Bundle entry points (UMD, `export: 'default'`)\n- `website/src/convert.ts` — Cloudflare Worker API (defuddle.md)\n\n## Environments\n\n- **Browser** (`defuddle`, `defuddle/full`) — Native DOM. Used by extensions and web apps.\n- **Node.js** (`defuddle/node`) — Accepts any DOM `Document` (linkedom, JSDOM, happy-dom, etc.). Async API.\n- **CLI** (`src/cli.ts`) — linkedom. Supports `--markdown` and `--json` flags.\n- **Cloudflare Worker** (`website/src/convert.ts`) — linkedom, most constrained DOM.\n\n## Build and test\n\n- `npm run build` — Build all bundles\n- `npm test` — Run Vitest\n\n### Testing across environments\n\nAlways use `curl` when testing the Worker or defuddle.md — do not open URLs in a browser.\n\n1. **Worker** — run with `cd website && npx wrangler dev`, then:\n   ```bash\n   curl http://localhost:8787/https://stephango.com/saw\n   ```\n2. **defuddle.md** — same pattern against production:\n   ```bash\n   curl https://defuddle.md/https://stephango.com/saw\n   ```\n3. **CLI**: `npx defuddle parse https://stephango.com/saw --markdown`\n4. **Vitest fixtures**: HTML files in `tests/fixtures/` with expected output in `tests/expected/`\n\n## Debugging content extraction\n\n### Pipeline order\n\n1. Flatten shadow DOM (`flattenShadowRoots`)\n2. Resolve React streaming SSR (`resolveStreamedContent`)\n3. Find main content (auto-detection or `contentSelector`)\n4. `standardizeFootnotes` — runs before removals because CSS sidenotes use `display:none`\n5. `standardizeCallouts` — converts GitHub alerts, Bootstrap alerts, callout asides to `blockquote[data-callout]` before selector removal strips `.alert` etc.\n6. `removeSmallImages`\n7. `removeHiddenElements`\n8. `removeLowScoring`\n9. `removeBySelector` — exact and partial selectors from `src/constants.ts`\n10. `removeByContentPattern` — content-based removal (read time, boilerplate, article cards)\n11. `standardizeContent` — HTML normalization\n12. Resolve relative URLs\n\n### Pipeline toggles\n\n```typescript\nnew Defuddle(document, {\n  removeSmallImages: false,\n  removeHiddenElements: false,\n  removeLowScoring: false,\n  removeExactSelectors: false,\n  removePartialSelectors: false,\n  removeContentPatterns: false,\n  standardize: false,  // disables standardizeFootnotes and standardizeContent\n  includeReplies: false, // excludes replies from extractors like Reddit, HN, GitHub, Twitter/X\n}).parse();\n```\n\n### Debug mode\n\n```typescript\nconst result = new Defuddle(document, { debug: true }).parse();\nresult.debug.contentSelector; // CSS path of chosen content element\nresult.debug.removals;        // array of {step, selector, reason, text}\n```\n\nDebug mode preserves class/id/data-* attributes and skips div flattening. Use `contentSelector` to bypass auto-detection when it picks the wrong element.\n\n### Debugging strategy\n\n1. Check `result.debug.removals` for unexpected entries\n2. Disable steps one at a time to find which one removes the content\n3. For selector issues, check `EXACT_SELECTORS` and `PARTIAL_SELECTORS` in `src/constants.ts`\n4. Elements inside `<pre>` or `<code>` are protected from selector removal\n5. After fixing, create a minimal fixture in `tests/fixtures/` with expected output in `tests/expected/` to prevent regressions. Anonymize fixtures — replace real names, emails, URLs, and identifying content with generic placeholders. **Verify the fixture fails before applying the fix** — run `npm test` with the fix reverted (or with the expected output reflecting the buggy behavior) to confirm the fixture actually exercises the bug. A fixture that passes on both old and new code proves nothing.\n\n### Rules\n\n- **Never use `innerHTML` directly.** Always use `parseHTML()` from `src/utils/dom.ts` which parses via `<template>` elements (no script execution, no resource loading).\n- **Sanitize URLs** — `javascript:`, `blob:`, and non-image `data:` URLs must be stripped from `href`/`src` attributes; `data:image/*` is allowed except as an iframe `src`. `srcdoc` must be stripped from iframes. `on*` event handler attributes must be removed. See `isDangerousUrl()` in `src/utils/dom.ts` and `_stripUnsafeElements()` in `src/defuddle.ts`, which runs on the main extraction output regardless of which pipeline steps are enabled.\n- **Never interpolate page-derived values into HTML strings unescaped.** Extractors that build HTML from template literals (e.g. `` `<img src=\"${src}\" alt=\"${alt}\">` ``) must wrap every interpolated value in `escapeHtml()` from `src/utils/dom.ts`, or build the node via `createElement`/`setAttribute` + `serializeHTML()`. An unescaped `\"` lets attacker-controlled values (image `alt`/`src`, `og:image`, descriptions) inject new attributes (XSS, e.g. GHSA-jg4p-g6xj-4qmf). Downstream sanitization is a backstop, not a substitute — escape at the point of interpolation. Cover new extractors with a poisoned-attribute case in `tests/extractor-xss.test.ts`.\n\n### Common pitfalls\n- **UMD exports**: `export: 'default'` in webpack means named exports must be static properties on the default class (see `src/index.full.ts`).\n- **Live HTMLCollections**: `getElementsByTagName` returns live collections. Convert to static arrays with `Array.from()` before mutating the DOM.\n"},"files":{"CLAUDE.md":"# Defuddle\n\nExtracts main content from web pages as clean HTML.\n\n## Project structure\n\n- `src/defuddle.ts` — Core parsing pipeline\n- `src/standardize.ts` — HTML normalization (headings, code blocks, footnotes)\n- `src/scoring.ts` — Content scoring to remove non-content blocks\n- `src/constants.ts` — Exact/partial selectors for clutter removal\n- `src/elements/` — Element-specific rules (code, footnotes, math)\n- `src/extractors/` — Site-specific extractors\n- `src/utils/dom.ts` — DOM utilities (`parseHTML`, `serializeHTML`)\n- `src/index.ts` / `src/index.full.ts` — Bundle entry points (UMD, `export: 'default'`)\n- `website/src/convert.ts` — Cloudflare Worker API (defuddle.md)\n\n## Environments\n\n- **Browser** (`defuddle`, `defuddle/full`) — Native DOM. Used by extensions and web apps.\n- **Node.js** (`defuddle/node`) — Accepts any DOM `Document` (linkedom, JSDOM, happy-dom, etc.). Async API.\n- **CLI** (`src/cli.ts`) — linkedom. Supports `--markdown` and `--json` flags.\n- **Cloudflare Worker** (`website/src/convert.ts`) — linkedom, most constrained DOM.\n\n## Build and test\n\n- `npm run build` — Build all bundles\n- `npm test` — Run Vitest\n\n### Testing across environments\n\nAlways use `curl` when testing the Worker or defuddle.md — do not open URLs in a browser.\n\n1. **Worker** — run with `cd website && npx wrangler dev`, then:\n   ```bash\n   curl http://localhost:8787/https://stephango.com/saw\n   ```\n2. **defuddle.md** — same pattern against production:\n   ```bash\n   curl https://defuddle.md/https://stephango.com/saw\n   ```\n3. **CLI**: `npx defuddle parse https://stephango.com/saw --markdown`\n4. **Vitest fixtures**: HTML files in `tests/fixtures/` with expected output in `tests/expected/`\n\n## Debugging content extraction\n\n### Pipeline order\n\n1. Flatten shadow DOM (`flattenShadowRoots`)\n2. Resolve React streaming SSR (`resolveStreamedContent`)\n3. Find main content (auto-detection or `contentSelector`)\n4. `standardizeFootnotes` — runs before removals because CSS sidenotes use `display:none`\n5. `standardizeCallouts` — converts GitHub alerts, Bootstrap alerts, callout asides to `blockquote[data-callout]` before selector removal strips `.alert` etc.\n6. `removeSmallImages`\n7. `removeHiddenElements`\n8. `removeLowScoring`\n9. `removeBySelector` — exact and partial selectors from `src/constants.ts`\n10. `removeByContentPattern` — content-based removal (read time, boilerplate, article cards)\n11. `standardizeContent` — HTML normalization\n12. Resolve relative URLs\n\n### Pipeline toggles\n\n```typescript\nnew Defuddle(document, {\n  removeSmallImages: false,\n  removeHiddenElements: false,\n  removeLowScoring: false,\n  removeExactSelectors: false,\n  removePartialSelectors: false,\n  removeContentPatterns: false,\n  standardize: false,  // disables standardizeFootnotes and standardizeContent\n  includeReplies: false, // excludes replies from extractors like Reddit, HN, GitHub, Twitter/X\n}).parse();\n```\n\n### Debug mode\n\n```typescript\nconst result = new Defuddle(document, { debug: true }).parse();\nresult.debug.contentSelector; // CSS path of chosen content element\nresult.debug.removals;        // array of {step, selector, reason, text}\n```\n\nDebug mode preserves class/id/data-* attributes and skips div flattening. Use `contentSelector` to bypass auto-detection when it picks the wrong element.\n\n### Debugging strategy\n\n1. Check `result.debug.removals` for unexpected entries\n2. Disable steps one at a time to find which one removes the content\n3. For selector issues, check `EXACT_SELECTORS` and `PARTIAL_SELECTORS` in `src/constants.ts`\n4. Elements inside `<pre>` or `<code>` are protected from selector removal\n5. After fixing, create a minimal fixture in `tests/fixtures/` with expected output in `tests/expected/` to prevent regressions. Anonymize fixtures — replace real names, emails, URLs, and identifying content with generic placeholders. **Verify the fixture fails before applying the fix** — run `npm test` with the fix reverted (or with the expected output reflecting the buggy behavior) to confirm the fixture actually exercises the bug. A fixture that passes on both old and new code proves nothing.\n\n### Rules\n\n- **Never use `innerHTML` directly.** Always use `parseHTML()` from `src/utils/dom.ts` which parses via `<template>` elements (no script execution, no resource loading).\n- **Sanitize URLs** — `javascript:`, `blob:`, and non-image `data:` URLs must be stripped from `href`/`src` attributes; `data:image/*` is allowed except as an iframe `src`. `srcdoc` must be stripped from iframes. `on*` event handler attributes must be removed. See `isDangerousUrl()` in `src/utils/dom.ts` and `_stripUnsafeElements()` in `src/defuddle.ts`, which runs on the main extraction output regardless of which pipeline steps are enabled.\n- **Never interpolate page-derived values into HTML strings unescaped.** Extractors that build HTML from template literals (e.g. `` `<img src=\"${src}\" alt=\"${alt}\">` ``) must wrap every interpolated value in `escapeHtml()` from `src/utils/dom.ts`, or build the node via `createElement`/`setAttribute` + `serializeHTML()`. An unescaped `\"` lets attacker-controlled values (image `alt`/`src`, `og:image`, descriptions) inject new attributes (XSS, e.g. GHSA-jg4p-g6xj-4qmf). Downstream sanitization is a backstop, not a substitute — escape at the point of interpolation. Cover new extractors with a poisoned-attribute case in `tests/extractor-xss.test.ts`.\n\n### Common pitfalls\n- **UMD exports**: `export: 'default'` in webpack means named exports must be static properties on the default class (see `src/index.full.ts`).\n- **Live HTMLCollections**: `getElementsByTagName` returns live collections. Convert to static arrays with `Array.from()` before mutating the DOM.\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# Defuddle\n\nExtracts main content from web pages as clean HTML.\n\n## Project structure\n\n- `src/defuddle.ts` — Core parsing pipeline\n- `src/standardize.ts` — HTML normalization (headings, code blocks, footnotes)\n- `src/scoring.ts` — Content scoring to remove non-content blocks\n- `src/constants.ts` — Exact/partial selectors for clutter removal\n- `src/elements/` — Element-specific rules (code, footnotes, math)\n- `src/extractors/` — Site-specific extractors\n- `src/utils/dom.ts` — DOM utilities (`parseHTML`, `serializeHTML`)\n- `src/index.ts` / `src/index.full.ts` — Bundle entry points (UMD, `export: 'default'`)\n- `website/src/convert.ts` — Cloudflare Worker API (defuddle.md)\n\n## Environments\n\n- **Browser** (`defuddle`, `defuddle/full`) — Native DOM. Used by extensions and web apps.\n- **Node.js** (`defuddle/node`) — Accepts any DOM `Document` (linkedom, JSDOM, happy-dom, etc.). Async API.\n- **CLI** (`src/cli.ts`) — linkedom. Supports `--markdown` and `--json` flags.\n- **Cloudflare Worker** (`website/src/convert.ts`) — linkedom, most constrained DOM.\n\n## Build and test\n\n- `npm run build` — Build all bundles\n- `npm test` — Run Vitest\n\n### Testing across environments\n\nAlways use `curl` when testing the Worker or defuddle.md — do not open URLs in a browser.\n\n1. **Worker** — run with `cd website && npx wrangler dev`, then:\n   ```bash\n   curl http://localhost:8787/https://stephango.com/saw\n   ```\n2. **defuddle.md** — same pattern against production:\n   ```bash\n   curl https://defuddle.md/https://stephango.com/saw\n   ```\n3. **CLI**: `npx defuddle parse https://stephango.com/saw --markdown`\n4. **Vitest fixtures**: HTML files in `tests/fixtures/` with expected output in `tests/expected/`\n\n## Debugging content extraction\n\n### Pipeline order\n\n1. Flatten shadow DOM (`flattenShadowRoots`)\n2. Resolve React streaming SSR (`resolveStreamedContent`)\n3. Find main content (auto-detection or `contentSelector`)\n4. `standardizeFootnotes` — runs before removals because CSS sidenotes use `display:none`\n5. `standardizeCallouts` — converts GitHub alerts, Bootstrap alerts, callout asides to `blockquote[data-callout]` before selector removal strips `.alert` etc.\n6. `removeSmallImages`\n7. `removeHiddenElements`\n8. `removeLowScoring`\n9. `removeBySelector` — exact and partial selectors from `src/constants.ts`\n10. `removeByContentPattern` — content-based removal (read time, boilerplate, article cards)\n11. `standardizeContent` — HTML normalization\n12. Resolve relative URLs\n\n### Pipeline toggles\n\n```typescript\nnew Defuddle(document, {\n  removeSmallImages: false,\n  removeHiddenElements: false,\n  removeLowScoring: false,\n  removeExactSelectors: false,\n  removePartialSelectors: false,\n  removeContentPatterns: false,\n  standardize: false,  // disables standardizeFootnotes and standardizeContent\n  includeReplies: false, // excludes replies from extractors like Reddit, HN, GitHub, Twitter/X\n}).parse();\n```\n\n### Debug mode\n\n```typescript\nconst result = new Defuddle(document, { debug: true }).parse();\nresult.debug.contentSelector; // CSS path of chosen content element\nresult.debug.removals;        // array of {step, selector, reason, text}\n```\n\nDebug mode preserves class/id/data-* attributes and skips div flattening. Use `contentSelector` to bypass auto-detection when it picks the wrong element.\n\n### Debugging strategy\n\n1. Check `result.debug.removals` for unexpected entries\n2. Disable steps one at a time to find which one removes the content\n3. For selector issues, check `EXACT_SELECTORS` and `PARTIAL_SELECTORS` in `src/constants.ts`\n4. Elements inside `<pre>` or `<code>` are protected from selector removal\n5. After fixing, create a minimal fixture in `tests/fixtures/` with expected output in `tests/expected/` to prevent regressions. Anonymize fixtures — replace real names, emails, URLs, and identifying content with generic placeholders. **Verify the fixture fails before applying the fix** — run `npm test` with the fix reverted (or with the expected output reflecting the buggy behavior) to confirm the fixture actually exercises the bug. A fixture that passes on both old and new code proves nothing.\n\n### Rules\n\n- **Never use `innerHTML` directly.** Always use `parseHTML()` from `src/utils/dom.ts` which parses via `<template>` elements (no script execution, no resource loading).\n- **Sanitize URLs** — `javascript:`, `blob:`, and non-image `data:` URLs must be stripped from `href`/`src` attributes; `data:image/*` is allowed except as an iframe `src`. `srcdoc` must be stripped from iframes. `on*` event handler attributes must be removed. See `isDangerousUrl()` in `src/utils/dom.ts` and `_stripUnsafeElements()` in `src/defuddle.ts`, which runs on the main extraction output regardless of which pipeline steps are enabled.\n- **Never interpolate page-derived values into HTML strings unescaped.** Extractors that build HTML from template literals (e.g. `` `<img src=\"${src}\" alt=\"${alt}\">` ``) must wrap every interpolated value in `escapeHtml()` from `src/utils/dom.ts`, or build the node via `createElement`/`setAttribute` + `serializeHTML()`. An unescaped `\"` lets attacker-controlled values (image `alt`/`src`, `og:image`, descriptions) inject new attributes (XSS, e.g. GHSA-jg4p-g6xj-4qmf). Downstream sanitization is a backstop, not a substitute — escape at the point of interpolation. Cover new extractors with a poisoned-attribute case in `tests/extractor-xss.test.ts`.\n\n### Common pitfalls\n- **UMD exports**: `export: 'default'` in webpack means named exports must be static properties on the default class (see `src/index.full.ts`).\n- **Live HTMLCollections**: `getElementsByTagName` returns live collections. Convert to static arrays with `Array.from()` before mutating the DOM.\n","category":"root","tokens":1431}]}