# Technical Documentation: peteromallet/dataclaw

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/peteromallet/dataclaw)

### Overview (`README.md`)

# DataClaw

> **This is a performance art project.** Anthropic built their models on the world's freely shared information, then introduced increasingly [dystopian data policies](https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks) to stop anyone else from doing the same with their data - pulling up the ladder behind them. DataClaw lets you throw the ladder back down. The dataset it produces is yours to share.

Turn your Claude Code, Codex, and other coding-agent conversation history into structured data and publish it to Hugging Face with a single command. DataClaw parses session logs, redacts secrets and PII, and uploads the result as a ready-to-use dataset.



Every export is tagged **`dataclaw`** on Hugging Face. Together, they may someday form a growing [distributed dataset](https://huggingface.co/datasets?other=dataclaw) of real-world human-AI coding collaboration.

## Install

### Mac app

<p align="center">
  <a href="https://github.com/peteromallet/dataclaw/releases/latest/download/DataClaw-macOS-Apple-Silicon.dmg">
    <img alt="Download DataClaw for Apple Silicon Macs" src="https://img.shields.io/badge/Download%20for%20Mac-Apple%20Silicon-111111?style=for-the-badge&logo=apple&logoColor=white">
  </a>
  <a href="https://github.com/peteromallet/dataclaw/releases/latest">
    <img alt="View GitHub Releases" src="https://img.shields.io/badge/View%20Releases-GitHub-0969da?style=for-the-badge&logo=github&logoColor=white">
  </a>
</p>

A menu-bar app for Apple Silicon Macs. Download the DMG, drag `DataClaw.app` to Applications, and launch it from Applications or Spotlight. The app bundles everything it needs — no Python or CLI install required.

#### Opening the app for the first time

The build is currently unsigned (Apple Developer ID signing is being set up), so macOS blocks the first launch. This is expected, one-time, and takes about 20 seconds:



1. Double-click **DataClaw.app** → a dialog says it can't be opened → click **Done** (not **Move to Trash**).
2. Open **System Settings** and type **Privacy & Security** in the search field (or: Apple menu → System Settings).
3. Scroll to the **Security** section — you'll see *"DataClaw" was blocked to protect your Mac* → click **Open Anyway**.
4. In the confirmation dialog click **Open Anyway** again and authenticate with Touch ID or your password. DataClaw launches and lives in the menu bar; subsequent launches are normal.

### CLI

For the terminal workflow, Intel Macs, or driving DataClaw with a coding agent:

```bash
pip install -U dataclaw
```

## Give this to your agent

Paste this into Claude Code, Codex, or any coding agent:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## What gets exported

- User messages - Including voice transcripts and images
- Assistant responses
- Assistant thinking - Opt out with `--no-thinking`
- Tool calls - Tool name, inputs, outputs
- Token usage - Input/output tokens per session
- Metadata - Model name, git branch, timestamps

### Privacy & Redaction

DataClaw applies multiple layers of protection:

1. Username redaction - Your OS username + any configured usernames replaced with stable hashes
2. Secret redaction - Regex patterns catch JWT tokens, API keys (Anthropic, OpenAI, HF, GitHub, AWS, etc.), database passwords, private keys, Discord webhooks, and more
3. Entropy analysis - Long high-entropy strings in quotes are flagged as potential secrets
4. Email redaction - Regex pattern catches email addresses
5. Custom redaction - You can configure additional strings to redact
6. Structured redaction - All session fields, tool inputs and outputs, nested metadata, dictionary keys, and embedded JSON containers are checked. Secret-named fields such as `API_KEY_BACKUP` also provide context for opaque values.

**This is NOT foolproof.** Always review your exported data before publishing.
Automated redaction cannot catch everything - especially service-specific
identifiers, third-party PII, or secrets in unusual formats. Binary attachments and
base64 payloads are preserved; this redactor does not decode attachments or read
text from images. Embedded JSON decoding is limited to eight nested wrappers.
Credential-shaped examples may also be redacted conservatively.

We recommend converting the exported jsonl into human-readable yaml using `dataclaw jsonl-to-yaml`,
then use tools such as [trufflehog](https://github.com/trufflesecurity/trufflehog) and [gitleaks](https://github.com/gitleaks/gitleaks) to scan it.
You can also compare the exported jsonl with a previous baseline using `dataclaw diff-jsonl`.

To help improve redaction, report issues: https://github.com/peteromallet/dataclaw/issues

## Finding datasets on Hugging Face

All repos are tagged `dataclaw`.

- **Browse all:** [huggingface.co/datasets?other=dataclaw](https://huggingface.co/datasets?other=dataclaw)
- **Load one:**
  ```python
  from datasets import load_dataset
  ds = load_dataset("alice/my-personal-codex-data", split="train")
  ```
- **Combine several:**
  ```python
  from datasets import load_dataset, concatenate_datasets
  repos = ["alice/my-personal-codex-data", "bob/my-personal-codex-data"]
  ds = concatenate_datasets([load_dataset(r, split="train") for r in repos])
  ```

The auto-generated HF README includes:
- Model distribution (which models, how many sessions each)
- Total token counts
- Project count
- Last updated timestamp

## Contributing

**Missing data:** If you found any data not exported, please report an issue. You can ask your coding agent to analyze the data, export it in this repo, and open a PR.

**Better scheme:** If you need to clean the data and want to propose a better scheme, feel free to open an issue.

**New provider:** If you use a new coding agent, you can ask it to read this repo and export its data as a new provider. Take Claude Code and Codex parsers as examples because they are the most well maintained. When you finish, ask the following questions:
- Did you follow the schema the existing parsers emit? It's fine to add custom fields in `messages[].content_parts` and `tool_uses[].output.raw`.
- Did you export all data, especially:
  - tool call inputs and outputs
  - long inputs and outputs that may be saved somewhere else
  - binary content (may be encoded as base64) such as images, in both user messages and tool calls. We do not apply anonymizer on binary content
  - subagents
- Does the coding agent automatically delete old sessions? How to prevent this?

## Code Quality

<p align="center">
  <img src="scorecard.png" alt="Code Quality Scorecard">
</p>

## License

MIT

---

### RELEASING (`docs/RELEASING.md`)

# Releasing DataClaw

1. Edit no version files by hand.
2. From an up-to-date `main`, run `make release VERSION=x.y.z`.
3. `scripts/release.py` verifies a clean tree on the default branch, checks that `x.y.z` is stable semver and greater than the current version, bumps `pyproject.toml`, syncs derived files, commits `release: vx.y.z`, creates annotated tag `vx.y.z`, and pushes `main` plus tags.
4. The tag push runs `.github/workflows/release.yml`; CI builds the macOS app, uploads assets, and creates the GitHub release with generated notes.
5. Watch https://github.com/peteromallet/dataclaw/actions/workflows/release.yml and verify the new assets at https://github.com/peteromallet/dataclaw/releases/latest.
6. In `release.yml`, checkout reads the tagged tree, the tag guard compares `github.ref_name` to `pyproject.toml`, then both signed and unsigned paths run `pnpm -C app tauri build`.
7. At that build, Tauri reads the app version from `app/src-tauri/Cargo.toml` because `tauri.conf.json` has no `version`; CI keeps Cargo synced to `pyproject.toml`, so the app bundle cannot drift from the tag.

Version derivation:

- `pyproject.toml` `[project].version` is the only hand-edited source of truth.
- `app/package.json` is rewritten by `python scripts/sync_version.py`.
- `app/src-tauri/Cargo.toml` is rewritten by `python scripts/sync_version.py`.
- `app/src-tauri/Cargo.lock` `dataclaw-app` entry is rewritten by `python scripts/sync_version.py`.
- `app/src-tauri/tauri.conf.json` intentionally has no `version`; Tauri v2 uses `Cargo.toml` when the config version is omitted.

Guards:

- PR CI runs `python scripts/sync_version.py --check` and fails if any derived file drifts or if `tauri.conf.json` grows a version field.
- Release CI fails before building if pushed tag `vx.y.z` does not match `pyproject.toml`.
- `README.md` download links point at `releases/latest`; `CHANGELOG.md` keeps historical release headings only.
- There is no `dataclaw.__version__`, CLI `--version`, or `pyinstaller.spec` version copy.

---

### Claude Code Vs Cchv Gaps (`docs/claude-code-vs-cchv-gaps.md`)

# Claude Code Gaps vs CCHV

## Scope

This note compares Claude Code handling in:

- DataClaw: `~/dataclaw`
- Claude Code History Viewer (CCHV): `~/claude-code-history-viewer`

The goal is to identify Claude Code data that CCHV captures more faithfully than DataClaw today.

## Summary

CCHV retains substantially more Claude-log structure than DataClaw.

The biggest current DataClaw gaps are:

1. It flattens the raw message graph into a simplified conversation schema.
2. It preserves much more Claude tool-result structure than before, but still does not keep the full raw message/tool object model that CCHV exposes.
3. It drops Claude `summary` entries and rename-derived session metadata.
4. It drops per-message metadata such as `uuid`, `parentUuid`, `stop_reason`, and raw `usage`.
5. It merges all Claude subagent files in one session directory into a single synthetic `:subagents` export entry, while CCHV keeps per-file session granularity.

## Detailed Findings

### 1. CCHV preserves raw Claude message identity and threading; DataClaw does not

CCHV's Claude message model keeps raw message identity and graph fields such as:

- `uuid`
- `parentUuid`
- raw `sessionId`
- `isSidechain`

References:

- CCHV raw/model fields: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:24-47`
- CCHV exported message fields: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:95-143`

By contrast, DataClaw exports a simplified session object with only:

- `session_id`
- `model`
- `git_branch`
- `start_time`
- `end_time`
- `messages`
- `stats`

References:

- DataClaw session result shape: `~/dataclaw/dataclaw/parsers/common.py:54-69`
- DataClaw Claude parser only converts `user` and `assistant` entries into simplified messages: `~/dataclaw/dataclaw/parsers/claude.py:222-265`

Practical consequence:

- DataClaw loses the original message DAG / parent-child chain and raw entry identity.

### 2. CCHV keeps raw `toolUse` / `toolUseResult`; DataClaw now preserves more structure but still normalizes them

CCHV stores raw Claude tool structures directly:

- `toolUse`
- `toolUseResult`

References:

- CCHV raw tool fields: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:40-67`
- CCHV exported tool fields: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:108-143`

DataClaw now:

- extracts assistant `tool_use` blocks
- parses tool input into anonymized structured fields
- preserves normalized result text in `output.text`
- preserves extra structured result data in `output.raw`
- avoids duplicate edit/result payloads where possible

References:

- DataClaw tool-result extraction and `output.raw` preservation: `~/dataclaw/dataclaw/parsers/claude.py:86-245`
- DataClaw assistant tool export shape: `~/dataclaw/dataclaw/parsers/claude.py:373-422`

Practical consequence:

- DataClaw now keeps many structured tool-result details such as `stderr`, file objects, image payloads, and extra nested result metadata.
- It still does not preserve the full raw Claude message/tool object model that CCHV exposes, and it intentionally drops some redundant edit-result fields (`oldString`, `newString`, `structuredPatch`) plus duplicate `create` file content.

### 3. CCHV keeps per-message metadata; DataClaw mostly aggregates or drops it

CCHV keeps per-message metadata including:

- `usage`
- `stop_reason`
- `costUSD`
- `durationMs`

References:

- CCHV message content metadata: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:13-21`
- CCHV message-level exported metadata: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:117-127`

DataClaw only aggregates selected token counts into session stats and does not export message-level `usage` or `stop_reason`.

References:

- DataClaw token aggregation: `~/dataclaw/dataclaw/parsers/claude.py:251-260`
- DataClaw exported session shape has only session-level `stats`: `~/dataclaw/dataclaw/parsers/common.py:54-69`

Practical consequence:

- DataClaw loses per-message token usage and stop metadata that CCHV preserves.

### 4. CCHV captures Claude `summary` entries and rename-derived session names; DataClaw drops them

CCHV explicitly reads Claude `summary` entries into session metadata and also extracts rename information from `system/local_command` messages.

References:

- Summary ingestion during metadata extraction: `~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:318-323`
- Summary fallback in phase-2 scanning: `~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:445-455`
- Rename extraction from system/local command content: `~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:593-619`
- Session metadata fields that expose this: `~/claude-code-history-viewer/src-tauri/src/models/session.rs:51-72`

DataClaw's Claude parser only processes `user` and `assistant` entries, so Claude `summary` entries are currently ignored.

References:

- DataClaw only handles `user` / `assistant`: `~/dataclaw/dataclaw/parsers/claude.py:241-265`

Practical consequence:

- DataClaw loses Claude-generated session summaries and rename-derived display names that CCHV keeps.

### 5. CCHV keeps subagent files as distinct sessions; DataClaw merges them per session directory

CCHV recursively scans all JSONL files under the project path and builds session metadata per file path.

References:

- Recursive JSONL discovery: `~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:806-811`
- CCHV session metadata distinguishes file-path identity from raw session ID via `session_id` vs `actual_session_id`: `~/claude-code-history-viewer/src-tauri/src/models/session.rs:51-72`

DataClaw now exports Claude subagents, but it merges all `subagents/agent-*.jsonl` files in a session directory into one synthetic export session and suffixes the session ID with `:subagents` when a root session also exists.

References:

- DataClaw subagent discovery: `~/dataclaw/dataclaw/parsers/claude.py:147-162`
- DataClaw subagent merge behavior: `~/dataclaw/dataclaw/parsers/claude.py:165-218`

Practical consequence:

- DataClaw preserves more than before, but still loses per-subagent-file granularity that CCHV retains.

### 6. CCHV preserves raw `message.content` JSON blocks; DataClaw converts them into a narrow schema

CCHV preserves raw `message.content` JSON for loaded messages.

References:

- CCHV content model: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:13-21`
- CCHV message content field: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:104-105`

DataClaw instead extracts only:

- user text
- assistant text
- assistant thinking text
- assistant tool uses

References:

- DataClaw user-content extraction: `~/dataclaw/dataclaw/parsers/claude.py:268-276`
- DataClaw assistant-content extraction: `~/dataclaw/dataclaw/parsers/claude.py:279-328`

Practical consequence:

- DataClaw drops raw Claude content-block fidelity, including structured user `tool_result` content and other non-text block details.

### 7. CCHV has partial support for more Claude event types than DataClaw, but its viewer still filters some of them

CCHV's raw model knows about additional Claude fields and event payloads for:

- `summary`
- `system`
- `progress`
- `file-history-snapshot`
- `queue-operation`

References:

- CCHV raw event fields: `~/claude-code-history-viewer/src-tauri/src/models/message.rs:23-92`

However, CCHV's session message viewer excludes some event types and hides some system subtypes:

- excluded from the viewer: `progress`, `queue-operation`, `file-history-snapshot`, `last-prompt`, `pr-link`
- hidden system subtypes: `stop_hook_summary`, `turn_duration`

References:

- Excluded message types: `~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:563-590`

DataClaw is stricter still: its Claude export path ignores all non-`user` / non-`assistant` entries entirely.

References:

- DataClaw entry handling: `~/dataclaw/dataclaw/parsers/claude.py:241-265`

Practical consequence:

- CCHV is not fully lossless, but it still has broader Claude-log awareness than DataClaw.

## Observed In Real Claude Logs On This Machine

The following real Claude files on this machine contain data shapes that matter for the comparison:

- Root session example with `file-history-snapshot`, `isMeta`, `permissionMode`, raw `tool_result`, `progress`, and `thinking` blocks:
  `~/.claude/projects/-home-wd-transformers-qwen3-moe-fused/5801999e-5607-4d50-9b68-8bf91a1b9252.jsonl:1-51`

- Subagent example with `isSidechain: true` and `agentId`:
  `~/.claude/projects/-home-wd-transformers-qwen3-moe-fused/5801999e-5607-4d50-9b68-8bf91a1b9252/subagents/agent-ac4d48d.jsonl:1-48`

- Real Claude `summary` entries exist in session logs:
  `~/.claude/projects/-home-wd-rocm-systems/050b046a-5e97-40f3-88c6-4611e385f672.jsonl:393-1441`

These machine logs confirm that the gaps above are not theoretical; the omitted or flattened fields are present in actual Claude Code data.

## Bottom Line

If the goal is a training/export dataset with simple, normalized conversation rows, DataClaw's current Claude export shape is reasonable.

If the goal is Claude-log fidelity, CCHV currently retains more of the original Claude structure than DataClaw, especially around:

- raw message identity / threading
- raw message/tool object fidelity
- per-message metadata
- summaries / rename metadata
- per-file subagent granularity

---

### Codex Vs Cchv Gaps (`docs/codex-vs-cchv-gaps.md`)

# Codex Gaps vs CCHV

## Scope

This note compares Codex handling in:

- DataClaw: `~/dataclaw`
- Claude Code History Viewer (CCHV): `~/claude-code-history-viewer`

The goal is to identify Codex data that CCHV captures more faithfully than DataClaw today.

## Summary

CCHV preserves more of the raw Codex rollout structure than DataClaw.

The biggest current DataClaw gaps are:

1. It ignores `response_item.type == "message"` entries, which means it drops Codex developer/context messages and some wrapper messages.
2. It drops Codex progress/system events such as `task_started`, `task_complete`, `context_compacted`, and `turn_aborted`.
3. It only keeps session-level token totals, while CCHV attaches token usage to individual assistant messages.
4. It exports a smaller session schema and omits some session-list metadata that CCHV computes.

There is no evidence on this machine that Codex is storing separate child-agent/subagent logs outside the rollout JSONL files that DataClaw reads.

## Detailed Findings

### 1. CCHV keeps `response_item.type == "message"`; DataClaw drops them

CCHV converts all Codex `response_item` messages, including `developer`, `user`, and `assistant` roles.

References:

- CCHV message loading loop: `~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:267-393`
- CCHV message conversion for `response_item.type == "message"`: `~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:603-620`

DataClaw's Codex parser mainly handles these `response_item` variants:

- `function_call`
- `custom_tool_call`
- `reasoning`

It also now extracts user `input_image` parts from `response_item.type == "message"` and exports them via `messages[].content_parts`.

It still does not ingest most other `response_item.type == "message"` text content.

References:

- DataClaw `handle_response_item(...)`: `~/dataclaw/dataclaw/parsers/codex.py:403-452`
- DataClaw Codex user image extraction: `~/dataclaw/dataclaw/parsers/codex.py:293-399`

Practical consequence:

- DataClaw drops Codex developer messages and wrapper/context messages that are stored as normal `response_item` messages.

On this machine, real Codex rollout files contain:

- `321` developer `response_item` messages
- `1194` user `response_item` messages
- `2554` assistant `response_item` messages

Example real file with dropped developer and wrapper user messages:

- `~/.codex/sessions/2026/03/28/rollout-2026-03-28T09-51-22-019d3223-8607-7ff3-bf8f-6f7f2ca14fe4.jsonl:3-6`

Important nuance:

- Much of the assistant/user conversation text is duplicated by `event_msg.agent_message` and `event_msg.user_message`, so the biggest practical loss is the extra wrapper/context material rather than ordinary user/assistant chat text.

### Important Counterpoint: DataClaw now preserves real Codex image input on this machine

This is an important implemented improvement relative to the earlier state of the parser.

DataClaw now preserves Codex user image input as `messages[].content_parts` when it appears in:

- `response_item.payload.content[].type == "input_image"`
- or `event_msg.user_message.local_images` as a local-file fallback

References:

- DataClaw Codex image extraction: `~/dataclaw/dataclaw/parsers/codex.py:293-399`

Observed in real Codex data on this machine:

- `~/.codex/sessions/2026/04/02/rollout-2026-04-02T08-26-37-019d4b95-bc24-7992-af85-e47738953690.jsonl:6-7`

So user image input is no longer a current Codex gap in DataClaw.

### 2. CCHV keeps Codex progress/system events; DataClaw drops them

CCHV explicitly converts these Codex event types into messages:

- `task_started`
- `task_complete`
- `context_compacted`
- `turn_aborted`
- `agent_reasoning`
- `agent_message`
- `user_message`

References:

- CCHV Codex event conversion: `~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:838-1027`

DataClaw only keeps this smaller subset:

- `token_count`
- `agent_reasoning`
- `user_message`
- `agent_message`

References:

- DataClaw event handling: `~/dataclaw/dataclaw/parsers/codex.py:215-230`

Practical consequence:

- DataClaw drops Codex task/progress/system boundaries that CCHV exposes.

Observed in real Codex logs on this machine:

- `task_started`: `62`
- `task_complete`: `43`
- `context_compacted`: `78`
- `turn_aborted`: `99`

Examples:

- `task_started` and later `task_complete` in one real rollout:
  `~/.codex/sessions/2026/03/27/rollout-2026-03-27T14-05-48-019d2de6-1cd3-7cc1-89bc-fc30f7f0314f.jsonl:2,304`
- `turn_aborted` in a real rollout:
  `~/.codex/sessions/2026/03/28/rollout-2026-03-28T09-51-22-019d3223-8607-7ff3-bf8f-6f7f2ca14fe4.jsonl:101`

### 3. CCHV attaches per-message token usage; DataClaw only keeps session totals

CCHV reads Codex `token_count` events, computes deltas, and attaches usage to the last assistant message.

References:

- CCHV token handling in Codex loader: `~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:336-376`

DataClaw only updates session-level maxima/totals and stores them in `stats`.

References:

- DataClaw token aggregation: `~/dataclaw/dataclaw/parsers/codex.py:332-341`
- DataClaw session result shape: `~/dataclaw/dataclaw/parsers/common.py:56-71`

Practical consequence:

- DataClaw loses per-message token usage that CCHV preserves.

### 4. CCHV keeps more session-list metadata than DataClaw exports

CCHV computes extra Codex session metadata including:

- `file_path`
- `message_count`
- `has_tool_use`
- first-user-message `summary`

References:

- CCHV session info extraction: `~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:441-560`

DataClaw exports a smaller normalized session schema with only:

- `session_id`
- `model`
- `git_branch`
- `start_time`
- `end_time`
- `messages`
- `stats`

References:

- DataClaw normalized session shape: `~/dataclaw/dataclaw/parsers/common.py:56-71`

Practical consequence:

- DataClaw omits some Codex session-level metadata that CCHV exposes in its session browser/index.

### 5. There is no evidence here of extra Codex subagent logs that CCHV captures and DataClaw misses

DataClaw builds the Codex project index from every session JSONL file under:

- `~/.codex/sessions/**/*.jsonl`
- `~/.codex/archived_sessions/*.jsonl`

References:

- DataClaw Codex session discovery: `~/dataclaw/dataclaw/parsers/codex.py:74-88`

On this machine:

- `~/.codex/sessions` contains `78` rollout JSONL files
- `~/.codex/state_5.sqlite` contains `78` rows in `threads`
- every `threads.rollout_path` matches one of those JSONL files
- `thread_spawn_edges` has `0` rows
- `agent_jobs` has `0` rows
- `agent_job_items` has `0` rows

Practical consequence:

- There is no extra Codex child-thread/subagent graph on this machine that CCHV captures and DataClaw misses.

This is different from the Claude case, where separate sidechain files existed on disk.

## Important Counterpoint: DataClaw keeps some Codex tool metadata that CCHV normalizes away

This is not a pure one-way comparison.

For Codex tool outputs, DataClaw preserves some parsed metadata such as:

- `exit_code`
- `wall_time`
- `duration_seconds`

References:

- DataClaw Codex tool-result parsing: `~/dataclaw/dataclaw/parsers/codex.py:122-172`

CCHV normalizes tool outputs more aggressively and mainly keeps the extracted payload text.

References:

- CCHV tool output normalization: `~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:1142-1159`

So CCHV has higher Codex message/event fidelity overall, but DataClaw is not strictly worse on every Codex field.

## Bottom Line

Compared with CCHV, DataClaw currently loses more Codex rollout structure around:

- `response_item` messages, especially developer/context wrappers
- progress/system events (`task_started`, `task_complete`, `context_compacted`, `turn_aborted`)
- per-message token usage
- some session-list metadata

But for the real Codex data on this machine, there is no separate Codex subagent/session graph outside the rollout JSONL files, so DataClaw is not missing hidden child-session files in the way Claude previously was.

---

### Cursor Vs Cchv Gaps (`docs/cursor-vs-cchv-gaps.md`)

# Cursor Gaps vs CCHV

## Scope

This note compares Cursor handling in:

- DataClaw: `~/dataclaw`
- Claude Code History Viewer (CCHV): `~/claude-code-history-viewer`

There is no real Cursor data on this machine, so this comparison is based on code only.

## Summary

Based on the current implementations, CCHV appears to preserve more Cursor-specific structure than DataClaw.

The clearest current DataClaw gaps are:

1. It drops user image attachments.
2. It does not preserve `isThought` semantics for assistant thought bubbles.
3. It flattens Cursor bubble content into a smaller normalized schema and loses content-array ordering/interleaving.
4. It does not preserve some Cursor session metadata that CCHV loads from workspace composer data.
5. It may be less robust than CCHV when Cursor JSON contains embedded control characters.

## Detailed Findings

### 1. CCHV keeps user image attachments; DataClaw drops them

CCHV reads `bubble.images` for user bubbles and preserves `data:image/...;base64,...` attachments as image content blocks.

References:

- CCHV user-bubble conversion: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:439-492`
- Image handling in user bubbles: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:452-476`

DataClaw's Cursor parser only exports user `text` and ignores `images`.

References:

- DataClaw user-bubble handling: `~/dataclaw/dataclaw/parsers/cursor.py:251-262`

Practical consequence:

- If Cursor stores user screenshots or pasted image attachments in `bubble.images`, CCHV will preserve them and DataClaw will not.

### 2. CCHV preserves `isThought` as thinking; DataClaw does not

CCHV checks `bubble.isThought` on assistant bubbles and emits a `thinking` content block when present.

References:

- CCHV assistant-bubble conversion: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:494-571`
- `isThought` handling: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:542-553`

DataClaw does not check `isThought`. It only exports assistant thinking when a separate `bubble.thinking.text` field exists.

References:

- DataClaw assistant bubble with tool path: `~/dataclaw/dataclaw/parsers/cursor.py:317-321`
- DataClaw assistant bubble without tool path: `~/dataclaw/dataclaw/parsers/cursor.py:333-347`

Practical consequence:

- A Cursor bubble represented as plain `text` plus `isThought: true` would be preserved by CCHV as thought text, but would become normal assistant `content` in DataClaw.

### 3. CCHV keeps a structured content array; DataClaw flattens it

CCHV converts Cursor bubbles into a structured content array containing items such as:

- `text`
- `image`
- `thinking`
- `tool_use`
- `tool_result`

References:

- CCHV user-bubble content array: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:445-490`
- CCHV assistant-bubble content array: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:500-569`

DataClaw flattens Cursor bubbles into the normalized DataClaw schema:

- user `content`
- assistant `content`
- assistant `thinking`
- assistant `tool_uses`

References:

- DataClaw normalized session schema: `~/dataclaw/dataclaw/parsers/common.py:56-71`
- DataClaw Cursor parser: `~/dataclaw/dataclaw/parsers/cursor.py:251-356`

Practical consequence:

- Even when the same information is mostly present, DataClaw loses original content-array ordering/interleaving.
- For example, an assistant bubble that conceptually contains `tool_use` followed by `tool_result` and then additional text is normalized into `tool_uses[]` plus top-level `content`, not preserved as an ordered content array.

### 4. CCHV keeps more Cursor session metadata than DataClaw exports

CCHV reads workspace-level Cursor composer metadata and keeps fields such as:

- composer `name`
- `createdAt`
- `lastUpdatedAt`
- `unifiedMode`
- `isArchived`
- derived `has_tool_use`
- summary/title

References:

- CCHV session loading from workspace DB: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:116-187`
- Workspace composer metadata loader: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:352-380`

DataClaw exports only the normalized session schema with:

- `session_id`
- `model`
- `git_branch`
- `start_time`
- `end_time`
- `messages`
- `stats`

References:

- DataClaw session schema: `~/dataclaw/dataclaw/parsers/common.py:56-71`

Practical consequence:

- Cursor-specific session title/summary, mode, archived state, and related metadata are not preserved in DataClaw export.

### 5. CCHV may be more robust to malformed Cursor JSON

CCHV sanitizes embedded control characters before parsing Cursor JSON values.

References:

- CCHV Cursor JSON sanitizer: `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:383-397`

DataClaw calls `json.loads(...)` directly on Cursor DB values without a similar sanitization pass.

References:

- DataClaw project-index parse path: `~/dataclaw/dataclaw/parsers/cursor.py:85-86,112-113`
- DataClaw session/bubble parse path: `~/dataclaw/dataclaw/parsers/cursor.py:191-192,211-212`

Practical consequence:

- If real Cursor DB rows contain control characters or otherwise slightly malformed JSON, CCHV may still load them while DataClaw may skip them.

## Ambiguous Difference: Tool Input Field Source

CCHV and DataClaw normalize Cursor tool calls from slightly different raw fields.

CCHV uses:

- `toolFormerData.rawArgs`

References:

- `~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:516-530`

DataClaw uses:

- `toolFormerData.params`
- and a nested `tools[].parameters` unwrapping path

References:

- `~/dataclaw/dataclaw/parsers/cursor.py:272-284`

Without real Cursor data on this machine, this note does not claim that one side is definitively more faithful than the other here. It is simply an implementation difference worth noting.

## Bottom Line

Based on code only, the strongest likely Cursor gaps in DataClaw compared with CCHV are:

- user image attachments
- `isThought` / thought-bubble semantics
- richer ordered content-array structure
- extra session metadata from workspace composer state
- parser robustness for control-character-containing Cursor JSON

---

### Gemini Vs Cchv Gaps (`docs/gemini-vs-cchv-gaps.md`)

# Gemini CLI Gaps vs CCHV

## Scope

This note compares Gemini CLI handling in:

- DataClaw: `~/dataclaw`
- Claude Code History Viewer (CCHV): `~/claude-code-history-viewer`

The goal is to identify Gemini CLI data that CCHV captures more faithfully than DataClaw today, while also noting important cases where DataClaw retains data that CCHV skips.

## Summary

CCHV preserves more Gemini message/event structure than DataClaw in several places.

The biggest current DataClaw gaps are:

1. It drops Gemini `info` / `warning` / `error` messages.
2. It drops `resultDisplay` tool UI output, including file-diff previews and tool-status strings.
3. It only keeps session-level token totals, while CCHV keeps per-message Gemini token usage.
4. It does not preserve top-level Gemini session metadata such as `summary` and `kind`.

Important counterpoint:

- CCHV explicitly skips Gemini sessions whose top-level `kind` is `subagent`, while DataClaw currently exports them as normal sessions.

## Detailed Findings

### 1. CCHV keeps `info` / `warning` / `error` messages; DataClaw drops them

CCHV converts Gemini message records with types:

- `user`
- `gemini`
- `info`
- `warning`
- `error`

References:

- CCHV Gemini message dispatch: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:464-485`
- CCHV system-message conversion: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:630-653`

DataClaw only exports:

- `user`
- `gemini`

References:

- DataClaw Gemini parser message handling: `~/dataclaw/dataclaw/parsers/gemini.py:311-375`

Practical consequence:

- DataClaw drops Gemini informational and error messages that CCHV exposes as system messages.

Observed in real Gemini data on this machine:

- `835` `info` messages
- `29` `error` messages

Example real file:

- `~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-24T08-56-51cb7147.json:10,16,1111`

### 2. CCHV keeps `resultDisplay`; DataClaw ignores it

CCHV converts Gemini `toolCalls[].resultDisplay` into extra content blocks.

References:

- CCHV includes `resultDisplay` during Gemini tool-call conversion: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:592-597`
- CCHV `extract_result_display(...)`: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:893-928`

DataClaw's Gemini parser never reads `resultDisplay`.

References:

- DataClaw Gemini tool-call parsing: `~/dataclaw/dataclaw/parsers/gemini.py:162-282`

Practical consequence:

- DataClaw drops user-visible tool UI output, including:
  - short status strings such as `Found 4 matching file(s)`
  - read-file previews such as `Read lines 50-150 ...`
  - file-diff previews stored in `resultDisplay.fileDiff`
  - potential subagent-progress markers (`isSubagentProgress`) if they appear

Observed in real Gemini data on this machine:

- `4386` string `resultDisplay` values
- `854` object `resultDisplay` values containing:
  - `fileDiff`
  - `fileName`
  - `filePath`
  - `originalContent`
  - `newContent`
  - `diffStat`
  - `isNewFile`

Example real file with file-diff previews:

- `~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-28T01-43-f9f3aa2a.json:305-306,350-351,395-396,440-441`

### Important Counterpoint: DataClaw now preserves non-text Gemini user content parts

CCHV converts Gemini content parts such as:

- `inlineData` image/document blocks
- `fileData` URL-backed document blocks
- plain text parts
- `functionCall`
- `functionResponse`
- `executableCode`
- `codeExecutionResult`

References:

- CCHV content conversion helpers: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:660-867`

DataClaw now preserves Gemini user content parts via `messages[].content_parts`, including:

- `inlineData` image/document parts
- `fileData` URL-backed document parts
- `functionCall`
- `functionResponse`

It keeps the canonical joined text in `messages[].content` and stores the structured non-text parts separately.

References:

- DataClaw Gemini user content parsing: `~/dataclaw/dataclaw/parsers/gemini.py:324-430`
- DataClaw Gemini user message export: `~/dataclaw/dataclaw/parsers/gemini.py:463-474`

Practical consequence:

- This is no longer a primary Gemini gap in DataClaw.
- DataClaw now preserves user attachments and structured function parts while still exporting large base64/blob payloads verbatim.

Observed in real Gemini data on this machine:

- real `inlineData` image attachments exist in user messages, for example:
  `~/.gemini/tmp/rocm-systems/chats/session-2026-03-06T03-33-68bc726c.json:4764,4794,5006,5036`

These include large base64 image payloads that CCHV maps to image/document blocks.

### 3. CCHV keeps per-message token usage; DataClaw only keeps session totals

CCHV stores Gemini per-message token usage derived directly from each Gemini response record's `tokens` field.

References:

- CCHV Gemini usage extraction: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:601-608,616-627`

DataClaw only aggregates token counts into session-level `stats`.

References:

- DataClaw Gemini token aggregation: `~/dataclaw/dataclaw/parsers/gemini.py:340-343`
- DataClaw normalized session shape: `~/dataclaw/dataclaw/parsers/common.py:56-71`

Practical consequence:

- DataClaw loses per-message Gemini usage, including cached-input attribution on individual assistant responses.

### 4. CCHV keeps more Gemini session metadata than DataClaw exports

CCHV extracts Gemini session metadata including:

- `session_id`
- `kind`
- `start_time`
- `last_updated`
- `message_count`
- `has_tool_use`
- `summary`

References:

- CCHV Gemini lightweight metadata extraction: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:388-457`

DataClaw exports a smaller normalized session shape and does not preserve top-level Gemini `summary` or `kind`.

References:

- DataClaw Gemini metadata initialization: `~/dataclaw/dataclaw/parsers/gemini.py:300-308`
- DataClaw normalized session shape: `~/dataclaw/dataclaw/parsers/common.py:56-71`

Practical consequence:

- DataClaw loses Gemini session metadata that CCHV surfaces in its session index/browser.

Observed in real Gemini data on this machine:

- `122` Gemini session files have top-level `summary`
- top-level `kind` values present include:
  - `main`
  - `subagent`

## Important Counterpoint: CCHV skips Gemini `kind == "subagent"` sessions, but DataClaw exports them

This is an important Gemini difference in the opposite direction.

CCHV explicitly skips Gemini sessions whose top-level `kind` is `subagent` in both:

- project/session listing
- search

References:

- CCHV session listing skip: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:167-169`
- CCHV project scan skip: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:85-88`
- CCHV search skip: `~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:283-286`

DataClaw does not filter by `kind`, so it includes Gemini subagent sessions if they exist as chat files.

References:

- DataClaw Gemini discovery and parse paths read all `session-*.json` files: `~/dataclaw/dataclaw/parsers/gemini.py:123-159`

Observed in real Gemini data on this machine:

- there is exactly one real Gemini chat file with `kind: "subagent"`:
  `~/.gemini/tmp/tmp/chats/session-2026-03-05T03-59-51c63ffc.json:82`
- DataClaw successfully parses and exports that session.

So, unlike the Claude comparison, Gemini is not a simple one-way story where CCHV always preserves more.

## Observed In Real Gemini Data On This Machine

The following real Gemini structures exist on this machine and matter for the comparison:

- `info` / `error` message types:
  `~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-24T08-56-51cb7147.json:10,16,1111`

- `resultDisplay.fileDiff` edit previews:
  `~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-28T01-43-f9f3aa2a.json:305-306,350-351,395-396,440-441`

- user `inlineData` image attachments:
  `~/.gemini/tmp/rocm-systems/chats/session-2026-03-06T03-33-68bc726c.json:4764,4794,5006,5036`

- content-part `functionResponse` blocks:
  `~/.gemini/tmp/rocm-systems/chats/session-2026-03-06T03-33-68bc726c.json:122,147,172,231`

- a real Gemini subagent session:
  `~/.gemini/tmp/tmp/chats/session-2026-03-05T03-59-51c63ffc.json:82`

These are all real, present data shapes, not just theoretical parser code paths.

## Bottom Line

Compared with CCHV, DataClaw currently loses more Gemini fidelity around:

- `info` / `warning` / `error` messages
- `resultDisplay` tool UI output
- per-message token usage
- top-level session metadata like `summary` and `kind`

But CCHV also has one notable Gemini omission that DataClaw does not:

- CCHV skips Gemini `kind == "subagent"` sessions from its normal session views, while DataClaw exports them.

And DataClaw now also preserves structured Gemini user content parts such as `inlineData`, `fileData`, `functionCall`, and `functionResponse`, so those are no longer primary Gemini gaps.

---

### Hermes Provider Notes (`docs/hermes-provider-notes.md`)

# Hermes Provider Notes

## Provider contract

- Providers are registered in `dataclaw/providers.py` through `ModuleProvider.from_module`.
- A parser module must expose `SOURCE`, a module-level source path selected by `source_path_attr`, and these functions: `discover_projects`, `parse_project_sessions`, `build_export_session_tasks`, and `parse_export_session_task`.
- Parsed sessions use the shared `make_session_result` shape from `dataclaw/parsers/common.py`: `session_id`, `model`, `git_branch`, `start_time`, `end_time`, `messages`, and `stats`. `collect_project_sessions` adds `project` and `source`.
- Built-in parsers accept an `Anonymizer`, but they do not rewrite strings during parsing. Export applies `secrets.transform_session` recursively with the provider's `NON_ANON_STRING_KEYS`. Hermes follows that same flow.
- `include_thinking` controls whether parser-specific reasoning/thinking fields are emitted as message `thinking`.
- Export parallelism is provider-driven: `build_export_session_tasks` creates lightweight `ExportSessionTask` records and `parse_export_session_task` reparses one session in a worker process.

## Hermes state.db

Path inspected read-only: `~/.hermes/state.db`.

Tables of interest:

- `sessions(id, source, user_id, model, model_config, system_prompt, parent_session_id, started_at REAL, ended_at, end_reason, message_count, tool_call_count, input_tokens, output_tokens, title, cache_read_tokens, cache_write_tokens, reasoning_tokens, billing_*, estimated_cost_usd, actual_cost_usd, ..., api_call_count)`
- `messages(id, session_id, role, content, tool_call_id, tool_calls, tool_name, timestamp REAL, token_count, finish_reason, reasoning, reasoning_content, reasoning_details, codex_reasoning_items, codex_message_items)`
- FTS5 tables exist for search and are ignored.

Real counts on 2026-06-05:

- `sessions`: 10,033 total.
- By `sessions.source`: `cli` 10,026 sessions, `telegram` 6 sessions, `cron` 1 session.
- `message_count = 0`: 1,261 sessions. `message_count = 1`: 33 sessions.
- `parent_session_id` is populated on 917 sessions. The common export schema has no parent-chain field, so the provider exports each session independently.
- `messages.role`: `tool` 175,231 rows, `assistant` 127,614 rows, `user` 9,721 rows.
- Non-empty assistant `tool_calls`: 118,534 rows.
- Non-empty reasoning fields: `reasoning` 2 rows and `reasoning_content` 2 rows. `reasoning_details`, `codex_reasoning_items`, and `codex_message_items` were empty in the inspected DB.

Important timestamp detail: Hermes `started_at`, `ended_at`, and message `timestamp` values are Unix seconds, not milliseconds.

Observed message structure:

- `user` rows carry user-visible text in `content`.
- `assistant` rows carry assistant text in `content` and OpenAI-style function call lists in `tool_calls`.
- Each `tool_calls` item is a dict with keys like `id`, `call_id`, `type`, `response_item_id`, and `function`; `function` has `name` and JSON-string `arguments`.
- `tool` rows carry tool output in `content`, keyed by `tool_call_id`. Tool rows generally do not repeat the tool name, so the parser attaches them to the matching assistant tool call.

## Hermes session JSON files

Path inspected: `~/.hermes/sessions/`.

Two file families exist:

- `session_<id>.json` with keys like `session_id`, `model`, `base_url`, `platform`, `session_start`, `last_updated`, `system_prompt`, `tools`, `message_count`, and `messages`.
- `request_dump_<id>_<timestamp>.json`, which are raw/debug API request dumps.

Coverage comparison:

- `session_*.json`: 12,490 files.
- DB sessions: 10,033 rows.
- Overlap by session id: 3,825.
- DB-only sessions: 6,208.
- JSON-only sessions: 8,665.

Source-of-truth decision: use `state.db`. It is canonical for current Hermes history, has normalized tool-result rows, includes richer reasoning columns, and covers thousands of sessions absent from the JSON files. The JSON files are useful for historical/debug inspection but are not complete enough to drive export.

## Project grouping

Hermes has no per-project directory. The parser groups by `sessions.source`, yielding display names like `hermes:cli`. This mirrors the Cursor-style single-store approach: build a project index from stable metadata in the store, then parse sessions by id.

Discovery ignores sessions with `message_count <= 0`; parsing still skips any session that produces no normalized messages.

---

### Opencode Vs Cchv Gaps (`docs/opencode-vs-cchv-gaps.md`)

# OpenCode Gaps vs CCHV

## Scope

This note compares OpenCode handling in:

- DataClaw: `~/dataclaw`
- Claude Code History Viewer (CCHV): `~/claude-code-history-viewer`

The goal is to identify OpenCode data that CCHV captures more faithfully than DataClaw today, with references to both codebases and to the real OpenCode data stored on this machine.

## Summary

CCHV preserves substantially more OpenCode part-level structure than DataClaw.

The biggest current DataClaw gaps are:

1. It drops `patch`, `step-start`, `step-finish`, and `compaction` parts.
2. It loses per-message usage and cost derived from `step-finish` parts.
3. It loses message threading via `parentID`.
4. It flattens OpenCode content into a smaller normalized schema and drops part-level structure that CCHV keeps.

## Detailed Findings

### Important Counterpoint: DataClaw now preserves OpenCode user `file` parts

CCHV processes OpenCode `part.type == "file"` and preserves:

- image `data:` URIs as image blocks
- image URLs as image blocks
- non-image file references as file text blocks

References:

- CCHV OpenCode part processing: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1204-1240`

DataClaw now preserves OpenCode user `file` parts via `messages[].content_parts`, including:

- image `data:` URIs as image blocks with verbatim base64 payloads
- non-image `file://...` references as document URL blocks

References:

- DataClaw OpenCode file-source parsing: `~/dataclaw/dataclaw/parsers/opencode.py:211-245`
- DataClaw OpenCode user message extraction: `~/dataclaw/dataclaw/parsers/opencode.py:248-272`

Practical consequence:

- This is no longer a primary OpenCode gap in DataClaw for user file/image parts.
- DataClaw now preserves the real OpenCode base64 image payloads found on this machine.

Observed in real OpenCode data on this machine:

- querying `~/.local/share/opencode/opencode.db` showed `99` `file` parts
- `7` of those were image parts with `mime: image/png`
- those image parts used `data:image/png;base64,...` URLs

Example real image part from the database:

- DB file: `~/.local/share/opencode/opencode.db`
- session: `ses_3321dc5e9ffeljot1d6UlwCGOB`
- message: `msg_ccde78ea3001CsoqVaRwira24N`
- part payload contains:
  - `type: "file"`
  - `mime: "image/png"`
  - `url: "data:image/png;base64,..."`

### 1. CCHV keeps `patch`, `step-start`, `step-finish`, and `compaction` parts; DataClaw drops them

CCHV explicitly handles these OpenCode part types:

- `patch`
- `step-start`
- `step-finish`
- `compaction`

References:

- `step-finish` and `compaction`: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1130-1181`
- `patch`: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1182-1202`
- `step-start`: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1242-1249`

DataClaw's OpenCode parser does not handle any of these part types.

References:

- DataClaw assistant extraction only branches on `text`, `reasoning`, and `tool`: `~/dataclaw/dataclaw/parsers/opencode.py:235-265`

Practical consequence:

- DataClaw drops patch summaries, step boundaries, and compaction markers that CCHV shows.

Observed in real OpenCode data on this machine:

- `patch`: `2349` parts
- `step-start`: `9572` parts
- `step-finish`: `9451` parts
- `compaction`: `32` parts

Example real parts from the database:

- Patch part:
  - DB file: `~/.local/share/opencode/opencode.db`
  - session: `ses_3abf6a2e7ffeRw4KIgQY5d0prz`
  - message: `msg_c540a13e8001dgyT9upME85T5v`
  - part payload: `{"type":"patch","files":["~/ComfyUI/comfy/ldm/wan/model.py"],...}`

- Compaction part:
  - DB file: `~/.local/share/opencode/opencode.db`
  - session: `ses_3afeba227ffeLYbMdEzqlMWeKD`
  - message: `msg_c51049d72001i65GspUCMGRSZB`
  - part payload: `{"type":"compaction","auto":true}`

### 2. CCHV keeps per-message token usage and cost; DataClaw only keeps session totals

CCHV derives OpenCode per-message usage and cost from both:

- message-level fields
- `step-finish` parts

References:

- message-level usage/cost extraction: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:822-883`
- `step-finish` token/cost accumulation: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1130-1171`

DataClaw only accumulates OpenCode token counts into session-level `stats`, and does not keep per-message usage or cost.

References:

- DataClaw token accumulation: `~/dataclaw/dataclaw/parsers/opencode.py:180-186`
- DataClaw normalized session schema: `~/dataclaw/dataclaw/parsers/common.py:56-71`

Practical consequence:

- DataClaw loses per-message usage/cost fidelity that CCHV preserves.

Observed in real OpenCode data on this machine:

- many messages have no `tokens` in `message.data`, but real `step-finish` parts do have:
  - `tokens.input`
  - `tokens.output`
  - `tokens.cache.read`
  - `tokens.cache.write`
  - `cost`

Example `step-finish` payload from the database:

- DB file: `~/.local/share/opencode/opencode.db`
- contains values like:
  - `{"type":"step-finish","reason":"tool-calls","cost":0,"tokens":{"input":11294,"output":83,"cache":{"read":0,"write":0}}}`

### 3. CCHV keeps message threading via `parentID`; DataClaw drops it

CCHV maps OpenCode `parentID` to `parent_uuid` in the loaded message model.

References:

- OpenCode DB path message metadata extraction: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:827-830`
- JSON fallback path does the same: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:416-420`

DataClaw's normalized export schema does not preserve message parent links.

References:

- DataClaw session schema: `~/dataclaw/dataclaw/parsers/common.py:56-71`

Practical consequence:

- DataClaw loses OpenCode message threading / parent-child structure that CCHV keeps.

Observed in real OpenCode data on this machine:

- querying `~/.local/share/opencode/opencode.db` showed that many OpenCode messages include `parentID`
- at inspection time, `9697` messages had a non-empty `parentID`

### 4. CCHV keeps a richer OpenCode content array; DataClaw flattens it

CCHV turns OpenCode parts into a structured content array with items like:

- `text`
- `thinking`
- `tool_use`
- `tool_result`
- `image`
- patch summaries
- step summaries

References:

- CCHV `process_parts(...)`: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1049-1260`

DataClaw flattens OpenCode into a smaller schema:

- user `content`
- assistant `content`
- assistant `thinking`
- assistant `tool_uses`

References:

- DataClaw user content extraction: `~/dataclaw/dataclaw/parsers/opencode.py:210-223`
- DataClaw assistant content extraction: `~/dataclaw/dataclaw/parsers/opencode.py:226-277`

Practical consequence:

- DataClaw loses OpenCode part-level fidelity beyond the normalized text/thinking/tool schema.

### 5. Potential additional gap: CCHV has a JSON-storage fallback path; DataClaw only reads SQLite

CCHV loads OpenCode messages from SQLite first, then falls back to JSON storage files under `storage/` when needed.

References:

- CCHV SQLite-first, JSON-fallback loading: `~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:333-497`

DataClaw only reads the SQLite database at:

- `~/.local/share/opencode/opencode.db`

References:

- DataClaw OpenCode DB-only parser: `~/dataclaw/dataclaw/parsers/opencode.py:103-194`

Practical consequence:

- If some OpenCode data exists only in JSON storage and not in SQLite, CCHV can see it but DataClaw cannot.

This note does not claim that such DB-missing session content was observed on this machine; it is an implementation difference worth noting.

## Observed In Real OpenCode Data On This Machine

At inspection time, querying `~/.local/share/opencode/opencode.db` showed these real OpenCode part types:

- `tool`: `12112`
- `step-start`: `9572`
- `step-finish`: `9451`
- `reasoning`: `8641`
- `text`: `3886`
- `patch`: `2349`
- `file`: `99`
- `compaction`: `32`

The most important real payload classes for the comparison were:

- image file parts with base64 `data:` URIs
- patch parts with modified file lists
- step-finish parts with tokens/cost
- compaction markers
- pervasive `parentID` message links

## Bottom Line

Compared with CCHV, DataClaw currently loses substantial OpenCode fidelity around:

- patch / step / compaction parts
- per-message usage and cost
- parent-child message threading
- part-level content structure generally

DataClaw now preserves OpenCode user file/image parts, including real base64 image payloads. The largest remaining practical OpenCode gaps on this machine are patch / step / compaction parts and per-message usage / cost.

---

### Reliability Merge And Privacy Filter Plan (`docs/reliability-merge-and-privacy-filter-plan.md`)

# DataClaw reliability: merge-on-upload + privacy filter — design & plan

> Produced from a multi-subagent audit (2026-05-30). Every claim is cited file:line.
> Two goals: (1) uploads must **merge**, never destroy, prior data; (2) the **model
> privacy filter** the UI promises must actually run.

---

## Part A — Incremental / merge-on-upload

### Current behavior (verified)
Upload is a **wholesale overwrite**. `push_to_huggingface` (`dataclaw/_cli/exporting.py:651`)
calls `api.upload_file(path_in_repo="conversations.jsonl", ...)` (`:669`) with the locally
re-generated full file. No download, no merge. The remote dataset always equals the latest
local scan, so any session not reproducible locally **right now** is silently deleted
(second machine, log rotation, narrower `--source`, new excluded project, cleared `~/.claude`).

### Intent residue found
- `last_export_cutoff` — written by Rust (`app/src-tauri/src/dataclaw.rs:239-261`), **read by
  nobody**. Half-built incremental-cutoff scaffold.
- `dataclaw/jsonl_tools.py` — has `IDENTITY_FIELDS=("source","project","session_id","start_time")`
  (`:26`), `identity_key()` (`:221`), `index_jsonl()` (`:239`), `diff_jsonl_files()` (`:685`).
  **No merge function; not wired to upload** — `diff-jsonl` is an offline CLI command only.
- Data-safety intent was actually shipped as **trust gates** (session-shrink `review.py:616`,
  redaction-drift `commands.py:780`), all comparing to **local** `last_export` — not remote.

### The correct unit
Each JSONL line is one **session** record with a `messages` array. "New messages since last
upload" = same `identity_key`, more messages, different record hash. The merge unit is the
**session**, not the message.

### Recommended design: download → merge (union) → re-redact → re-gate → reupload

1. **Download** prior remote `conversations.jsonl` via `hf_hub_download(repo_id,
   "conversations.jsonl", repo_type="dataset")`. **Fail closed**: only a confirmed 404
   (`EntryNotFoundError`) or missing-repo (`RepositoryNotFoundError`) counts as "empty remote";
   any network/auth/HTTP error **aborts the push** (never overwrite remote with local-only).
2. **Union by `identity_key`** (read records RAW, not via `index_jsonl` — that normalizes and
   strips `originalFile`, see H3):
   - remote-only → carry forward (never drop)
   - local-only → add
   - both → keep superset (more messages; tie-break larger bytes, then later `end_time`)
   - **Invariant `merged_total >= remote_total`** ⇒ shrink is impossible by construction.
3. **🔴 RE-REDACT carried-forward remote records through the CURRENT redactor** before writing
   them (see "Critical correctness requirement" below). This is mandatory, not optional.
4. **Re-run the confirm gates on the MERGED file and re-hash it** so reviewed-bytes ==
   shipped-bytes (preserve the SHA invariant at `commands.py:365-376`).
5. **Upload the merged union** (not the local export) with `parent_commit=<repo head sha>`
   for optimistic concurrency; on 412 conflict re-download+re-merge (cap ~5 attempts).
   Update `meta["sessions"]`/`last_export.sessions` to `merged_total` so metadata and the
   shrink gate stay consistent (H5).

Hook point: `push_to_huggingface` upload block (`exporting.py:666-675`). Merge fn belongs in
`jsonl_tools.py` next to `identity_key`/`index_jsonl`. Draft code exists in the agent transcript.

### 🔴 Critical correctness requirement (the naive merge is a PRIVACY REGRESSION)
Remote records were redacted under **whatever policy was current when they were uploaded**.
Carrying them forward verbatim **re-publishes old, looser-policy redaction** and **bypasses
every gate** (gates run in `confirm()` on the local file, before the merge). Adversarial review
verdict on the naive union: scenarios (a) new redact_string, (b) superset-by-count picking the
older copy, (c) future model filter, (d) gate bypass → **leak, leak, leak, total-bypass**.

**Fix:** re-run `secrets.transform_session` (+ `Anonymizer`) over every carried-forward record
with the **current** config before reupload. This is **idempotent and pseudonym-stable**
(verified): secrets become the constant `[REDACTED]` which matches no pattern; pseudonyms are
deterministic `user_+sha256(name)[:8]` and the hash token doesn't re-match the username regex.
Then re-gate + re-hash the merged file.

### Additional holes the merge must handle
- **H1 `start_time` key drift** (`jsonl_tools.py:26`, `parsers/common.py:157-164`): format drift
  (`Z` vs `+00:00`, `None` vs backfilled) → same session under two keys → **duplicate + leak**.
  Fix: canonicalize `start_time` before keying, or key on `(source, project, session_id)`.
- **H2 anonymized `project` in key** (`secrets.py:572-576`): `project` is anonymized in-place and
  is part of the key → key drift. Strip `project` from key or key on post-anon identity.
- **H3 `normalize_for_diff` strips `originalFile`** (`jsonl_tools.py:234-235`): don't read remote
  records via the diff-normalizing loader or you destroy `originalFile` content.
- **H4 schema drift**: old remote records may lack fields the current walker expects → missed
  secrets or crash. Re-redaction must be schema-tolerant.
- **H5 non-atomic upload + stale `last_export.sessions`**: 3 separate `upload_file` commits;
  update session count to merged total.

### Trust gates
Keep the existing **local** gates unchanged (they guard the human confirm step). Do **not** add a
remote-shrink gate — the union invariant makes remote shrink impossible. But the merged file
**must** pass the gates after merge+re-redaction (step 4).

---

## Part B — Model-based privacy filter

### It was real, and was lost (not reverted)
A complete **848-line `dataclaw/privacy_filter.py`** + `tests/test_privacy_filter.py` was
committed on `5d0a741` ("Add macOS app release flow", a side branch) and **never merged to main**.
A *different* same-titled commit (`42b21d1`) landed on main without it. The file was on disk as
late as the 2026-05-16 desloppify scan, then lost untracked. The `pii` deps
(`transformers`, `torch`, `accelerate`, `tokenizers`) were dropped from HEAD `pyproject.toml`.

Recover with: `git show 5d0a741:dataclaw/privacy_filter.py` and `:tests/test_privacy_filter.py`.

### What it does (and quality)
`transformers.pipeline(task="token-classification", aggregation_strategy="simple")` NER with real
CPU/MPS device handling (`resolve_device`/`_auto_device`), 480-token chunking, min_score 0.85,
oversized-session blanket-redact guard, reverse-order span splicing. **Core logic is sound.**

Bugs / gaps (line refs on the `5d0a741` version):
- 🔴 **Field-walk mismatch (must fix):** it walks `messages[].content/thinking` + `tool_uses[].input/output`
  but **NOT `messages[].content_parts`**, which HEAD's `secrets.transform_session` *does* walk
  (`secrets.py:590`). PII in `content_parts` would pass untouched. Add `content_parts` to the
  field loops in `scan_session`/`redact_session`/`_redact_oversized_session` + a test.
- 🔴 **`MODEL_ID = "openai/privacy-filter"` is a placeholder** — does not exist on the Hub.
- Minor: cross-chunk-boundary entities may be split/under-redacted (low severity); `min_score`
  discarded in `_load` then re-applied in `scan_text` (cosmetic); `dtype=` kwarg needs transformers ≥4.45.

### Model decision (drop-in for the existing pipeline call)
- **Default: `openai/privacy-filter`** — the model the original implementation
  intended. It is a REAL, published model (Apache-2.0, ~300k downloads,
  token-classification, safetensors + ONNX) — the earlier "placeholder that doesn't
  exist" claim was WRONG. It drops into the existing
  `pipeline(task="token-classification", aggregation_strategy="simple")` call with no
  adapter code and detects PII as intended.
- Overridable via the `privacy_filter.model` config key / `DATACLAW_PRIVACY_FILTER_MODEL`
  env var, so swapping is trivial.
- Alternatives if ever needed: `lakshyakh93/deberta_finetuned_pii` (MIT, broad labels,
  pure drop-in). Avoid GLiNER / Presidio (need adapters / heavier deps).

### Rewiring against HEAD `_cli` structure
- **Mutation happens only at EXPORT time**, inline per session, right after
  `secrets.transform_session` (`exporting.py:201` parallel worker, `:292` serial). Confirm
  (`review.py:625`) is **read-only** and hashes the file (`review.py:830`); publish enforces that
  hash (`commands.py:365-376`). **So the model edits MUST run at export, before the hash lock** —
  not at confirm.
- Keep the dict/text functions (`redact_session`, `redact_text`, `scan_text`, `_load`,
  `resolve_device`, oversized guard); **discard** the shard/jsonl/manifest layer (HEAD has no
  run_dir/manifest pipeline — it was dropped).
- Read config (free-form dict, no schema in `config.py`):
  `privacy_filter.{enabled (default False), device, min_score=0.85, include_tool_io=True, roles={user}}`.
  Rust already writes `privacy_filter.enabled/.device` (`dataclaw.rs:571,580`) — this closes the gap.
- Emit `privacy_filter_*` events via `progress_callback` → lights up the **already-present** dead
  Dashboard `model_privacy` stage (`Dashboard.tsx:209-210`, handlers `:434-516`). `mechanical_pii`
  stage maps to the existing regex layer.

### Deps / packaging
- Restore `[project.optional-dependencies] pii = ["transformers>=4.57","torch>=2.3","accelerate","tokenizers"]`
  + the `pii` pytest marker. Keep as an **extra** so the lazy-import graceful-degradation holds.
- **Do NOT bundle torch into the Mac sidecar by default** (~1.5–2.5 GB; `pyinstaller.spec` has no
  torch hooks). Default `enabled=false`. Treat the model stage as power-user / `dataclaw[pii]`, or
  a separate heavy build. **Download the model at first run** (HF cache), don't bundle weights.
- **Graceful degradation:** if enabled but torch/model/network unavailable → **warn and continue
  with mechanical-only redaction**, do not abort the export.

---

## Prioritized action list
1. **Merge-on-upload with re-redaction + re-gate** (Part A). The data-destruction bug *and* the
   privacy regression it would introduce — do them together; one is unsafe without the other.
2. **Atomic export write** (temp+fsync+rename in `exporting.py:493`) — independent, stops silently
   truncated datasets being published. (From the first audit.)
3. **Privacy filter recovery + content_parts fix + real MODEL_ID + export-time rewire** (Part B).
4. **Source enum reconcile** (drop `hermes`, add `cursor`, validate in Rust, fail-closed in Python)
   and **harden secret patterns + salt the anonymizer**. (From the first audit.)
5. Real cross-process run lock; batch the 3 HF uploads into one `create_commit`.

## Merge-quality review (4-perspective sense-check) + fixes shipped

A focused review of the update/merge mechanism (correctness, reliability, elegance, scale)
found the design **fundamentally right and reliable** (fail-closed, convergent, self-healing,
optimistic-concurrency correct) but flagged a scale problem the carry-forward re-redaction fix
introduced. Shipped in `619ef63`:

- **Keystone — redaction-policy stamp.** Each record is stamped with
  `redaction_policy_version()` (hash of redact strings/usernames + model config + code
  version). The carry-forward redactor skips re-redaction for records already stamped current,
  re-redacting only stale ones. Turns a steady-state push from **O(total history)** (was
  re-running the PII model over the whole dataset every push) into **~O(new data)**, while
  preserving tighten-only (policy change bumps the version → one-time full re-scan).
- **Skip-unchanged.** If the merged file is byte-identical to the downloaded remote, skip all
  uploads — no more empty-commit churn from the metadata timestamp.
- **Corrupt-line guard.** `_load_raw_records` preserves unparseable JSONL lines verbatim
  instead of raising — a corrupt remote line can't drop data or wedge all future pushes.

### Known/deferred from the merge review (not yet done)
- **Memory**: merge materializes both full files in RAM → OOM risk past ~1–2 GB (streaming merge).
- **Network**: full re-download + re-upload every push (sharding — larger change).
- **Compaction semantics**: "more messages wins" keeps the stale pre-compaction copy when a
  conversation was legitimately summarized (keeps data, doesn't lose it; judgment call).
- **Atomicity**: 3 uploads are separate commits (self-heals; `create_commit` would make atomic).
- Model-load-status publish-time signal; `git_branch` redaction; dead UI telemetry. (From earlier audits.)

## Open verification items (couldn't run live)
- `huggingface_hub` version: confirm `upload_file(parent_commit=...)` support and that conflicts
  surface as HTTP 412; confirm `hf_hub_download` error classes.
- Live Hub availability + offset behavior of the two recommended `MODEL_ID`s.

---

--- METRICS ---
- Files Extracted: 9
- Estimated Token Budget: ~16967 tokens
- Canonical Reference: https://codewiki.google/github.com/peteromallet/dataclaw