dataclaw Documentation

Agent harness to publish your agent chat history as Huggingface datasets.

2,119 stars Python Python 9 files · ~17,050 tokens
RAW Doc

Technical Documentation: peteromallet/dataclaw

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

DataClaw

This is a performance art project. Anthropic built their models on the world's freely shared information, then introduced increasingly dystopian data policies to stop anyone else from doing the same with their data - pulling up the ladder behind them. DataClaw lets you throw the ladder back down. The dataset it produces is yours to share.

Turn your Claude Code, Codex, and other coding-agent conversation history into structured data and publish it to Hugging Face with a single command. DataClaw parses session logs, redacts secrets and PII, and uploads the result as a ready-to-use dataset.

Every export is tagged dataclaw on Hugging Face. Together, they may someday form a growing distributed dataset of real-world human-AI coding collaboration.

Install

Mac app

A menu-bar app for Apple Silicon Macs. Download the DMG, drag DataClaw.app to Applications, and launch it from Applications or Spotlight. The app bundles everything it needs — no Python or CLI install required.

Opening the app for the first time

The build is currently unsigned (Apple Developer ID signing is being set up), so macOS blocks the first launch. This is expected, one-time, and takes about 20 seconds:

  1. Double-click DataClaw.app → a dialog says it can't be opened → click Done (not Move to Trash).
  2. Open System Settings and type Privacy & Security in the search field (or: Apple menu → System Settings).
  3. Scroll to the Security section — you'll see "DataClaw" was blocked to protect your Mac → click Open Anyway.
  4. In the confirmation dialog click Open Anyway again and authenticate with Touch ID or your password. DataClaw launches and lives in the menu bar; subsequent launches are normal.

CLI

For the terminal workflow, Intel Macs, or driving DataClaw with a coding agent:

bash
pip install -U dataclaw

Give this to your agent

Paste this into Claude Code, Codex, or any coding agent:

text
/* Detailed source-code truncated for AI context efficiency. */

What gets exported

  • User messages - Including voice transcripts and images
  • Assistant responses
  • Assistant thinking - Opt out with --no-thinking
  • Tool calls - Tool name, inputs, outputs
  • Token usage - Input/output tokens per session
  • Metadata - Model name, git branch, timestamps

Privacy & Redaction

DataClaw applies multiple layers of protection:

  1. Username redaction - Your OS username + any configured usernames replaced with stable hashes
  2. Secret redaction - Regex patterns catch JWT tokens, API keys (Anthropic, OpenAI, HF, GitHub, AWS, etc.), database passwords, private keys, Discord webhooks, and more
  3. Entropy analysis - Long high-entropy strings in quotes are flagged as potential secrets
  4. Email redaction - Regex pattern catches email addresses
  5. Custom redaction - You can configure additional strings to redact
  6. Structured redaction - All session fields, tool inputs and outputs, nested metadata, dictionary keys, and embedded JSON containers are checked. Secret-named fields such as API_KEY_BACKUP also provide context for opaque values.

This is NOT foolproof. Always review your exported data before publishing.
Automated redaction cannot catch everything - especially service-specific
identifiers, third-party PII, or secrets in unusual formats. Binary attachments and
base64 payloads are preserved; this redactor does not decode attachments or read
text from images. Embedded JSON decoding is limited to eight nested wrappers.
Credential-shaped examples may also be redacted conservatively.

We recommend converting the exported jsonl into human-readable yaml using dataclaw jsonl-to-yaml,
then use tools such as trufflehog and gitleaks to scan it.
You can also compare the exported jsonl with a previous baseline using dataclaw diff-jsonl.

To help improve redaction, report issues: https://github.com/peteromallet/dataclaw/issues

Finding datasets on Hugging Face

All repos are tagged dataclaw.

  • Browse all: huggingface.co/datasets?other=dataclaw
  • Load one:
    python
    from datasets import load_dataset
ds = load_dataset("alice/my-personal-codex-data", split="train")
  • Combine several:
    python
    from datasets import load_dataset, concatenate_datasets
repos = ["alice/my-personal-codex-data", "bob/my-personal-codex-data"]
ds = concatenate_datasets([load_dataset(r, split="train") for r in repos])

The auto-generated HF README includes:

  • Model distribution (which models, how many sessions each)
  • Total token counts
  • Project count
  • Last updated timestamp

Contributing

Missing data: If you found any data not exported, please report an issue. You can ask your coding agent to analyze the data, export it in this repo, and open a PR.

Better scheme: If you need to clean the data and want to propose a better scheme, feel free to open an issue.

New provider: If you use a new coding agent, you can ask it to read this repo and export its data as a new provider. Take Claude Code and Codex parsers as examples because they are the most well maintained. When you finish, ask the following questions:

  • Did you follow the schema the existing parsers emit? It's fine to add custom fields in messages[].content_parts and tool_uses[].output.raw.
  • Did you export all data, especially:
    • tool call inputs and outputs
    • long inputs and outputs that may be saved somewhere else
    • binary content (may be encoded as base64) such as images, in both user messages and tool calls. We do not apply anonymizer on binary content
    • subagents
  • Does the coding agent automatically delete old sessions? How to prevent this?

Code Quality

License

MIT

RELEASING (`docs/RELEASING.md`)

Releasing DataClaw

  1. Edit no version files by hand.
  2. From an up-to-date main, run make release VERSION=x.y.z.
  3. scripts/release.py verifies a clean tree on the default branch, checks that x.y.z is stable semver and greater than the current version, bumps pyproject.toml, syncs derived files, commits release: vx.y.z, creates annotated tag vx.y.z, and pushes main plus tags.
  4. The tag push runs .github/workflows/release.yml; CI builds the macOS app, uploads assets, and creates the GitHub release with generated notes.
  5. Watch https://github.com/peteromallet/dataclaw/actions/workflows/release.yml and verify the new assets at https://github.com/peteromallet/dataclaw/releases/latest.
  6. In release.yml, checkout reads the tagged tree, the tag guard compares github.ref_name to pyproject.toml, then both signed and unsigned paths run pnpm -C app tauri build.
  7. At that build, Tauri reads the app version from app/src-tauri/Cargo.toml because tauri.conf.json has no version; CI keeps Cargo synced to pyproject.toml, so the app bundle cannot drift from the tag.

Version derivation:

  • pyproject.toml [project].version is the only hand-edited source of truth.
  • app/package.json is rewritten by python scripts/sync_version.py.
  • app/src-tauri/Cargo.toml is rewritten by python scripts/sync_version.py.
  • app/src-tauri/Cargo.lock dataclaw-app entry is rewritten by python scripts/sync_version.py.
  • app/src-tauri/tauri.conf.json intentionally has no version; Tauri v2 uses Cargo.toml when the config version is omitted.

Guards:

  • PR CI runs python scripts/sync_version.py --check and fails if any derived file drifts or if tauri.conf.json grows a version field.
  • Release CI fails before building if pushed tag vx.y.z does not match pyproject.toml.
  • README.md download links point at releases/latest; CHANGELOG.md keeps historical release headings only.
  • There is no dataclaw.__version__, CLI --version, or pyinstaller.spec version copy.

Claude Code Vs Cchv Gaps (`docs/claude-code-vs-cchv-gaps.md`)

Claude Code Gaps vs CCHV

Scope

This note compares Claude Code handling in:

  • DataClaw: ~/dataclaw
  • Claude Code History Viewer (CCHV): ~/claude-code-history-viewer

The goal is to identify Claude Code data that CCHV captures more faithfully than DataClaw today.

Summary

CCHV retains substantially more Claude-log structure than DataClaw.

The biggest current DataClaw gaps are:

  1. It flattens the raw message graph into a simplified conversation schema.
  2. It preserves much more Claude tool-result structure than before, but still does not keep the full raw message/tool object model that CCHV exposes.
  3. It drops Claude summary entries and rename-derived session metadata.
  4. It drops per-message metadata such as uuid, parentUuid, stop_reason, and raw usage.
  5. It merges all Claude subagent files in one session directory into a single synthetic :subagents export entry, while CCHV keeps per-file session granularity.

Detailed Findings

1. CCHV preserves raw Claude message identity and threading; DataClaw does not

CCHV's Claude message model keeps raw message identity and graph fields such as:

  • uuid
  • parentUuid
  • raw sessionId
  • isSidechain

References:

  • CCHV raw/model fields: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:24-47
  • CCHV exported message fields: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:95-143

By contrast, DataClaw exports a simplified session object with only:

  • session_id
  • model
  • git_branch
  • start_time
  • end_time
  • messages
  • stats

References:

  • DataClaw session result shape: ~/dataclaw/dataclaw/parsers/common.py:54-69
  • DataClaw Claude parser only converts user and assistant entries into simplified messages: ~/dataclaw/dataclaw/parsers/claude.py:222-265

Practical consequence:

  • DataClaw loses the original message DAG / parent-child chain and raw entry identity.

2. CCHV keeps raw `toolUse` / `toolUseResult`; DataClaw now preserves more structure but still normalizes them

CCHV stores raw Claude tool structures directly:

  • toolUse
  • toolUseResult

References:

  • CCHV raw tool fields: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:40-67
  • CCHV exported tool fields: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:108-143

DataClaw now:

  • extracts assistant tool_use blocks
  • parses tool input into anonymized structured fields
  • preserves normalized result text in output.text
  • preserves extra structured result data in output.raw
  • avoids duplicate edit/result payloads where possible

References:

  • DataClaw tool-result extraction and output.raw preservation: ~/dataclaw/dataclaw/parsers/claude.py:86-245
  • DataClaw assistant tool export shape: ~/dataclaw/dataclaw/parsers/claude.py:373-422

Practical consequence:

  • DataClaw now keeps many structured tool-result details such as stderr, file objects, image payloads, and extra nested result metadata.
  • It still does not preserve the full raw Claude message/tool object model that CCHV exposes, and it intentionally drops some redundant edit-result fields (oldString, newString, structuredPatch) plus duplicate create file content.

3. CCHV keeps per-message metadata; DataClaw mostly aggregates or drops it

CCHV keeps per-message metadata including:

  • usage
  • stop_reason
  • costUSD
  • durationMs

References:

  • CCHV message content metadata: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:13-21
  • CCHV message-level exported metadata: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:117-127

DataClaw only aggregates selected token counts into session stats and does not export message-level usage or stop_reason.

References:

  • DataClaw token aggregation: ~/dataclaw/dataclaw/parsers/claude.py:251-260
  • DataClaw exported session shape has only session-level stats: ~/dataclaw/dataclaw/parsers/common.py:54-69

Practical consequence:

  • DataClaw loses per-message token usage and stop metadata that CCHV preserves.

4. CCHV captures Claude `summary` entries and rename-derived session names; DataClaw drops them

CCHV explicitly reads Claude summary entries into session metadata and also extracts rename information from system/local_command messages.

References:

  • Summary ingestion during metadata extraction: ~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:318-323
  • Summary fallback in phase-2 scanning: ~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:445-455
  • Rename extraction from system/local command content: ~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:593-619
  • Session metadata fields that expose this: ~/claude-code-history-viewer/src-tauri/src/models/session.rs:51-72

DataClaw's Claude parser only processes user and assistant entries, so Claude summary entries are currently ignored.

References:

  • DataClaw only handles user / assistant: ~/dataclaw/dataclaw/parsers/claude.py:241-265

Practical consequence:

  • DataClaw loses Claude-generated session summaries and rename-derived display names that CCHV keeps.

5. CCHV keeps subagent files as distinct sessions; DataClaw merges them per session directory

CCHV recursively scans all JSONL files under the project path and builds session metadata per file path.

References:

  • Recursive JSONL discovery: ~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:806-811
  • CCHV session metadata distinguishes file-path identity from raw session ID via session_id vs actual_session_id: ~/claude-code-history-viewer/src-tauri/src/models/session.rs:51-72

DataClaw now exports Claude subagents, but it merges all subagents/agent-*.jsonl files in a session directory into one synthetic export session and suffixes the session ID with :subagents when a root session also exists.

References:

  • DataClaw subagent discovery: ~/dataclaw/dataclaw/parsers/claude.py:147-162
  • DataClaw subagent merge behavior: ~/dataclaw/dataclaw/parsers/claude.py:165-218

Practical consequence:

  • DataClaw preserves more than before, but still loses per-subagent-file granularity that CCHV retains.

6. CCHV preserves raw `message.content` JSON blocks; DataClaw converts them into a narrow schema

CCHV preserves raw message.content JSON for loaded messages.

References:

  • CCHV content model: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:13-21
  • CCHV message content field: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:104-105

DataClaw instead extracts only:

  • user text
  • assistant text
  • assistant thinking text
  • assistant tool uses

References:

  • DataClaw user-content extraction: ~/dataclaw/dataclaw/parsers/claude.py:268-276
  • DataClaw assistant-content extraction: ~/dataclaw/dataclaw/parsers/claude.py:279-328

Practical consequence:

  • DataClaw drops raw Claude content-block fidelity, including structured user tool_result content and other non-text block details.

7. CCHV has partial support for more Claude event types than DataClaw, but its viewer still filters some of them

CCHV's raw model knows about additional Claude fields and event payloads for:

  • summary
  • system
  • progress
  • file-history-snapshot
  • queue-operation

References:

  • CCHV raw event fields: ~/claude-code-history-viewer/src-tauri/src/models/message.rs:23-92

However, CCHV's session message viewer excludes some event types and hides some system subtypes:

  • excluded from the viewer: progress, queue-operation, file-history-snapshot, last-prompt, pr-link
  • hidden system subtypes: stop_hook_summary, turn_duration

References:

  • Excluded message types: ~/claude-code-history-viewer/src-tauri/src/commands/session/load.rs:563-590

DataClaw is stricter still: its Claude export path ignores all non-user / non-assistant entries entirely.

References:

  • DataClaw entry handling: ~/dataclaw/dataclaw/parsers/claude.py:241-265

Practical consequence:

  • CCHV is not fully lossless, but it still has broader Claude-log awareness than DataClaw.

Observed In Real Claude Logs On This Machine

The following real Claude files on this machine contain data shapes that matter for the comparison:

  • Root session example with file-history-snapshot, isMeta, permissionMode, raw tool_result, progress, and thinking blocks:
    ~/.claude/projects/-home-wd-transformers-qwen3-moe-fused/5801999e-5607-4d50-9b68-8bf91a1b9252.jsonl:1-51

  • Subagent example with isSidechain: true and agentId:
    ~/.claude/projects/-home-wd-transformers-qwen3-moe-fused/5801999e-5607-4d50-9b68-8bf91a1b9252/subagents/agent-ac4d48d.jsonl:1-48

  • Real Claude summary entries exist in session logs:
    ~/.claude/projects/-home-wd-rocm-systems/050b046a-5e97-40f3-88c6-4611e385f672.jsonl:393-1441

These machine logs confirm that the gaps above are not theoretical; the omitted or flattened fields are present in actual Claude Code data.

Bottom Line

If the goal is a training/export dataset with simple, normalized conversation rows, DataClaw's current Claude export shape is reasonable.

If the goal is Claude-log fidelity, CCHV currently retains more of the original Claude structure than DataClaw, especially around:

  • raw message identity / threading
  • raw message/tool object fidelity
  • per-message metadata
  • summaries / rename metadata
  • per-file subagent granularity

Codex Vs Cchv Gaps (`docs/codex-vs-cchv-gaps.md`)

Codex Gaps vs CCHV

Scope

This note compares Codex handling in:

  • DataClaw: ~/dataclaw
  • Claude Code History Viewer (CCHV): ~/claude-code-history-viewer

The goal is to identify Codex data that CCHV captures more faithfully than DataClaw today.

Summary

CCHV preserves more of the raw Codex rollout structure than DataClaw.

The biggest current DataClaw gaps are:

  1. It ignores response_item.type == "message" entries, which means it drops Codex developer/context messages and some wrapper messages.
  2. It drops Codex progress/system events such as task_started, task_complete, context_compacted, and turn_aborted.
  3. It only keeps session-level token totals, while CCHV attaches token usage to individual assistant messages.
  4. It exports a smaller session schema and omits some session-list metadata that CCHV computes.

There is no evidence on this machine that Codex is storing separate child-agent/subagent logs outside the rollout JSONL files that DataClaw reads.

Detailed Findings

1. CCHV keeps `response_item.type == "message"`; DataClaw drops them

CCHV converts all Codex response_item messages, including developer, user, and assistant roles.

References:

  • CCHV message loading loop: ~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:267-393
  • CCHV message conversion for response_item.type == "message": ~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:603-620

DataClaw's Codex parser mainly handles these response_item variants:

  • function_call
  • custom_tool_call
  • reasoning

It also now extracts user input_image parts from response_item.type == "message" and exports them via messages[].content_parts.

It still does not ingest most other response_item.type == "message" text content.

References:

  • DataClaw handle_response_item(...): ~/dataclaw/dataclaw/parsers/codex.py:403-452
  • DataClaw Codex user image extraction: ~/dataclaw/dataclaw/parsers/codex.py:293-399

Practical consequence:

  • DataClaw drops Codex developer messages and wrapper/context messages that are stored as normal response_item messages.

On this machine, real Codex rollout files contain:

  • 321 developer response_item messages
  • 1194 user response_item messages
  • 2554 assistant response_item messages

Example real file with dropped developer and wrapper user messages:

  • ~/.codex/sessions/2026/03/28/rollout-2026-03-28T09-51-22-019d3223-8607-7ff3-bf8f-6f7f2ca14fe4.jsonl:3-6

Important nuance:

  • Much of the assistant/user conversation text is duplicated by event_msg.agent_message and event_msg.user_message, so the biggest practical loss is the extra wrapper/context material rather than ordinary user/assistant chat text.

Important Counterpoint: DataClaw now preserves real Codex image input on this machine

This is an important implemented improvement relative to the earlier state of the parser.

DataClaw now preserves Codex user image input as messages[].content_parts when it appears in:

  • response_item.payload.content[].type == "input_image"
  • or event_msg.user_message.local_images as a local-file fallback

References:

  • DataClaw Codex image extraction: ~/dataclaw/dataclaw/parsers/codex.py:293-399

Observed in real Codex data on this machine:

  • ~/.codex/sessions/2026/04/02/rollout-2026-04-02T08-26-37-019d4b95-bc24-7992-af85-e47738953690.jsonl:6-7

So user image input is no longer a current Codex gap in DataClaw.

2. CCHV keeps Codex progress/system events; DataClaw drops them

CCHV explicitly converts these Codex event types into messages:

  • task_started
  • task_complete
  • context_compacted
  • turn_aborted
  • agent_reasoning
  • agent_message
  • user_message

References:

  • CCHV Codex event conversion: ~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:838-1027

DataClaw only keeps this smaller subset:

  • token_count
  • agent_reasoning
  • user_message
  • agent_message

References:

  • DataClaw event handling: ~/dataclaw/dataclaw/parsers/codex.py:215-230

Practical consequence:

  • DataClaw drops Codex task/progress/system boundaries that CCHV exposes.

Observed in real Codex logs on this machine:

  • task_started: 62
  • task_complete: 43
  • context_compacted: 78
  • turn_aborted: 99

Examples:

  • task_started and later task_complete in one real rollout:
    ~/.codex/sessions/2026/03/27/rollout-2026-03-27T14-05-48-019d2de6-1cd3-7cc1-89bc-fc30f7f0314f.jsonl:2,304
  • turn_aborted in a real rollout:
    ~/.codex/sessions/2026/03/28/rollout-2026-03-28T09-51-22-019d3223-8607-7ff3-bf8f-6f7f2ca14fe4.jsonl:101

3. CCHV attaches per-message token usage; DataClaw only keeps session totals

CCHV reads Codex token_count events, computes deltas, and attaches usage to the last assistant message.

References:

  • CCHV token handling in Codex loader: ~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:336-376

DataClaw only updates session-level maxima/totals and stores them in stats.

References:

  • DataClaw token aggregation: ~/dataclaw/dataclaw/parsers/codex.py:332-341
  • DataClaw session result shape: ~/dataclaw/dataclaw/parsers/common.py:56-71

Practical consequence:

  • DataClaw loses per-message token usage that CCHV preserves.

4. CCHV keeps more session-list metadata than DataClaw exports

CCHV computes extra Codex session metadata including:

  • file_path
  • message_count
  • has_tool_use
  • first-user-message summary

References:

  • CCHV session info extraction: ~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:441-560

DataClaw exports a smaller normalized session schema with only:

  • session_id
  • model
  • git_branch
  • start_time
  • end_time
  • messages
  • stats

References:

  • DataClaw normalized session shape: ~/dataclaw/dataclaw/parsers/common.py:56-71

Practical consequence:

  • DataClaw omits some Codex session-level metadata that CCHV exposes in its session browser/index.

5. There is no evidence here of extra Codex subagent logs that CCHV captures and DataClaw misses

DataClaw builds the Codex project index from every session JSONL file under:

  • ~/.codex/sessions/**/*.jsonl
  • ~/.codex/archived_sessions/*.jsonl

References:

  • DataClaw Codex session discovery: ~/dataclaw/dataclaw/parsers/codex.py:74-88

On this machine:

  • ~/.codex/sessions contains 78 rollout JSONL files
  • ~/.codex/state_5.sqlite contains 78 rows in threads
  • every threads.rollout_path matches one of those JSONL files
  • thread_spawn_edges has 0 rows
  • agent_jobs has 0 rows
  • agent_job_items has 0 rows

Practical consequence:

  • There is no extra Codex child-thread/subagent graph on this machine that CCHV captures and DataClaw misses.

This is different from the Claude case, where separate sidechain files existed on disk.

Important Counterpoint: DataClaw keeps some Codex tool metadata that CCHV normalizes away

This is not a pure one-way comparison.

For Codex tool outputs, DataClaw preserves some parsed metadata such as:

  • exit_code
  • wall_time
  • duration_seconds

References:

  • DataClaw Codex tool-result parsing: ~/dataclaw/dataclaw/parsers/codex.py:122-172

CCHV normalizes tool outputs more aggressively and mainly keeps the extracted payload text.

References:

  • CCHV tool output normalization: ~/claude-code-history-viewer/src-tauri/src/providers/codex.rs:1142-1159

So CCHV has higher Codex message/event fidelity overall, but DataClaw is not strictly worse on every Codex field.

Bottom Line

Compared with CCHV, DataClaw currently loses more Codex rollout structure around:

  • response_item messages, especially developer/context wrappers
  • progress/system events (task_started, task_complete, context_compacted, turn_aborted)
  • per-message token usage
  • some session-list metadata

But for the real Codex data on this machine, there is no separate Codex subagent/session graph outside the rollout JSONL files, so DataClaw is not missing hidden child-session files in the way Claude previously was.

Cursor Vs Cchv Gaps (`docs/cursor-vs-cchv-gaps.md`)

Cursor Gaps vs CCHV

Scope

This note compares Cursor handling in:

  • DataClaw: ~/dataclaw
  • Claude Code History Viewer (CCHV): ~/claude-code-history-viewer

There is no real Cursor data on this machine, so this comparison is based on code only.

Summary

Based on the current implementations, CCHV appears to preserve more Cursor-specific structure than DataClaw.

The clearest current DataClaw gaps are:

  1. It drops user image attachments.
  2. It does not preserve isThought semantics for assistant thought bubbles.
  3. It flattens Cursor bubble content into a smaller normalized schema and loses content-array ordering/interleaving.
  4. It does not preserve some Cursor session metadata that CCHV loads from workspace composer data.
  5. It may be less robust than CCHV when Cursor JSON contains embedded control characters.

Detailed Findings

1. CCHV keeps user image attachments; DataClaw drops them

CCHV reads bubble.images for user bubbles and preserves data:image/...;base64,... attachments as image content blocks.

References:

  • CCHV user-bubble conversion: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:439-492
  • Image handling in user bubbles: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:452-476

DataClaw's Cursor parser only exports user text and ignores images.

References:

  • DataClaw user-bubble handling: ~/dataclaw/dataclaw/parsers/cursor.py:251-262

Practical consequence:

  • If Cursor stores user screenshots or pasted image attachments in bubble.images, CCHV will preserve them and DataClaw will not.

2. CCHV preserves `isThought` as thinking; DataClaw does not

CCHV checks bubble.isThought on assistant bubbles and emits a thinking content block when present.

References:

  • CCHV assistant-bubble conversion: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:494-571
  • isThought handling: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:542-553

DataClaw does not check isThought. It only exports assistant thinking when a separate bubble.thinking.text field exists.

References:

  • DataClaw assistant bubble with tool path: ~/dataclaw/dataclaw/parsers/cursor.py:317-321
  • DataClaw assistant bubble without tool path: ~/dataclaw/dataclaw/parsers/cursor.py:333-347

Practical consequence:

  • A Cursor bubble represented as plain text plus isThought: true would be preserved by CCHV as thought text, but would become normal assistant content in DataClaw.

3. CCHV keeps a structured content array; DataClaw flattens it

CCHV converts Cursor bubbles into a structured content array containing items such as:

  • text
  • image
  • thinking
  • tool_use
  • tool_result

References:

  • CCHV user-bubble content array: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:445-490
  • CCHV assistant-bubble content array: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:500-569

DataClaw flattens Cursor bubbles into the normalized DataClaw schema:

  • user content
  • assistant content
  • assistant thinking
  • assistant tool_uses

References:

  • DataClaw normalized session schema: ~/dataclaw/dataclaw/parsers/common.py:56-71
  • DataClaw Cursor parser: ~/dataclaw/dataclaw/parsers/cursor.py:251-356

Practical consequence:

  • Even when the same information is mostly present, DataClaw loses original content-array ordering/interleaving.
  • For example, an assistant bubble that conceptually contains tool_use followed by tool_result and then additional text is normalized into tool_uses[] plus top-level content, not preserved as an ordered content array.

4. CCHV keeps more Cursor session metadata than DataClaw exports

CCHV reads workspace-level Cursor composer metadata and keeps fields such as:

  • composer name
  • createdAt
  • lastUpdatedAt
  • unifiedMode
  • isArchived
  • derived has_tool_use
  • summary/title

References:

  • CCHV session loading from workspace DB: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:116-187
  • Workspace composer metadata loader: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:352-380

DataClaw exports only the normalized session schema with:

  • session_id
  • model
  • git_branch
  • start_time
  • end_time
  • messages
  • stats

References:

  • DataClaw session schema: ~/dataclaw/dataclaw/parsers/common.py:56-71

Practical consequence:

  • Cursor-specific session title/summary, mode, archived state, and related metadata are not preserved in DataClaw export.

5. CCHV may be more robust to malformed Cursor JSON

CCHV sanitizes embedded control characters before parsing Cursor JSON values.

References:

  • CCHV Cursor JSON sanitizer: ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:383-397

DataClaw calls json.loads(...) directly on Cursor DB values without a similar sanitization pass.

References:

  • DataClaw project-index parse path: ~/dataclaw/dataclaw/parsers/cursor.py:85-86,112-113
  • DataClaw session/bubble parse path: ~/dataclaw/dataclaw/parsers/cursor.py:191-192,211-212

Practical consequence:

  • If real Cursor DB rows contain control characters or otherwise slightly malformed JSON, CCHV may still load them while DataClaw may skip them.

Ambiguous Difference: Tool Input Field Source

CCHV and DataClaw normalize Cursor tool calls from slightly different raw fields.

CCHV uses:

  • toolFormerData.rawArgs

References:

  • ~/claude-code-history-viewer/src-tauri/src/providers/cursor.rs:516-530

DataClaw uses:

  • toolFormerData.params
  • and a nested tools[].parameters unwrapping path

References:

  • ~/dataclaw/dataclaw/parsers/cursor.py:272-284

Without real Cursor data on this machine, this note does not claim that one side is definitively more faithful than the other here. It is simply an implementation difference worth noting.

Bottom Line

Based on code only, the strongest likely Cursor gaps in DataClaw compared with CCHV are:

  • user image attachments
  • isThought / thought-bubble semantics
  • richer ordered content-array structure
  • extra session metadata from workspace composer state
  • parser robustness for control-character-containing Cursor JSON

Gemini Vs Cchv Gaps (`docs/gemini-vs-cchv-gaps.md`)

Gemini CLI Gaps vs CCHV

Scope

This note compares Gemini CLI handling in:

  • DataClaw: ~/dataclaw
  • Claude Code History Viewer (CCHV): ~/claude-code-history-viewer

The goal is to identify Gemini CLI data that CCHV captures more faithfully than DataClaw today, while also noting important cases where DataClaw retains data that CCHV skips.

Summary

CCHV preserves more Gemini message/event structure than DataClaw in several places.

The biggest current DataClaw gaps are:

  1. It drops Gemini info / warning / error messages.
  2. It drops resultDisplay tool UI output, including file-diff previews and tool-status strings.
  3. It only keeps session-level token totals, while CCHV keeps per-message Gemini token usage.
  4. It does not preserve top-level Gemini session metadata such as summary and kind.

Important counterpoint:

  • CCHV explicitly skips Gemini sessions whose top-level kind is subagent, while DataClaw currently exports them as normal sessions.

Detailed Findings

1. CCHV keeps `info` / `warning` / `error` messages; DataClaw drops them

CCHV converts Gemini message records with types:

  • user
  • gemini
  • info
  • warning
  • error

References:

  • CCHV Gemini message dispatch: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:464-485
  • CCHV system-message conversion: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:630-653

DataClaw only exports:

  • user
  • gemini

References:

  • DataClaw Gemini parser message handling: ~/dataclaw/dataclaw/parsers/gemini.py:311-375

Practical consequence:

  • DataClaw drops Gemini informational and error messages that CCHV exposes as system messages.

Observed in real Gemini data on this machine:

  • 835 info messages
  • 29 error messages

Example real file:

  • ~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-24T08-56-51cb7147.json:10,16,1111

2. CCHV keeps `resultDisplay`; DataClaw ignores it

CCHV converts Gemini toolCalls[].resultDisplay into extra content blocks.

References:

  • CCHV includes resultDisplay during Gemini tool-call conversion: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:592-597
  • CCHV extract_result_display(...): ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:893-928

DataClaw's Gemini parser never reads resultDisplay.

References:

  • DataClaw Gemini tool-call parsing: ~/dataclaw/dataclaw/parsers/gemini.py:162-282

Practical consequence:

  • DataClaw drops user-visible tool UI output, including:
    • short status strings such as Found 4 matching file(s)
    • read-file previews such as Read lines 50-150 ...
    • file-diff previews stored in resultDisplay.fileDiff
    • potential subagent-progress markers (isSubagentProgress) if they appear

Observed in real Gemini data on this machine:

  • 4386 string resultDisplay values
  • 854 object resultDisplay values containing:
    • fileDiff
    • fileName
    • filePath
    • originalContent
    • newContent
    • diffStat
    • isNewFile

Example real file with file-diff previews:

  • ~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-28T01-43-f9f3aa2a.json:305-306,350-351,395-396,440-441

Important Counterpoint: DataClaw now preserves non-text Gemini user content parts

CCHV converts Gemini content parts such as:

  • inlineData image/document blocks
  • fileData URL-backed document blocks
  • plain text parts
  • functionCall
  • functionResponse
  • executableCode
  • codeExecutionResult

References:

  • CCHV content conversion helpers: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:660-867

DataClaw now preserves Gemini user content parts via messages[].content_parts, including:

  • inlineData image/document parts
  • fileData URL-backed document parts
  • functionCall
  • functionResponse

It keeps the canonical joined text in messages[].content and stores the structured non-text parts separately.

References:

  • DataClaw Gemini user content parsing: ~/dataclaw/dataclaw/parsers/gemini.py:324-430
  • DataClaw Gemini user message export: ~/dataclaw/dataclaw/parsers/gemini.py:463-474

Practical consequence:

  • This is no longer a primary Gemini gap in DataClaw.
  • DataClaw now preserves user attachments and structured function parts while still exporting large base64/blob payloads verbatim.

Observed in real Gemini data on this machine:

  • real inlineData image attachments exist in user messages, for example:
    ~/.gemini/tmp/rocm-systems/chats/session-2026-03-06T03-33-68bc726c.json:4764,4794,5006,5036

These include large base64 image payloads that CCHV maps to image/document blocks.

3. CCHV keeps per-message token usage; DataClaw only keeps session totals

CCHV stores Gemini per-message token usage derived directly from each Gemini response record's tokens field.

References:

  • CCHV Gemini usage extraction: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:601-608,616-627

DataClaw only aggregates token counts into session-level stats.

References:

  • DataClaw Gemini token aggregation: ~/dataclaw/dataclaw/parsers/gemini.py:340-343
  • DataClaw normalized session shape: ~/dataclaw/dataclaw/parsers/common.py:56-71

Practical consequence:

  • DataClaw loses per-message Gemini usage, including cached-input attribution on individual assistant responses.

4. CCHV keeps more Gemini session metadata than DataClaw exports

CCHV extracts Gemini session metadata including:

  • session_id
  • kind
  • start_time
  • last_updated
  • message_count
  • has_tool_use
  • summary

References:

  • CCHV Gemini lightweight metadata extraction: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:388-457

DataClaw exports a smaller normalized session shape and does not preserve top-level Gemini summary or kind.

References:

  • DataClaw Gemini metadata initialization: ~/dataclaw/dataclaw/parsers/gemini.py:300-308
  • DataClaw normalized session shape: ~/dataclaw/dataclaw/parsers/common.py:56-71

Practical consequence:

  • DataClaw loses Gemini session metadata that CCHV surfaces in its session index/browser.

Observed in real Gemini data on this machine:

  • 122 Gemini session files have top-level summary
  • top-level kind values present include:
    • main
    • subagent

Important Counterpoint: CCHV skips Gemini `kind == "subagent"` sessions, but DataClaw exports them

This is an important Gemini difference in the opposite direction.

CCHV explicitly skips Gemini sessions whose top-level kind is subagent in both:

  • project/session listing
  • search

References:

  • CCHV session listing skip: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:167-169
  • CCHV project scan skip: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:85-88
  • CCHV search skip: ~/claude-code-history-viewer/src-tauri/src/providers/gemini.rs:283-286

DataClaw does not filter by kind, so it includes Gemini subagent sessions if they exist as chat files.

References:

  • DataClaw Gemini discovery and parse paths read all session-*.json files: ~/dataclaw/dataclaw/parsers/gemini.py:123-159

Observed in real Gemini data on this machine:

  • there is exactly one real Gemini chat file with kind: "subagent":
    ~/.gemini/tmp/tmp/chats/session-2026-03-05T03-59-51c63ffc.json:82
  • DataClaw successfully parses and exports that session.

So, unlike the Claude comparison, Gemini is not a simple one-way story where CCHV always preserves more.

Observed In Real Gemini Data On This Machine

The following real Gemini structures exist on this machine and matter for the comparison:

  • info / error message types:
    ~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-24T08-56-51cb7147.json:10,16,1111

  • resultDisplay.fileDiff edit previews:
    ~/.gemini/tmp/comfyui-featherops/chats/session-2026-03-28T01-43-f9f3aa2a.json:305-306,350-351,395-396,440-441

  • user inlineData image attachments:
    ~/.gemini/tmp/rocm-systems/chats/session-2026-03-06T03-33-68bc726c.json:4764,4794,5006,5036

  • content-part functionResponse blocks:
    ~/.gemini/tmp/rocm-systems/chats/session-2026-03-06T03-33-68bc726c.json:122,147,172,231

  • a real Gemini subagent session:
    ~/.gemini/tmp/tmp/chats/session-2026-03-05T03-59-51c63ffc.json:82

These are all real, present data shapes, not just theoretical parser code paths.

Bottom Line

Compared with CCHV, DataClaw currently loses more Gemini fidelity around:

  • info / warning / error messages
  • resultDisplay tool UI output
  • per-message token usage
  • top-level session metadata like summary and kind

But CCHV also has one notable Gemini omission that DataClaw does not:

  • CCHV skips Gemini kind == "subagent" sessions from its normal session views, while DataClaw exports them.

And DataClaw now also preserves structured Gemini user content parts such as inlineData, fileData, functionCall, and functionResponse, so those are no longer primary Gemini gaps.

Hermes Provider Notes (`docs/hermes-provider-notes.md`)

Hermes Provider Notes

Provider contract

  • Providers are registered in dataclaw/providers.py through ModuleProvider.from_module.
  • A parser module must expose SOURCE, a module-level source path selected by source_path_attr, and these functions: discover_projects, parse_project_sessions, build_export_session_tasks, and parse_export_session_task.
  • Parsed sessions use the shared make_session_result shape from dataclaw/parsers/common.py: session_id, model, git_branch, start_time, end_time, messages, and stats. collect_project_sessions adds project and source.
  • Built-in parsers accept an Anonymizer, but they do not rewrite strings during parsing. Export applies secrets.transform_session recursively with the provider's NON_ANON_STRING_KEYS. Hermes follows that same flow.
  • include_thinking controls whether parser-specific reasoning/thinking fields are emitted as message thinking.
  • Export parallelism is provider-driven: build_export_session_tasks creates lightweight ExportSessionTask records and parse_export_session_task reparses one session in a worker process.

Hermes state.db

Path inspected read-only: ~/.hermes/state.db.

Tables of interest:

  • sessions(id, source, user_id, model, model_config, system_prompt, parent_session_id, started_at REAL, ended_at, end_reason, message_count, tool_call_count, input_tokens, output_tokens, title, cache_read_tokens, cache_write_tokens, reasoning_tokens, billing_*, estimated_cost_usd, actual_cost_usd, ..., api_call_count)
  • messages(id, session_id, role, content, tool_call_id, tool_calls, tool_name, timestamp REAL, token_count, finish_reason, reasoning, reasoning_content, reasoning_details, codex_reasoning_items, codex_message_items)
  • FTS5 tables exist for search and are ignored.

Real counts on 2026-06-05:

  • sessions: 10,033 total.
  • By sessions.source: cli 10,026 sessions, telegram 6 sessions, cron 1 session.
  • message_count = 0: 1,261 sessions. message_count = 1: 33 sessions.
  • parent_session_id is populated on 917 sessions. The common export schema has no parent-chain field, so the provider exports each session independently.
  • messages.role: tool 175,231 rows, assistant 127,614 rows, user 9,721 rows.
  • Non-empty assistant tool_calls: 118,534 rows.
  • Non-empty reasoning fields: reasoning 2 rows and reasoning_content 2 rows. reasoning_details, codex_reasoning_items, and codex_message_items were empty in the inspected DB.

Important timestamp detail: Hermes started_at, ended_at, and message timestamp values are Unix seconds, not milliseconds.

Observed message structure:

  • user rows carry user-visible text in content.
  • assistant rows carry assistant text in content and OpenAI-style function call lists in tool_calls.
  • Each tool_calls item is a dict with keys like id, call_id, type, response_item_id, and function; function has name and JSON-string arguments.
  • tool rows carry tool output in content, keyed by tool_call_id. Tool rows generally do not repeat the tool name, so the parser attaches them to the matching assistant tool call.

Hermes session JSON files

Path inspected: ~/.hermes/sessions/.

Two file families exist:

  • session_.json with keys like session_id, model, base_url, platform, session_start, last_updated, system_prompt, tools, message_count, and messages.
  • request_dump__<timestamp>.json, which are raw/debug API request dumps.

Coverage comparison:

  • session_*.json: 12,490 files.
  • DB sessions: 10,033 rows.
  • Overlap by session id: 3,825.
  • DB-only sessions: 6,208.
  • JSON-only sessions: 8,665.

Source-of-truth decision: use state.db. It is canonical for current Hermes history, has normalized tool-result rows, includes richer reasoning columns, and covers thousands of sessions absent from the JSON files. The JSON files are useful for historical/debug inspection but are not complete enough to drive export.

Project grouping

Hermes has no per-project directory. The parser groups by sessions.source, yielding display names like hermes:cli. This mirrors the Cursor-style single-store approach: build a project index from stable metadata in the store, then parse sessions by id.

Discovery ignores sessions with message_count <= 0; parsing still skips any session that produces no normalized messages.

Opencode Vs Cchv Gaps (`docs/opencode-vs-cchv-gaps.md`)

OpenCode Gaps vs CCHV

Scope

This note compares OpenCode handling in:

  • DataClaw: ~/dataclaw
  • Claude Code History Viewer (CCHV): ~/claude-code-history-viewer

The goal is to identify OpenCode data that CCHV captures more faithfully than DataClaw today, with references to both codebases and to the real OpenCode data stored on this machine.

Summary

CCHV preserves substantially more OpenCode part-level structure than DataClaw.

The biggest current DataClaw gaps are:

  1. It drops patch, step-start, step-finish, and compaction parts.
  2. It loses per-message usage and cost derived from step-finish parts.
  3. It loses message threading via parentID.
  4. It flattens OpenCode content into a smaller normalized schema and drops part-level structure that CCHV keeps.

Detailed Findings

Important Counterpoint: DataClaw now preserves OpenCode user `file` parts

CCHV processes OpenCode part.type == "file" and preserves:

  • image data: URIs as image blocks
  • image URLs as image blocks
  • non-image file references as file text blocks

References:

  • CCHV OpenCode part processing: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1204-1240

DataClaw now preserves OpenCode user file parts via messages[].content_parts, including:

  • image data: URIs as image blocks with verbatim base64 payloads
  • non-image file://... references as document URL blocks

References:

  • DataClaw OpenCode file-source parsing: ~/dataclaw/dataclaw/parsers/opencode.py:211-245
  • DataClaw OpenCode user message extraction: ~/dataclaw/dataclaw/parsers/opencode.py:248-272

Practical consequence:

  • This is no longer a primary OpenCode gap in DataClaw for user file/image parts.
  • DataClaw now preserves the real OpenCode base64 image payloads found on this machine.

Observed in real OpenCode data on this machine:

  • querying ~/.local/share/opencode/opencode.db showed 99 file parts
  • 7 of those were image parts with mime: image/png
  • those image parts used data:image/png;base64,... URLs

Example real image part from the database:

  • DB file: ~/.local/share/opencode/opencode.db
  • session: ses_3321dc5e9ffeljot1d6UlwCGOB
  • message: msg_ccde78ea3001CsoqVaRwira24N
  • part payload contains:
    • type: "file"
    • mime: "image/png"
    • url: "data:image/png;base64,..."

1. CCHV keeps `patch`, `step-start`, `step-finish`, and `compaction` parts; DataClaw drops them

CCHV explicitly handles these OpenCode part types:

  • patch
  • step-start
  • step-finish
  • compaction

References:

  • step-finish and compaction: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1130-1181
  • patch: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1182-1202
  • step-start: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1242-1249

DataClaw's OpenCode parser does not handle any of these part types.

References:

  • DataClaw assistant extraction only branches on text, reasoning, and tool: ~/dataclaw/dataclaw/parsers/opencode.py:235-265

Practical consequence:

  • DataClaw drops patch summaries, step boundaries, and compaction markers that CCHV shows.

Observed in real OpenCode data on this machine:

  • patch: 2349 parts
  • step-start: 9572 parts
  • step-finish: 9451 parts
  • compaction: 32 parts

Example real parts from the database:

  • Patch part:

    • DB file: ~/.local/share/opencode/opencode.db
    • session: ses_3abf6a2e7ffeRw4KIgQY5d0prz
    • message: msg_c540a13e8001dgyT9upME85T5v
    • part payload: {"type":"patch","files":["~/ComfyUI/comfy/ldm/wan/model.py"],...}

  • Compaction part:

    • DB file: ~/.local/share/opencode/opencode.db
    • session: ses_3afeba227ffeLYbMdEzqlMWeKD
    • message: msg_c51049d72001i65GspUCMGRSZB
    • part payload: {"type":"compaction","auto":true}

2. CCHV keeps per-message token usage and cost; DataClaw only keeps session totals

CCHV derives OpenCode per-message usage and cost from both:

  • message-level fields
  • step-finish parts

References:

  • message-level usage/cost extraction: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:822-883
  • step-finish token/cost accumulation: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1130-1171

DataClaw only accumulates OpenCode token counts into session-level stats, and does not keep per-message usage or cost.

References:

  • DataClaw token accumulation: ~/dataclaw/dataclaw/parsers/opencode.py:180-186
  • DataClaw normalized session schema: ~/dataclaw/dataclaw/parsers/common.py:56-71

Practical consequence:

  • DataClaw loses per-message usage/cost fidelity that CCHV preserves.

Observed in real OpenCode data on this machine:

  • many messages have no tokens in message.data, but real step-finish parts do have:
    • tokens.input
    • tokens.output
    • tokens.cache.read
    • tokens.cache.write
    • cost

Example step-finish payload from the database:

  • DB file: ~/.local/share/opencode/opencode.db
  • contains values like:
    • {"type":"step-finish","reason":"tool-calls","cost":0,"tokens":{"input":11294,"output":83,"cache":{"read":0,"write":0}}}

3. CCHV keeps message threading via `parentID`; DataClaw drops it

CCHV maps OpenCode parentID to parent_uuid in the loaded message model.

References:

  • OpenCode DB path message metadata extraction: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:827-830
  • JSON fallback path does the same: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:416-420

DataClaw's normalized export schema does not preserve message parent links.

References:

  • DataClaw session schema: ~/dataclaw/dataclaw/parsers/common.py:56-71

Practical consequence:

  • DataClaw loses OpenCode message threading / parent-child structure that CCHV keeps.

Observed in real OpenCode data on this machine:

  • querying ~/.local/share/opencode/opencode.db showed that many OpenCode messages include parentID
  • at inspection time, 9697 messages had a non-empty parentID

4. CCHV keeps a richer OpenCode content array; DataClaw flattens it

CCHV turns OpenCode parts into a structured content array with items like:

  • text
  • thinking
  • tool_use
  • tool_result
  • image
  • patch summaries
  • step summaries

References:

  • CCHV process_parts(...): ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:1049-1260

DataClaw flattens OpenCode into a smaller schema:

  • user content
  • assistant content
  • assistant thinking
  • assistant tool_uses

References:

  • DataClaw user content extraction: ~/dataclaw/dataclaw/parsers/opencode.py:210-223
  • DataClaw assistant content extraction: ~/dataclaw/dataclaw/parsers/opencode.py:226-277

Practical consequence:

  • DataClaw loses OpenCode part-level fidelity beyond the normalized text/thinking/tool schema.

5. Potential additional gap: CCHV has a JSON-storage fallback path; DataClaw only reads SQLite

CCHV loads OpenCode messages from SQLite first, then falls back to JSON storage files under storage/ when needed.

References:

  • CCHV SQLite-first, JSON-fallback loading: ~/claude-code-history-viewer/src-tauri/src/providers/opencode.rs:333-497

DataClaw only reads the SQLite database at:

  • ~/.local/share/opencode/opencode.db

References:

  • DataClaw OpenCode DB-only parser: ~/dataclaw/dataclaw/parsers/opencode.py:103-194

Practical consequence:

  • If some OpenCode data exists only in JSON storage and not in SQLite, CCHV can see it but DataClaw cannot.

This note does not claim that such DB-missing session content was observed on this machine; it is an implementation difference worth noting.

Observed In Real OpenCode Data On This Machine

At inspection time, querying ~/.local/share/opencode/opencode.db showed these real OpenCode part types:

  • tool: 12112
  • step-start: 9572
  • step-finish: 9451
  • reasoning: 8641
  • text: 3886
  • patch: 2349
  • file: 99
  • compaction: 32

The most important real payload classes for the comparison were:

  • image file parts with base64 data: URIs
  • patch parts with modified file lists
  • step-finish parts with tokens/cost
  • compaction markers
  • pervasive parentID message links

Bottom Line

Compared with CCHV, DataClaw currently loses substantial OpenCode fidelity around:

  • patch / step / compaction parts
  • per-message usage and cost
  • parent-child message threading
  • part-level content structure generally

DataClaw now preserves OpenCode user file/image parts, including real base64 image payloads. The largest remaining practical OpenCode gaps on this machine are patch / step / compaction parts and per-message usage / cost.

Reliability Merge And Privacy Filter Plan (`docs/reliability-merge-and-privacy-filter-plan.md`)

DataClaw reliability: merge-on-upload + privacy filter — design & plan

Produced from a multi-subagent audit (2026-05-30). Every claim is cited file:line.
Two goals: (1) uploads must merge, never destroy, prior data; (2) the model
privacy filter the UI promises must actually run.

Part A — Incremental / merge-on-upload

Current behavior (verified)

Upload is a wholesale overwrite. push_to_huggingface (dataclaw/_cli/exporting.py:651)
calls api.upload_file(path_in_repo="conversations.jsonl", ...) (:669) with the locally
re-generated full file. No download, no merge. The remote dataset always equals the latest
local scan, so any session not reproducible locally right now is silently deleted
(second machine, log rotation, narrower --source, new excluded project, cleared ~/.claude).

Intent residue found

  • last_export_cutoff — written by Rust (app/src-tauri/src/dataclaw.rs:239-261), read by
    nobody    . Half-built incremental-cutoff scaffold.
  • dataclaw/jsonl_tools.py — has IDENTITY_FIELDS=("source","project","session_id","start_time")
    (:26), identity_key() (:221), index_jsonl() (:239), diff_jsonl_files() (:685).
    No merge function; not wired to upload — diff-jsonl is an offline CLI command only.
  • Data-safety intent was actually shipped as trust gates (session-shrink review.py:616,
    redaction-drift commands.py:780), all comparing to local last_export — not remote.

The correct unit

Each JSONL line is one session record with a messages array. "New messages since last
upload" = same identity_key, more messages, different record hash. The merge unit is the
session, not the message.

  1. Download prior remote conversations.jsonl via hf_hub_download(repo_id, "conversations.jsonl", repo_type="dataset"). Fail closed: only a confirmed 404
    (EntryNotFoundError) or missing-repo (RepositoryNotFoundError) counts as "empty remote";
    any network/auth/HTTP error aborts the push (never overwrite remote with local-only).
  2. Union by identity_key (read records RAW, not via index_jsonl — that normalizes and
    strips originalFile, see H3):
    • remote-only → carry forward (never drop)
    • local-only → add
    • both → keep superset (more messages; tie-break larger bytes, then later end_time)
    • Invariant merged_total >= remote_total ⇒ shrink is impossible by construction.
  3. 🔴 RE-REDACT carried-forward remote records through the CURRENT redactor before writing
    them (see "Critical correctness requirement" below). This is mandatory, not optional.
  4. Re-run the confirm gates on the MERGED file and re-hash it so reviewed-bytes ==
    shipped-bytes (preserve the SHA invariant at commands.py:365-376).
  5. Upload the merged union (not the local export) with parent_commit=<repo head sha>
    for optimistic concurrency; on 412 conflict re-download+re-merge (cap ~5 attempts).
    Update meta["sessions"]/last_export.sessions to merged_total so metadata and the
    shrink gate stay consistent (H5).

Hook point: push_to_huggingface upload block (exporting.py:666-675). Merge fn belongs in
jsonl_tools.py next to identity_key/index_jsonl. Draft code exists in the agent transcript.

🔴 Critical correctness requirement (the naive merge is a PRIVACY REGRESSION)

Remote records were redacted under whatever policy was current when they were uploaded.
Carrying them forward verbatim re-publishes old, looser-policy redaction and bypasses
every gate (gates run in confirm() on the local file, before the merge). Adversarial review
verdict on the naive union: scenarios (a) new redact_string, (b) superset-by-count picking the
older copy, (c) future model filter, (d) gate bypass → leak, leak, leak, total-bypass.

Fix: re-run secrets.transform_session (+ Anonymizer) over every carried-forward record
with the current config before reupload. This is idempotent and pseudonym-stable
(verified): secrets become the constant [REDACTED] which matches no pattern; pseudonyms are
deterministic user_+sha256(name)[:8] and the hash token doesn't re-match the username regex.
Then re-gate + re-hash the merged file.

Additional holes the merge must handle

  • H1 start_time key drift (jsonl_tools.py:26, parsers/common.py:157-164): format drift
    (Z vs +00:00, None vs backfilled) → same session under two keys → duplicate + leak.
    Fix: canonicalize start_time before keying, or key on (source, project, session_id).
  • H2 anonymized project in key (secrets.py:572-576): project is anonymized in-place and
    is part of the key → key drift. Strip project from key or key on post-anon identity.
  • H3 normalize_for_diff strips originalFile (jsonl_tools.py:234-235): don't read remote
    records via the diff-normalizing loader or you destroy originalFile content.
  • H4 schema drift: old remote records may lack fields the current walker expects → missed
    secrets or crash. Re-redaction must be schema-tolerant.
  • H5 non-atomic upload + stale last_export.sessions: 3 separate upload_file commits;
    update session count to merged total.

Trust gates

Keep the existing local gates unchanged (they guard the human confirm step). Do not add a
remote-shrink gate — the union invariant makes remote shrink impossible. But the merged file
must pass the gates after merge+re-redaction (step 4).

Part B — Model-based privacy filter

It was real, and was lost (not reverted)

A complete 848-line dataclaw/privacy_filter.py + tests/test_privacy_filter.py was
committed on 5d0a741 ("Add macOS app release flow", a side branch) and never merged to main.
A different same-titled commit (42b21d1) landed on main without it. The file was on disk as
late as the 2026-05-16 desloppify scan, then lost untracked. The pii deps
(transformers, torch, accelerate, tokenizers) were dropped from HEAD pyproject.toml.

Recover with: git show 5d0a741:dataclaw/privacy_filter.py and :tests/test_privacy_filter.py.

What it does (and quality)

transformers.pipeline(task="token-classification", aggregation_strategy="simple") NER with real
CPU/MPS device handling (resolve_device/_auto_device), 480-token chunking, min_score 0.85,
oversized-session blanket-redact guard, reverse-order span splicing. Core logic is sound.

Bugs / gaps (line refs on the 5d0a741 version):

  • 🔴 Field-walk mismatch (must fix): it walks messages[].content/thinking + tool_uses[].input/output
    but NOT messages[].content_parts, which HEAD's secrets.transform_session does walk
    (secrets.py:590). PII in content_parts would pass untouched. Add content_parts to the
    field loops in scan_session/redact_session/_redact_oversized_session + a test.
  • 🔴 MODEL_ID = "openai/privacy-filter" is a placeholder — does not exist on the Hub.
  • Minor: cross-chunk-boundary entities may be split/under-redacted (low severity); min_score
    discarded in _load then re-applied in scan_text (cosmetic); dtype= kwarg needs transformers ≥4.45.

Model decision (drop-in for the existing pipeline call)

  • Default: openai/privacy-filter — the model the original implementation
    intended. It is a REAL, published model (Apache-2.0, ~300k downloads,
    token-classification, safetensors + ONNX) — the earlier "placeholder that doesn't
    exist" claim was WRONG. It drops into the existing
    pipeline(task="token-classification", aggregation_strategy="simple") call with no
    adapter code and detects PII as intended.
  • Overridable via the privacy_filter.model config key / DATACLAW_PRIVACY_FILTER_MODEL
    env var, so swapping is trivial.
  • Alternatives if ever needed: lakshyakh93/deberta_finetuned_pii (MIT, broad labels,
    pure drop-in). Avoid GLiNER / Presidio (need adapters / heavier deps).

Rewiring against HEAD `_cli` structure

  • Mutation happens only at EXPORT time, inline per session, right after
    secrets.transform_session (exporting.py:201 parallel worker, :292 serial). Confirm
    (review.py:625) is read-only and hashes the file (review.py:830); publish enforces that
    hash (commands.py:365-376). So the model edits MUST run at export, before the hash lock —
    not at confirm.
  • Keep the dict/text functions (redact_session, redact_text, scan_text, _load,
    resolve_device, oversized guard); discard the shard/jsonl/manifest layer (HEAD has no
    run_dir/manifest pipeline — it was dropped).
  • Read config (free-form dict, no schema in config.py):
    privacy_filter.{enabled (default False), device, min_score=0.85, include_tool_io=True, roles={user}}.
    Rust already writes privacy_filter.enabled/.device (dataclaw.rs:571,580) — this closes the gap.
  • Emit privacy_filter_* events via progress_callback → lights up the already-present dead
    Dashboard model_privacy stage (Dashboard.tsx:209-210, handlers :434-516). mechanical_pii
    stage maps to the existing regex layer.

Deps / packaging

  • Restore [project.optional-dependencies] pii = ["transformers>=4.57","torch>=2.3","accelerate","tokenizers"]
    • the pii pytest marker. Keep as an extra so the lazy-import graceful-degradation holds.
  • Do NOT bundle torch into the Mac sidecar by default (~1.5–2.5 GB; pyinstaller.spec has no
    torch hooks). Default enabled=false. Treat the model stage as power-user / dataclaw[pii], or
    a separate heavy build. Download the model at first run (HF cache), don't bundle weights.
  • Graceful degradation: if enabled but torch/model/network unavailable → warn and continue
    with mechanical-only redaction    , do not abort the export.

Prioritized action list

  1. Merge-on-upload with re-redaction + re-gate (Part A). The data-destruction bug and the
    privacy regression it would introduce — do them together; one is unsafe without the other.
  2. Atomic export write (temp+fsync+rename in exporting.py:493) — independent, stops silently
    truncated datasets being published. (From the first audit.)
  3. Privacy filter recovery + content_parts fix + real MODEL_ID + export-time rewire (Part B).
  4. Source enum reconcile (drop hermes, add cursor, validate in Rust, fail-closed in Python)
    and harden secret patterns + salt the anonymizer. (From the first audit.)
  5. Real cross-process run lock; batch the 3 HF uploads into one create_commit.

Merge-quality review (4-perspective sense-check) + fixes shipped

A focused review of the update/merge mechanism (correctness, reliability, elegance, scale)
found the design fundamentally right and reliable (fail-closed, convergent, self-healing,
optimistic-concurrency correct) but flagged a scale problem the carry-forward re-redaction fix
introduced. Shipped in 619ef63:

  • Keystone — redaction-policy stamp. Each record is stamped with
    redaction_policy_version() (hash of redact strings/usernames + model config + code
    version). The carry-forward redactor skips re-redaction for records already stamped current,
    re-redacting only stale ones. Turns a steady-state push from O(total history) (was
    re-running the PII model over the whole dataset every push) into ~O(new data), while
    preserving tighten-only (policy change bumps the version → one-time full re-scan).
  • Skip-unchanged. If the merged file is byte-identical to the downloaded remote, skip all
    uploads — no more empty-commit churn from the metadata timestamp.
  • Corrupt-line guard. _load_raw_records preserves unparseable JSONL lines verbatim
    instead of raising — a corrupt remote line can't drop data or wedge all future pushes.

Known/deferred from the merge review (not yet done)

  • Memory: merge materializes both full files in RAM → OOM risk past ~1–2 GB (streaming merge).
  • Network: full re-download + re-upload every push (sharding — larger change).
  • Compaction semantics: "more messages wins" keeps the stale pre-compaction copy when a
    conversation was legitimately summarized (keeps data, doesn't lose it; judgment call).
  • Atomicity: 3 uploads are separate commits (self-heals; create_commit would make atomic).
  • Model-load-status publish-time signal; git_branch redaction; dead UI telemetry. (From earlier audits.)

Open verification items (couldn't run live)

  • huggingface_hub version: confirm upload_file(parent_commit=...) support and that conflicts
    surface as HTTP 412; confirm hf_hub_download error classes.
  • Live Hub availability + offset behavior of the two recommended MODEL_IDs.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →