{"owner":"rerun-io","repo":"rerun","hasSkills":true,"totalSkillsCount":19,"totalTokensCount":41332,"categories":["anthropic-skill","project-spec","root-instruction","claude-rule","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["skills/rerun-urdf/SKILL.md","skills/assemble-changelog/SKILL.md","skills/rerun-chunk-processing/SKILL.md","skills/rerun-parquet/SKILL.md","skills/rerun-blueprint/SKILL.md","skills/rerun-mp4/SKILL.md","skills/rerun-data-model/SKILL.md","skills/rerun-lerobot/SKILL.md","skills/rerun-mcap/SKILL.md","skills/rerun-catalog-queries/SKILL.md","ARCHITECTURE.md","AGENTS.md","CLAUDE.md","DESIGN.md","rerun_py/ARCHITECTURE.md","crates/build/re_dev_tools/README.md","crates/build/re_build_tools/README.md","crates/build/re_dev_tools/src/build_web_viewer/README.md","crates/build/re_dev_tools/src/build_search_index/README.md"],"skills":{"skills/rerun-urdf/SKILL.md":"---\nname: rerun-urdf\ndescription: Drive the Rerun URDF API (rerun.urdf.UrdfTree) to ingest a URDF as a Transform3D layer on a robot recording. Read when logging a robot model, running forward kinematics from joint states, composing a fixed chain for sensor extrinsics, or when the transform tree will not connect from the data alone. Builds on rerun-chunk-processing (stream/lens mechanics) and rerun-data-model (entity paths, timeline, base-vs-layer).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun URDF ingestion\n\nA URDF gives you a robot's geometry and its kinematic tree.\nIngesting it means two API calls on one `rerun.urdf.UrdfTree`: stream the static model, then drive it with forward kinematics from joint states. The transforms it produces are derived, so they are a **layer**, never base (the `URDF + joints (computed)` row of the `rerun-data-model` table).\n\nThis skill is the `UrdfTree` API and the two judgment calls it cannot make for\nyou: how your joint values map to URDF joints, and how the disconnected frames\nin the scene connect to one root. The stream/lens plumbing (`LazyChunkStream`,\n`DeriveLens`, `Selector`, writing and optimizing RRDs) is in\n`rerun-chunk-processing`; reach for it, do not re-derive it. Nothing below is\ntied to a data format: where your joint names, joint values, and calibration\ncome from is yours to wire in.\n\n## Getting the URDF\n\nDatasets rarely ship one, and the robot's own repo is not always the fastest route.\nIn order:\n\n- [`robot_descriptions.py`](https://github.com/robot-descriptions/robot_descriptions.py) — `pip install robot_descriptions`, then `from robot_descriptions import ur5e_description; ur5e_description.URDF_PATH`.\n It downloads and caches the upstream description and hands you a plain path with the meshes already next to it.\n Check its index first; it covers most of the table below.\n- The vendor repo, when the description is newer than the package's pin or is not in it.\n- [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) — MJCF, not URDF, so it needs conversion. Worth it only when nothing else exists.\n\n| Robot | Source | Notes |\n| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |\n| SO-100 / SO-101 (LeRobot) | [TheRobotStudio/SO-ARM100](https://github.com/TheRobotStudio/SO-ARM100) | `Simulation/SO100/so100.urdf`, `Simulation/SO101/so101_new_calib.urdf`; plain URDF, relative mesh paths |\n| ALOHA (ViperX 300, WidowX) | [Interbotix/interbotix_ros_manipulators](https://github.com/Interbotix/interbotix_ros_manipulators) | xacro |\n| Franka Panda / FR3 | [frankaemika/franka_description](https://github.com/frankaemika/franka_description) | xacro |\n| Universal Robots UR3 to UR20 | [UniversalRobots/Universal_Robots_ROS2_Description](https://github.com/UniversalRobots/Universal_Robots_ROS2_Description) | xacro, arm parameters in YAML |\n| Kinova Gen3 | [Kinovarobotics/ros2_kortex](https://github.com/Kinovarobotics/ros2_kortex) | xacro |\n| UFactory xArm / Lite 6 | [xArm-Developer/xarm_ros2](https://github.com/xArm-Developer/xarm_ros2) | xacro |\n| Unitree Z1, G1, H1, Go2 | [unitreerobotics/unitree_ros](https://github.com/unitreerobotics/unitree_ros) | plain URDF per robot |\n\nTwo things bite before FK ever runs:\n\n- **xacro is not URDF.** First run `xacro arm.urdf.xacro > arm.urdf` (`pip install xacro`), passing whatever arguments the file requires (gripper variant, arm size).\n A xacro file handed to `UrdfTree.from_file_path` fails to parse.\n- **`package://` mesh URIs.** The importer resolves them by scanning `ROS_PACKAGE_PATH` (ROS 1) and `AMENT_PREFIX_PATH` (ROS 2) for the package, and otherwise treats the rest of the URI as relative to the URDF's own directory.\n Outside a sourced ROS workspace, either export one of those variables or lay the meshes out relative to the URDF.\n Meshes that fail to resolve give you a kinematically correct but invisible robot.\n\n## The API\n\n```python\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(\n urdf_path,\n entity_path_prefix=\"robot\", # links log under /robot/)`. The `recording_id` must equal the\nbase segment id or the layer never attaches; `application_id` is discarded on\nregistration.\n\n## Mapping joint state to URDF joints (you supply this; the data will not)\n\n`compute_joint_transform_batches` is only as right as the `names`/`values` you\nhand it, and the mapping is not in the URDF. Three things go wrong silently:\n\n- **Order.** Build an explicit `names` array aligned to the `values` you read.\n Never assume your message's field order matches the URDF's `` order.\n- **Count.** The URDF's non-`fixed` joint count rarely equals your reported value\n count. A gripper sent as one value is often two prismatic joints in the URDF;\n a mimic joint may be omitted from the message. Reconcile explicitly, and use\n the API to do it: iterate `urdf.joints()`, partition by `joint_type` and\n `mimic`. A joint with `mimic` set derives its value from the driver joint as\n `driver * multiplier + offset`; feed it that, not a message field. Confirm\n the count against `urdf.joints()`, not the message length.\n- **Units.** URDF joints are radians (revolute) and meters (prismatic). Convert\n if your source differs.\n\nGet any of these wrong and FK runs and writes a confident, wrong pose.\n\n### Verify joint mapping and limits\n\nJoint values in the source data aren't guaranteed to be within URDF joint limits.\nThis can happen with any format (MCAP, Parquet, etc).\nIt is therefore useful to check whether joint values in the data match the URDF model's limits.\nOut-of-range values can indicate issues like wrong indexing or model mismatch and should be clarified before calculating the forward kinematics.\n\nTo get limits of a URDF model:\n\n```python\nlimits = {joint.name: (joint.limit_lower, joint.limit_upper) for joint in urdf.joints() if joint.joint_type != \"fixed\"}\n# then compare each column's min/max against its joint's range\n```\n\n`scripts/check_joint_mapping.py` is that check worked out for one source shape — a LeRobot-style parquet column of per-frame joint values.\nRead it as the example to copy and adapt for your own format, and run it directly if your data is already a LeRobot-style parquet file:\n\n```bash\nuv run scripts/check_joint_mapping.py --column observation.state --degrees\n```\n\n**Important:** this is a validation meant to be done before full processing — don't run this for _every_ file in a larger dataset.\nSelect a single representative file or a small subset for this verification.\n\nFor each joint it tries the usual candidate corrections (identity, negate, `% 360`, ±90°/±180° offsets, and their combinations) and reports which ones land inside that joint's limits.\nIt reports the fit, not the answer: a joint with two surviving candidates is **ambiguous**, and no amount of range-checking will separate them.\nValidate ambiguous mappings against known poses or geometry for the dataset.\n\n**Read the failures by shape.**\nA whole column outside its range means a wrong sign, unit, or offset — a bug, and the pose will be confidently wrong.\nA few frames a few degrees past a limit is calibration slack — expected, and `clamp=True` is the fix, not a workaround.\n`clamp=True` warns once per offending row, so run it while working out the mapping: a flood of warnings is a mapping smell, and a thin trickle at the extremes of one joint is a healthy recording.\n\n## Make the joint states readable first\n\nWhere the joint values come from is not this skill's problem, but a dead joint\nsource produces an empty FK layer **with no error**: a source path that\nmatched nothing, a decoder that yielded zero rows, or a reader that dropped the\nmessage silently. Whatever the source, confirm the joint-state stream yields\nrows before debugging FK. The importer skill for your source format covers its\nown empty-stream failure modes.\n\n## How transforms compose (reason about this before logging anything)\n\nRerun resolves a pose by chaining transforms from a frame up to a root. There\nare two ways an edge in that chain gets defined, and a URDF ingest mixes both:\n\n- **By entity-path hierarchy.** A `Transform3D` on `/a/b` with no frame names is\n the transform of `/a/b` relative to its parent path `/a`. Composition follows\n the entity tree.\n- **By explicit frame graph.** A `Transform3D` that carries `parent_frame` and\n `child_frame` defines an edge between two **named frames**, independent of\n where in the entity tree it is logged. URDF FK uses this: every joint\n transform names its parent and child link frames.\n\nSo a URDF ingest is a graph of named frames. An edge exists only if some\n`Transform3D` names that exact `parent_frame -> child_frame` pair. A frame with\nno incoming edge is a root. The viewer renders every root at the world origin,\nwhich is why two unconnected robots silently overlap instead of erroring.\n\n## Resolving the transform forest (the part the data cannot always give you)\n\nA URDF is **one tree rooted at its base link**. FK and fixed joints supply every\nedge inside that tree. A real scene is a **forest** of roots the URDF never\nconnects: a world or scene frame, each robot's base, every camera or sensor\nframe. The edges that join those roots come from calibration (extrinsics in a\nsidecar, a TF static publisher, a hand-eye result), not from the URDF, and some\nare simply absent. A single connected tree is not always solvable. Resolve it\ndeliberately:\n\n1. **Enumerate every frame.** Iterate `urdf.joints()` and collect the\n `parent_link`/`child_link` pairs (the in-tree edges, frame-prefixed);\n `urdf.root_link()` is that URDF's root. Add every sensor/world frame the\n scene needs (from the `rerun-data-model` table). Decide the one intended\n root.\n2. **Classify each edge by source.** In-URDF edges (FK joints, `fixed` joints)\n come from the URDF plus joint values. Inter-root edges (root to each robot\n base, root to each fixed sensor, an arm link to a wrist-mounted camera) come\n from calibration and you must log them yourself.\n3. **Compose fixed chains from the URDF** when you need the transform between two\n links joined only by `fixed` joints (a camera bracket, a tool mount): walk\n parent links across `fixed` joints via `urdf.joints()`, turning each joint's\n `origin_xyz`/`origin_rpy` into a homogeneous matrix and multiplying along\n the chain. If the walk cannot reach\n the target link, **stop and say so** (\"no fixed chain from A to B; stuck at\n C\"). A broken chain is a wrong pose, not a missing one.\n4. **Build the edge set and find the roots.** Collect every `parent_frame ->\nchild_frame` pair you will log (URDF + calibration). Walk parents from each\n frame; any frame that does not reach the intended root is an unconnected root\n and names a **missing edge**. This is a pure graph check you can run before\n writing the RRD.\n5. **Resolve every missing edge, or fail loudly.** For each one:\n - If calibration supplies the transform, log it (next section).\n - If the data does not, the tree is unsolvable. Do **not** leave the frame\n disconnected (it collapses onto the origin and reads as one merged scene).\n Either abort and name the missing edge, or log identity and emit a loud\n warning naming the assumed edge. State which you did.\n6. **Match frame names across sources.** FK-derived `parent_frame`/`child_frame`\n must equal the names `urdf.stream()` emits for the static geometry, and your\n calibration edges must use those same names, or links float off the mesh. The\n `frame_prefix` is what keeps them identical; reuse it everywhere.\n\nA correct ingest has exactly one root, and a path from every frame to it.\n\n## Logging a connection correctly\n\nA connecting edge is a **static** `Transform3D` carrying the bridging frame\nnames. Static (no time index) so it holds for the whole recording; the frame\nnames, not the entity path, are what create the graph edge. This\n`Chunk.from_columns` is the rare **sidecar exception** — a calibration transform\nno reader or FK lens can produce; do not generalize it to transforms a reader\nemits (a `frame_transforms` topic → `Transform3D`) or that FK derives:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import Chunk, LazyChunkStream\n\n# world -> this robot's base, from your calibration (translation + xyzw quaternion)\nedge = Chunk.from_columns(\n \"/world/robot_base\", # any sensible bridging entity path\n indexes=[], # no index == static\n columns=rr.Transform3D.columns(\n translation=[translation],\n quaternion=[quaternion_xyzw],\n parent_frame=[\"world\"], # must match the root frame name\n child_frame=[\"arm_base_link\"], # must match the URDF root frame (prefixed)\n ),\n)\nedges = LazyChunkStream.from_iter([edge]) # merge alongside model + FK streams\n```\n\nLog a fixed-chain result (step 3) the same way, with `parent_frame` and\n`child_frame` set to the two link frames the chain spans. Merge all edge chunks\ninto the same recording as the model and FK streams so they share the graph.\n\n## Minimal shape (one robot, generic joint source)\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens, LazyChunkStream, OptimizationProfile, Selector\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(urdf_path, entity_path_prefix=\"robot\", static_transform_entity_path=\"robot/tf_static\")\nmodel = urdf.stream(include_joint_transforms=True).drop(content=\"/robot/**/collision_geometries/**\")\n\njoints = source_joint_state_stream() # your reader; one message column of names+values\nfk = (\n joints\n .lenses(\n DeriveLens(JOINT_MSG_COMPONENT, output_entity=\"/tmp/batches\").to_component(\n \"rerun.urdf.JointTransformBatch\",\n Selector(\".\").pipe(lambda msgs: urdf.compute_joint_transform_batches(read_names(msgs), read_values(msgs))),\n ),\n content=JOINT_SOURCE_PATH,\n output_mode=\"forward_all\",\n )\n .lenses(\n DeriveLens(\"rerun.urdf.JointTransformBatch\", output_entity=\"/robot/transforms\", scatter=True)\n .to_component(rr.Transform3D.descriptor_translation(), Selector(\"[].translation\"))\n .to_component(rr.Transform3D.descriptor_quaternion(), Selector(\"[].quaternion\"))\n .to_component(rr.Transform3D.descriptor_parent_frame(), Selector(\"[].parent_frame\"))\n .to_component(rr.Transform3D.descriptor_child_frame(), Selector(\"[].child_frame\")),\n content=\"/tmp/batches\",\n output_mode=\"drop_unmatched\",\n )\n .filter(content=\"/robot/transforms\")\n)\n\nLazyChunkStream.merge(model, fk).collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n out_path,\n application_id=\"urdf\",\n recording_id=segment_id,\n)\n```\n\n`read_names`, `read_values`, `JOINT_MSG_COMPONENT`, and `JOINT_SOURCE_PATH` are\nthe only data-specific pieces, and the mapping section above is what makes them\ncorrect.\n\n## One URDF for a whole dataset (assets)\n\n`urdf.stream()` emits only static chunks — meshes and fixed-joint transforms, with no time index.\nThe model does not belong in a per-episode layer, where every segment pays for the same meshes.\nRegister it once as a dataset **asset**:\n\n```python\nurdf.stream().collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n model_rrd, application_id=\"urdf\", recording_id=\"so100_model\"\n)\nasset_id = dataset.register_asset(\"s3://bucket/so100_model.rrd\") # uri must be readable by the server\n```\n\nThe split is along static vs temporal: the **model** is an asset, and the **FK transforms** stay a per-segment layer keyed by `recording_id`.\nKeeping `include_joint_transforms=True` in the asset costs nothing and gives the rest pose for segments or time ranges the FK layer does not cover.\n\n- The viewer loads a dataset's assets alongside any segment it opens, and caches them across segments.\n- `dataset.segment_store(segment_id)` includes assets; pass `include_assets=False` to skip the extra manifest request per asset.\n- Dataframe queries never see assets — they stay on the dataset's own segments.\n\nWithout a catalog, the same split still pays: write the model RRD once and open it next to whichever episode you are looking at, rather than copying it into each one.\n\n## Gotchas that cause real failures\n\n1. Empty layer, no error: dead joint-state source (decoded to zero rows; see the importer skill for your format) or wrong `JOINT_SOURCE_PATH`/component name.\n2. Confident wrong pose: joint count, order, or units wrong.\n3. Layer writes but never attaches: `recording_id != segment_id`.\n4. Frames collide: two robots sharing a `frame_prefix`.\n5. Scene looks merged at the origin: unconnected roots logged as identity without a calibration edge.\n6. Catalog ingest rejects or misorders chunks: `OBJECT_STORE` optimization skipped.\n7. Layer opened in a running viewer never merges: opening an `.rrd` **by path** registers it as its own dataset and replaces its `application_id`, so it lands beside the segment instead of on it.\n To eyeball a layer locally, either merge model + FK + base into one RRD, or send both through a `RecordingStream(application_id, recording_id=segment_id)` pointed at the viewer with `rec.connect_grpc(url=\"rerun+http://127.0.0.1:9876/proxy\")`, which keeps the IDs and merges the stores.\n8. Robot invisible but transforms correct: meshes failed to resolve (see \"Getting the URDF\").\n9. No 3D view after adding a URDF to an existing recording: the auto-layout does not necessarily rebuild for the new entities. Send an explicit blueprint with `rrb.Spatial3DView(origin=\"/robot\")` (see `rerun-blueprint`).\n\n## References\n\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing` (FK two-lens pattern, two robots + scene URDFs, prefixes, recoloring, calibration offsets)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/animated_urdf` (classic logging API: `log_urdf_to_recording`, per-joint `compute_transform`)\n- [Catalog object model: assets](https://rerun.io/docs/concepts/query-and-transform/catalog-object-model#assets) (`register_asset`, the limits, and how assets are read back)\n- `rerun-data-model` (the mapping table this skill consumes)\n- the importer skill for your joint-state source format (making the source readable)\n- `rerun-chunk-processing` (lens/stream, write/optimize mechanics)\n","skills/assemble-changelog/SKILL.md":"---\nname: assemble-changelog\ndescription: >\n Assemble the per-PR changeset entries in docs/content/changelog/upcoming/\n into a release's changeset-0-XX.md and generate the detailed CHANGELOG.md sections.\n Use at release time when the user wants to build or finalize the changelog or\n changeset for a Rerun release, or merge the upcoming/ entries. Invoked via\n /assemble-changelog or \"assemble the changelog for 0.x.y\".\nuser_invocable: true\nallowed-tools: Bash Read Edit Write\n---\n\n# Assemble-changelog\n\nRelease-time changelog assembly for the Rerun repo.\n\nAlways work in the root of a standalone `rerun-io/rerun` checkout — normally the `prepare-release-0.x.y` branch, where the result is committed.\nThis is step 4 of [RELEASES.md](../../RELEASES.md); read it for the surrounding context.\n\nBefore doing any release work, verify that the current directory is the repository root and that its `origin` is `rerun-io/rerun`:\n\n```bash\ntest \"$(git rev-parse --show-toplevel)\" = \"$PWD\"\ngit remote get-url origin\n```\n\nDo not run the workflow in the reality monorepo, including from its `rerun/` directory.\nThe release scripts need the standalone repository's `0.x.y` tags and resolve `(#N)` commit references against `rerun-io/rerun`.\nRunning them against reality can silently resolve reality PR numbers to unrelated Rerun PRs.\n\nIf either precondition is not met, stop before running any release command.\nTell the user that the skill requires the root of a standalone `rerun-io/rerun` checkout, and ask them to restart it there.\nDo not clone a repository, fetch tags, or switch branches for the user.\n\nResolve the target version from `$ARGUMENTS` (e.g. `0.34.0`). If absent, read it\nfrom `Cargo.toml` (`version = \"0.x.y-…\"`) and confirm with the user.\n\n## Workflow\n\n### 1. Assemble `upcoming/` → the release changeset\n\nThe curated entries live one-file-per-PR in `docs/content/changelog/upcoming/*.md`\n(skip `_template.md`). Each declares `type: highlight|misc|breaking|feature` in its\nfrontmatter. Merge them into `docs/content/changelog/changeset-0-XX.md`, creating that file from\n`docs/content/changelog/_template.md` if it does not exist yet (set `title` to the version — keep it\nquoted, e.g. `title: \"0.36\"`, so YAML keeps it a string — and `order` one lower than the previous release):\n\n- `highlight` → one `### ` subsection each under `## Highlights`.\n- `feature` → one `### ` subsection each under `## New features`.\n- `misc` → exactly one `- ` list item each under `## Other`. Fold its `### ` heading and body into the item, preserving links. Do not use headings or prose outside the list items.\n- `breaking` → one `### ` subsection each under `## Breaking changes`. If none, write `None.`.\n\nKeep the sections in that order and omit `Highlights` and `Other` if they have no entries.\nThe changelog is user-facing (it's part of the website), so it leads with what's new;\nthe verbose, developer-only breaking-change migration guides go last so most readers\ndon't have to scroll past them.\n\nTailor the output to the release type:\n\n- Patch release (`0.x.Y`, Y > 0) → typically only bug fixes. Skip `Highlights` and\n `New features` (there usually won't be `upcoming/` entries anyway); include `Other`\n only if there are miscellaneous entries, and keep `Breaking changes` only if there are any.\n- Minor release (`0.X.0`) → the full template: highlights, new features, other items, breaking changes.\n\nPreserve each entry's prose and structure (migration guides, tables, `snippet:` directives,\nscreenshots, links), except that each miscellaneous entry becomes its single list item.\nDe-duplicate overlapping entries and order breaking changes most-impactful first.\nDrop the per-entry frontmatter.\n\nRelative doc links in entries were written as if from `changelog/` (e.g.\n`../reference/migration/...`), which is correct once merged — keep them as-is.\n\nImages and videos must live on `static.rerun.io` (upload with `scripts/upload_image.py`), never as\nGitHub attachments: attachments uploaded in the private monorepo cannot be shown on the website.\n\nFeature leads write their `## Highlights` entries per PR, so read the section as a whole and\nflag to the human anything that does not hang together, rather than rewriting it yourself.\n\nFinally, point the `redirect:` frontmatter in `docs/content/changelog.md` at\n`changelog/changeset-0-XX`: CI's `scripts/ci/check_changelog_redirect.py` requires the\nnewest changeset to be the redirect target, so the repoint must land together with the\nnew changeset.\n\n### 2. Resolve release blockers\n\nEnsure that every non-template file from `upcoming/` was merged into the changeset, then search the assembled changeset for unresolved placeholders:\n\n```bash\nrg -n 'TODO\\([^)]+\\)' docs/content/changelog/changeset-0-XX.md # NOLINT\n```\n\nResolve every match before continuing.\nAn unresolved `TODO(name)` blocks the release.\n\n### 3. Generate the summary and detail sections into CHANGELOG.md\n\n```bash\npixi run uvpy scripts/generate_changelog.py --version 0.x.y\n```\n\nEdit PR titles/labels to improve the output, then copy the result into `CHANGELOG.md`\n(drop the trailing \"Chronological changes\" section; replace the placeholder video/blogpost\nlines as previous releases did). Spot-check a few entries against the actual PRs:\npolluted titles (old, unrelated PRs; `thanks @…` for core team members) mean a PR-number\nlookup misfired — see the warning at the top.\nRead the generated section for wrong contributor attribution and typos.\nIf the release ships without Rust crates (`publish-crates=false`), say in the `CHANGELOG.md`\nsection that this version is not on crates.io.\n\nDo this *after* step 1: the script reads the assembled changeset and emits a summary of it\n(section headings + links to the changeset on the website), rather than inlining its prose.\n`CHANGELOG.md` therefore never duplicates the changeset — if the changeset is missing, the\nscript emits an unresolved placeholder instead.\n\n### 4. Polish the CHANGELOG.md section\n\nThe generated section is a starting point; edit it by hand so readers can find their way to the details.\n\n- **Order the overview by impact.**\n Put the biggest new capabilities first, then integrations and APIs, then format support, then polish, then niche or developer-facing items.\n- **Group and order the details.**\n The script emits each details subsection (🐍 Python API, 🪳 Bug fixes, 🌁 Viewer improvements, …) in commit order.\n Within each subsection, put entries that touch the same feature next to each other (e.g. all MCP/ViewerControl entries, all state timeline entries, all hangs and deadlocks), order the groups by their most impactful entry, and order entries within a group by impact.\n Only reorder lines; don't move entries between subsections or drop them.\n- **Make entries clickable.**\n Wherever possible, link an entry (in the overview *and* in the details) to the docs that show how to use the feature: the how-to guide, the archetype/view reference page, the CLI or MCP reference section, the Python/JS ref site, or docs.rs.\n Reuse the links the changeset already contains, converted to `https://rerun.io/docs/`, and check that each target file and `#anchor` heading exists.\n Link external names too (e.g. a third-party tool to its docs).\n In detail entries, link the key term, not the whole line, so the trailing commit/PR link stays distinct.\n- **Link each breaking change** to its subheading in the changeset: `https://rerun.io/docs/changelog/changeset-0-XX#`.\n Anchors are the lowercased heading with backticks and punctuation dropped and spaces turned into hyphens (`rrd::optimize` → `rrdoptimize`).\n- **Name the actual break.**\n A breaking-change heading (in the changeset and in `CHANGELOG.md`) must say what broke, e.g. \"`WriteChunks` removed\", not the new feature that replaces it (\"Catalog staging\").\n Lead the subsection with the removal and migration-guide link, then the replacement.\n- **Flag experimental and unstable features.**\n Check every new feature in the changeset and `CHANGELOG.md` against these markers (`PREV` is the previous minor release tag, e.g. `0.38.0`):\n\n ```bash\n # Non-blueprint types marked unstable whose definitions changed this cycle\n git diff --name-only $PREV HEAD -- crates/build/re_type_definitions | grep -v /blueprint/ | xargs rg -l 'state = \"unstable\"'\n # Python APIs in the `rerun.experimental` module\n git log --oneline $PREV..HEAD -- rerun_py/rerun_sdk/rerun/experimental\n # Viewer features behind a flag in `ExperimentalAppOptions`\n git diff $PREV HEAD -- crates/viewer_support/re_viewer_context/src/app_options.rs\n # Docs and docstrings that call something experimental\n git diff --name-only $PREV HEAD -- docs/content rerun_py/rerun_sdk | xargs rg -l -i 'is experimental'\n ```\n\n A file showing up in the first command does not mean the type is new: cross-reference the hits with the actual entries.\n For each matching feature, say so where readers will see it: prefix the changeset heading with \"Experimental\" (as `### Experimental 3D gaussian splat support` did in 0.36), and add a sentence saying it may change in future releases (for unstable types: that the data may not stay backwards compatible).\n If the feature is behind a flag, say how to enable it.\n In the `CHANGELOG.md` overview, move all of these entries into a final `#### 🧪 Experimental and unstable` subsection at the end of \"Overview & highlights\", with a one-line note that they may change and that unstable types may not stay backwards compatible.\n Order that subsection by impact too, and don't repeat \"Experimental\" in its entries.\n Blueprint types (`rerun.blueprint.*`, under `re_type_definitions/rerun/blueprint/`) are an exception: they are all marked unstable, which only means the blueprint data isn't backwards compatible.\n That alone does not make a feature experimental, so a new view or blueprint setting is only experimental if one of the other markers applies to it (or to the data archetype it shows).\n Likewise, a Viewer feature is not experimental just because it is configured through an experimental API.\n- **Add media to the headline features** in the overview: a screenshot (or a still frame plus a link to the video) indented under the bullet, using the `` markup that `pixi run upload-image` prints.\n Aim for a few images (roughly three to five), covering the most visual headline features: a picture sells a new view or renderer feature far better than a bullet does.\n Start from the media already in the changeset and the PR descriptions; if a visual feature has none, ask the user for a screenshot rather than skipping it.\n Don't add images to every entry: API, CLI, and bug-fix items rarely benefit.\n All media must live on `static.rerun.io`.\n Reuse a PR's media only if it is already on `static.rerun.io` or publicly accessible: `github.com/user-attachments/…` links from the private monorepo return 404 for the public.\n Download those with `curl -L -H \"Authorization: token $(gh auth token)\" ` and upload them with `pixi run upload-image --name `.\n GitHub does not render `` from external hosts, so for a video, extract a representative still (`ffmpeg -ss -i video.mp4 -frames:v 1 still.png`), upload it, and link the `.mp4` below it.\n Look at the frames and pick one without a cursor over the subject.\n\n### 5. Empty the inbox\n\nDelete the merged `upcoming/*.md` entries, keeping `_template.md` and any entries not included in this release.\nNever add redirects for these temporary entries.\n\n## Checklist before declaring done\n\n- [ ] Every non-template `upcoming/` entry is represented in the changeset.\n- [ ] No `TODO(name)` remains in the changeset.\n- [ ] No summaries or other prose were synthesized for existing entries.\n- [ ] The `CHANGELOG.md` overview is ordered by impact, a few of its most visual headline features have `static.rerun.io` images, and its breaking changes link to the changeset subheadings.\n- [ ] Every overview and detail entry that has relevant docs links to them.\n- [ ] Each details subsection groups entries by feature, most impactful first.\n- [ ] Every experimental or unstable feature (feature flag, `rerun.experimental`, `#[rerun(state = \"unstable\")]` on a non-blueprint type, or documented as experimental) is labeled as such in the changeset, and listed under the overview's final \"Experimental and unstable\" subsection.\n- [ ] `pixi run lint-rerun CHANGELOG.md docs/content/changelog/changeset-0-XX.md` passes.\n- [ ] Images and videos are hosted on `static.rerun.io`, not as GitHub attachments.\n- [ ] `## Highlights` reads well as a whole, or the human has been told where it doesn't.\n- [ ] The `CHANGELOG.md` section has correct contributor attribution and no typos.\n- [ ] `upcoming/` contains only `_template.md` and entries deferred to a later release.\n- [ ] `python scripts/ci/check_changelog_redirect.py` passes (redirect points at this changeset).\n- [ ] `python scripts/ci/check_doc_redirects.py --base origin/main` passes (`upcoming/` entries are exempt and DO NOT need a redirect).\n\n## Notes\n\n- This skill lives in `skills/assemble-changelog` in the standalone Rerun repository.\n- Synced commits in `rerun-io/rerun` carry a `Source-Ref` trailer (the reality merge\n commit); `generate_changelog.py` resolves it back to the originating reality PR for\n correct titles, labels, and contributors.\n- The *next* release's changeset is not pre-created: an empty changeset for an\n unreleased version would make `check_changelog_redirect.py` fail, since it requires the newest\n `changeset-0-xx.md` to be the redirect target. During a cycle, `upcoming/` is the only in-flight\n artifact.\n","skills/rerun-chunk-processing/SKILL.md":"---\nname: rerun-chunk-processing\ndescription: \"Core mechanics of the Rerun Chunk Processing API (rerun.chunk) — LazyChunkStream pipelines, Chunk, lenses (MutateLens/DeriveLens/Selector), RrdReader, writing optimized RRDs. Read BEFORE writing any ingestion/conversion/preprocessing code (convert an MCAP, build a recording from a dataset, preprocess an .rrd, port an old converter): it mandates reader+lens pipelines and steers away from hand-built chunks — no Chunk.from_columns for data a reader/lens can produce, no per-message rr.log, no manual pa.array assembly. Source-specific knowledge lives in the importer skills (rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot); read rerun-data-model first to decide what the data should become.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun chunk processing\n\nThe pipeline layer between raw data and an RRD: readers produce `Chunk`s,\nstreams transform them, terminal calls execute. This skill is the generic\nmechanics only. Decide the data model first (`rerun-data-model`), then pick the\nimporter skill for each source:\n\n| Source | Reader | Skill |\n| ----------------------------------------- | --------------------------------------- | --------------- |\n| MCAP file (ROS2, protobuf, Foxglove) | `McapReader(path).stream()` | `rerun-mcap` |\n| URDF robot model (+ joint states → FK) | `UrdfTree.from_file_path(...).stream()` | `rerun-urdf` |\n| Parquet table (trajectories, sensor logs) | `ParquetReader(path).stream()` | `rerun-parquet` |\n| mp4 camera video | `Mp4Reader(path).stream()` | `rerun-mp4` |\n| HDF5 file (groups → entities) | `Hdf5Reader(path).stream()` | none yet |\n| LeRobot dataset directory | built-in importer, then `RrdReader` | `rerun-lerobot` |\n| Existing RRD | `RrdReader(path)` | here, below |\n| Sidecar files (JSON calib, metadata) | `Chunk.from_columns` + `from_iter` | here, below |\n\nThe API is `rerun.chunk`; when\nbehavior matters, check the installed surface:\n`python -c \"from rerun.chunk import LazyChunkStream; help(LazyChunkStream)\"`.\n\n## Decision rule: where does each component come from?\n\nDefault: **a reader produces the chunks; lenses shape them.** Walk this before\nwriting any conversion code — most \"build it by hand\" instincts are wrong here:\n\n1. **Source a reader supports?** Use the reader's `.stream()`; never hand-parse\n and re-log. MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`,\n mp4→`Mp4Reader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`,\n LeRobot dir→`log_file_from_path`.\n2. **A decoder already emits the archetype?** Foxglove gives `Transform3D`,\n `Pinhole`, `VideoStream` (real sample bytes) ready-made — **pass it through**,\n do not re-derive. Only custom-protobuf topics arrive as `:message` and\n need a lens (see `rerun-mcap`).\n3. **Fix an existing component in place** (swapped resolution, recolor, unit\n convert)? `MutateLens`, `output_mode=\"forward_unmatched\"`.\n4. **Derive a new component/entity** (FK→`/tf`, scalars from a message)?\n `DeriveLens`. To scatter one row into N (a joint batch → per-joint `/tf`),\n use the **two-lens pair**: derive the batch with `output_mode=\"forward_all\"`\n (keeps the originals, e.g. the joint states), then a second\n `DeriveLens` with `scatter=True` and `output_mode=\"drop_unmatched\"` (emits\n only the scattered rows). See the `robot_data_preprocessing` example.\n5. **Genuine sidecar** no reader or lens can produce (JSON calibration offsets,\n hand-measured extrinsics, external metadata)? `Chunk.from_columns` + `from_iter`.\n6. Finish with `LazyChunkStream.merge(...)` →\n `.collect(optimize=OptimizationProfile.OBJECT_STORE)` →\n `write_rrd(application_id, recording_id)`.\n\nWhy this order: the pipeline stays lazy, columnar, multithreaded, and\n`OBJECT_STORE`-optimizable. A hand-built row loop or an out-of-lens `pa.array`\nthrows all of that away — that is the path we are deliberately avoiding.\n\n## Anti-patterns (use a reader + lens instead)\n\nIf you are writing the left, stop and use the right:\n\n- **`for`-loop building rows/components** → a lens with a `Selector(...).pipe(...)`\n PyArrow-compute callback.\n- **`rr.init` + `rr.log` per message for conversion** → that is _live_ logging;\n for ingestion, read with a reader and `write_rrd`.\n- **`chunk.to_record_batch()` + `pc.filter` then rebuilding via\n `Chunk.from_columns`** (row-thinning by hand) → `stream.drop(content=...)`,\n `.split(...)`, or a `MutateLens` returning a filtered `pa.array`.\n- **`pa.array` / `pa.RecordBatch` / `np.frombuffer` assembled OUTSIDE a lens** →\n move the transform inside a `MutateLens`/`DeriveLens` selector callback.\n- **`rr.send_columns` hand-assembled from a custom parser** → use the matching\n reader; it produces chunks directly.\n- **Parsing MCAP/URDF with a non-Rerun library then re-logging** → `McapReader`\n / `UrdfTree`.\n- **`Chunk.from_columns` for data a reader already decodes** (`Pinhole`\n intrinsics, `VideoStream`, `Transform3D` from a transforms topic) → keep it in\n the reader stream; fix with a `MutateLens` if needed.\n\nA wall of `pyarrow.compute` \"missing-attribute\" type errors (`pc.filter`,\n`pc.list_element`) usually means `pc.*` calls sit in module-level helpers instead\nof inside `Selector.pipe` lens callbacks. Refactor into a lens before suppressing\nthe checker — the errors are a smell that the hand-building should not exist.\n\n**Porting an existing converter?** Hand-built converters predate decoder\nimprovements and are not ground truth. Re-verify the decoder output (step 2) and\ncheck every `Chunk.from_columns` / for-loop against this list before copying.\n\n## Core model\n\n- `LazyChunkStream` is a lazy pipeline DAG, not a collection. Building\n filters, lenses, maps, splits, and merges reads no source data.\n- Execution starts at terminal calls: `write_rrd(...)`, `collect()`,\n `to_chunks()`, or iterating the stream.\n- Execution is streaming, multithreaded, and mostly GIL-free. Prefer\n stream/lens operations and PyArrow compute over Python row loops.\n- **Move semantics**: builder calls (`filter`, `drop`, `lenses`, `map`,\n `flat_map`) consume the input stream; reusing a consumed stream raises.\n Reassign after each step. Terminal calls do not consume, but each terminal\n call re-executes the whole pipeline; `collect()` once if that is too costly.\n- `ChunkStore` is materialized in memory (`stream.collect()`,\n `ChunkStore.from_chunks`). `LazyStore` is manifest-indexed, loads chunks on\n demand (`RrdReader(path).store()`, catalog segment stores). Both have\n `schema()`, `summary()`, `stream()`, and `write_rrd(...)`.\n\n## Stream composition\n\n```python\nfrom rerun.chunk import Chunk, LazyChunkStream, OptimizationProfile\n```\n\n- `stream.filter(content=, has_timeline=, is_static=, components=)` keeps the\n matching portion of each chunk; `stream.drop(...)` is its complement, same\n keyword filters. `content` takes an entity-path glob or a list of them.\n- `stream.map(fn)` applies `Chunk -> Chunk`; `stream.flat_map(fn)` applies\n `Chunk -> Iterable[Chunk]`. Escape hatches for chunk-level Python logic;\n prefer lenses for columnar work.\n- `stream.split(content=, ...)` returns `(matching, non_matching)`; both\n branches share the same upstream.\n- `LazyChunkStream.merge(*streams)` fans in any number of sources.\n- `LazyChunkStream.from_iter(chunks)` wraps hand-built chunks.\n\n```python\nstream = source_stream() # any importer skill\nstream = stream.drop(content=\"/video_raw/**\")\nstream = stream.lenses(fix_lens, content=\"/cam/**\", output_mode=\"forward_unmatched\")\nmerged = LazyChunkStream.merge(stream, sidecar_stream)\nmerged.write_rrd(out_path, application_id=\"my_app\", recording_id=recording_id)\n```\n\n## Hand-built chunks — sidecar only\n\nUse `Chunk.from_columns` ONLY for data no reader or lens can emit — JSON/CSV\ncalibration, frame offsets, external metadata. If a reader\n(`McapReader`/`UrdfTree`/`ParquetReader`/`Hdf5Reader`) decodes the topic or a\nlens can derive it, that is the idiomatic path; do not hand-assemble it here.\nIn the `robot_data_preprocessing` example the _only_ hand-built chunk is the JSON\noffsets sidecar; the camera fix, FK→`/tf`, meshes, and recolor are all\nreaders + lenses.\n\n`Chunk.from_columns(entity_path, indexes, columns)` mirrors\n`rr.send_columns(...)` and accepts the same archetype `.columns(...)` helpers.\nEmpty `indexes` means static.\n\n```python\nchunk = Chunk.from_columns(\n \"/tf_static/robot_offsets\",\n indexes=[], # static\n columns=rr.Transform3D.columns(\n translation=translations,\n quaternion=quaternions_xyzw,\n parent_frame=parents,\n child_frame=children,\n ),\n)\nsidecar_stream = LazyChunkStream.from_iter([chunk])\n```\n\n`rr.AnyValues.columns(...)` covers non-standard metadata fields. For\ninspection, a `Chunk` exposes `entity_path`, `num_rows`, `is_static`,\n`timeline_names`, `to_record_batch()`, and `format()` (human-readable table).\n\n### Recording properties\n\nRecording properties are the one exception here: do NOT hand-build them.\n`Chunk.from_property(name, values)` mirrors `rr.send_property` and lands a\nstatic chunk on `/__properties/`.\n\nDo this:\n\n```python\nchunk = Chunk.from_property(\n \"episode\",\n rr.AnyValues(task=\"pick\", duration_sec=12.4),\n)\n```\n\ninstead of hand-assembling the same chunk:\n\n```python\nChunk.from_columns(\n f\"/__properties/{name}\",\n indexes=[],\n columns=rr.AnyValues.columns(**{name: [value]}), # type: ignore[arg-type]\n)\n```\n\n- `/__properties/` is Rerun's internal layout; the second version hardcodes it.\n- A dynamic field name makes `AnyValues.columns()` need a\n `# type: ignore[arg-type]`. That suppression is the signal to switch.\n\n## Lenses\n\nLenses reshape, fix, or derive components without iterating rows. Apply with\n`stream.lenses(lenses, output_mode=..., content=...)`.\n\n- `MutateLens(component, selector, keep_row_ids=False)` modifies an existing\n component in place.\n- `DeriveLens(component, output_entity=None, scatter=False)` creates new\n columns, optionally at another entity. Chain `.to_component(descriptor,\nselector)` per output; `.to_timeline(name, \"sequence\" | \"duration_ns\" |\n\"timestamp_ns\", selector)` extracts a time column from the data itself.\n `scatter=True` explodes one input row into N output rows (one per list\n element).\n- Scope with `content=` whenever the same component name exists under multiple\n entities.\n\nOutput modes, and **the default is `drop_unmatched`**:\n\n- `drop_unmatched` (default): only lens outputs survive. Right for derive-only\n intermediate streams; silently discards everything else if applied broadly.\n- `forward_unmatched`: lens outputs plus the original components no lens\n consumed. Right for targeted fixes that preserve the rest of the stream.\n- `forward_all`: lens outputs plus every original component, including a\n `DeriveLens` input. Same as `forward_unmatched` for mutate lenses.\n\nIn-place fix (keep Arrow type and length intact):\n\n```python\nstream = stream.lenses(\n MutateLens(\n \"Pinhole:resolution\",\n Selector(\".\").pipe(\n lambda res: pa.array(\n [(h, w) for w, h in res.to_pylist()],\n type=res.type,\n )\n ),\n ),\n content=[\"/external/cam_low\", \"/external/cam_high\"],\n output_mode=\"forward_unmatched\",\n)\n```\n\nDerive with unit conversion (PyArrow compute, no Python loop):\n\n```python\nDeriveLens(\"schemas.proto.JointState:message\", output_entity=\"/joints_deg/waist\").to_component(\n rr.Scalars.descriptor_scalars(),\n Selector(\".joint_positions\").pipe(lambda arr: pc.multiply(pc.list_element(arr, 0), 180.0 / math.pi)),\n)\n```\n\n## Selector grammar\n\n`Selector(\"\")` navigates nested Arrow data, jq-style:\n\n- `.` current value; `.field` struct field\n- `[]` iterate list elements; `[N]` index a list\n- `?` suppress errors / skip missing optionals; `!` assert non-null\n- `|` pipe one expression into another\n\n`.pipe(fn)` chains a Python/PyArrow transform (or another Selector).\n`.execute(array)` runs it eagerly; `.execute_per_row(array)` guarantees the\noutput row count matches the input (use inside lens callbacks that must stay\nrow-aligned).\n\n## Writing RRDs\n\n- `stream.write_rrd(path, application_id=..., recording_id=...)` executes and\n writes in one streaming pass.\n- `stream.collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(...)`\n materializes, optimizes chunk layout, then writes. Memory scales with the\n materialized chunks.\n- Profiles: `OBJECT_STORE` (large chunks, for storage/query/catalog) and\n `LIVE` (small chunks, low-latency viewer).\n- Multiple physical RRDs form one logical recording when they share a\n `recording_id`; use this to separate base data, model/URDF data, and layers.\n\n**Always use `OptimizationProfile.OBJECT_STORE`** when the RRD is headed for a\nRerun catalog or Hub, unless explicitly asked otherwise.\n\n## Chunk API vs logging API\n\n- Logging (`rr.log`, `rr.send_columns`, `RecordingStream`) is for live logging\n from user code; chunk processing is for ingestion, conversion, and\n postprocessing existing recordings.\n- Logging → chunks: write an RRD, read it back with `RrdReader`.\n `RrdReader(path)` lists `recordings()` / `blueprints()` (each a `StoreEntry`\n with `kind`, `application_id`, `recording_id`); `.stream(store=entry)` for\n sequential passes, `.store(store=entry)` for indexed access.\n- Chunks → logging: `rerun.send_chunks(chunks, recording=...)`\n accepts a `Chunk`, `LazyChunkStream`, `LazyStore`, `ChunkStore`, or any\n iterable of chunks. The source store's `application_id`/`recording_id` are\n **not** preserved; the active recording's identity wins.\n\n## Common gotchas\n\n- The default lens `output_mode` is `drop_unmatched`; forgetting to set\n `forward_unmatched` on a targeted fix silently drops the rest of the stream.\n- Do not reuse a consumed `LazyChunkStream`; reassign or `split` deliberately.\n- Scope lenses with `content=`; the same component name often exists under\n many entities.\n- Preserve Arrow array type and length in `MutateLens` transforms.\n- For catalog layers, the layer `recording_id` must equal the segment id.\n- This is `rerun.chunk`; pin-check signatures when upgrading.\n\n## References\n\n- End-to-end example (MCAP + URDF + JSON sidecar, lenses, merge, optimize):\n `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- Docs: `https://rerun.io/docs/concepts/logging-and-ingestion/chunk-processing-api`,\n `https://rerun.io/docs/concepts/query-and-transform/lenses`\n","skills/rerun-parquet/SKILL.md":"---\nname: rerun-parquet\ndescription: Ingest tabular Parquet files into Rerun chunk streams with rerun.experimental.ParquetReader. Read when converting trajectory or sensor tables (LeRobot-style parquet, exported logs) into entities and components — column grouping, timeline/index columns, static columns, and lenses (DeriveLens) that assemble the typed components (Transform3D, Scalars) from the reader's grouped struct/scalar output. Builds on rerun-chunk-processing and rerun-data-model.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun parquet ingestion\n\n`ParquetReader` is a **pure reader**: it maps a flat table onto the Rerun\nmodel by turning raw columns into grouped, time-indexed chunks of struct and\nscalar components. Column-name prefixes become entities, grouped columns\nbecome a single struct component, designated columns become timelines. The\nreader does **not** assemble archetypes anymore — mapping struct fields into\ntyped Rerun components (Transform3D, Scalars, Points3D) is done with lenses on\nthe reader's `.stream()`. The whole reader job is configuration; fill in the\n`rerun-data-model` mapping table first, then express it through `stream()`.\nStream mechanics after `.stream()` are in `rerun-chunk-processing`.\n\n**The whole table is configuration, not code.** If you find yourself building\n`Chunk.from_columns` from a parquet, or munging it in pandas first, stop —\n`ParquetReader` plus a lens almost certainly expresses it. Anything the reader\ncannot express (per-row entity routing, derived values, unit conversion)\nbelongs in lenses downstream, not in pre-pandas munging; keep the pipeline\ncolumnar.\n\n## The API\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nreader = ParquetReader(table_path) # a lightweight handle over the file\nstream = reader.stream(\n entity_path_prefix=\"/world\", # prepended to every entity path\n column_grouping=\"prefix\", # \"prefix\" | \"individual\" | \"explicit_prefixes\"\n delimiter=\"_\", # split for column_grouping=\"prefix\"\n prefixes=None, # required for \"explicit_prefixes\"\n use_structs=True, # pack grouped columns into one struct component\n static_columns=[\"robot_type\"], # constant-per-file values, logged static\n index_columns=[IndexColumn.timestamp(\"timestamp\", input_unit=\"us\"), IndexColumn.sequence(\"frame_index\")],\n)\n```\n\nThe constructor takes only `path`; all loading options live on `stream()`, and every one is keyword-only.\nEach `stream()` call is independent, so one reader can drive several differently-configured streams over the same file.\nThe config is validated against the file's schema at `stream()` (missing index/static columns raise `ValueError` there, not mid-iteration).\nThere is no `column_rules` kwarg — typed-component assembly moved to lenses (below).\n\n## What the reader emits\n\nThe reader turns the table into chunks, one chunk per group, then leaves the\ndata as generic struct/scalar components for lenses to map. The naming is the\nkey thing the rest of the pipeline keys off:\n\n- **A grouped multi-column prefix `X`** → entity `/X`, with a single struct\n component named **`data`**. The struct's fields are the column names with the\n prefix (and delimiter, for `\"prefix\"`) stripped. So `A_pos_x`, `A_quat_w`\n under prefix `A` land as struct `data` with fields `pos_x`, `quat_w` on\n entity `/A`.\n- **A lone column with no group** → its own entity named after the column, and\n a raw component named after the column — *not* a `data` struct. So a `speed`\n column becomes entity `/speed`, component `speed`.\n- **A `/__properties` metadata chunk** built from the parquet file's schema\n metadata. You typically drop it right after `.stream()`:\n\n ```python\n stream = reader.stream().drop(content=\"/__properties/**\")\n ```\n\n## Column grouping: which columns share an entity\n\n- `\"prefix\"` (default): split each column name on `delimiter`, group by the\n first segment. `gripper_pos_x`, `gripper_pos_y` → entity `/gripper`, struct\n `data{pos_x, pos_y}`.\n- `\"explicit_prefixes\"`: group by the exact strings in `prefixes`, tried\n longest-first; the prefix is stripped from each struct field name (a raw\n string match, no delimiter — `foo` + `a` → field `a`). Columns matching no\n prefix become individual groups. Use this when names contain the delimiter\n ambiguously (`observation.state` vs `observation.images.top`: pass the full\n prefixes).\n- `\"individual\"`: every column is its own chunk/entity with a raw component\n named after the column — no struct packing at all, even for columns sharing a\n prefix. `use_structs` is ignored here. Rarely the model you want; reach for\n it only as a debugging baseline.\n\n`use_structs=True` (default) packs a group's columns into a single Arrow\nstruct component (the `data` field) for `\"prefix\"`/`\"explicit_prefixes\"`;\n`False` emits one component per column (the pre-struct flat layout, what\nqueries see as separate columns).\n\n## Timelines: `index_columns`\n\nEach entry is built with `IndexColumn`: `IndexColumn.timestamp(name, input_unit=...)` (since epoch), `IndexColumn.duration(name, input_unit=...)` (elapsed), or `IndexColumn.sequence(name)` (ordinal int).\n\n- `input_unit` describes what the raw integers in the column *are* (`\"ns\"` default,\n `\"us\"`, `\"ms\"`, `\"s\"`); Rerun rescales to ns internally. Sequences take no unit.\n\n**If omitted, a synthetic `row_index` sequence timeline is generated.** That\nis almost never the timeline you want to query or align against; always name\nthe real time columns. Stamp both a timestamp and a sequence timeline when the\ntable has both (multi-rate alignment, see `rerun-data-model`).\n\n## Static columns: `static_columns`\n\nListed columns are constant across all rows; they are emitted once as a single\nstatic (timeless) chunk, separate from the temporal data. A listed column that\nactually varies raises an error when the stream runs — that error is a\ndata-quality signal, not a reason to drop the static declaration.\n\n## Typed components via lenses\n\nThe reader's grouped output is generic struct (`data`) and scalar data. A\n`DeriveLens` reads that struct's fields, packs and casts them into real Rerun\ncomponents, and writes them to an output entity — this is what the old\n`column_rules` API used to do, now done downstream on the stream.\n\nConstruct a lens against the reader's struct component (`\"data\"` for grouped\nprefixes, or the column name for a lone/individual column), then add one or\nmore `.to_*` builder methods. Each builder returns a fresh lens, so they chain.\n\n| Builder | Produces | Argument order |\n|---|---|---|\n| `to_translation(x, y, z)` | `Transform3D:translation` | x, y, z |\n| `to_quaternion(x, y, z, w)` | `Transform3D:quaternion` | x, y, z, w (xyzw) |\n| `to_scale(x, y, z)` | `Transform3D:scale` | x, y, z |\n| `to_rotation_axis_angle(axis_x, axis_y, axis_z, angle)` | `Transform3D:rotation_axis_angle` | axis_x, axis_y, axis_z, angle (radians) |\n| `to_scalars(*fields)` | `Scalars:scalars` | one or more field names |\n| `to_packed_component(component, *fields)` | the given component | descriptor, then field names |\n| `to_component(component, selector)` | the given component | descriptor, then a `Selector` |\n| `to_timeline(name, type, selector)` | a timeline (not a component) | name, `\"sequence\"`/`\"duration_ns\"`/`\"timestamp_ns\"`, selector |\n\n`to_packed_component` packs the named struct fields (in order, at least one\nrequired) into the fixed-size list the component expects, and by default\n**auto-casts `f64`→`f32`** to match component types. The `to_translation`,\n`to_quaternion`, `to_scale` helpers are convenience wrappers over it, so they\nall auto-cast. `to_rotation_axis_angle` builds a `Struct{axis, angle}` and\nhard-casts axis and angle to `f32` internally. `to_scalars` with a single field\nemits a plain scalar per row (not a 1-element list); with multiple fields it\nemits one scalar series per field at the same entity.\n\nApply lenses with `.stream().lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")`:\n\n- `content` is a pre-filter on the *source* entity path — it scopes which\n chunks the lens may touch. Out-of-scope chunks pass through unchanged. Set it\n to the reader's grouped entity (e.g. `\"/A\"`).\n- `output_mode` decides the fate of in-scope-but-unmatched chunks:\n `\"drop_unmatched\"` (default, keep only lens output), `\"forward_unmatched\"`\n (output replaces matched, other originals survive), or `\"forward_all\"`\n (output plus all originals).\n- The lens's own `output_entity=` sets the *destination* entity — independent\n of `content`, which gates the input side.\n\nEnd-to-end Transform3D example. The reader groups `A_*` columns into a `data`\nstruct at `/A`; the lens reads the prefix-stripped field names (`pos_x`,\n`quat_w`), packs and casts them, and writes a full `Transform3D` to `/pose`:\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nlens = (\n DeriveLens(\"data\", output_entity=\"/pose\")\n .to_translation(\"pos_x\", \"pos_y\", \"pos_z\")\n .to_quaternion(\"quat_x\", \"quat_y\", \"quat_z\", \"quat_w\")\n)\n\nchunks = (\n ParquetReader(table_path)\n .stream(index_columns=[IndexColumn.sequence(\"frame_index\")])\n .lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")\n .to_chunks()\n)\n```\n\nChaining several `.to_*` on **one lens with a shared `output_entity`**\naccumulates multiple component columns into the same archetype at that entity —\nabove, both `Transform3D:translation` and `Transform3D:quaternion` land on\n`/pose`, forming a complete `Transform3D`. For a generic fixed-size-list\ncomponent, pass the descriptor to `to_packed_component`:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens\nfrom rerun.experimental import ParquetReader\n\nlens = DeriveLens(\"data\", output_entity=\"/points\").to_packed_component(\n rr.Points3D.descriptor_positions(), \"x\", \"y\", \"z\"\n)\n```\n\n## Selectors\n\nLens field paths use `Selector`, a jq-like grammar over Arrow columns\n(`.field` to access a struct field, `[]` to iterate a list, `[N]` to index, `?`\nto suppress errors on absent fields, `!` to assert non-null, `|` to pipe, and\n`pack(.x, .y, .z)` to zip paths into a fixed-size list). The `to_*` helpers\nbuild these selectors for you; reach for `to_component(component, Selector(\".x\"))`\nwhen you need a custom field path. Field paths reference the\n**prefix-stripped** struct field names — the lens sees `pos_x`, not `A_pos_x`.\n\n## Gotchas\n\n1. No `index_columns` → synthetic `row_index` timeline only. Queries that\n expect a timestamp timeline find nothing.\n2. The `unit` is the raw column's unit, not a desired output unit; a\n microsecond column declared `\"ns\"` lands 1000x in the past.\n3. `static_columns` raises if a listed column actually varies; that error is a\n data-quality signal, not a reason to drop the static declaration.\n Uniformity is data-dependent, so it is raised lazily when the stream runs;\n a listed column that doesn't *exist* raises eagerly at `stream()`.\n4. A grouped prefix's struct component is named **`data`** — that is the\n `input_component` string a `DeriveLens` matches against. A lone or\n `\"individual\"` column is instead a raw component named after the column.\n5. Selector field paths reference the **prefix-stripped** struct field names\n (`pos_x`, not `gripper_pos_x`).\n6. Drop the `/__properties` metadata chunk the reader emits from parquet schema\n metadata: `.stream().drop(content=\"/__properties/**\")`.\n7. Quaternion column order is x, y, z, w in `to_quaternion`; check the source's\n convention before wiring fields.\n8. `to_packed_component` (and the transform helpers built on it) auto-casts\n `f64`→`f32` to match component types; this is usually what you want for\n parquet's double columns.\n9. Anything the reader cannot express (per-row entity routing, derived values,\n unit conversion) belongs in lenses downstream, not in pre-pandas munging;\n keep the pipeline columnar (`rerun-chunk-processing`).\n\n## References\n\n- Lens builder source with full docstrings: `rerun/chunk/_lens.py` in\n the installed `rerun-sdk` package (`to_translation`, `to_quaternion`,\n `to_scale`, `to_rotation_axis_angle`, `to_scalars`, `to_packed_component`,\n `to_component`, `to_timeline`).\n- Reader source: `rerun/experimental/_parquet_reader.py`, or\n `python -c \"from rerun.experimental import ParquetReader; help(ParquetReader)\"`\n- Canonical worked examples: the integration tests\n `rerun_py/tests/integration/test_parquet_reader.py` (grouping, index/static\n columns, and the Transform3D / Points3D / Scalars lens flows) and\n `rerun_py/tests/integration/test_lazy_chunk_stream.py` (lens application,\n `content`/`output_mode`, selectors).\n- `rerun-lerobot` — LeRobot datasets store episodes as parquet; that skill\n covers the built-in importer route vs reading the parquet directly with\n this reader.\n- `rerun-data-model` (mapping decisions), `rerun-chunk-processing` (stream\n mechanics after `.stream()`)\n","skills/rerun-blueprint/SKILL.md":"---\nname: rerun-blueprint\ndescription: Change how the Rerun viewer looks by writing a blueprint, for a recording or for a table. Read this whenever asked to change the look of the viewer, such as laying out a recording or dataset, designing a default blueprint, deciding which views show which entities, or making a segment or catalog table look nice by hiding columns, changing column order or filtering rows. Covers archetype-to-view mapping, layout reasoning, the rrb construction API, the contents grammar, table blueprints, and the screenshot loop.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun blueprint\n\n**To change how the viewer looks, change the blueprint.** This holds for a\nrecording's layout, views and their properties, and for a table's columns, row\nfilters and cards. Don't click through the viewer's menus to do it. With the\nviewer MCP, read and write the blueprint with `rerun_get_blueprint` and\n`rerun_set_blueprint` rather than the low-level click tools.\n\nA blueprint decides how a recording is shown, and the recording decides what exists.\nRead the data, design a first layout, then **iterate from screenshots** until it\nreads at a glance. The constructors are discoverable (`dir(rrb)`,\n`help(rrb.Spatial3DView)`); this skill covers what you can't guess. Import as\n`import rerun.blueprint as rrb`.\n\n## Tables\n\nA dataset's segment table and catalog tables have a table blueprint of their\nown, separate from a recording's. It sets which columns show, their order and\nnames, the row filters, and whether rows show as a table or as cards. When asked\nto make a table look nice, hide columns or filter rows, write this blueprint.\n\nThis hides two columns and shows three others first. Listed columns come first,\nthe rest keep their default order and visibility:\n\n```python\nblueprint = rrb.TableBlueprint(\n table_layout=rrb.table.TableLayout(\n columns=[\n rrb.table.Column(\"recording link\"),\n rrb.table.Column(\"property:episode:current_task\", name=\"Task\"),\n rrb.table.Column(\"property:episode:success\"),\n rrb.table.Column(\"property:episode:ext1_cam_serial\", visible=False),\n rrb.table.Column(\"property:episode:ext2_cam_serial\", visible=False),\n ],\n ),\n)\nblueprint.save(\"my_app\", path)\ndataset.register_blueprint(path.as_uri(), segment_table=True)\n```\n\n`Column` takes the column's name as the table has it, such as\n`property:episode:uuid`, not its header. Registering makes it the default for\neveryone who opens the table, and the server must be able to read `path`.\n\nRow filters are SQL expressions, one per string, all of which must be true for a\nrow: `rrb.TableBlueprint(filters=[\"\\\"task\\\" ILIKE '%pick%'\", '\"duration\" > 10'])`.\nDouble quotes for column names, single quotes for strings, and\n`any_match(\"column\", x -> …)` for list columns.\n\nWith the viewer MCP, change the table on screen directly:\n`rerun_get_table_schema` names the columns, `rerun_get_blueprint` with\n`{\"table\": {}}` reads the table blueprint, and `rerun_set_blueprint` with\n`{\"table\": {}}` replaces it. `rerun_set_blueprint` returns an error for a filter\nthat fails. A column's entry sits at `/table/layouts/table/columns/`,\nwith each `:` and space in the name escaped as `\\:` and `\\ `, which JSON writes\nas `\\\\:` and `\\\\ `. The [MCP docs](https://rerun.io/docs/reference/viewer/mcp#table-blueprints)\nhave a worked example of the JSON. For the whole Python API, including cards\nand segment previews, see\n[Configure table layouts and segment previews](https://rerun.io/docs/howto/visualization/configure-table-blueprints).\n\n## 1. Read the data\n\nEnumerate every `(entity_path, archetype)` pair first — the archetype picks the\nview, the entity path scopes it. For a dataset (see `rerun-catalog-queries`):\n\n```python\nfor c in dataset.schema().component_columns():\n print(c.entity_path, c.archetype, c.component_name)\n```\n\nFor a local recording, stream it with `RrdReader` and read `entity_path` and\nthe archetype off each chunk (see `rerun-chunk-processing`).\n\nThe dataset schema alone doesn't list every entity a view can show. Assets are\nstatic data such as URDF robot meshes, which the viewer applies to every segment,\nbut they don't appear in the segment entities. When `rerun_get_dataset_schema`\nlists `asset_ids` next to `entities`, call `rerun_get_dataset_assets_schema` to\nget the entities in each asset, and read it before choosing between a 2D and a\n3D view or preview. For example, `asset_ids: [\"urdf-model-left\", \"urdf-model-right\"]`\nmeans the robot models can go in a `Spatial3DView`.\n\n## 2. Map archetype to view\n\n| Archetype family | View |\n| ------------------------------------------------------------------------------------- | --------------------------------------------------- |\n| `Points2D`, `Image`, `EncodedImage`, `Boxes2D`, `LineStrips2D`, `Pinhole` projections | `Spatial2DView` |\n| `Points3D`, `Mesh3D`, `Boxes3D`, `LineStrips3D`, `Transform3D`, `Asset3D` | `Spatial3DView` |\n| `Scalars`, `SeriesLines`, `SeriesPoints` | `TimeSeriesView` |\n| `TextLog` / `TextDocument` / `BarChart` | `TextLogView` / `TextDocumentView` / `BarChartView` |\n| `Tensor`, `DepthImage` (heatmap) | `TensorView` |\n| `GeoPoints`, `GeoLineStrings` | `MapView` |\n| `GraphNodes`, `GraphEdges` | `GraphView` |\n| tabular / catalog data | `DataframeView` |\n\nAsset entities, such as `Asset3D` and URDF meshes, go in a `Spatial3DView`.\n\n## 3. Group entities into views\n\nThis step is just deciding which entities share a view. Sizing and placement are\nmechanics you tune by eye in the screenshot loop, not rules to settle up front.\n\n- **Group by shared path prefix.** Entities under a common prefix usually belong\n in one view. Don't give every entity its own pane: dozens of raw message\n entities go in one `DataframeView`, or get left out, rather than a pane each.\n- **One spatial frame per spatial view.** A whole `Transform3D` tree goes in one\n `Spatial3DView` at the common ancestor; a `Pinhole` camera gets its own\n `Spatial2DView` rooted at the camera so images inherit the projection.\n- **Collapse scalars.** Many `Scalars` under a prefix → one `TimeSeriesView` over\n the prefix, not one each. Split only when value ranges clash.\n- **`column_shares`/`row_shares` are relative weights** on a container's\n children, equal by default. Tune them from the screenshots.\n\n## 4. Construct it\n\nContainers (`Grid`, `Horizontal`, `Vertical`, `Tabs`) hold views. Default a flat\nset to `Grid`; use the others for a deliberate split, `Tabs` for alternatives\ncompeting for one slot (left/right/depth cameras). **Always set an explicit\n`origin` and `name`** — `origin` defaults to `/`, which dumps the whole tree into\none view (the usual cause of an unreadable blob).\n\n```python\nblueprint = rrb.Blueprint(\n rrb.Horizontal(\n rrb.Spatial3DView(origin=\"/world\", name=\"Scene\"),\n rrb.Vertical(\n rrb.Spatial2DView(origin=\"/world/camera\", name=\"Camera\"),\n rrb.TimeSeriesView(origin=\"/sensors\", name=\"Sensors\"),\n ),\n column_shares=[3, 2],\n ),\n rrb.TextLogView(origin=\"/logs\", name=\"Logs\"),\n collapse_panels=True,\n)\n```\n\n`contents` defaults to `\"$origin/**\"`. Scope a view with include/exclude rules,\ne.g. `contents=[\"+ $origin/**\", \"- $origin/internal/**\"]`. A bare line is an\ninclude; `/**` is the only wildcard (matches a subtree). Most-specific rule wins,\nties go to the last, unmatched paths are excluded.\n\n**Coordinate frames.** A spatial view only renders entities it can place relative\nto its target frame. `Transform3D`/`Pinhole` logged on entities compose down the\ntree, so `origin` is enough. But **named frames** (`CoordinateFrame(frame=...)`,\ncommon in ROS/MCAP) live in a separate frame graph — point the view at a frame\nthe data occupies via `spatial_information=rrb.SpatialInformation(target_frame=\"\", apply_latest_at=True)`.\n- A view back blank? The importer may not have finished (bump the settle) or the\n cursor sits before the data (add `rrb.TimePanel(play_state=rrb.components.PlayState.Following)`).\n- `rrb` views and `ViewerClient` (`rerun.experimental`) are unstable. Check\n `help()` if a constructor argument is rejected.\n\n## See also\n\n- `rerun-data-model` — entities, archetypes, timelines.\n- `rerun-catalog-queries` — enumerate entities in a dataset.\n","skills/rerun-mp4/SKILL.md":"---\nname: rerun-mp4\ndescription: Ingest .mp4 video into Rerun chunk streams with rerun.experimental.Mp4Reader. Read when converting video into a VideoStream, choosing stream vs asset mode, transcoding (B-frames, output codec, GOP size) through FFmpeg, or aligning video PTS onto a recording's wall-clock timeline. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (where the video belongs in the recording).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun mp4 ingestion\n\n`Mp4Reader` turns one `.mp4` file into a lazy chunk stream on **one entity**:\ncompressed video samples, no decode to pixels, nothing re-encoded unless it has\nto be. Everything is configured on the constructor; `stream()` takes no\narguments. Stream mechanics after `.stream()` (filter, map, merge, collect,\nwrite) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.experimental import Mp4Reader, Mp4TranscodeOptions\n\nreader = Mp4Reader(video_path, entity_path=\"/camera/front\") # mode=\"stream\" by default\nstream = reader.stream() # lazy: nothing is decoded yet\n```\n\nEvery parameter after `path` is keyword-only. One reader handles one file — for\nseveral cameras, build one reader per file and `LazyChunkStream.merge(...)` them\n(see below).\n\n## Two modes\n\n| `mode` | emits | when |\n| -------------------- | ------------------------------------------------------- | ----------------------------------------------------------------- |\n| `\"stream\"` (default) | a codec chunk, per-GOP sample chunks, a keyframe marker | almost always — every frame is time-indexed and queryable |\n| `\"asset\"` | the whole file as one blob, plus a frame index | the codec cannot be a `VideoStream`, or the source starts mid-GOP |\n\nReach for stream mode; the next section covers exactly what it emits. Asset mode\ncopies the entire file into the recording as one blob, so nothing downstream can\nlook at a single frame without the whole asset. It is the fallback for codecs a\n`VideoStream` cannot carry (`mp4v`, image-sequence mp4), and it is what `rerun\nvideo.mp4` and the built-in file importer use.\n\n## What stream mode emits\n\nFor `tests/assets/video/Big_Buck_Bunny_1080_1s_h264_nobframes.mp4` (30 frames,\none GOP) with `entity_path=\"/camera/front\"`, the whole output is three chunks:\n\n```\n[0] entity=/camera/front static=True rows=1 timelines=[] cols=['VideoStream:codec']\n[1] entity=/camera/front static=False rows=30 timelines=['video'] cols=['VideoStream:sample']\n[2] entity=/camera/front static=False rows=1 timelines=['video'] cols=['VideoStream:is_keyframe']\n```\n\n- **One static chunk, one row**, holding `VideoStream:codec`.\n It carries **no timeline** — code that walks the stream must\n handle that (`if chunk.is_static: continue`).\n- **One temporal chunk per GOP**: a keyframe plus every sample that depends on\n it, up to (not including) the next keyframe. Samples only — the keyframe flags\n are not a column here.\n- **One trailing `is_keyframe` marker chunk**, holding a *sparse* `True` row at\n each keyframe's time and nothing else. Keeping it out of the sample chunks is\n what lets a keyframe-only query skip the sample payload, and what\n `collect(optimize=…)` accepts as canonical. It is list-per-row, so a row reads\n back as `[True]`, not `True`.\n- **Timeline `video`**, `duration[ns]`, values are the mp4 PTS from the start of\n the video.\n- **Every chunk on the same entity.** `entity_path=None` (the default) derives\n it from the file's *absolute* path, so `foo/video.mp4` run from `/data`\n becomes `/data/foo/video.mp4`. Pass `entity_path` explicitly in any real\n pipeline.\n\nA short clip is often a single GOP, so \"three chunks total\" is the normal shape —\nnot a sign that something was dropped. `chunk_by_gop=False` emits one Rerun\nchunk per sample instead (the marker chunk is unaffected). That is a debugging\nshape — one chunk per frame is a poor storage layout, and it also makes a\nfollowing optimize pass **6× more expensive** (measured below) — so leave the\ndefault alone unless you are inspecting individual samples.\n\nDistinguish the three by their columns, not by position or `is_static`: the codec\nchunk is the static one, sample chunks carry `VideoStream:sample`, and the marker\nchunk carries only `VideoStream:is_keyframe`.\n\nSupported stream-mode codecs are exactly the five `VideoCodec` values: H264,\nH265, AV1, VP8, VP9.\n\n## Transcoding through FFmpeg\n\n`VideoStream` cannot yet model DTS != PTS, so an H.264/H.265 source with\ncontainer-level B-frame reordering **cannot be emitted directly**. The reader\nhandles that itself: it re-encodes through FFmpeg with `-bf 0`, streams the\nresult back as a fragmented mp4, and turns each fragment into one GOP chunk, so\nonly one GOP is resident at a time. This is automatic and invisible in the API —\nthe output has the same shape as a clean source. It does need an `ffmpeg`\nexecutable. Asset mode is unaffected by B-frames.\n\n`Mp4TranscodeOptions` (stream mode only) additionally *requests* a transcode:\n\n| field | effect |\n| ----------------- | -------------------------------------------------------------------------- |\n| `output_codec` | re-encode to another `VideoCodec`; the emitted `VideoStream:codec` follows |\n| `gop_size` | force a keyframe every N frames — the knob for seek cost in the viewer |\n| `try_gpu` | best-effort hardware encode, NVENC / VideoToolbox only |\n| `ffmpeg_override` | use this `ffmpeg` instead of the one on `PATH` |\n\nRequesting the `output_codec` the source already uses is a **no-op**: it stays on\nthe direct, no-FFmpeg path. With `chunk_by_gop=True`, `gop_size=N` makes every\nGOP chunk but the last hold exactly N samples — a 30-frame clip at `gop_size=10`\ngives 10, 10, 10. `try_gpu` realistically covers H264/H265 plus AV1 on newer\nNVIDIA; VP8/VP9 always fall back to software, and it does nothing unless a\ntranscode is already happening.\n\n```python\nMp4Reader(\n video_path,\n entity_path=\"/camera/front\",\n # ~1s GOPs on a 60fps source, so seeking in the viewer stays snappy.\n transcode=Mp4TranscodeOptions(gop_size=64),\n).stream()\n```\n\n## Timelines: mp4 PTS is not your recording's clock\n\nThe emitted times are always PTS — elapsed nanoseconds from the start of *that\nvideo*. In a multi-sensor recording, that is almost never the timeline you want\nto align on.\n\n- `timeline_name=\"real_time\"` renames the timeline.\n- `timeline_type=\"timestamp\"` only **retypes** the same PTS values as\n nanoseconds since the Unix epoch. The reader does not shift them, so on its\n own it renders the video near 1970. It is meaningful only paired with a retag\n step.\n- **Retag with `stream.map(...)`.** Samples arrive in presentation order\n (B-frames are already stripped), so a running cursor maps sample `i` to\n `capture_times_ns[i]` — but the cursor must skip the keyframe marker chunk,\n which holds no samples. Map that one through the PTS the samples were already\n assigned:\n\n```python\nSAMPLE_COL = \"VideoStream:sample\"\n\n\ndef _reindex_to_capture_times(stream, capture_times_ns, timeline_name):\n cursor = 0\n pts_to_time = {}\n\n def _retag(chunk):\n nonlocal cursor\n if chunk.is_static: # the codec chunk carries no timeline\n return chunk\n batch = chunk.to_record_batch()\n col_index = batch.schema.get_field_index(timeline_name)\n old_field = batch.schema.field(col_index)\n old_pts = np.asarray(batch.column(col_index).cast(pa.int64()))\n\n if SAMPLE_COL in batch.schema.names:\n # Clamp in case the decoder yields a slightly different frame count.\n indices = np.clip(np.arange(cursor, cursor + chunk.num_rows), 0, len(capture_times_ns) - 1)\n cursor += chunk.num_rows\n new_times_ns = capture_times_ns[indices]\n pts_to_time.update(zip(old_pts.tolist(), new_times_ns.tolist()))\n else:\n # The sparse keyframe marker, emitted after every sample chunk.\n new_times_ns = np.array([pts_to_time[pts] for pts in old_pts.tolist()], dtype=np.int64)\n\n times = pa.array(new_times_ns.astype(\"datetime64[ns]\"))\n # `metadata=` is load-bearing — see gotcha 3.\n new_field = pa.field(old_field.name, times.type, nullable=old_field.nullable, metadata=old_field.metadata)\n return Chunk.from_record_batch(batch.set_column(col_index, new_field, times))[0]\n\n return stream.map(_retag)\n```\n\nThe cursor makes this order-dependent, so keep the retag on the single reader's\nstream, before any merge. This is the DROID loader's pattern — see the\nreferences.\n\nAny `map`/`flat_map` that assumes \"every non-static chunk is samples, in order\"\nhas this same bug: it advances over the marker chunk and stamps it with whatever\ntime comes next, silently moving the keyframe markers off their samples.\n\n## Several cameras into one recording\n\nOne reader per file, distinct entity paths, then merge:\n\n```python\nstreams = [\n Mp4Reader(path, entity_path=f\"/camera/{name}\", timeline_name=\"real_time\", timeline_type=\"timestamp\").stream()\n for name, path in cameras.items()\n]\n(\n LazyChunkStream\n .merge(*streams)\n .collect(optimize=OptimizationProfile.OBJECT_STORE)\n .write_rrd(out_path, application_id=app_id, recording_id=segment_id)\n)\n```\n\n`OBJECT_STORE` GOP-rebatches the video and preserves the reader's keyframe\nmarker as-is; no `fix_keyframe=True` is needed. If you *do* see\n`skipping GoP rebatching … is_keyframe data is incorrect`, something upstream\nrewrote the marker (see gotcha 4) — `fix_keyframe=True` re-derives it from the\nencoded samples as an escape hatch.\n\n## How the reader's chunks relate to optimize's\n\nThe reader and `collect(optimize=…)` agree on **where GOPs start** but not on\n**how many GOPs share a chunk**, and that is by design:\n\n- The reader emits the **finest GOP-aligned partition**: exactly one chunk per\n GOP. It cannot do better, because it does not know which profile the data is\n headed for.\n- Optimize then applies the size policy, **merging consecutive GOPs** up to the\n profile's `max_bytes` / `max_rows`. It never splits a GOP across chunks, so\n every boundary it keeps is one the reader already produced.\n\nSo optimize's partition is a pure *coarsening* of the reader's: same samples in\nthe same order, every optimized chunk a run of whole reader GOPs, keyframe marker\nuntouched. Where each GOP already sits near the profile's budget the two come out\nidentical (a 30-frame clip at `gop_size=10` is 3 chunks either way). Where GOPs\nare small they diverge sharply — a 12-GOP clip becomes **1** chunk under\n`OBJECT_STORE` and **4** under `LIVE`.\n\nThe practical consequence: **do not pre-merge or re-chunk the reader's output**\nto \"help\" optimize. Hand it the per-GOP stream and let the profile decide. And if\nyou skip optimize entirely (writing straight through `send_chunks`, as the DROID\nloader does), you are storing the finest partition — correct, but more chunks\nthan object storage wants.\n\n### The optimize pass is cheap on GOP-chunked input\n\nRunning optimize over the reader's output is not redundant work worth avoiding.\nOn an 89 MB / 1800-frame / 60-GOP H.264 file:\n\n| | `collect()` | `collect(optimize=OBJECT_STORE)` | delta |\n| ----------------------------- | ----------- | -------------------------------- | ------------------- |\n| `chunk_by_gop=True` (default) | 11.1 ms | 16.3 ms | **+5.2 ms** (1.5×) |\n| `chunk_by_gop=False` | 14.2 ms | 48.6 ms | **+34.4 ms** (3.4×) |\n\nNeither half of the pass is expensive on GOP-aligned input:\n\n- **Detecting GOP starts is header-only.** `build_sample_index` runs\n `detect_gop_start` on every sample, but that reads a few bytes of each sample's\n header — it never decodes. The whole optimize delta above (5.2 ms) is smaller\n than a Python loop calling the same detector over the same 1800 samples\n (6.0 ms), which is mostly FFI overhead.\n- **Rebuilding the chunks copies nothing.** `chunk_from_gop` calls\n `taken(0..n)`, and `re_arrow_util::take_array` returns the array **uncopied**\n when the indices are consecutive from zero over the whole array. One chunk per\n GOP means that is always the case, so the multi-MB sample buffers are never\n duplicated — hence 89 MB \"rebuilt\" in 5 ms.\n\nThat fast path is exactly what `chunk_by_gop=False` gives up: each GOP then spans\n30 source chunks, so `chunk_from_gop` has to `concat_and_sort` them and the sample\nbytes really are copied. `LIVE` also costs more than `OBJECT_STORE` (+13.2 ms vs\n+5.2 ms on the same file), because its much smaller chunk budget makes it attempt\ncompaction it cannot complete — a GOP is never split.\n\nRetag each camera *before* the merge — the retag above walks chunks with a\ncursor, so it must see one video's chunks in order.\n\n## Gotchas\n\n1. **Errors surface on the first pull, not at construction.** The constructor\n only checks that the file exists and validates its arguments; `stream()`\n builds a lazy pipeline and also succeeds. Codec support, keyframe layout, and\n FFmpeg availability are checked when the stream runs, so wrap the\n *consumption* (`to_chunks()`, `send_chunks`, iteration), not the\n `Mp4Reader(…)` call.\n2. **A codec outside the five `VideoCodec` values cannot be a `VideoStream`.**\n `mp4v` (MPEG-4 Part 2) is the one you will actually meet in robot datasets;\n it raises `RuntimeError: MP4 error: MP4 demux: Video track uses unsupported\ncodec \"mp4v\"`. Asset mode does accept the file, but only partly: the blob\n chunk is emitted and the `VideoFrameReference` index chunk is **skipped with\n a warning**, because the frame timestamps cannot be read either — so you get\n the bytes and no timeline. Skipping that camera and recording the fact as a\n recording property (what the DROID loader does) is usually better than a\n timeline-less blob, and either beats failing a whole episode over one camera.\n3. **A `map`/`flat_map` that rewrites the time column must carry the field\n metadata over.** `pa.field(...)` without `metadata=old_field.metadata` drops\n `rerun:kind: 'index'`, and the rebuilt chunk silently becomes **static** —\n no error, no timeline, and the samples land outside time entirely.\n4. **The trailing marker chunk is temporal but holds no samples**, so any\n per-chunk logic keyed on `not chunk.is_static` will process it as if it were\n samples. Key on the `VideoStream:sample` column instead. See the retag above.\n5. **Asset mode is capped at ~2 GiB** by Arrow's i32 offsets, and duplicates the\n file's bytes into the RRD.\n6. `mode=\"asset\"` rejects both `chunk_by_gop=False` and `transcode=` with\n `ValueError` — those are stream-mode-only knobs.\n7. `timeline_type=\"timestamp\"` on its own shifts nothing. Without a retag, the\n video sits at the epoch.\n8. **The reader emits compressed samples, never pixels.** Thumbnails, CLIP\n embeddings, or anything else needing RGB have to decode the file separately\n (OpenCV, PyAV); the `VideoStream` chunks cannot supply them.\n9. Every `stream()` call re-decodes the file from scratch, and every terminal\n call re-runs the pipeline. `collect()` once when you need more than one pass.\n10. Handle the static codec chunk explicitly in any `map`/`flat_map` — it has no\n timeline and no samples, and blind indexing into a time column will fail on\n it.\n11. **Samples before the first keyframe** are rejected in stream mode (a decoder\n cannot start mid-GOP); use asset mode for such a file.\n\n## References\n\n- Canonical worked examples: `rerun_py/tests/integration/test_mp4_reader.py`\n (both modes, `chunk_by_gop`, entity paths, `timeline_type`, transcode\n transforms, and the error cases).\n- Rust core: `crates/data_flow/re_mp4_reader/` (`stream.rs` for the GOP/transcode\n path, `asset.rs` for the blob+index path), with\n `crates/data_flow/re_mp4_reader/tests/stream.rs` covering codec pairs and GOP\n spacing.\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-data-model` (where\n video, calibration, and thumbnails belong in the recording).\n","skills/rerun-data-model/SKILL.md":"---\nname: rerun-data-model\ndescription: \"How raw multimodal robot data maps onto the Rerun data model. Read FIRST, before modeling or converting a dataset — and whenever you are about to convert/ingest/preprocess robot data into an .rrd or build a Rerun recording, even if not asked for the data model. Resolves the entity-vs-component, property-vs-component-vs-layer, and static-vs-temporal decisions and routes to the mechanism (do it with readers and lenses, not hand-built chunks or per-message rr.log): rerun-chunk-processing and the importer skills rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun data model\n\nThe hard part of ingesting a dataset is the **modeling decision**, not the API call.\nGet the model right and any mechanism works; get it wrong and queries, views, and training all break.\nThis skill is just the decisions.\nFor mechanism details see `rerun-chunk-processing` (pipeline mechanics) and the importer skills it routes to: `rerun-mcap`,`rerun-urdf`,`rerun-parquet`, `rerun-mp4`, `rerun-lerobot`.\nFor exact signatures, the docs at `rerun.io/docs/concepts/logging-and-ingestion`.\n\n**Before writing conversion code, fill in the mapping table below.** It is the design, and a human can review it in seconds.\n\n## Pick the mechanism before you model the bytes\n\nModeling decides _what_ each datum becomes; this decides _how_ it gets there — and the default is **a reader + lenses, not hand-built chunks**:\n\n- **Does a reader exist for this source?** MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`, LeRobot dir→`log_file_from_path`. **Yes →** `reader.stream()` + lenses; the reader produces the chunks, you do not.\n- **No, and it is genuine external metadata** (JSON calibration, offsets) or a specific custom use case that can't be covered by a generic reader**→** `Chunk.from_columns`.\n- **Otherwise**: consider if you are about to hand-build something a reader or lens should produce and ask for clarification.\n\nFor MCAP specifically: a Foxglove- or ROS-decodable file emits archetypes like `Transform3D`, `Pinhole`, and `VideoStream` for certain supported message schemas ready-made — pass those through, never re-derive them; only custom protobuf signal topics need lenses. The full decision tree and the anti-pattern list are in `rerun-chunk-processing`.\n\n## The model\n\n```\nDataset → Segment → Layer → Recording → Entity → Component → Chunk\n```\n\n- **Entity** = a thing, named by a path (`/robot/arm/camera`).\n **Component** = one typed field on it (positions, image, a scalar).\n **Archetype** = a standard bundle of components that log and render together (`Points3D`, `Image`, `Transform3D`).\n- Every value you log sits on two axes: **where** (entity path + component) and **when** (which timeline, or _static_ = all time).\n- **Segment** = one episode; on registration its `recording_id` becomes the `segment_id`. **Layer** = an extra `.rrd` on top of a segment. It attaches by matching the segment's `recording_id` and nothing else: that shared id is the whole layering mechanism.\n\n## The decisions\n\n**Entity vs component on an existing entity**\nNew entity if it has its own spatial frame (`Transform3D`/`Pinhole`), its own annotation context, or should be shown/cleared/shared independently (each robot link, each camera, each sensor). Same entity + extra component for auxiliary data at the same instances (per-point confidence).\nUse `AnyValues` for non-standard fields.\n\n**Property vs component vs layer** (the most common confusion)\n\n- **Component**: per-timestamp signal on the timeline (joint angle, image).\n- **Segment property**: one-per-episode metadata for catalog filter/search (operator, robot, site, task, date). Not per timestamp.\n- **Layer**: a whole derived `.rrd` over a base segment (FK transforms, point clouds, gripper state, labels, quality scores). Queryable as if part of base.\n\n**Static vs temporal**\nStatic belongs to all timelines and shadows any temporal value of the same component on the same entity. Use it for invariants (calibration, coordinate frames, robot meshes, annotation context, a video asset). Never make per-frame data static.\n\n**Which timeline**\nA `timestamp` timeline (ns since epoch) for cross-sensor clock alignment, a `sequence` timeline for frame/ordinal alignment; stamp on both when useful.\n**Do not resample to a common rate.** Latest-at reconciles multi-rate streams at query time by holding each component's last sample (no interpolation).\n\n**Base vs layer**\nBase = faithful conversion of the raw streams, nothing computed.\nLayer = anything derived (FK from URDF + joint states, clouds from depth + intrinsics).\nKeep them separate `.rrd`s.\n\n## The mapping table (produce before coding)\n\n| Source (topic/column/key) | Entity path | Archetype | Component(s) | Timeline | Static/temporal | Base/layer | Property? |\n| ------------------------- | -------------------- | ------------------------------------- | -------------------- | ------------- | --------------------------- | ---------- | --------- |\n| mcap `/joint_states` | `/robot/` | `Scalars` | `scalars` | `sensor_time` | temporal | base | no |\n| `cam0/color.mp4` | `/camera/cam0/video` | `VideoStream` | codec+sample | `video_time` | codec static, samples temporal | base | no |\n| `calibration.json` | `/camera/cam0` | `Pinhole` (+`Transform3D` extrinsics) | `image_from_camera` | — | static | base | no |\n| URDF + joints (computed) | `/robot/` on the CLI) at the dataset _directory_ and it ingests episodes, camera videos, and state/action tables with no conversion code.\n- **Converting?** `rr.experimental.LeRobotReader` reads the dataset one episode at a time as a lazy chunk stream, which is what you want for per-episode RRDs and for lenses.\n\nThe download step needs `huggingface_hub`.\n\n## Step 0: pick a dataset and check the HF token\n\nBoth scripts below live in `scripts/` next to this file, and run standalone through `uv` — they declare `huggingface_hub` inline, so there is no environment to set up.\n\nTwo facts decide what a dataset costs to fetch, and the Hub search results show neither.\nGet both, for every candidate, in one call:\n\n```bash\nuv run scripts/find_lerobot_dataset.py --robot-type --want 5\n```\n\n**`codebase_version` decides whether a partial download is possible at all.**\n\n- **v2.1 and earlier** store one parquet and one mp4 per episode (`video_path` ends in `episode_{episode_index:06d}.mp4`), so \"just the first N episodes\" downloads only those episodes.\n- **v3.0** concatenates every episode into a few large files (`video_path` ends in `file-{file_index:03d}.mp4`), so any subset costs the whole download.\n If the user asked for a few episodes, prefer a v2.1 dataset — for the _download_ only.\n `LeRobotReader` reads a single episode out of either version once the files are local.\n\n**How camera frames are stored decides the size**, and it varies by two orders of magnitude.\nThe `frames` and `MB/ep` columns report it:\n\n- `mp4`: a `video`-dtype feature, one file per episode beside the parquet.\n- `inline`: an `image`-dtype feature, raw frames inside the episode parquet, which is far larger for the same footage.\n\nBoth scripts report the actual size for the dataset in hand, so use that number rather than guessing.\nTell the user the size before you start the download, and prefer `mp4` unless they asked for something only the other set has.\n\n`HF_TOKEN` is not required for public datasets, but without it the Hub rate-limits and throttles downloads.\nIf it is unset, say so and point the user at to create a read token, then `export HF_TOKEN=hf_…` (or `hf auth login`).\nDo not block on it — carry on unauthenticated and let the user decide.\n\n```bash\n[ -n \"$HF_TOKEN\" ] || echo \"HF_TOKEN unset — downloads will be throttled\"\n```\n\n## Step 1: dataset -> one combined RRD\n\nOnly needed if you want every episode in one file — to hand someone a single RRD, or to reprocess data you cannot re-read from the dataset directory.\nFor per-episode segments, skip to Step 2 and let `LeRobotReader` read the dataset directly.\n\n```python\nfrom huggingface_hub import snapshot_download\nimport rerun as rr\n\ndataset_dir = snapshot_download(repo_id=repo_id, repo_type=\"dataset\", local_dir=dest)\n\nwith rr.RecordingStream(\"rerun_example_lerobot\") as rec:\n rec.save(str(combined_rrd))\n rec.log_file_from_path(str(dataset_dir)) # the built-in importer\n```\n\nThe importer emits one recording per episode (recording ids like `episode_1`), plus a metadata-only root recording, all into the single RRD.\n\n`rr.RecordingStream` + `log_file_from_path` here is the **importer bootstrap** — the one place `RecordingStream` is correct in an ingestion pipeline (it drives the built-in importer, not per-message logging). Do not generalize it to `rr.log`-per-message loops; for everything after import, reprocess with a reader + lenses (see `rerun-chunk-processing`: Chunk API vs logging API).\n\n### Downloading only the first few episodes\n\n```bash\nuv run scripts/fetch_lerobot_episodes.py --episodes 5 # prints the dataset directory\n```\n\nIt reports the download size before starting, downloads `meta/*` plus the wanted `episode_%06d` files, trims `meta/` to match, verifies that exactly the requested episodes landed, and prints what it got.\nIts last line is the dataset directory; do not follow it with `find` or `du`.\nDo the equivalent by hand and three things bite:\n\n- The importer iterates the episode list in `meta/`, so an untrimmed `meta/` warns once per absent episode.\n- `snapshot_download` writes into an existing directory, so episodes left by an earlier run with a different N survive and inflate the result.\n- Both failures are invisible until you count the recordings in the viewer, which is why the script asserts instead of leaving you to look.\n\n### Just looking at it in the viewer?\n\nSkip the RRD entirely — the viewer imports the dataset directory itself.\nPass the plain path (`rerun /path/to/dataset`, or `open_url` over MCP).\nEach episode arrives as its own recording, with `recording_id` `episode_0`, `episode_1`, ….\n\n## Step 2: one RRD per episode\n\nCatalog segments are one-recording-per-file, and `recording_id` becomes the segment id on registration.\n\n`LeRobotReader` goes straight there, without building the combined RRD of Step 1 first:\n\n```python\nreader = rr.experimental.LeRobotReader(dataset_dir) # v2 or v3\nfor episode in reader.episodes():\n episode_id = f\"episode_{episode:05d}\" # zero-padded; see below\n reader.stream(episode).write_rrd(\n rrd_dir / f\"{episode_id}.rrd\",\n application_id=\"my_dataset\",\n recording_id=episode_id,\n )\n```\n\nStreaming is lazy end to end, so memory is bounded by chunk size rather than by episode or dataset size, and video is cut to the episode's time window.\nB-frame-free video (AV1, LeRobot's default codec) streams directly; a stream that must be re-encoded — H.264 with B-frames, or a window starting mid-GOP — needs `ffmpeg` on the `PATH`.\n`stream()` also takes `entity_path_prefix`, `timeline`, and `video_mode=\"skip\"` to drop video entirely.\n\nTo add lenses, drop topics, or fix data, put them between the stream and the write — see `rerun-chunk-processing`.\n\n### Splitting an RRD you already have\n\nIf the combined RRD of Step 1 already exists — or the data came from somewhere other than the dataset directory — split it with `RrdReader` instead:\n\n```python\nreader = rr.chunk.RrdReader(str(combined_rrd))\nfor entry in reader.recordings():\n store = reader.store(store=entry)\n if not store.schema().entity_paths(): # skip the metadata-only root recording\n continue\n episode_id = zero_pad(entry.recording_id) # episode_1 -> episode_00001\n with rr.RecordingStream(\"rerun_example_lerobot\", recording_id=episode_id, send_properties=False) as rec:\n rec.save(str(rrd_dir / f\"{episode_id}.rrd\"))\n rec.send_chunks(store)\n```\n\nTwo non-obvious moves:\n\n- **Zero-pad the episode id.** `episode_10` sorts before `episode_2`\n lexicographically; segment tables and viewers sort lexicographically. Pad to\n a fixed width when re-assigning `recording_id`.\n- **`send_properties=False`** on the new stream, so the copy doesn't inject\n fresh recording properties on top of the copied chunks.\n\n`send_chunks` does not preserve the source store's identity; the new stream's\n`recording_id` wins, which is exactly what makes the rename work.\n\nIf episodes need cleanup (drop topics, fix data, add derived components), run the store through lenses between read and write: `reader.stream(store=entry).drop(...).lenses(...)` then `collect().write_rrd(..., recording_id=episode_id)` (see `rerun-chunk-processing`).\n\nComputed layers and per-episode properties then follow the standard patterns in `rerun-data-model` (layer `recording_id` must equal the episode segment id).\n\n## Step 3: add the robot model (URDF)\n\nA LeRobot dataset ships joint values, never a robot.\nWithout a URDF you get camera views and joint plots but no 3D arm, which is usually not what the user pictured when they asked to look at the data.\nAdding one is a forward-kinematics layer — `rerun-urdf` owns that; this section is only the LeRobot-specific wiring.\n\n- **The URDF.** `meta/info.json` `robot_type` names it (`so100`, `so101`, `aloha`, …); `rerun-urdf` has the source table.\n- **The joint names.** `meta/info.json` `features.observation.state.names` lists them in column order.\n Map each source name explicitly to the corresponding URDF joint name; do not assume that removing a prefix produces a valid URDF name.\n- **The values.** `observation.state` is the measured pose and the one to run FK on; `action` is the commanded pose and belongs on the plots, not the robot.\n- **The units and calibration.** Confirm the units, sign, and offset against the dataset and robot documentation.\n Run `rerun-urdf`'s `check_joint_mapping.py` against the episode parquets before trusting a pose.\n\nThe FK source is the per-episode parquet, read directly rather than out of the imported RRD:\n\n```python\nParquetReader(str(episode_parquet)).stream(\n column_grouping=\"individual\",\n index_columns=[IndexColumn.sequence(\"frame_index\")],\n).drop(content=\"/__properties/**\")\n```\n\n`frame_index` is the index the importer uses too, so the FK transforms line up with the video and plots.\nMerge the URDF model stream plus the FK stream into each episode's RRD, keeping the episode's `recording_id` (see `rerun-urdf`'s \"Minimal shape\").\nFor a whole dataset, the model belongs in a shared asset rather than in every segment — also `rerun-urdf`.\n\n## Gotchas\n\n1. `log_file_from_path` must target the dataset **root directory**, not a file inside it. So does `LeRobotReader`.\n2. Unpadded episode ids sort incorrectly downstream; pad before registering.\n3. The combined RRD contains a metadata-only root recording; skip stores with no entity paths or you register an empty segment.\n4. A partially-downloaded dataset imports fine, but warns once per episode listed in `meta/` and missing on disk.\n5. Downloading a subset into a directory an earlier run already populated leaves the older episodes in place; start from an empty directory.\n6. `image`-dtype (inline) datasets can be ~100x larger per episode than `mp4` ones; check before downloading, not after.\n7. Check signs and units - e.g. an `observation.state` in degrees could produce wrong FK when fed into a robot model that expects radians.\n\n## References\n\n- `rerun-chunk-processing` (`LazyChunkStream`, lenses, `RrdReader`)\n- `rerun-urdf` (adding the robot model and the joint-value mapping)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/dataloader` `prepare_dataset.py` (download → import → split → register, complete and runnable) and `train.py` (training-side consumption via `rerun.experimental.dataloader`)\n","skills/rerun-mcap/SKILL.md":"---\nname: rerun-mcap\ndescription: Ingest MCAP files into Rerun chunk streams with rerun.chunk.McapReader. Read when converting an MCAP recording, selecting topics or decoders, decoding custom protobuf messages, or when an MCAP-derived stream comes out empty. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (what the topics should become).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun MCAP ingestion\n\n`McapReader` turns an MCAP file into a lazy chunk stream: one entity per topic\nat the topic's path, message payloads decoded by pluggable decoders. This\nskill covers the reader's options, what each topic becomes, and the failure\nmodes that yield an empty stream with no error. Stream mechanics (filter, drop,\nlenses, merge, write) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.chunk import McapReader\n\nreader = McapReader(mcap_path) # see help(McapReader) for the full option set\nstream = reader.stream()\n```\n\nA URDF embedded in the MCAP can be ingested as well (then see `rerun-urdf`).\n\n## What a topic becomes\n\nWith the default decoders (`decoders=None`), the message **schema name** decides what a topic becomes — **pass archetypes through, lens only the raw `:message` topics**:\n\n| MCAP schema name | decodes to | what to do |\n| --- | --- | --- |\n| `foxglove.FrameTransforms` | `Transform3D` | pass through; do **not** hand-build |\n| `foxglove.CameraCalibration` | `Pinhole` | pass through |\n| `foxglove.CompressedVideo` | `VideoStream` (real sample bytes) + `CoordinateFrame` | pass through |\n| other `foxglove.*` well-known types | the matching archetype | pass through |\n| ros2 well-known types (`ros2msg` / `ros2_reflection`) | archetype | pass through |\n| your own `schemas.proto.*` / custom protobuf | one `:message` struct | attach semantics with a `DeriveLens` + `Selector` |\n\nSo a camera topic already arrives as `Pinhole`, its video as `VideoStream`, and a `frame_transforms` topic as `Transform3D` — only custom messages (e.g. a custom joint states schema, a custom gripper status enum) come through reflection or raw only and need lenses.\n\nThe `foxglove` decoder does the schema→archetype mapping; because foxglove messages are protobuf-_encoded_ it rides on the `protobuf` decoder, so keep `decoders=None` (verified: `decoders=[\"protobuf\"]` alone leaves `foxglove.CameraCalibration` a raw `:message`; adding `foxglove` makes it a `Pinhole`). Confirm on your file: `McapReader(path).stream()`, then read `McapSchema:name` and a few `Chunk.format()` before deciding anything is missing or needs rebuilding.\n\n- Entity path = topic name (`/sensors/joint_states` stays `/sensors/joint_states`).\n Filter early: `McapReader(path).stream().filter(content=\"/sensors/**\")`.\n- A reflection-decoded message lands as one struct component named `:message`.\n Navigate it with `Selector` (`Selector(\".joint_positions\")`) inside lenses; this is how custom messages\n get Rerun semantics attached (see the DeriveLens patterns in `rerun-chunk-processing`).\n- Topic regexes use RE2 syntax and are **not anchored**: `cam` matches\n `/external/cam_low` and `/camera_info`.\n Anchor explicitly (`^/external/cam`) when it matters. Prefer reader-level topic filtering over `.filter(...)`\n when you can, so excluded topics are never decoded at all.\n\n## When to use the low-level `mcap` package instead\n\n`McapReader` keeps payloads in columnar chunk streams; that is almost always what you want.\nDrop to `mcap.reader.make_reader` only when you need raw record metadata without payloads, or when you need to rewrite the container itself (re-registering schemas, channels, and messages).\n\n## Gotchas\n\n1. Empty stream, no error: a topic regex that matched nothing, or a channel\n whose decoder produced no rows. Check `Chunk.format()` on a few chunks of\n `reader.stream().to_chunks()` against a tiny test file, or compare topic\n names with the `mcap` CLI / package first.\n2. Topic regexes are unanchored RE2; excludes run after includes.\n3. `timeline_type=\"timestamp\"` interprets MCAP log times as wall-clock ns\n since epoch. If the recording's clock is wrong, fix it at the reader with\n `timestamp_offset_ns` rather than mutating timestamps downstream.\n4. Decoder subsets silently skip topics no decoder claims; when a topic is\n missing, retry with `decoders=None` to rule out decoder selection.\n5. Example fix-lenses are dataset-specific. Before copying a `MutateLens` like\n the `Pinhole:resolution` swap from the `robot_data_preprocessing` example,\n read the raw component from `McapReader(path).stream()` and confirm the defect\n exists in _your_ data — applied blindly it corrupts correct calibration (a\n correct 648×480 flipped to 480×648).\n6. `foxglove` derives both the camera's `Pinhole:child_frame` and the video's\n `CoordinateFrame:frame` from each message's `.frame_id` (plus an image-plane\n suffix), so they **match** when the calibration and video topics share a\n `frame_id`. Only when those topics carry _different_ `frame_id`s does the\n video frame diverge and orphan the video from its image plane — re-home it\n then with a per-camera `MutateLens` on `CoordinateFrame:frame`.\n\n## References\n\n- End-to-end MCAP pipeline: `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-urdf` (FK from joint-state topics), `rerun-data-model` (modeling decisions)\n","skills/rerun-catalog-queries/SKILL.md":"---\nname: rerun-catalog-queries\ndescription: Performance patterns and gotchas for querying a Rerun catalog from Python. Reach for this when exploring an unfamiliar dataset, when a CatalogClient/dataset query is unexpectedly slow, or when shaping a per-segment / per-episode pipeline that hits the catalog from many places.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash\n---\n\n# Rerun catalog queries\n\nPractical performance patterns for querying a Rerun catalog\nfrom Python (`rerun.catalog.CatalogClient` →\n`dataset.reader(...)` → DataFusion `DataFrame`). The DataFusion side of\nthe stack is covered by the **`datafusion-python`** skill — load that\nfor `DataFrame` / `SessionContext` / expression-API references. This\nskill focuses on catalog-specific behaviors and the round-trip costs\nthat catch teams off guard.\n\n---\n\n## Connect with a `rerun://` URL\n\n```python\nfrom rerun.catalog import CatalogClient\n\nclient = CatalogClient(\"rerun+https://api.example.cloud.rerun.io:443\")\ndataset = client.get_dataset(id=dataset_id)\n```\n\n`CatalogClient` takes a `rerun://` or `rerun+https://` URL, or\n`rerun+http://` for a local server. Do not pass a plain `https://` URL.\nA segment URL such as\n`rerun://host:443/dataset/?segment_id=` gives\nthe server, the dataset id and the segment id.\n\nThe client reads the token from `REDAP_TOKEN`, or from the stored\ncredentials of `rerun auth login`, so `token=` is usually not needed.\n\n---\n\n## The query cost model in one sentence\n\nEvery materialization of a `dataset.reader(...)` DataFrame is **one\ncloud round-trip**, with most of the cost being the network/decode\npair, not the compute. Plan for round-trip count and payload bytes, in\nthat order.\n\nA typical catalog round-trip is **a few seconds** even for a tiny\nresult. So:\n- 30 segments × 1 query each ≈ 90s. (Naive per-segment loop.)\n- 30 segments × 4 queries each ≈ 6 minutes. (A splitter that runs\n `count` + `collect_column` for both starts and stops.)\n- 1 query covering all 30 segments ≈ 3s.\n\nThe same fan-out happens along the **entity** axis: 10 entities ×\n1 `filter_contents([one_entity]).reader()` each ≈ 30s, vs one\n`filter_contents([all_entities]).reader()` ≈ 3s. Push as much as you\ncan into one round-trip — across segments and across entities.\n\n---\n\n## Always apply `filter_contents` and time-window filters before `.reader(...)`\n\nThe single biggest lever:\n\n- `filter_contents([entity_globs])` restricts which entity-path columns\n the reader produces. Without it, every entity in the dataset is\n read.\n- Time-window filters push down to storage and dramatically reduce\n bytes scanned when they compare the bare index column with a literal:\n `df.filter(col(index) >= lit(start)).filter(col(index) <= lit(end))`.\n Integer, timestamp and duration literals of any unit work. A `cast`\n or arithmetic on the index column is not pushed down.\n\nCombined: `dataset.filter_segments(seg).filter_contents(entities).reader(...)\n.filter(in_window).select(...)`.\n\n---\n\n## `df.cache()` is your friend for repeated probes\n\nWhen the same materialization gets used by multiple downstream\nfilter/count/collect calls, materialize once with `DataFrame.cache()`\nand operate on the cached frame:\n\n```python\ncached = (\n dataset\n .filter_segments(seg)\n .filter_contents([entity])\n .reader(index=index_col)\n .select(col(index_col).cast(pa.int64()).alias(index_col), value.alias(\"v\"))\n .cache() # one network round-trip, materializes into in-memory batches\n)\nstarts = cached.filter(col(\"v\") == start_val).collect_column(index_col)\nstops = cached.filter(stop_pred(col(\"v\"))).collect_column(index_col)\n```\n\nWithout `cache()`, each `count()` / `collect_column()` re-executes the\nwhole reader chain.\n\n**When NOT to cache.** `cache()` forces materialization into Arrow\nbatches, breaking laziness. If downstream code keeps composing more\nDataFusion ops on top (joins, windows, further filters) and only\nmaterializes once at the end, caching mid-pipeline turns one execution\ninto two and pre-empts whatever physical-plan optimizations the engine\ncould have done across the boundary. Reach for `cache()` when the\nconsumers are terminal (`count()`, `collect_column()`, `to_arrow_table()`),\nnot when they're another lazy `DataFrame`.\n\n---\n\n## Cross-segment batching: drop `filter_segments`, group by `rerun_segment_id`\n\nFor pipelines that need the same query on many segments, omit\n`filter_segments(...)` entirely and pull a single cross-segment table.\nEvery reader row carries a `rerun_segment_id` column — group locally:\n\n```python\ndf = dataset.filter_contents(entities).reader(index=index_col)\ncached = df.select(\n \"rerun_segment_id\",\n col(index_col).cast(pa.int64()).alias(index_col),\n value.alias(\"v\"),\n).cache()\n\n# Now N filter/aggregate calls are local, not network.\nstarts = cached.filter(col(\"v\") == start_val).select(\"rerun_segment_id\", index_col).to_arrow_table()\n```\n\nTrigger / event columns are tiny enough that pulling all segments at\nonce dominates per-segment looping by an order of magnitude.\n\n---\n\n## Per-entity fan-out within a segment\n\nSymmetric to cross-segment batching, along the entity axis. If you\nneed data from N entities of a single segment, **don't** loop:\n\n```python\n# Anti-pattern: N reader setups, N round-trips.\nfor entity in entities:\n df = dataset.filter_segments(seg).filter_contents([entity]).reader(index=ix)\n ...\n```\n\nInstead pull all N at once and project per entity locally. Per \"Reader\nrow layout\" below, every row carries data for one entity and NULLs\nfor the others, so `col(\"::\").is_not_null()`\nis the per-entity filter:\n\n```python\nshared = (\n dataset\n .filter_segments(seg)\n .filter_contents(sorted(set(entities)))\n .reader(index=ix)\n .filter(col(ix).cast(pa.int64()).between(start_ns, end_ns))\n)\n# Each downstream consumer narrows to its entity's rows lazily.\nsrc_a = shared.filter(col(f\"{ent_a}:{comp_a}\").is_not_null()).select(ix, f\"{ent_a}:{comp_a}\")\nsrc_b = shared.filter(col(f\"{ent_b}:{comp_b}\").is_not_null()).select(ix, f\"{ent_b}:{comp_b}\")\n```\n\nDataFusion can share the underlying scan across the per-entity\nprojections when it builds the physical plan, so this stays a single\ncatalog round-trip even though there are N logical consumers. Works\ninside generators that build per-source DataFusion plans (resampling,\nbracket lookup, nearest-in-time joins) — collapse the network fan-out\nwithout changing the per-source logic.\n\nA trap when refactoring: if a downstream query uses a reader column's\nfully-qualified name (`col(f\"{entity}:{archetype}:{component}\")`),\nyou don't need to alias the column in `shared`. The shared reader's\noutput schema preserves native column names, so existing per-entity\nprojection helpers keep working unchanged.\n\n---\n\n## `count()` is *not* free\n\nCounter-intuitive: `df.count()` and `df.aggregate([], [F.count(col)])`\ndo not always push down. Aggregate plans can force the engine to\nmaterialize the underlying column data server-side, then count on the\nclient. `F.count(col)` over wide entity-columns can ship full struct or\nblob payloads to count nullity.\n\nAlternatives, in order of preference for \"is anything here\":\n\n| Need | Use |\n|---|---|\n| \"any row in this filter?\" | `df.select(col(index)).limit(1).to_arrow_table().num_rows > 0` — server short-circuits on first match |\n| \"count rows in a tiny window\" | `df.filter(window).select(col(index)).count()` after the time filter |\n| \"count *each* entity in a wide query\" | per-entity `limit(1)` probes, threaded — *not* one big `count(col)` aggregate |\n\nA trap that fooled us: assuming `bool_or(col.is_not_null())` would only\nneed nullity buffers. It does not — the operator still touches payload\ndata on most plans.\n\n---\n\n## `using_index_values` + `fill_latest_at` is great for resampling, not for presence\n\n```python\n.reader(index=index_col, using_index_values=targets, fill_latest_at=True)\n```\n\n- Returns one row per target timestamp, each entity column carrying its\n latest non-null value at-or-before the target.\n- Excellent for nearest-prior resampling (no DataFusion required).\n- **Don't** use it as a presence check. Two reasons:\n 1. Semantics are \"ever emitted before T\", not \"emitted in [start, T]\".\n 2. The server still ships the full struct/blob payload for every\n entity to compute the latest-known value; a downstream\n `is_not_null()` projection runs post-transfer and doesn't reduce\n wire bytes.\n\nFor a strict in-window presence check, prefer per-entity `limit(1)`\nover the time-filtered reader, run concurrently.\n\n---\n\n## Explore an unfamiliar dataset with `dataset.info()`\n\n```python\ninfo = dataset.info()\nprint(info) # compact text overview\n```\n\nIt reports the segment count and size, each layer with how many\nsegments have it, each timeline with its kind, value range and segment\ncoverage, and the property column names. It reads only the segment\ntable and the schema, no recording data, so it stays cheap on large\ndatasets.\n\nUse it to choose `reader(index=…)` and to bound time-window filters\nbefore the first real query. `TimelineInfo.min` / `.max` are numpy\nvalues that work directly as literals:\n\n```python\nts = next(t for t in info.timelines if t.name == \"capture_time\")\ndf = dataset.reader(index=ts.name).filter(col(ts.name) >= lit(ts.min))\n```\n\nProperties found here can be filtered via `dataset.segment_table()`,\nand layer names tell you what `register(…, layer_name=…)` already\nattached.\n\n---\n\n## Schema introspection is cheap; use it before probing\n\n```python\nschema = dataset.filter_segments(seg).schema()\navailable = {(c.entity_path, c.component) for c in schema.component_columns()}\n```\n\nThis is a single round-trip and tells you which `(entity, component)`\npairs the segment registered. If a column isn't in the schema, you can\ndrop it from your manifest without any further cloud queries. This is\noften a complete substitute for \"does this entity have events\"\nprobing.\n\nCaveat: schema presence ≠ events. The MCAP records the topic schema\neven for unused topics. If your pipeline cares about distinguishing\n\"registered but never emitted\" vs \"registered with events\", you have\nto probe — see \"is anything here\" patterns above.\n\n---\n\n## Reader row layout: entities are columns, not rows\n\nEvery row from `dataset..reader(...)` corresponds to a single\nevent on a single entity. Other entities' columns are null on that\nrow. Implications:\n\n- `select(\"rerun_segment_id\", \"::\")`\n works — quote the entity column when using SQL.\n- There is **no** `rerun_entity_path` row attribute. To attribute rows\n to entities you either filter to one entity at a time, or pick a\n per-entity column (e.g. `:McapChannel:id`) whose non-null pattern\n identifies the source.\n- `df.count()` returns *total events across all entities*, not\n per-entity counts.\n\n---\n\n## Field access on a null struct returns `0.0`, not null\n\nA DataFusion gotcha that bites pipelines reading struct messages:\n\n```python\ncol(\"/some/entity:msg.MyType:message\")[0][\"sub\"][\"x\"]\n# When the parent struct is null on a row, this evaluates to 0.0\n# (and \"\" for strings), not null.\n```\n\nWrap struct-walk projections with a null guard:\n\n```python\nparent = col(\"/some/entity:msg.MyType:message\")\nleaf = parent[0][\"sub\"][\"x\"]\nguard = parent.is_null() | parent[0].is_null() | parent[0][\"sub\"].is_null()\nexpr = F.when(guard, lit(None)).otherwise(leaf)\n```\n\n**Only apply this to struct sources.** For scalar / blob columns\n(`Scalars:scalars`, `EncodedImage:blob`, etc.) the wrap is a no-op at\nbest and at worst rewrites the plan in ways that change downstream\njoin behavior. Gate the guard on whether the source actually walks\nthrough a struct boundary.\n\n---\n\n## Common debug recipe\n\nWhen a query stage is slower than expected:\n\n1. **Count round-trips.** Wrap each `to_arrow_table()` /\n `collect_column()` / `count()` with `time.perf_counter()`. Each is\n a round-trip. If you see N segment queries, that's N × few-seconds\n minimum.\n2. **Split build vs materialize timing.** Time the lazy DataFrame\n construction *separately* from the terminal `to_arrow_table()` call.\n If \"build\" takes seconds, something inside is materializing eagerly\n (an Arrow round-trip in a join helper, a `cache()` in a generator,\n a `.collect()` hidden in a chained-join utility). A correctly lazy\n plan should build in ~milliseconds regardless of result size.\n3. **Measure bytes.** `tbl.nbytes` after `to_arrow_table()` reveals\n when \"I projected `is_not_null()`\" actually shipped megabytes. If\n bytes are large despite a small projection, the operator didn't\n push down.\n4. **Cross-segment first, then cross-entity.** If the per-segment query\n is fundamentally the same (just scoped by id), drop `filter_segments`\n and group by `rerun_segment_id` locally. If you also have a\n per-entity loop within a segment, collapse it the same way (one\n `filter_contents([all])` reader, per-entity `is_not_null()` filters\n downstream).\n5. **Cache before terminal re-use.** If two `count()` / `collect_column()`\n calls share the same reader, `df.cache()` between them. Don't cache\n if the consumers are themselves lazy DataFrames being composed\n further — caching breaks plan-wide optimization.\n6. **Window first.** Always push the time filter before any projection\n or aggregate that touches payload columns.\n\n---\n\n## See also\n\n- `datafusion-python` skill — DataFrame API, SQL parity, expression\n building, common pitfalls (boolean operators, immutability, etc.).\n Not installed? Ask the user to install it globally:\n `npx skills add apache/datafusion-python`.\n","ARCHITECTURE.md":"# Rerun architecture\n\nThis document describes the technical architecture of Rerun.\n\n## See also\n\n- [`BUILD.md`](BUILD.md)\n- [`TESTING.md`](TESTING.md)\n- [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK\n- [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)\n- [`CODE_STYLE.md`](CODE_STYLE.md)\n- [`CONTRIBUTING.md`](CONTRIBUTING.md)\n- [`RELEASES.md`](RELEASES.md)\n\n## The major components\n\n### Logging APIs\n\nIt all starts with logging. You can log rich data (point clouds, images, etc.) with either our Python SDK or our Rust SDK.\n\nThe logging SDK:s encodes the data using Apache Arrow (see more below).\n\nThe logging data can be written to disk as `.rrd` files, or transmitted over gRPC to either a Rerun Viewer or a Rerun Server.\n\n### Rerun Viewer\n\nThe Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser.\n\n#### Native Viewer\n\nThe easiest way to launch the Viewer is directly from the logging API with `rr.init(\"rerun_example_app\", spawn=True)`. However, the standalone Viewer can also be run from the command line, for example to view an `.rrd` file: `rerun mydata.rrd`.\n\n#### Web viewer\n\nYou can try running the Viewer in a browser using `rr.serve()` in Python, or using `rerun --web-viewer mydata.rrd`.\n\nThe web viewer consists of just a few small files - a thin `.html`, a `.wasm` blob, and an auto-generated `.js` bridge for the wasm. These files are served using the [`re_web_viewer_server`](https://github.com/rerun-io/rerun/tree/latest/crates/top/re_web_viewer_server) crate.\n\nThe web viewer can load `.rrd` files (just drag-drop them into the browser), or read logging data streamed over gRPC.\n\n### `.rrd` files\n\n`.rrd` (\"**R**e**r**un **D**ata\") is just a bunch of log messages appended one after the other to a file.\n\nNOTE: `.rrd` files do not yet offer full backwards or forwards compatibility. However, the current version of Rerun will always be able to open `.rrd` files generated by the *previous* version. This is not guaranteed to work for files generated with older versions.\n\n## Technologies we use\n\n### Apache Arrow\n\n[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_chunk_store`](crates/store/re_chunk_store/README.md).\n\nIn Rust, we use the [`arrow` crate](https://crates.io/crates/arrow).\n\n### `wgpu`\n\nThe Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web.\n\nOn web builds, we use WebGPU when available on the Web, but automatically fall back to a WebGL based emulation layer (with a more limited feature set).\n\nWe have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/viewer_support/re_renderer/README.md).\n\n### `egui`\n\nThe GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode).\n\nWe use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web.\n\n### Wasm\n\nWasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.\nThe Rerun Viewer can be compiled to Wasm and run in a browser.\n\nThreading support in Wasm is nascent, so care must be taken that we don't spawn any threads when compiling for `wasm32`.\n\nWasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:\n\n- Access files\n- Read environment variables\n- Get the current time (use [`instant`](https://crates.io/crates/instant) instead)\n- Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)\n- etc\n\n## Immediate mode\n\nThe Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch.\n\nIn fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.\n\nThe advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.\n\nImmediate mode is also a forcing function, forcing us to relentlessly optimize our code.\nThis leads to a very responsive GUI, where there is no \"hiccups\" when switching data source or doing time scrubbing.\n\nOf course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.\n\n## Crates\n\nHere is an overview of the crates included in the project, and how they depend on each other:\n\n\n\nEach band is one folder under `crates/`, with `rerun_py` and `rerun_c` folded into the top band.\nEvery arrow points downwards, because the folders are layers: a crate may only depend on its own folder or one below it.\n[`scripts/check_crate_layers.py`](scripts/check_crate_layers.py) enforces that in CI.\nA `dev-dependency` is exempt, since a test may reach anywhere.\n\nSee [`crates/README.md`](crates/README.md) for the layers as a list, and for which one a new crate belongs in.\nThe tables below give a one-line description of every crate, one table per folder.\n\nOnly the arrows *inside* a band are drawn.\n\nThe diagram and the crate tables below are both generated from `cargo metadata`, so neither has to be written by hand.\nA crate's description is its `description` field in its own `Cargo.toml`, so there is only one place to write it.\nRegenerate both after adding, removing, or renaming a crate:\n\n```sh\npixi run crate-graph\n```\n\n**Run that command yourself whenever you add or remove a crate.**\nNothing checks it for you: `pixi run crate-graph-check` is disabled in CI, because graphviz lays the same graph out slightly differently on macOS and on Linux, so whoever regenerated the diagram last decided whether the check passed.\n\nPass `--edges all` to `scripts/generate_crate_graph.py` for a version with every dependency drawn, cross-band ones included.\n\n\n\n### Test support\n\n[`crates/tests`](./crates/tests) — only tests depend on these, so they sit at the top of the diagram.\n\n| Crate | Description |\n| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |\n| re_redap_tests | Official test suite for the Rerun Data Protocol |\n| re_renderer_tests | Snapshot tests for `re_renderer`'s rendering primitives. |\n| re_test_context | A common context used for tests. |\n| re_test_mocks | In-process server doubles (`MockOtlpCollector`, `MockPostHog`) used by tests that need to capture outbound OTel/PostHog traffic. |\n| re_test_viewport | A viewport used for tests |\n| re_web_tests | Discovers and runs Rerun web tests |\n\n### SDK / CLI / Wasm\n\n[`crates/top`](./crates/top) — the entry points for our users, and the viewer app itself.\n\n| Crate | Description |\n| -------------------- | ---------------------------------------------------------------------------------- |\n| re_sdk | Rerun logging SDK |\n| re_viewer | The Rerun viewer |\n| re_viewer_mcp | MCP server that allows llm agents to use the Rerun Viewer. |\n| re_web_viewer_server | Serves the Rerun web viewer (Wasm and HTML) over HTTP |\n| rerun | The Rerun Rust SDK: log images, point clouds, etc, and visualize them effortlessly |\n| rerun-cli | The Rerun command-line tool: the `rerun` binary, including the native Viewer |\n| rerun_c | Rerun C SDK |\n| rerun_py | The Rerun Python SDK |\n\n### Views\n\n[`crates/views`](./crates/views) — the visualizations a user can put in the viewport.\nA sibling of `crates/panels`: neither depends on the other.\n\n| Crate | Description |\n| ---------------------- | --------------------------------------------------------------------------- |\n| re_view_audio | A view that shows the waveform of an audio asset and plays it back. |\n| re_view_bar_chart | A view that shows a single bar chart. |\n| re_view_dataframe | A view that shows the data contained in entities in a table. |\n| re_view_graph | A view that shows a graph (node-link diagram). |\n| re_view_map | A view that shows a map. |\n| re_view_spatial | Views that show entities in a 2D or 3D spatial relationship. |\n| re_view_state_timeline | A view that shows state transitions as horizontal lanes over time. |\n| re_view_tensor | A view dedicated to visualizing tensors with arbitrary dimensionality. |\n| re_view_text_document | A simple view that shows a single text box. |\n| re_view_text_log | A view that shows text entries in a table and scrolls with the active time. |\n| re_view_time_series | A view that shows plots over Rerun timelines. |\n\n### Panels\n\n[`crates/panels`](./crates/panels) — the panels the app is assembled from, and the widgets they are built out of.\nA sibling of `crates/views`: neither depends on the other.\n\n| Crate | Description |\n| ------------------ | ----------------------------------------------------------------------------------------- |\n| re_agent_ui | Chat UI for driving a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_blueprint_tree | The UI for the blueprint tree in the left panel. |\n| re_chunk_store_ui | Display the contents of chunk stores. |\n| re_memory_view | Flamegraph visualization for memory usage trees |\n| re_recording_panel | The UI for the recording panel. |\n| re_redap_browser | The UI and communication to implement the in-viewer redap server browser. |\n| re_selection_panel | The UI for the selection panel. |\n| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |\n| re_viewport | The central viewport panel of the Rerun viewer. |\n\n### Viewer support\n\n[`crates/viewer_support`](./crates/viewer_support) — the UI and rendering machinery the views are built on.\n\n| Crate | Description |\n| ---------------------- | ---------------------------------------------------------------------------------------------------------- |\n| re_arrow_ui | Show arrow data in a tree of rerun list_items and format arrow with syntax highlighting. |\n| re_component_fallbacks | Handles registering various component fallbacks to `FallbackProviderRegistry`. |\n| re_component_ui | Provides ui editors for Rerun component data for registration with the Rerun Viewer component ui registry. |\n| re_context_menu | Support crate for context menu and actions. |\n| re_data_ui | Provides ui elements for Rerun component data for the Rerun Viewer. |\n| re_dataframe_ui | Rich table widget over DataFusion. |\n| re_gamepad | Gamepad input handling for the Rerun viewer. |\n| re_plot | Shared utilities for plot-based views. |\n| re_renderer | A wgpu based renderer for all your visualization needs. |\n| re_renderer_examples | Examples for the re_renderer crate. |\n| re_time_ruler | Time ruler widget shared by the time panel and time-aware views. |\n| re_ui | Rerun GUI theme and helpers, built around egui |\n| re_view | Types & utilities for defining view classes and communicating with the viewport. |\n| re_viewer_context | Rerun viewer state that is shared with the viewer's code components. |\n| re_viewport_blueprint | The data model describing the layout of the viewport. |\n\n### Application-level store\n\n[`crates/store_app`](./crates/store_app) — the queryable state a viewer or a server works with.\n\n| Crate | Description |\n| ------------- | --------------------------------------- |\n| re_dataframe | High-level query APIs |\n| re_datafusion | High-level query APIs |\n| re_entity_db | In-memory storage of Rerun entities |\n| re_query | High-level query APIs |\n| re_server | An ephemeral local Rerun catalog server |\n| re_tf | Rerun spatial transform processing |\n\n### Data flow\n\n[`crates/data_flow`](./crates/data_flow) — getting data in and out: clients, servers, and file importers.\n\n| Crate | Description |\n| ----------------- | --------------------------------------------------------------------------------- |\n| re_catalog_client | High-level client for Rerun catalogs |\n| re_data_source | Handles loading of Rerun data |\n| re_grpc_client | Client for the legacy StoreHub API (`re_grpc_server`). |\n| re_grpc_server | Server for the legacy StoreHub API |\n| re_hdf5 | Core HDF5-to-chunk loading logic for Rerun |\n| re_importer | Handles importing of Rerun data from file using importer plugins |\n| re_lenses | Lenses are an API for extracting, transforming, and restructuring component data. |\n| re_lerobot | Core LeRobot-dataset-to-chunk loading logic for Rerun |\n| re_mcap | Convert MCAP into Rerun-compatible data |\n| re_mp4_reader | Core mp4-to-chunk loading logic for Rerun |\n| re_parquet | Core parquet-to-chunk loading logic for Rerun |\n| re_redap_client | Official gRPC client for the Rerun Data Protocol |\n\n### Data model & chunk store\n\n[`crates/store`](./crates/store) — the data model, and the in-memory store that holds it.\n\n| Crate | Description |\n| ------------------ | --------------------------------------------------------------------------------------------- |\n| re_arrow_json | Rerun data as JSON, and back |\n| re_chunk | A chunk of Rerun data, encoded using Arrow. Used for logging, transport, storage and compute. |\n| re_chunk_index | Indexes of Rerun chunks: RRD manifests and the chunk providers that serve them |\n| re_chunk_optimizer | Analysis and memory-bounded optimization of Rerun chunk layouts |\n| re_chunk_store | A storage engine for Rerun's Chunks |\n| re_lenses_core | Core lens types and composable Arrow array transformations. |\n| re_log_channel | An in-memory channel of Rerun data messages. |\n| re_log_encoding | Encode/decode RRD streams: the physical byte layout of .rrd files |\n| re_log_msg | The messages that carry Rerun data between the SDK, `.rrd` files and the viewer. |\n| re_log_types | The basic building blocks of the Rerun log format: entity paths, timelines, and store ids. |\n| re_protos | Rerun remote gRPC/protobuf API types |\n| re_sdk_types | The standard Rerun data types, component types, and archetypes. |\n| re_sorbet | Rerun arrow metadata definitions |\n| re_types_core | The core traits and types that power Rerun's data model. |\n| re_uri | Parsing and constructing Rerun URIs |\n\n### Build support\n\n[`crates/build`](./crates/build) — crates that run at build time.\n\n| Crate | Description |\n| ------------------------ | --------------------------------------------------------------------------------------- |\n| re_build_tools | build.rs helpers for generating build info |\n| re_dev_tools | Various tools for rerun development |\n| re_protos_builder | Code generation for Rerun's Protobuf and gRPC definitions. |\n| re_type_definitions | Rerun's type definitions: a subset of Rust that generates the Rust, Python and C++ SDKs |\n| re_types_builder | Generates code for Rerun's SDKs from the type definitions. |\n| re_types_builder_macros | The `#[rerun_type]` attribute macro used by Rerun's IDL definitions. |\n| re_types_builder_prelude | The vocabulary that Rerun's IDL definitions are written against. |\n\n### Utilities\n\n[`crates/utils`](./crates/utils) — small crates that depend on nothing outside `crates/utils`.\n\n| Crate | Description |\n| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| re_agent | Talk to a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_analytics | Rerun's analytics SDK |\n| re_arrow_util | Helpers for working with arrow. |\n| re_async | Async runtime abstractions for native and WebAssembly targets. |\n| re_audio | Rerun audio decoding and playback utilities. |\n| re_auth | Authentication helpers for Rerun |\n| re_backoff | Utilities for retrying operations with backoff. |\n| re_build_info | Information about the build. Use together with re_build_tools |\n| re_byte_size | Calculate the heap-allocated size of values at runtime. |\n| re_byte_size_derive | Derive macro for the `SizeBytes` trait from `re_byte_size`. |\n| re_capabilities | Capability tokens for the Rerun code base. |\n| re_case | Case conversions, the way Rerun likes them |\n| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |\n| re_dump_puffin | Convert puffin profiler recordings (.puffin files) into JSON for analysis |\n| re_error | Helpers for handling errors. |\n| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |\n| re_grpc_headers | Rerun gRPC header conventions: well-known header names, the `RerunVersionInterceptor`, and the tower `Layer` machinery that propagates them across requests and responses. |\n| re_int | Small numeric helper traits: saturating integer casts and unsigned absolute value |\n| re_log | Helpers for setting up and doing text logging in the Rerun crates. |\n| re_memory | Run-time memory tracking and profiling. |\n| re_mutex | A wrapper around `parking_lot::Mutex` useful for debugging deadlocks. |\n| re_perf_telemetry | In and out of process performance profiling utilities for Rerun & Redap |\n| re_quota_channel | A channel that applies backpressure based on byte size. |\n| re_ros_msg | Parse and decode ROS messages |\n| re_rvl | Support for compressed depth data using the RVL format. |\n| re_span | An integer range that always has a non-negative length |\n| re_string_interner | Yet another string interning library |\n| re_tracing | Helpers for tracing/spans/flamegraphs and such. |\n| re_tuid | 128-bit Time-based Unique Identifier |\n| re_video | Rerun video processing utilities. |\n| re_web | Utilities for interacting with Web APIs |\n| re_workspace_hack | Unifies third-party dependency features across the workspace to avoid recompiles. Managed by cargo-hakari. |\n\n\n\n### Dependencies and docs\n\nIn order to get a dependency graph for our in-house crates and their docs, we recommend you run:\n\n```\ncargo install cargo-depgraph\ncargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > /tmp/rerun-deps.png\nopen /tmp/rerun-deps.png\n```\n\nand:\n\n```\ncargo doc --no-deps --open\n```\n\nand then browse through the `re_*` crates.\n","AGENTS.md":"# CLAUDE.md\n\nGuidance for LLMs working in this repo.\n\n## Project overview\n\nRerun is a time-aware multimodal data stack and visualizations tool used in robotics, spatial AI, computer vision, and similar domains. It provides SDKs (Python, Rust, C++) for logging rich data (images, point clouds, tensors, etc.) and a Viewer for visualization.\n\n## Build system\n\nWe use `pixi` for task management and dependency installation. Check `pixi.toml` for a full list of tasks.\n\n### Essential commands\n\n**Building:**\n- `pixi run py-build` - Build Python SDK into local .venv (uses uv)\n- `pixi run rerun-build` - Build native viewer (without web viewer)\n- `pixi run rerun-build-web` - Build web viewer (wasm)\n- `pixi run cpp-build-all` - Build all C++ artifacts\n\n**Running:**\n- `pixi run rerun` - Run the viewer\n- `pixi run uvpy script.py` - Run Python scripts with rerun SDK\n- `cargo run -p ` - Run specific Rust example (e.g., `cargo run -p dna`)\n\n**Code generation:**\n- `pixi run codegen` - Generate Rust/Python/C++ code from the `re_type_definitions` crate\n\n**Formatting:**\n- `pixi run rs-fmt` - Format all Rust files. Always run this after making changes.\n- `pixi run py-fmt` - Format Python files\n- `pixi run cpp-fmt` - Format C++ files\n- `pixi run toml-fmt` - Format TOML files\n\n**Testing:**\n- `cargo clippy -p ` - Run rust checks before building\n- `cargo nextest run --all-features --no-fail-fast -p ` - Run tests for specific crate\n - Example: `cargo nextest run --all-features --no-fail-fast -p re_view_spatial`\n- Use `cargo nextest` (not `cargo test`) for better output + parallelism\n- Always use `--all-features` unless specific reason not to\n- Use `--no-fail-fast` to gather all failures in single run\n\n**Snapshots:**\n- **`insta` snapshots**: Text-based, run with regular Rust tests. On failure: `cargo insta review` (install: `cargo install cargo-insta`)\n- **Image comparison tests**: Render image vs checked-in reference. Uses `egui_kittest`'s `Harness::snapshot` + `TestContext` for mocking viewer.\n - Results saved to `tests/snapshots/`, failures produce `diff.png`\n - Update refs: `UPDATE_SNAPSHOTS=1`\n - Update from failed CI run: `./scripts/update_snapshots_from_ci.sh`\n - Best practices: see [egui_kittest README](https://github.com/emilk/egui/tree/master/crates/egui_kittest#snapshot-testing)\n\n### Long-running commands (for agents)\n\nBuilds and test runs here take a long time, so run them in the background.\nA background command notifies you when it exits, so never start a second command to watch the first one.\nNo `sleep`-and-`tail` pollers, no `until` loops on the log.\nStart the build, do something else, and read its output once the notification arrives.\n\n### Driving a running egui UI (for agents)\n\nTwo MCP servers let an agent see and click a running UI instead of guessing from code:\n\n- `rerun viewer-mcp` drives a running Rerun Viewer over gRPC. See `docs/content/reference/viewer/mcp.md`.\n- `egui-mcp` drives *any* egui app started with `EGUI_INSPECTION=1`, including our example apps and headless `egui_kittest` harnesses.\n It exposes `attach`, `widget_tree`, `click`, `type_text`, `screenshot`, `wait_for`, and friends.\n Both are configured in the repo-root `.mcp.json`; install with `cargo install egui_mcp`.\n\nTypical loop: start the app with `EGUI_INSPECTION=1`, call `attach` (host `127.0.0.1`, port `5719`), `widget_tree` to find widgets, `click`/`type_text`, then `screenshot` with a `save_path` and look at the image.\n`egui-mcp` needs the app to paint frames: windowed apps must not be occluded on macOS, so prefer a headless harness where one exists.\nA headless `egui_kittest` harness can opt in with `egui_inspection::attach_from_env(&harness.ctx, label)`.\nExample: `EGUI_INSPECTION=1 cargo run -p re_agent_ui --example agent_app -- --headless`.\n\n## Code generation system\n\n**Critical: Never edit generated files directly.** All generated files are marked \"DO NOT EDIT\" at the top.\n\n### Type definition flow\n\n```\nre_type_definitions → pixi run codegen → Generated code (Rust/Python/C++) + docs (docs/content/reference/types/)\n```\n\n- Type definitions live in `crates/build/re_type_definitions/rerun/`\n - `encodings/*.def.rs` - Low-level types (Vec3D, Mat4x4, etc.)\n - `components/*.def.rs` - Component types (Position3D, Color, etc.)\n - `archetypes/*.def.rs` - Archetypes (Points3D, Image, etc.)\n - `blueprint/*.def.rs` - Blueprint system types\n- Codegen implementation is in `crates/build/re_types_builder/`\n- After modifying a definition, run `pixi run codegen` to regenerate code\n\n### Extension pattern\n\nTo add custom functionality to generated types, create `_ext` files:\n- Rust: `filename_ext.rs` (automatically imported by codegen)\n- Python: `filename_ext.py` (mixed in with generated class)\n- C++: `filename_ext.cpp` (compiled and included automatically, parts of it may be marked for copy into the header by codegen)\n\n## Code conventions\n\n### General\n\n- use `…` instead of `...` \n- validate various custom conventions via `pixi run lint-rerun ` (not passing any file will check everything)\n- Use `format!(\"{x}\")` over `format!(\"{}, x)` (same in log calls etc)\n- Don't write trivial comments that add nothing new\n- In error and log messages, put the error first and any file path at the end (e.g. `Failed to import: {err}\\nFile path: {path}`), never in the middle.\n Paths can be long or sensitive, so trailing placement makes them easy to strip when copy-pasting.\n- Prose style (em vs en dash, sentence endings, casing) — see [`DESIGN.md`](DESIGN.md). In short: spaced em dash ` — `, never unspaced `word—word`, and don't use `–` as a sentence dash (it's for numeric ranges only) \n- One sentence per line in markdown files.\n Markdown joins consecutive lines into a paragraph, so rendering is unchanged — but diffs become much easier to review.\n\n## Architecture overview\n\n### Crate organization\n\n```\ncrates/\n├── build/ # Code generation (re_types_builder)\n├── store/ # Data types, storage, querying\n├── top/ # User-facing SDKs and CLI\n└── viewer/ # Viewer UI and rendering\n```\n\nFor more details about the architecture see `ARCHITECTURE.md`.\n\n**When adding, removing, or renaming a crate**, run `pixi run crate-graph`.\nIt regenerates both the crate dependency diagram (`crate_graph.svg`) and the crate tables in `ARCHITECTURE.md` from `cargo metadata`.\nA crate's one-line description comes from the `description` field of its `Cargo.toml`.\n\n### Type system hierarchy\n\nThe type system has three levels (generated from `re_type_definitions`):\n\n1. **Encodings** (`rerun.encodings.*`) - Basic types like Vec3D, Color\n2. **Components** (`rerun.components.*`) - Named semantic wrappers (Position3D, Radius)\n3. **Archetypes** (`rerun.archetypes.*`) - Collections of components (Points3D, Image)\n\nEach archetype specifies:\n- Required components (must be provided)\n- Recommended components (have good defaults)\n- Optional components (purely optional)\n\nExample: `Points3D` archetype requires `positions`, recommends `colors` and `radii`, allows optional `labels`.\n\n### Data flow\n\n```\nSDK (log archetype)\n ↓ encode to Apache Arrow\nLogMsg (encoded data)\n ↓ transport (gRPC/file/memory)\nre_chunk_store (indexed time series DB)\n ↓ query\nViewer (immediate mode rendering)\n```\n\n### Blueprint system\n\nThe blueprint is the viewer's configuration layer:\n- Stored as a separate store (`re_entity_db`) with \"blueprint\" timeline\n- Defines: view layout, visibility, per-entity overrides, view properties\n- Uses the same type system as logged data\n- Basic blueprint path hierarchy: `/viewport/`, `/view/{uuid}/`, `/container/{uuid}/`\n\n### Visualizers\n\nEach view type (Spatial3D, TimeSeries, etc.) has registered visualizers:\n- Determine which entities/archetypes can be visualized\n- Execute per-frame: query data → process → generate render commands\n- Examples: Points3DVisualizer, LineStripsVisualizer, MeshVisualizer\n\nThe viewer uses **immediate mode**: every frame, query the store and re-render from scratch.\n\n## Documentation snippets\n\nSee [`docs/snippets/README.md`](docs/snippets/README.md) for running, building, finding snippets. Config in [`docs/snippets/snippets.toml`](docs/snippets/snippets.toml).\n\n## Python development workflow\n\nPython uses a separate uv-managed .venv (not pixi's conda env):\n\n```bash\npixi run py-build # Build rerun-sdk into .venv\npixi run uvpy script.py # Run Python scripts via uv\npixi run uv run script.py # Explicit uv run\n```\n\nThe `uv` wrapper script unsets `CONDA_PREFIX` to ensure isolation from pixi's environment.\n\n## Important notes\n\n- **PyO3 Configuration**: If you see PyO3 config errors, run `pixi run ensure-pyo3-build-cfg`\n- **git-lfs**: Required for test snapshots. Install with your package manager and run `git lfs install`\n- **Immediate Mode**: The entire viewer is rendered from scratch each frame (no state management callbacks)\n- **Arrow Native**: Data is stored, transmitted, and queried as Apache Arrow arrays\n- **Multi-language**: Changes to `re_type_definitions` affect Rust, Python, and C++ simultaneously\n\n## Python docstring formatting\n\nPython API docs are built with **MkDocs + mkdocstrings** (NOT Sphinx). Never use reStructuredText (rST) syntax in Python docstrings or documentation. Use markdown instead:\n\n- **Cross-references:** Use `[`ClassName`][]` (mkdocstrings syntax), NOT `:class:`ClassName`` / `:func:` / `:meth:` (rST roles)\n- **Warnings/notes:** Use MkDocs admonitions (`!!! warning` with indented body), NOT `.. warning::` (rST directives)\n- **Deprecation notices:** Use the `@deprecated` decorator (mkdocstrings renders it automatically). Do NOT duplicate in the docstring with `.. deprecated::` or `**Deprecated:**`\n- **Code blocks:** Use markdown fenced blocks (`` ``` ``), NOT `.. code-block::`\n- **Parameter docs:** Use numpy-style sections (`Parameters`, `Returns` with `----------`), which is what the codebase already uses\n\n## Documentation system\n\nSee [`docs/README.md`](docs/README.md) for the full documentation architecture.\n\nThe docs span multiple sites: the main docs at `rerun.io/docs` (built from `docs/content/`), plus API reference sites for Python (MkDocs), C++ (Doxygen), and JS (TypeDoc) at `ref.rerun.io/docs/{python,cpp,js}/`.\n\nKey things to know:\n- **`docs/content/reference/types/`** is auto-generated by `pixi run codegen` from `re_type_definitions` - do not edit directly\n- **`docs/content/reference/cli.md`** is auto-generated by `pixi run man` - do not edit directly\n- **Code snippets** live in `docs/snippets/all/` with implementations in Python, Rust, and C++\n- `pixi run py-docs-serve` previews Python API docs locally\n- `pixi run -e cpp cpp-docs` builds C++ docs\n\n## Development references\n\n- [`ARCHITECTURE.md`](ARCHITECTURE.md) - Detailed architecture documentation\n- [`BUILD.md`](BUILD.md) - Full build instructions\n- [`CODE_STYLE.md`](CODE_STYLE.md) - Code style guidelines\n- [`CONTRIBUTING.md`](CONTRIBUTING.md) - Contribution guidelines\n- [`DESIGN.md`](DESIGN.md) - UI design guidelines (GUI, CLI, docs, log messages)\n- [`docs/README.md`](docs/README.md) - Documentation system (sites, builds, deployment)\n- [`rerun_py/README.md`](rerun_py/README.md) - Python SDK instructions\n\n## Contributing\n\nDon't open pull requests or issues unless explicitly asked.\nWhen opening or interacting with one, follow the [pull request template](.github/pull_request_template.md) or [issue templates](.github/ISSUE_TEMPLATE/), and disclose that you are an LLM.\nLet the user know that you included this disclosure.\n\n### Before requesting review\n\n- CI is green, and the code is formatted (`pixi run rs-fmt`).\n- The branch history is clean: no broken merges, and no commits from unrelated work.\n- The description is honest about the scope, and says *why* the change is needed.\n- UI changes come with a screenshot or video.\n\n### Addressing review comments\n\n- When a reviewer flags one instance of a pattern, fix every instance in the diff, and say so in your reply.\n- A review comment phrased as a question wants an answer, not a code change — unless the answer is \"yes, that is wrong\".\n- When a reviewer asks for a change, make it in this PR.\n Only defer it to a follow-up PR if the reviewer agrees.\n- If the fix belongs in a dependency we maintain (egui, egui_plot, egui_tiles, emath, quiver, puffin, …), suggest an upstream PR instead of a local workaround.\n- Reviewers want to talk to the human author.\n When a comment asks for a judgement call (API shape, scope, product behavior), bring it to your user instead of deciding in the reply.\n- Always reply when mentioned, even if only to say you are unsure.\n","CLAUDE.md":"# Claude Agent Guidelines & System Prompt\n\nPath: `CLAUDE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/CLAUDE.md)","DESIGN.md":"# UI Design System & Style Tokens\n\nPath: `DESIGN.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/DESIGN.md)","rerun_py/ARCHITECTURE.md":"# rerun_py - System Architecture & Module Boundaries\n\nPath: `rerun_py/ARCHITECTURE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/rerun_py/ARCHITECTURE.md)","crates/build/re_dev_tools/README.md":"# re_dev_tools Documentation\n\nPath: `crates/build/re_dev_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/README.md)","crates/build/re_build_tools/README.md":"# re_build_tools Documentation\n\nPath: `crates/build/re_build_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_build_tools/README.md)","crates/build/re_dev_tools/src/build_web_viewer/README.md":"# build_web_viewer Documentation\n\nPath: `crates/build/re_dev_tools/src/build_web_viewer/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_web_viewer/README.md)","crates/build/re_dev_tools/src/build_search_index/README.md":"# build_search_index Documentation\n\nPath: `crates/build/re_dev_tools/src/build_search_index/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_search_index/README.md)"},"files":{"skills/rerun-urdf/SKILL.md":"---\nname: rerun-urdf\ndescription: Drive the Rerun URDF API (rerun.urdf.UrdfTree) to ingest a URDF as a Transform3D layer on a robot recording. Read when logging a robot model, running forward kinematics from joint states, composing a fixed chain for sensor extrinsics, or when the transform tree will not connect from the data alone. Builds on rerun-chunk-processing (stream/lens mechanics) and rerun-data-model (entity paths, timeline, base-vs-layer).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun URDF ingestion\n\nA URDF gives you a robot's geometry and its kinematic tree.\nIngesting it means two API calls on one `rerun.urdf.UrdfTree`: stream the static model, then drive it with forward kinematics from joint states. The transforms it produces are derived, so they are a **layer**, never base (the `URDF + joints (computed)` row of the `rerun-data-model` table).\n\nThis skill is the `UrdfTree` API and the two judgment calls it cannot make for\nyou: how your joint values map to URDF joints, and how the disconnected frames\nin the scene connect to one root. The stream/lens plumbing (`LazyChunkStream`,\n`DeriveLens`, `Selector`, writing and optimizing RRDs) is in\n`rerun-chunk-processing`; reach for it, do not re-derive it. Nothing below is\ntied to a data format: where your joint names, joint values, and calibration\ncome from is yours to wire in.\n\n## Getting the URDF\n\nDatasets rarely ship one, and the robot's own repo is not always the fastest route.\nIn order:\n\n- [`robot_descriptions.py`](https://github.com/robot-descriptions/robot_descriptions.py) — `pip install robot_descriptions`, then `from robot_descriptions import ur5e_description; ur5e_description.URDF_PATH`.\n It downloads and caches the upstream description and hands you a plain path with the meshes already next to it.\n Check its index first; it covers most of the table below.\n- The vendor repo, when the description is newer than the package's pin or is not in it.\n- [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) — MJCF, not URDF, so it needs conversion. Worth it only when nothing else exists.\n\n| Robot | Source | Notes |\n| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |\n| SO-100 / SO-101 (LeRobot) | [TheRobotStudio/SO-ARM100](https://github.com/TheRobotStudio/SO-ARM100) | `Simulation/SO100/so100.urdf`, `Simulation/SO101/so101_new_calib.urdf`; plain URDF, relative mesh paths |\n| ALOHA (ViperX 300, WidowX) | [Interbotix/interbotix_ros_manipulators](https://github.com/Interbotix/interbotix_ros_manipulators) | xacro |\n| Franka Panda / FR3 | [frankaemika/franka_description](https://github.com/frankaemika/franka_description) | xacro |\n| Universal Robots UR3 to UR20 | [UniversalRobots/Universal_Robots_ROS2_Description](https://github.com/UniversalRobots/Universal_Robots_ROS2_Description) | xacro, arm parameters in YAML |\n| Kinova Gen3 | [Kinovarobotics/ros2_kortex](https://github.com/Kinovarobotics/ros2_kortex) | xacro |\n| UFactory xArm / Lite 6 | [xArm-Developer/xarm_ros2](https://github.com/xArm-Developer/xarm_ros2) | xacro |\n| Unitree Z1, G1, H1, Go2 | [unitreerobotics/unitree_ros](https://github.com/unitreerobotics/unitree_ros) | plain URDF per robot |\n\nTwo things bite before FK ever runs:\n\n- **xacro is not URDF.** First run `xacro arm.urdf.xacro > arm.urdf` (`pip install xacro`), passing whatever arguments the file requires (gripper variant, arm size).\n A xacro file handed to `UrdfTree.from_file_path` fails to parse.\n- **`package://` mesh URIs.** The importer resolves them by scanning `ROS_PACKAGE_PATH` (ROS 1) and `AMENT_PREFIX_PATH` (ROS 2) for the package, and otherwise treats the rest of the URI as relative to the URDF's own directory.\n Outside a sourced ROS workspace, either export one of those variables or lay the meshes out relative to the URDF.\n Meshes that fail to resolve give you a kinematically correct but invisible robot.\n\n## The API\n\n```python\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(\n urdf_path,\n entity_path_prefix=\"robot\", # links log under /robot/)`. The `recording_id` must equal the\nbase segment id or the layer never attaches; `application_id` is discarded on\nregistration.\n\n## Mapping joint state to URDF joints (you supply this; the data will not)\n\n`compute_joint_transform_batches` is only as right as the `names`/`values` you\nhand it, and the mapping is not in the URDF. Three things go wrong silently:\n\n- **Order.** Build an explicit `names` array aligned to the `values` you read.\n Never assume your message's field order matches the URDF's `` order.\n- **Count.** The URDF's non-`fixed` joint count rarely equals your reported value\n count. A gripper sent as one value is often two prismatic joints in the URDF;\n a mimic joint may be omitted from the message. Reconcile explicitly, and use\n the API to do it: iterate `urdf.joints()`, partition by `joint_type` and\n `mimic`. A joint with `mimic` set derives its value from the driver joint as\n `driver * multiplier + offset`; feed it that, not a message field. Confirm\n the count against `urdf.joints()`, not the message length.\n- **Units.** URDF joints are radians (revolute) and meters (prismatic). Convert\n if your source differs.\n\nGet any of these wrong and FK runs and writes a confident, wrong pose.\n\n### Verify joint mapping and limits\n\nJoint values in the source data aren't guaranteed to be within URDF joint limits.\nThis can happen with any format (MCAP, Parquet, etc).\nIt is therefore useful to check whether joint values in the data match the URDF model's limits.\nOut-of-range values can indicate issues like wrong indexing or model mismatch and should be clarified before calculating the forward kinematics.\n\nTo get limits of a URDF model:\n\n```python\nlimits = {joint.name: (joint.limit_lower, joint.limit_upper) for joint in urdf.joints() if joint.joint_type != \"fixed\"}\n# then compare each column's min/max against its joint's range\n```\n\n`scripts/check_joint_mapping.py` is that check worked out for one source shape — a LeRobot-style parquet column of per-frame joint values.\nRead it as the example to copy and adapt for your own format, and run it directly if your data is already a LeRobot-style parquet file:\n\n```bash\nuv run scripts/check_joint_mapping.py --column observation.state --degrees\n```\n\n**Important:** this is a validation meant to be done before full processing — don't run this for _every_ file in a larger dataset.\nSelect a single representative file or a small subset for this verification.\n\nFor each joint it tries the usual candidate corrections (identity, negate, `% 360`, ±90°/±180° offsets, and their combinations) and reports which ones land inside that joint's limits.\nIt reports the fit, not the answer: a joint with two surviving candidates is **ambiguous**, and no amount of range-checking will separate them.\nValidate ambiguous mappings against known poses or geometry for the dataset.\n\n**Read the failures by shape.**\nA whole column outside its range means a wrong sign, unit, or offset — a bug, and the pose will be confidently wrong.\nA few frames a few degrees past a limit is calibration slack — expected, and `clamp=True` is the fix, not a workaround.\n`clamp=True` warns once per offending row, so run it while working out the mapping: a flood of warnings is a mapping smell, and a thin trickle at the extremes of one joint is a healthy recording.\n\n## Make the joint states readable first\n\nWhere the joint values come from is not this skill's problem, but a dead joint\nsource produces an empty FK layer **with no error**: a source path that\nmatched nothing, a decoder that yielded zero rows, or a reader that dropped the\nmessage silently. Whatever the source, confirm the joint-state stream yields\nrows before debugging FK. The importer skill for your source format covers its\nown empty-stream failure modes.\n\n## How transforms compose (reason about this before logging anything)\n\nRerun resolves a pose by chaining transforms from a frame up to a root. There\nare two ways an edge in that chain gets defined, and a URDF ingest mixes both:\n\n- **By entity-path hierarchy.** A `Transform3D` on `/a/b` with no frame names is\n the transform of `/a/b` relative to its parent path `/a`. Composition follows\n the entity tree.\n- **By explicit frame graph.** A `Transform3D` that carries `parent_frame` and\n `child_frame` defines an edge between two **named frames**, independent of\n where in the entity tree it is logged. URDF FK uses this: every joint\n transform names its parent and child link frames.\n\nSo a URDF ingest is a graph of named frames. An edge exists only if some\n`Transform3D` names that exact `parent_frame -> child_frame` pair. A frame with\nno incoming edge is a root. The viewer renders every root at the world origin,\nwhich is why two unconnected robots silently overlap instead of erroring.\n\n## Resolving the transform forest (the part the data cannot always give you)\n\nA URDF is **one tree rooted at its base link**. FK and fixed joints supply every\nedge inside that tree. A real scene is a **forest** of roots the URDF never\nconnects: a world or scene frame, each robot's base, every camera or sensor\nframe. The edges that join those roots come from calibration (extrinsics in a\nsidecar, a TF static publisher, a hand-eye result), not from the URDF, and some\nare simply absent. A single connected tree is not always solvable. Resolve it\ndeliberately:\n\n1. **Enumerate every frame.** Iterate `urdf.joints()` and collect the\n `parent_link`/`child_link` pairs (the in-tree edges, frame-prefixed);\n `urdf.root_link()` is that URDF's root. Add every sensor/world frame the\n scene needs (from the `rerun-data-model` table). Decide the one intended\n root.\n2. **Classify each edge by source.** In-URDF edges (FK joints, `fixed` joints)\n come from the URDF plus joint values. Inter-root edges (root to each robot\n base, root to each fixed sensor, an arm link to a wrist-mounted camera) come\n from calibration and you must log them yourself.\n3. **Compose fixed chains from the URDF** when you need the transform between two\n links joined only by `fixed` joints (a camera bracket, a tool mount): walk\n parent links across `fixed` joints via `urdf.joints()`, turning each joint's\n `origin_xyz`/`origin_rpy` into a homogeneous matrix and multiplying along\n the chain. If the walk cannot reach\n the target link, **stop and say so** (\"no fixed chain from A to B; stuck at\n C\"). A broken chain is a wrong pose, not a missing one.\n4. **Build the edge set and find the roots.** Collect every `parent_frame ->\nchild_frame` pair you will log (URDF + calibration). Walk parents from each\n frame; any frame that does not reach the intended root is an unconnected root\n and names a **missing edge**. This is a pure graph check you can run before\n writing the RRD.\n5. **Resolve every missing edge, or fail loudly.** For each one:\n - If calibration supplies the transform, log it (next section).\n - If the data does not, the tree is unsolvable. Do **not** leave the frame\n disconnected (it collapses onto the origin and reads as one merged scene).\n Either abort and name the missing edge, or log identity and emit a loud\n warning naming the assumed edge. State which you did.\n6. **Match frame names across sources.** FK-derived `parent_frame`/`child_frame`\n must equal the names `urdf.stream()` emits for the static geometry, and your\n calibration edges must use those same names, or links float off the mesh. The\n `frame_prefix` is what keeps them identical; reuse it everywhere.\n\nA correct ingest has exactly one root, and a path from every frame to it.\n\n## Logging a connection correctly\n\nA connecting edge is a **static** `Transform3D` carrying the bridging frame\nnames. Static (no time index) so it holds for the whole recording; the frame\nnames, not the entity path, are what create the graph edge. This\n`Chunk.from_columns` is the rare **sidecar exception** — a calibration transform\nno reader or FK lens can produce; do not generalize it to transforms a reader\nemits (a `frame_transforms` topic → `Transform3D`) or that FK derives:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import Chunk, LazyChunkStream\n\n# world -> this robot's base, from your calibration (translation + xyzw quaternion)\nedge = Chunk.from_columns(\n \"/world/robot_base\", # any sensible bridging entity path\n indexes=[], # no index == static\n columns=rr.Transform3D.columns(\n translation=[translation],\n quaternion=[quaternion_xyzw],\n parent_frame=[\"world\"], # must match the root frame name\n child_frame=[\"arm_base_link\"], # must match the URDF root frame (prefixed)\n ),\n)\nedges = LazyChunkStream.from_iter([edge]) # merge alongside model + FK streams\n```\n\nLog a fixed-chain result (step 3) the same way, with `parent_frame` and\n`child_frame` set to the two link frames the chain spans. Merge all edge chunks\ninto the same recording as the model and FK streams so they share the graph.\n\n## Minimal shape (one robot, generic joint source)\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens, LazyChunkStream, OptimizationProfile, Selector\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(urdf_path, entity_path_prefix=\"robot\", static_transform_entity_path=\"robot/tf_static\")\nmodel = urdf.stream(include_joint_transforms=True).drop(content=\"/robot/**/collision_geometries/**\")\n\njoints = source_joint_state_stream() # your reader; one message column of names+values\nfk = (\n joints\n .lenses(\n DeriveLens(JOINT_MSG_COMPONENT, output_entity=\"/tmp/batches\").to_component(\n \"rerun.urdf.JointTransformBatch\",\n Selector(\".\").pipe(lambda msgs: urdf.compute_joint_transform_batches(read_names(msgs), read_values(msgs))),\n ),\n content=JOINT_SOURCE_PATH,\n output_mode=\"forward_all\",\n )\n .lenses(\n DeriveLens(\"rerun.urdf.JointTransformBatch\", output_entity=\"/robot/transforms\", scatter=True)\n .to_component(rr.Transform3D.descriptor_translation(), Selector(\"[].translation\"))\n .to_component(rr.Transform3D.descriptor_quaternion(), Selector(\"[].quaternion\"))\n .to_component(rr.Transform3D.descriptor_parent_frame(), Selector(\"[].parent_frame\"))\n .to_component(rr.Transform3D.descriptor_child_frame(), Selector(\"[].child_frame\")),\n content=\"/tmp/batches\",\n output_mode=\"drop_unmatched\",\n )\n .filter(content=\"/robot/transforms\")\n)\n\nLazyChunkStream.merge(model, fk).collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n out_path,\n application_id=\"urdf\",\n recording_id=segment_id,\n)\n```\n\n`read_names`, `read_values`, `JOINT_MSG_COMPONENT`, and `JOINT_SOURCE_PATH` are\nthe only data-specific pieces, and the mapping section above is what makes them\ncorrect.\n\n## One URDF for a whole dataset (assets)\n\n`urdf.stream()` emits only static chunks — meshes and fixed-joint transforms, with no time index.\nThe model does not belong in a per-episode layer, where every segment pays for the same meshes.\nRegister it once as a dataset **asset**:\n\n```python\nurdf.stream().collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n model_rrd, application_id=\"urdf\", recording_id=\"so100_model\"\n)\nasset_id = dataset.register_asset(\"s3://bucket/so100_model.rrd\") # uri must be readable by the server\n```\n\nThe split is along static vs temporal: the **model** is an asset, and the **FK transforms** stay a per-segment layer keyed by `recording_id`.\nKeeping `include_joint_transforms=True` in the asset costs nothing and gives the rest pose for segments or time ranges the FK layer does not cover.\n\n- The viewer loads a dataset's assets alongside any segment it opens, and caches them across segments.\n- `dataset.segment_store(segment_id)` includes assets; pass `include_assets=False` to skip the extra manifest request per asset.\n- Dataframe queries never see assets — they stay on the dataset's own segments.\n\nWithout a catalog, the same split still pays: write the model RRD once and open it next to whichever episode you are looking at, rather than copying it into each one.\n\n## Gotchas that cause real failures\n\n1. Empty layer, no error: dead joint-state source (decoded to zero rows; see the importer skill for your format) or wrong `JOINT_SOURCE_PATH`/component name.\n2. Confident wrong pose: joint count, order, or units wrong.\n3. Layer writes but never attaches: `recording_id != segment_id`.\n4. Frames collide: two robots sharing a `frame_prefix`.\n5. Scene looks merged at the origin: unconnected roots logged as identity without a calibration edge.\n6. Catalog ingest rejects or misorders chunks: `OBJECT_STORE` optimization skipped.\n7. Layer opened in a running viewer never merges: opening an `.rrd` **by path** registers it as its own dataset and replaces its `application_id`, so it lands beside the segment instead of on it.\n To eyeball a layer locally, either merge model + FK + base into one RRD, or send both through a `RecordingStream(application_id, recording_id=segment_id)` pointed at the viewer with `rec.connect_grpc(url=\"rerun+http://127.0.0.1:9876/proxy\")`, which keeps the IDs and merges the stores.\n8. Robot invisible but transforms correct: meshes failed to resolve (see \"Getting the URDF\").\n9. No 3D view after adding a URDF to an existing recording: the auto-layout does not necessarily rebuild for the new entities. Send an explicit blueprint with `rrb.Spatial3DView(origin=\"/robot\")` (see `rerun-blueprint`).\n\n## References\n\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing` (FK two-lens pattern, two robots + scene URDFs, prefixes, recoloring, calibration offsets)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/animated_urdf` (classic logging API: `log_urdf_to_recording`, per-joint `compute_transform`)\n- [Catalog object model: assets](https://rerun.io/docs/concepts/query-and-transform/catalog-object-model#assets) (`register_asset`, the limits, and how assets are read back)\n- `rerun-data-model` (the mapping table this skill consumes)\n- the importer skill for your joint-state source format (making the source readable)\n- `rerun-chunk-processing` (lens/stream, write/optimize mechanics)\n","skills/assemble-changelog/SKILL.md":"---\nname: assemble-changelog\ndescription: >\n Assemble the per-PR changeset entries in docs/content/changelog/upcoming/\n into a release's changeset-0-XX.md and generate the detailed CHANGELOG.md sections.\n Use at release time when the user wants to build or finalize the changelog or\n changeset for a Rerun release, or merge the upcoming/ entries. Invoked via\n /assemble-changelog or \"assemble the changelog for 0.x.y\".\nuser_invocable: true\nallowed-tools: Bash Read Edit Write\n---\n\n# Assemble-changelog\n\nRelease-time changelog assembly for the Rerun repo.\n\nAlways work in the root of a standalone `rerun-io/rerun` checkout — normally the `prepare-release-0.x.y` branch, where the result is committed.\nThis is step 4 of [RELEASES.md](../../RELEASES.md); read it for the surrounding context.\n\nBefore doing any release work, verify that the current directory is the repository root and that its `origin` is `rerun-io/rerun`:\n\n```bash\ntest \"$(git rev-parse --show-toplevel)\" = \"$PWD\"\ngit remote get-url origin\n```\n\nDo not run the workflow in the reality monorepo, including from its `rerun/` directory.\nThe release scripts need the standalone repository's `0.x.y` tags and resolve `(#N)` commit references against `rerun-io/rerun`.\nRunning them against reality can silently resolve reality PR numbers to unrelated Rerun PRs.\n\nIf either precondition is not met, stop before running any release command.\nTell the user that the skill requires the root of a standalone `rerun-io/rerun` checkout, and ask them to restart it there.\nDo not clone a repository, fetch tags, or switch branches for the user.\n\nResolve the target version from `$ARGUMENTS` (e.g. `0.34.0`). If absent, read it\nfrom `Cargo.toml` (`version = \"0.x.y-…\"`) and confirm with the user.\n\n## Workflow\n\n### 1. Assemble `upcoming/` → the release changeset\n\nThe curated entries live one-file-per-PR in `docs/content/changelog/upcoming/*.md`\n(skip `_template.md`). Each declares `type: highlight|misc|breaking|feature` in its\nfrontmatter. Merge them into `docs/content/changelog/changeset-0-XX.md`, creating that file from\n`docs/content/changelog/_template.md` if it does not exist yet (set `title` to the version — keep it\nquoted, e.g. `title: \"0.36\"`, so YAML keeps it a string — and `order` one lower than the previous release):\n\n- `highlight` → one `### ` subsection each under `## Highlights`.\n- `feature` → one `### ` subsection each under `## New features`.\n- `misc` → exactly one `- ` list item each under `## Other`. Fold its `### ` heading and body into the item, preserving links. Do not use headings or prose outside the list items.\n- `breaking` → one `### ` subsection each under `## Breaking changes`. If none, write `None.`.\n\nKeep the sections in that order and omit `Highlights` and `Other` if they have no entries.\nThe changelog is user-facing (it's part of the website), so it leads with what's new;\nthe verbose, developer-only breaking-change migration guides go last so most readers\ndon't have to scroll past them.\n\nTailor the output to the release type:\n\n- Patch release (`0.x.Y`, Y > 0) → typically only bug fixes. Skip `Highlights` and\n `New features` (there usually won't be `upcoming/` entries anyway); include `Other`\n only if there are miscellaneous entries, and keep `Breaking changes` only if there are any.\n- Minor release (`0.X.0`) → the full template: highlights, new features, other items, breaking changes.\n\nPreserve each entry's prose and structure (migration guides, tables, `snippet:` directives,\nscreenshots, links), except that each miscellaneous entry becomes its single list item.\nDe-duplicate overlapping entries and order breaking changes most-impactful first.\nDrop the per-entry frontmatter.\n\nRelative doc links in entries were written as if from `changelog/` (e.g.\n`../reference/migration/...`), which is correct once merged — keep them as-is.\n\nImages and videos must live on `static.rerun.io` (upload with `scripts/upload_image.py`), never as\nGitHub attachments: attachments uploaded in the private monorepo cannot be shown on the website.\n\nFeature leads write their `## Highlights` entries per PR, so read the section as a whole and\nflag to the human anything that does not hang together, rather than rewriting it yourself.\n\nFinally, point the `redirect:` frontmatter in `docs/content/changelog.md` at\n`changelog/changeset-0-XX`: CI's `scripts/ci/check_changelog_redirect.py` requires the\nnewest changeset to be the redirect target, so the repoint must land together with the\nnew changeset.\n\n### 2. Resolve release blockers\n\nEnsure that every non-template file from `upcoming/` was merged into the changeset, then search the assembled changeset for unresolved placeholders:\n\n```bash\nrg -n 'TODO\\([^)]+\\)' docs/content/changelog/changeset-0-XX.md # NOLINT\n```\n\nResolve every match before continuing.\nAn unresolved `TODO(name)` blocks the release.\n\n### 3. Generate the summary and detail sections into CHANGELOG.md\n\n```bash\npixi run uvpy scripts/generate_changelog.py --version 0.x.y\n```\n\nEdit PR titles/labels to improve the output, then copy the result into `CHANGELOG.md`\n(drop the trailing \"Chronological changes\" section; replace the placeholder video/blogpost\nlines as previous releases did). Spot-check a few entries against the actual PRs:\npolluted titles (old, unrelated PRs; `thanks @…` for core team members) mean a PR-number\nlookup misfired — see the warning at the top.\nRead the generated section for wrong contributor attribution and typos.\nIf the release ships without Rust crates (`publish-crates=false`), say in the `CHANGELOG.md`\nsection that this version is not on crates.io.\n\nDo this *after* step 1: the script reads the assembled changeset and emits a summary of it\n(section headings + links to the changeset on the website), rather than inlining its prose.\n`CHANGELOG.md` therefore never duplicates the changeset — if the changeset is missing, the\nscript emits an unresolved placeholder instead.\n\n### 4. Polish the CHANGELOG.md section\n\nThe generated section is a starting point; edit it by hand so readers can find their way to the details.\n\n- **Order the overview by impact.**\n Put the biggest new capabilities first, then integrations and APIs, then format support, then polish, then niche or developer-facing items.\n- **Group and order the details.**\n The script emits each details subsection (🐍 Python API, 🪳 Bug fixes, 🌁 Viewer improvements, …) in commit order.\n Within each subsection, put entries that touch the same feature next to each other (e.g. all MCP/ViewerControl entries, all state timeline entries, all hangs and deadlocks), order the groups by their most impactful entry, and order entries within a group by impact.\n Only reorder lines; don't move entries between subsections or drop them.\n- **Make entries clickable.**\n Wherever possible, link an entry (in the overview *and* in the details) to the docs that show how to use the feature: the how-to guide, the archetype/view reference page, the CLI or MCP reference section, the Python/JS ref site, or docs.rs.\n Reuse the links the changeset already contains, converted to `https://rerun.io/docs/`, and check that each target file and `#anchor` heading exists.\n Link external names too (e.g. a third-party tool to its docs).\n In detail entries, link the key term, not the whole line, so the trailing commit/PR link stays distinct.\n- **Link each breaking change** to its subheading in the changeset: `https://rerun.io/docs/changelog/changeset-0-XX#`.\n Anchors are the lowercased heading with backticks and punctuation dropped and spaces turned into hyphens (`rrd::optimize` → `rrdoptimize`).\n- **Name the actual break.**\n A breaking-change heading (in the changeset and in `CHANGELOG.md`) must say what broke, e.g. \"`WriteChunks` removed\", not the new feature that replaces it (\"Catalog staging\").\n Lead the subsection with the removal and migration-guide link, then the replacement.\n- **Flag experimental and unstable features.**\n Check every new feature in the changeset and `CHANGELOG.md` against these markers (`PREV` is the previous minor release tag, e.g. `0.38.0`):\n\n ```bash\n # Non-blueprint types marked unstable whose definitions changed this cycle\n git diff --name-only $PREV HEAD -- crates/build/re_type_definitions | grep -v /blueprint/ | xargs rg -l 'state = \"unstable\"'\n # Python APIs in the `rerun.experimental` module\n git log --oneline $PREV..HEAD -- rerun_py/rerun_sdk/rerun/experimental\n # Viewer features behind a flag in `ExperimentalAppOptions`\n git diff $PREV HEAD -- crates/viewer_support/re_viewer_context/src/app_options.rs\n # Docs and docstrings that call something experimental\n git diff --name-only $PREV HEAD -- docs/content rerun_py/rerun_sdk | xargs rg -l -i 'is experimental'\n ```\n\n A file showing up in the first command does not mean the type is new: cross-reference the hits with the actual entries.\n For each matching feature, say so where readers will see it: prefix the changeset heading with \"Experimental\" (as `### Experimental 3D gaussian splat support` did in 0.36), and add a sentence saying it may change in future releases (for unstable types: that the data may not stay backwards compatible).\n If the feature is behind a flag, say how to enable it.\n In the `CHANGELOG.md` overview, move all of these entries into a final `#### 🧪 Experimental and unstable` subsection at the end of \"Overview & highlights\", with a one-line note that they may change and that unstable types may not stay backwards compatible.\n Order that subsection by impact too, and don't repeat \"Experimental\" in its entries.\n Blueprint types (`rerun.blueprint.*`, under `re_type_definitions/rerun/blueprint/`) are an exception: they are all marked unstable, which only means the blueprint data isn't backwards compatible.\n That alone does not make a feature experimental, so a new view or blueprint setting is only experimental if one of the other markers applies to it (or to the data archetype it shows).\n Likewise, a Viewer feature is not experimental just because it is configured through an experimental API.\n- **Add media to the headline features** in the overview: a screenshot (or a still frame plus a link to the video) indented under the bullet, using the `` markup that `pixi run upload-image` prints.\n Aim for a few images (roughly three to five), covering the most visual headline features: a picture sells a new view or renderer feature far better than a bullet does.\n Start from the media already in the changeset and the PR descriptions; if a visual feature has none, ask the user for a screenshot rather than skipping it.\n Don't add images to every entry: API, CLI, and bug-fix items rarely benefit.\n All media must live on `static.rerun.io`.\n Reuse a PR's media only if it is already on `static.rerun.io` or publicly accessible: `github.com/user-attachments/…` links from the private monorepo return 404 for the public.\n Download those with `curl -L -H \"Authorization: token $(gh auth token)\" ` and upload them with `pixi run upload-image --name `.\n GitHub does not render `` from external hosts, so for a video, extract a representative still (`ffmpeg -ss -i video.mp4 -frames:v 1 still.png`), upload it, and link the `.mp4` below it.\n Look at the frames and pick one without a cursor over the subject.\n\n### 5. Empty the inbox\n\nDelete the merged `upcoming/*.md` entries, keeping `_template.md` and any entries not included in this release.\nNever add redirects for these temporary entries.\n\n## Checklist before declaring done\n\n- [ ] Every non-template `upcoming/` entry is represented in the changeset.\n- [ ] No `TODO(name)` remains in the changeset.\n- [ ] No summaries or other prose were synthesized for existing entries.\n- [ ] The `CHANGELOG.md` overview is ordered by impact, a few of its most visual headline features have `static.rerun.io` images, and its breaking changes link to the changeset subheadings.\n- [ ] Every overview and detail entry that has relevant docs links to them.\n- [ ] Each details subsection groups entries by feature, most impactful first.\n- [ ] Every experimental or unstable feature (feature flag, `rerun.experimental`, `#[rerun(state = \"unstable\")]` on a non-blueprint type, or documented as experimental) is labeled as such in the changeset, and listed under the overview's final \"Experimental and unstable\" subsection.\n- [ ] `pixi run lint-rerun CHANGELOG.md docs/content/changelog/changeset-0-XX.md` passes.\n- [ ] Images and videos are hosted on `static.rerun.io`, not as GitHub attachments.\n- [ ] `## Highlights` reads well as a whole, or the human has been told where it doesn't.\n- [ ] The `CHANGELOG.md` section has correct contributor attribution and no typos.\n- [ ] `upcoming/` contains only `_template.md` and entries deferred to a later release.\n- [ ] `python scripts/ci/check_changelog_redirect.py` passes (redirect points at this changeset).\n- [ ] `python scripts/ci/check_doc_redirects.py --base origin/main` passes (`upcoming/` entries are exempt and DO NOT need a redirect).\n\n## Notes\n\n- This skill lives in `skills/assemble-changelog` in the standalone Rerun repository.\n- Synced commits in `rerun-io/rerun` carry a `Source-Ref` trailer (the reality merge\n commit); `generate_changelog.py` resolves it back to the originating reality PR for\n correct titles, labels, and contributors.\n- The *next* release's changeset is not pre-created: an empty changeset for an\n unreleased version would make `check_changelog_redirect.py` fail, since it requires the newest\n `changeset-0-xx.md` to be the redirect target. During a cycle, `upcoming/` is the only in-flight\n artifact.\n","skills/rerun-chunk-processing/SKILL.md":"---\nname: rerun-chunk-processing\ndescription: \"Core mechanics of the Rerun Chunk Processing API (rerun.chunk) — LazyChunkStream pipelines, Chunk, lenses (MutateLens/DeriveLens/Selector), RrdReader, writing optimized RRDs. Read BEFORE writing any ingestion/conversion/preprocessing code (convert an MCAP, build a recording from a dataset, preprocess an .rrd, port an old converter): it mandates reader+lens pipelines and steers away from hand-built chunks — no Chunk.from_columns for data a reader/lens can produce, no per-message rr.log, no manual pa.array assembly. Source-specific knowledge lives in the importer skills (rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot); read rerun-data-model first to decide what the data should become.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun chunk processing\n\nThe pipeline layer between raw data and an RRD: readers produce `Chunk`s,\nstreams transform them, terminal calls execute. This skill is the generic\nmechanics only. Decide the data model first (`rerun-data-model`), then pick the\nimporter skill for each source:\n\n| Source | Reader | Skill |\n| ----------------------------------------- | --------------------------------------- | --------------- |\n| MCAP file (ROS2, protobuf, Foxglove) | `McapReader(path).stream()` | `rerun-mcap` |\n| URDF robot model (+ joint states → FK) | `UrdfTree.from_file_path(...).stream()` | `rerun-urdf` |\n| Parquet table (trajectories, sensor logs) | `ParquetReader(path).stream()` | `rerun-parquet` |\n| mp4 camera video | `Mp4Reader(path).stream()` | `rerun-mp4` |\n| HDF5 file (groups → entities) | `Hdf5Reader(path).stream()` | none yet |\n| LeRobot dataset directory | built-in importer, then `RrdReader` | `rerun-lerobot` |\n| Existing RRD | `RrdReader(path)` | here, below |\n| Sidecar files (JSON calib, metadata) | `Chunk.from_columns` + `from_iter` | here, below |\n\nThe API is `rerun.chunk`; when\nbehavior matters, check the installed surface:\n`python -c \"from rerun.chunk import LazyChunkStream; help(LazyChunkStream)\"`.\n\n## Decision rule: where does each component come from?\n\nDefault: **a reader produces the chunks; lenses shape them.** Walk this before\nwriting any conversion code — most \"build it by hand\" instincts are wrong here:\n\n1. **Source a reader supports?** Use the reader's `.stream()`; never hand-parse\n and re-log. MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`,\n mp4→`Mp4Reader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`,\n LeRobot dir→`log_file_from_path`.\n2. **A decoder already emits the archetype?** Foxglove gives `Transform3D`,\n `Pinhole`, `VideoStream` (real sample bytes) ready-made — **pass it through**,\n do not re-derive. Only custom-protobuf topics arrive as `:message` and\n need a lens (see `rerun-mcap`).\n3. **Fix an existing component in place** (swapped resolution, recolor, unit\n convert)? `MutateLens`, `output_mode=\"forward_unmatched\"`.\n4. **Derive a new component/entity** (FK→`/tf`, scalars from a message)?\n `DeriveLens`. To scatter one row into N (a joint batch → per-joint `/tf`),\n use the **two-lens pair**: derive the batch with `output_mode=\"forward_all\"`\n (keeps the originals, e.g. the joint states), then a second\n `DeriveLens` with `scatter=True` and `output_mode=\"drop_unmatched\"` (emits\n only the scattered rows). See the `robot_data_preprocessing` example.\n5. **Genuine sidecar** no reader or lens can produce (JSON calibration offsets,\n hand-measured extrinsics, external metadata)? `Chunk.from_columns` + `from_iter`.\n6. Finish with `LazyChunkStream.merge(...)` →\n `.collect(optimize=OptimizationProfile.OBJECT_STORE)` →\n `write_rrd(application_id, recording_id)`.\n\nWhy this order: the pipeline stays lazy, columnar, multithreaded, and\n`OBJECT_STORE`-optimizable. A hand-built row loop or an out-of-lens `pa.array`\nthrows all of that away — that is the path we are deliberately avoiding.\n\n## Anti-patterns (use a reader + lens instead)\n\nIf you are writing the left, stop and use the right:\n\n- **`for`-loop building rows/components** → a lens with a `Selector(...).pipe(...)`\n PyArrow-compute callback.\n- **`rr.init` + `rr.log` per message for conversion** → that is _live_ logging;\n for ingestion, read with a reader and `write_rrd`.\n- **`chunk.to_record_batch()` + `pc.filter` then rebuilding via\n `Chunk.from_columns`** (row-thinning by hand) → `stream.drop(content=...)`,\n `.split(...)`, or a `MutateLens` returning a filtered `pa.array`.\n- **`pa.array` / `pa.RecordBatch` / `np.frombuffer` assembled OUTSIDE a lens** →\n move the transform inside a `MutateLens`/`DeriveLens` selector callback.\n- **`rr.send_columns` hand-assembled from a custom parser** → use the matching\n reader; it produces chunks directly.\n- **Parsing MCAP/URDF with a non-Rerun library then re-logging** → `McapReader`\n / `UrdfTree`.\n- **`Chunk.from_columns` for data a reader already decodes** (`Pinhole`\n intrinsics, `VideoStream`, `Transform3D` from a transforms topic) → keep it in\n the reader stream; fix with a `MutateLens` if needed.\n\nA wall of `pyarrow.compute` \"missing-attribute\" type errors (`pc.filter`,\n`pc.list_element`) usually means `pc.*` calls sit in module-level helpers instead\nof inside `Selector.pipe` lens callbacks. Refactor into a lens before suppressing\nthe checker — the errors are a smell that the hand-building should not exist.\n\n**Porting an existing converter?** Hand-built converters predate decoder\nimprovements and are not ground truth. Re-verify the decoder output (step 2) and\ncheck every `Chunk.from_columns` / for-loop against this list before copying.\n\n## Core model\n\n- `LazyChunkStream` is a lazy pipeline DAG, not a collection. Building\n filters, lenses, maps, splits, and merges reads no source data.\n- Execution starts at terminal calls: `write_rrd(...)`, `collect()`,\n `to_chunks()`, or iterating the stream.\n- Execution is streaming, multithreaded, and mostly GIL-free. Prefer\n stream/lens operations and PyArrow compute over Python row loops.\n- **Move semantics**: builder calls (`filter`, `drop`, `lenses`, `map`,\n `flat_map`) consume the input stream; reusing a consumed stream raises.\n Reassign after each step. Terminal calls do not consume, but each terminal\n call re-executes the whole pipeline; `collect()` once if that is too costly.\n- `ChunkStore` is materialized in memory (`stream.collect()`,\n `ChunkStore.from_chunks`). `LazyStore` is manifest-indexed, loads chunks on\n demand (`RrdReader(path).store()`, catalog segment stores). Both have\n `schema()`, `summary()`, `stream()`, and `write_rrd(...)`.\n\n## Stream composition\n\n```python\nfrom rerun.chunk import Chunk, LazyChunkStream, OptimizationProfile\n```\n\n- `stream.filter(content=, has_timeline=, is_static=, components=)` keeps the\n matching portion of each chunk; `stream.drop(...)` is its complement, same\n keyword filters. `content` takes an entity-path glob or a list of them.\n- `stream.map(fn)` applies `Chunk -> Chunk`; `stream.flat_map(fn)` applies\n `Chunk -> Iterable[Chunk]`. Escape hatches for chunk-level Python logic;\n prefer lenses for columnar work.\n- `stream.split(content=, ...)` returns `(matching, non_matching)`; both\n branches share the same upstream.\n- `LazyChunkStream.merge(*streams)` fans in any number of sources.\n- `LazyChunkStream.from_iter(chunks)` wraps hand-built chunks.\n\n```python\nstream = source_stream() # any importer skill\nstream = stream.drop(content=\"/video_raw/**\")\nstream = stream.lenses(fix_lens, content=\"/cam/**\", output_mode=\"forward_unmatched\")\nmerged = LazyChunkStream.merge(stream, sidecar_stream)\nmerged.write_rrd(out_path, application_id=\"my_app\", recording_id=recording_id)\n```\n\n## Hand-built chunks — sidecar only\n\nUse `Chunk.from_columns` ONLY for data no reader or lens can emit — JSON/CSV\ncalibration, frame offsets, external metadata. If a reader\n(`McapReader`/`UrdfTree`/`ParquetReader`/`Hdf5Reader`) decodes the topic or a\nlens can derive it, that is the idiomatic path; do not hand-assemble it here.\nIn the `robot_data_preprocessing` example the _only_ hand-built chunk is the JSON\noffsets sidecar; the camera fix, FK→`/tf`, meshes, and recolor are all\nreaders + lenses.\n\n`Chunk.from_columns(entity_path, indexes, columns)` mirrors\n`rr.send_columns(...)` and accepts the same archetype `.columns(...)` helpers.\nEmpty `indexes` means static.\n\n```python\nchunk = Chunk.from_columns(\n \"/tf_static/robot_offsets\",\n indexes=[], # static\n columns=rr.Transform3D.columns(\n translation=translations,\n quaternion=quaternions_xyzw,\n parent_frame=parents,\n child_frame=children,\n ),\n)\nsidecar_stream = LazyChunkStream.from_iter([chunk])\n```\n\n`rr.AnyValues.columns(...)` covers non-standard metadata fields. For\ninspection, a `Chunk` exposes `entity_path`, `num_rows`, `is_static`,\n`timeline_names`, `to_record_batch()`, and `format()` (human-readable table).\n\n### Recording properties\n\nRecording properties are the one exception here: do NOT hand-build them.\n`Chunk.from_property(name, values)` mirrors `rr.send_property` and lands a\nstatic chunk on `/__properties/`.\n\nDo this:\n\n```python\nchunk = Chunk.from_property(\n \"episode\",\n rr.AnyValues(task=\"pick\", duration_sec=12.4),\n)\n```\n\ninstead of hand-assembling the same chunk:\n\n```python\nChunk.from_columns(\n f\"/__properties/{name}\",\n indexes=[],\n columns=rr.AnyValues.columns(**{name: [value]}), # type: ignore[arg-type]\n)\n```\n\n- `/__properties/` is Rerun's internal layout; the second version hardcodes it.\n- A dynamic field name makes `AnyValues.columns()` need a\n `# type: ignore[arg-type]`. That suppression is the signal to switch.\n\n## Lenses\n\nLenses reshape, fix, or derive components without iterating rows. Apply with\n`stream.lenses(lenses, output_mode=..., content=...)`.\n\n- `MutateLens(component, selector, keep_row_ids=False)` modifies an existing\n component in place.\n- `DeriveLens(component, output_entity=None, scatter=False)` creates new\n columns, optionally at another entity. Chain `.to_component(descriptor,\nselector)` per output; `.to_timeline(name, \"sequence\" | \"duration_ns\" |\n\"timestamp_ns\", selector)` extracts a time column from the data itself.\n `scatter=True` explodes one input row into N output rows (one per list\n element).\n- Scope with `content=` whenever the same component name exists under multiple\n entities.\n\nOutput modes, and **the default is `drop_unmatched`**:\n\n- `drop_unmatched` (default): only lens outputs survive. Right for derive-only\n intermediate streams; silently discards everything else if applied broadly.\n- `forward_unmatched`: lens outputs plus the original components no lens\n consumed. Right for targeted fixes that preserve the rest of the stream.\n- `forward_all`: lens outputs plus every original component, including a\n `DeriveLens` input. Same as `forward_unmatched` for mutate lenses.\n\nIn-place fix (keep Arrow type and length intact):\n\n```python\nstream = stream.lenses(\n MutateLens(\n \"Pinhole:resolution\",\n Selector(\".\").pipe(\n lambda res: pa.array(\n [(h, w) for w, h in res.to_pylist()],\n type=res.type,\n )\n ),\n ),\n content=[\"/external/cam_low\", \"/external/cam_high\"],\n output_mode=\"forward_unmatched\",\n)\n```\n\nDerive with unit conversion (PyArrow compute, no Python loop):\n\n```python\nDeriveLens(\"schemas.proto.JointState:message\", output_entity=\"/joints_deg/waist\").to_component(\n rr.Scalars.descriptor_scalars(),\n Selector(\".joint_positions\").pipe(lambda arr: pc.multiply(pc.list_element(arr, 0), 180.0 / math.pi)),\n)\n```\n\n## Selector grammar\n\n`Selector(\"\")` navigates nested Arrow data, jq-style:\n\n- `.` current value; `.field` struct field\n- `[]` iterate list elements; `[N]` index a list\n- `?` suppress errors / skip missing optionals; `!` assert non-null\n- `|` pipe one expression into another\n\n`.pipe(fn)` chains a Python/PyArrow transform (or another Selector).\n`.execute(array)` runs it eagerly; `.execute_per_row(array)` guarantees the\noutput row count matches the input (use inside lens callbacks that must stay\nrow-aligned).\n\n## Writing RRDs\n\n- `stream.write_rrd(path, application_id=..., recording_id=...)` executes and\n writes in one streaming pass.\n- `stream.collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(...)`\n materializes, optimizes chunk layout, then writes. Memory scales with the\n materialized chunks.\n- Profiles: `OBJECT_STORE` (large chunks, for storage/query/catalog) and\n `LIVE` (small chunks, low-latency viewer).\n- Multiple physical RRDs form one logical recording when they share a\n `recording_id`; use this to separate base data, model/URDF data, and layers.\n\n**Always use `OptimizationProfile.OBJECT_STORE`** when the RRD is headed for a\nRerun catalog or Hub, unless explicitly asked otherwise.\n\n## Chunk API vs logging API\n\n- Logging (`rr.log`, `rr.send_columns`, `RecordingStream`) is for live logging\n from user code; chunk processing is for ingestion, conversion, and\n postprocessing existing recordings.\n- Logging → chunks: write an RRD, read it back with `RrdReader`.\n `RrdReader(path)` lists `recordings()` / `blueprints()` (each a `StoreEntry`\n with `kind`, `application_id`, `recording_id`); `.stream(store=entry)` for\n sequential passes, `.store(store=entry)` for indexed access.\n- Chunks → logging: `rerun.send_chunks(chunks, recording=...)`\n accepts a `Chunk`, `LazyChunkStream`, `LazyStore`, `ChunkStore`, or any\n iterable of chunks. The source store's `application_id`/`recording_id` are\n **not** preserved; the active recording's identity wins.\n\n## Common gotchas\n\n- The default lens `output_mode` is `drop_unmatched`; forgetting to set\n `forward_unmatched` on a targeted fix silently drops the rest of the stream.\n- Do not reuse a consumed `LazyChunkStream`; reassign or `split` deliberately.\n- Scope lenses with `content=`; the same component name often exists under\n many entities.\n- Preserve Arrow array type and length in `MutateLens` transforms.\n- For catalog layers, the layer `recording_id` must equal the segment id.\n- This is `rerun.chunk`; pin-check signatures when upgrading.\n\n## References\n\n- End-to-end example (MCAP + URDF + JSON sidecar, lenses, merge, optimize):\n `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- Docs: `https://rerun.io/docs/concepts/logging-and-ingestion/chunk-processing-api`,\n `https://rerun.io/docs/concepts/query-and-transform/lenses`\n","skills/rerun-parquet/SKILL.md":"---\nname: rerun-parquet\ndescription: Ingest tabular Parquet files into Rerun chunk streams with rerun.experimental.ParquetReader. Read when converting trajectory or sensor tables (LeRobot-style parquet, exported logs) into entities and components — column grouping, timeline/index columns, static columns, and lenses (DeriveLens) that assemble the typed components (Transform3D, Scalars) from the reader's grouped struct/scalar output. Builds on rerun-chunk-processing and rerun-data-model.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun parquet ingestion\n\n`ParquetReader` is a **pure reader**: it maps a flat table onto the Rerun\nmodel by turning raw columns into grouped, time-indexed chunks of struct and\nscalar components. Column-name prefixes become entities, grouped columns\nbecome a single struct component, designated columns become timelines. The\nreader does **not** assemble archetypes anymore — mapping struct fields into\ntyped Rerun components (Transform3D, Scalars, Points3D) is done with lenses on\nthe reader's `.stream()`. The whole reader job is configuration; fill in the\n`rerun-data-model` mapping table first, then express it through `stream()`.\nStream mechanics after `.stream()` are in `rerun-chunk-processing`.\n\n**The whole table is configuration, not code.** If you find yourself building\n`Chunk.from_columns` from a parquet, or munging it in pandas first, stop —\n`ParquetReader` plus a lens almost certainly expresses it. Anything the reader\ncannot express (per-row entity routing, derived values, unit conversion)\nbelongs in lenses downstream, not in pre-pandas munging; keep the pipeline\ncolumnar.\n\n## The API\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nreader = ParquetReader(table_path) # a lightweight handle over the file\nstream = reader.stream(\n entity_path_prefix=\"/world\", # prepended to every entity path\n column_grouping=\"prefix\", # \"prefix\" | \"individual\" | \"explicit_prefixes\"\n delimiter=\"_\", # split for column_grouping=\"prefix\"\n prefixes=None, # required for \"explicit_prefixes\"\n use_structs=True, # pack grouped columns into one struct component\n static_columns=[\"robot_type\"], # constant-per-file values, logged static\n index_columns=[IndexColumn.timestamp(\"timestamp\", input_unit=\"us\"), IndexColumn.sequence(\"frame_index\")],\n)\n```\n\nThe constructor takes only `path`; all loading options live on `stream()`, and every one is keyword-only.\nEach `stream()` call is independent, so one reader can drive several differently-configured streams over the same file.\nThe config is validated against the file's schema at `stream()` (missing index/static columns raise `ValueError` there, not mid-iteration).\nThere is no `column_rules` kwarg — typed-component assembly moved to lenses (below).\n\n## What the reader emits\n\nThe reader turns the table into chunks, one chunk per group, then leaves the\ndata as generic struct/scalar components for lenses to map. The naming is the\nkey thing the rest of the pipeline keys off:\n\n- **A grouped multi-column prefix `X`** → entity `/X`, with a single struct\n component named **`data`**. The struct's fields are the column names with the\n prefix (and delimiter, for `\"prefix\"`) stripped. So `A_pos_x`, `A_quat_w`\n under prefix `A` land as struct `data` with fields `pos_x`, `quat_w` on\n entity `/A`.\n- **A lone column with no group** → its own entity named after the column, and\n a raw component named after the column — *not* a `data` struct. So a `speed`\n column becomes entity `/speed`, component `speed`.\n- **A `/__properties` metadata chunk** built from the parquet file's schema\n metadata. You typically drop it right after `.stream()`:\n\n ```python\n stream = reader.stream().drop(content=\"/__properties/**\")\n ```\n\n## Column grouping: which columns share an entity\n\n- `\"prefix\"` (default): split each column name on `delimiter`, group by the\n first segment. `gripper_pos_x`, `gripper_pos_y` → entity `/gripper`, struct\n `data{pos_x, pos_y}`.\n- `\"explicit_prefixes\"`: group by the exact strings in `prefixes`, tried\n longest-first; the prefix is stripped from each struct field name (a raw\n string match, no delimiter — `foo` + `a` → field `a`). Columns matching no\n prefix become individual groups. Use this when names contain the delimiter\n ambiguously (`observation.state` vs `observation.images.top`: pass the full\n prefixes).\n- `\"individual\"`: every column is its own chunk/entity with a raw component\n named after the column — no struct packing at all, even for columns sharing a\n prefix. `use_structs` is ignored here. Rarely the model you want; reach for\n it only as a debugging baseline.\n\n`use_structs=True` (default) packs a group's columns into a single Arrow\nstruct component (the `data` field) for `\"prefix\"`/`\"explicit_prefixes\"`;\n`False` emits one component per column (the pre-struct flat layout, what\nqueries see as separate columns).\n\n## Timelines: `index_columns`\n\nEach entry is built with `IndexColumn`: `IndexColumn.timestamp(name, input_unit=...)` (since epoch), `IndexColumn.duration(name, input_unit=...)` (elapsed), or `IndexColumn.sequence(name)` (ordinal int).\n\n- `input_unit` describes what the raw integers in the column *are* (`\"ns\"` default,\n `\"us\"`, `\"ms\"`, `\"s\"`); Rerun rescales to ns internally. Sequences take no unit.\n\n**If omitted, a synthetic `row_index` sequence timeline is generated.** That\nis almost never the timeline you want to query or align against; always name\nthe real time columns. Stamp both a timestamp and a sequence timeline when the\ntable has both (multi-rate alignment, see `rerun-data-model`).\n\n## Static columns: `static_columns`\n\nListed columns are constant across all rows; they are emitted once as a single\nstatic (timeless) chunk, separate from the temporal data. A listed column that\nactually varies raises an error when the stream runs — that error is a\ndata-quality signal, not a reason to drop the static declaration.\n\n## Typed components via lenses\n\nThe reader's grouped output is generic struct (`data`) and scalar data. A\n`DeriveLens` reads that struct's fields, packs and casts them into real Rerun\ncomponents, and writes them to an output entity — this is what the old\n`column_rules` API used to do, now done downstream on the stream.\n\nConstruct a lens against the reader's struct component (`\"data\"` for grouped\nprefixes, or the column name for a lone/individual column), then add one or\nmore `.to_*` builder methods. Each builder returns a fresh lens, so they chain.\n\n| Builder | Produces | Argument order |\n|---|---|---|\n| `to_translation(x, y, z)` | `Transform3D:translation` | x, y, z |\n| `to_quaternion(x, y, z, w)` | `Transform3D:quaternion` | x, y, z, w (xyzw) |\n| `to_scale(x, y, z)` | `Transform3D:scale` | x, y, z |\n| `to_rotation_axis_angle(axis_x, axis_y, axis_z, angle)` | `Transform3D:rotation_axis_angle` | axis_x, axis_y, axis_z, angle (radians) |\n| `to_scalars(*fields)` | `Scalars:scalars` | one or more field names |\n| `to_packed_component(component, *fields)` | the given component | descriptor, then field names |\n| `to_component(component, selector)` | the given component | descriptor, then a `Selector` |\n| `to_timeline(name, type, selector)` | a timeline (not a component) | name, `\"sequence\"`/`\"duration_ns\"`/`\"timestamp_ns\"`, selector |\n\n`to_packed_component` packs the named struct fields (in order, at least one\nrequired) into the fixed-size list the component expects, and by default\n**auto-casts `f64`→`f32`** to match component types. The `to_translation`,\n`to_quaternion`, `to_scale` helpers are convenience wrappers over it, so they\nall auto-cast. `to_rotation_axis_angle` builds a `Struct{axis, angle}` and\nhard-casts axis and angle to `f32` internally. `to_scalars` with a single field\nemits a plain scalar per row (not a 1-element list); with multiple fields it\nemits one scalar series per field at the same entity.\n\nApply lenses with `.stream().lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")`:\n\n- `content` is a pre-filter on the *source* entity path — it scopes which\n chunks the lens may touch. Out-of-scope chunks pass through unchanged. Set it\n to the reader's grouped entity (e.g. `\"/A\"`).\n- `output_mode` decides the fate of in-scope-but-unmatched chunks:\n `\"drop_unmatched\"` (default, keep only lens output), `\"forward_unmatched\"`\n (output replaces matched, other originals survive), or `\"forward_all\"`\n (output plus all originals).\n- The lens's own `output_entity=` sets the *destination* entity — independent\n of `content`, which gates the input side.\n\nEnd-to-end Transform3D example. The reader groups `A_*` columns into a `data`\nstruct at `/A`; the lens reads the prefix-stripped field names (`pos_x`,\n`quat_w`), packs and casts them, and writes a full `Transform3D` to `/pose`:\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nlens = (\n DeriveLens(\"data\", output_entity=\"/pose\")\n .to_translation(\"pos_x\", \"pos_y\", \"pos_z\")\n .to_quaternion(\"quat_x\", \"quat_y\", \"quat_z\", \"quat_w\")\n)\n\nchunks = (\n ParquetReader(table_path)\n .stream(index_columns=[IndexColumn.sequence(\"frame_index\")])\n .lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")\n .to_chunks()\n)\n```\n\nChaining several `.to_*` on **one lens with a shared `output_entity`**\naccumulates multiple component columns into the same archetype at that entity —\nabove, both `Transform3D:translation` and `Transform3D:quaternion` land on\n`/pose`, forming a complete `Transform3D`. For a generic fixed-size-list\ncomponent, pass the descriptor to `to_packed_component`:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens\nfrom rerun.experimental import ParquetReader\n\nlens = DeriveLens(\"data\", output_entity=\"/points\").to_packed_component(\n rr.Points3D.descriptor_positions(), \"x\", \"y\", \"z\"\n)\n```\n\n## Selectors\n\nLens field paths use `Selector`, a jq-like grammar over Arrow columns\n(`.field` to access a struct field, `[]` to iterate a list, `[N]` to index, `?`\nto suppress errors on absent fields, `!` to assert non-null, `|` to pipe, and\n`pack(.x, .y, .z)` to zip paths into a fixed-size list). The `to_*` helpers\nbuild these selectors for you; reach for `to_component(component, Selector(\".x\"))`\nwhen you need a custom field path. Field paths reference the\n**prefix-stripped** struct field names — the lens sees `pos_x`, not `A_pos_x`.\n\n## Gotchas\n\n1. No `index_columns` → synthetic `row_index` timeline only. Queries that\n expect a timestamp timeline find nothing.\n2. The `unit` is the raw column's unit, not a desired output unit; a\n microsecond column declared `\"ns\"` lands 1000x in the past.\n3. `static_columns` raises if a listed column actually varies; that error is a\n data-quality signal, not a reason to drop the static declaration.\n Uniformity is data-dependent, so it is raised lazily when the stream runs;\n a listed column that doesn't *exist* raises eagerly at `stream()`.\n4. A grouped prefix's struct component is named **`data`** — that is the\n `input_component` string a `DeriveLens` matches against. A lone or\n `\"individual\"` column is instead a raw component named after the column.\n5. Selector field paths reference the **prefix-stripped** struct field names\n (`pos_x`, not `gripper_pos_x`).\n6. Drop the `/__properties` metadata chunk the reader emits from parquet schema\n metadata: `.stream().drop(content=\"/__properties/**\")`.\n7. Quaternion column order is x, y, z, w in `to_quaternion`; check the source's\n convention before wiring fields.\n8. `to_packed_component` (and the transform helpers built on it) auto-casts\n `f64`→`f32` to match component types; this is usually what you want for\n parquet's double columns.\n9. Anything the reader cannot express (per-row entity routing, derived values,\n unit conversion) belongs in lenses downstream, not in pre-pandas munging;\n keep the pipeline columnar (`rerun-chunk-processing`).\n\n## References\n\n- Lens builder source with full docstrings: `rerun/chunk/_lens.py` in\n the installed `rerun-sdk` package (`to_translation`, `to_quaternion`,\n `to_scale`, `to_rotation_axis_angle`, `to_scalars`, `to_packed_component`,\n `to_component`, `to_timeline`).\n- Reader source: `rerun/experimental/_parquet_reader.py`, or\n `python -c \"from rerun.experimental import ParquetReader; help(ParquetReader)\"`\n- Canonical worked examples: the integration tests\n `rerun_py/tests/integration/test_parquet_reader.py` (grouping, index/static\n columns, and the Transform3D / Points3D / Scalars lens flows) and\n `rerun_py/tests/integration/test_lazy_chunk_stream.py` (lens application,\n `content`/`output_mode`, selectors).\n- `rerun-lerobot` — LeRobot datasets store episodes as parquet; that skill\n covers the built-in importer route vs reading the parquet directly with\n this reader.\n- `rerun-data-model` (mapping decisions), `rerun-chunk-processing` (stream\n mechanics after `.stream()`)\n","skills/rerun-blueprint/SKILL.md":"---\nname: rerun-blueprint\ndescription: Change how the Rerun viewer looks by writing a blueprint, for a recording or for a table. Read this whenever asked to change the look of the viewer, such as laying out a recording or dataset, designing a default blueprint, deciding which views show which entities, or making a segment or catalog table look nice by hiding columns, changing column order or filtering rows. Covers archetype-to-view mapping, layout reasoning, the rrb construction API, the contents grammar, table blueprints, and the screenshot loop.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun blueprint\n\n**To change how the viewer looks, change the blueprint.** This holds for a\nrecording's layout, views and their properties, and for a table's columns, row\nfilters and cards. Don't click through the viewer's menus to do it. With the\nviewer MCP, read and write the blueprint with `rerun_get_blueprint` and\n`rerun_set_blueprint` rather than the low-level click tools.\n\nA blueprint decides how a recording is shown, and the recording decides what exists.\nRead the data, design a first layout, then **iterate from screenshots** until it\nreads at a glance. The constructors are discoverable (`dir(rrb)`,\n`help(rrb.Spatial3DView)`); this skill covers what you can't guess. Import as\n`import rerun.blueprint as rrb`.\n\n## Tables\n\nA dataset's segment table and catalog tables have a table blueprint of their\nown, separate from a recording's. It sets which columns show, their order and\nnames, the row filters, and whether rows show as a table or as cards. When asked\nto make a table look nice, hide columns or filter rows, write this blueprint.\n\nThis hides two columns and shows three others first. Listed columns come first,\nthe rest keep their default order and visibility:\n\n```python\nblueprint = rrb.TableBlueprint(\n table_layout=rrb.table.TableLayout(\n columns=[\n rrb.table.Column(\"recording link\"),\n rrb.table.Column(\"property:episode:current_task\", name=\"Task\"),\n rrb.table.Column(\"property:episode:success\"),\n rrb.table.Column(\"property:episode:ext1_cam_serial\", visible=False),\n rrb.table.Column(\"property:episode:ext2_cam_serial\", visible=False),\n ],\n ),\n)\nblueprint.save(\"my_app\", path)\ndataset.register_blueprint(path.as_uri(), segment_table=True)\n```\n\n`Column` takes the column's name as the table has it, such as\n`property:episode:uuid`, not its header. Registering makes it the default for\neveryone who opens the table, and the server must be able to read `path`.\n\nRow filters are SQL expressions, one per string, all of which must be true for a\nrow: `rrb.TableBlueprint(filters=[\"\\\"task\\\" ILIKE '%pick%'\", '\"duration\" > 10'])`.\nDouble quotes for column names, single quotes for strings, and\n`any_match(\"column\", x -> …)` for list columns.\n\nWith the viewer MCP, change the table on screen directly:\n`rerun_get_table_schema` names the columns, `rerun_get_blueprint` with\n`{\"table\": {}}` reads the table blueprint, and `rerun_set_blueprint` with\n`{\"table\": {}}` replaces it. `rerun_set_blueprint` returns an error for a filter\nthat fails. A column's entry sits at `/table/layouts/table/columns/`,\nwith each `:` and space in the name escaped as `\\:` and `\\ `, which JSON writes\nas `\\\\:` and `\\\\ `. The [MCP docs](https://rerun.io/docs/reference/viewer/mcp#table-blueprints)\nhave a worked example of the JSON. For the whole Python API, including cards\nand segment previews, see\n[Configure table layouts and segment previews](https://rerun.io/docs/howto/visualization/configure-table-blueprints).\n\n## 1. Read the data\n\nEnumerate every `(entity_path, archetype)` pair first — the archetype picks the\nview, the entity path scopes it. For a dataset (see `rerun-catalog-queries`):\n\n```python\nfor c in dataset.schema().component_columns():\n print(c.entity_path, c.archetype, c.component_name)\n```\n\nFor a local recording, stream it with `RrdReader` and read `entity_path` and\nthe archetype off each chunk (see `rerun-chunk-processing`).\n\nThe dataset schema alone doesn't list every entity a view can show. Assets are\nstatic data such as URDF robot meshes, which the viewer applies to every segment,\nbut they don't appear in the segment entities. When `rerun_get_dataset_schema`\nlists `asset_ids` next to `entities`, call `rerun_get_dataset_assets_schema` to\nget the entities in each asset, and read it before choosing between a 2D and a\n3D view or preview. For example, `asset_ids: [\"urdf-model-left\", \"urdf-model-right\"]`\nmeans the robot models can go in a `Spatial3DView`.\n\n## 2. Map archetype to view\n\n| Archetype family | View |\n| ------------------------------------------------------------------------------------- | --------------------------------------------------- |\n| `Points2D`, `Image`, `EncodedImage`, `Boxes2D`, `LineStrips2D`, `Pinhole` projections | `Spatial2DView` |\n| `Points3D`, `Mesh3D`, `Boxes3D`, `LineStrips3D`, `Transform3D`, `Asset3D` | `Spatial3DView` |\n| `Scalars`, `SeriesLines`, `SeriesPoints` | `TimeSeriesView` |\n| `TextLog` / `TextDocument` / `BarChart` | `TextLogView` / `TextDocumentView` / `BarChartView` |\n| `Tensor`, `DepthImage` (heatmap) | `TensorView` |\n| `GeoPoints`, `GeoLineStrings` | `MapView` |\n| `GraphNodes`, `GraphEdges` | `GraphView` |\n| tabular / catalog data | `DataframeView` |\n\nAsset entities, such as `Asset3D` and URDF meshes, go in a `Spatial3DView`.\n\n## 3. Group entities into views\n\nThis step is just deciding which entities share a view. Sizing and placement are\nmechanics you tune by eye in the screenshot loop, not rules to settle up front.\n\n- **Group by shared path prefix.** Entities under a common prefix usually belong\n in one view. Don't give every entity its own pane: dozens of raw message\n entities go in one `DataframeView`, or get left out, rather than a pane each.\n- **One spatial frame per spatial view.** A whole `Transform3D` tree goes in one\n `Spatial3DView` at the common ancestor; a `Pinhole` camera gets its own\n `Spatial2DView` rooted at the camera so images inherit the projection.\n- **Collapse scalars.** Many `Scalars` under a prefix → one `TimeSeriesView` over\n the prefix, not one each. Split only when value ranges clash.\n- **`column_shares`/`row_shares` are relative weights** on a container's\n children, equal by default. Tune them from the screenshots.\n\n## 4. Construct it\n\nContainers (`Grid`, `Horizontal`, `Vertical`, `Tabs`) hold views. Default a flat\nset to `Grid`; use the others for a deliberate split, `Tabs` for alternatives\ncompeting for one slot (left/right/depth cameras). **Always set an explicit\n`origin` and `name`** — `origin` defaults to `/`, which dumps the whole tree into\none view (the usual cause of an unreadable blob).\n\n```python\nblueprint = rrb.Blueprint(\n rrb.Horizontal(\n rrb.Spatial3DView(origin=\"/world\", name=\"Scene\"),\n rrb.Vertical(\n rrb.Spatial2DView(origin=\"/world/camera\", name=\"Camera\"),\n rrb.TimeSeriesView(origin=\"/sensors\", name=\"Sensors\"),\n ),\n column_shares=[3, 2],\n ),\n rrb.TextLogView(origin=\"/logs\", name=\"Logs\"),\n collapse_panels=True,\n)\n```\n\n`contents` defaults to `\"$origin/**\"`. Scope a view with include/exclude rules,\ne.g. `contents=[\"+ $origin/**\", \"- $origin/internal/**\"]`. A bare line is an\ninclude; `/**` is the only wildcard (matches a subtree). Most-specific rule wins,\nties go to the last, unmatched paths are excluded.\n\n**Coordinate frames.** A spatial view only renders entities it can place relative\nto its target frame. `Transform3D`/`Pinhole` logged on entities compose down the\ntree, so `origin` is enough. But **named frames** (`CoordinateFrame(frame=...)`,\ncommon in ROS/MCAP) live in a separate frame graph — point the view at a frame\nthe data occupies via `spatial_information=rrb.SpatialInformation(target_frame=\"\", apply_latest_at=True)`.\n- A view back blank? The importer may not have finished (bump the settle) or the\n cursor sits before the data (add `rrb.TimePanel(play_state=rrb.components.PlayState.Following)`).\n- `rrb` views and `ViewerClient` (`rerun.experimental`) are unstable. Check\n `help()` if a constructor argument is rejected.\n\n## See also\n\n- `rerun-data-model` — entities, archetypes, timelines.\n- `rerun-catalog-queries` — enumerate entities in a dataset.\n","skills/rerun-mp4/SKILL.md":"---\nname: rerun-mp4\ndescription: Ingest .mp4 video into Rerun chunk streams with rerun.experimental.Mp4Reader. Read when converting video into a VideoStream, choosing stream vs asset mode, transcoding (B-frames, output codec, GOP size) through FFmpeg, or aligning video PTS onto a recording's wall-clock timeline. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (where the video belongs in the recording).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun mp4 ingestion\n\n`Mp4Reader` turns one `.mp4` file into a lazy chunk stream on **one entity**:\ncompressed video samples, no decode to pixels, nothing re-encoded unless it has\nto be. Everything is configured on the constructor; `stream()` takes no\narguments. Stream mechanics after `.stream()` (filter, map, merge, collect,\nwrite) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.experimental import Mp4Reader, Mp4TranscodeOptions\n\nreader = Mp4Reader(video_path, entity_path=\"/camera/front\") # mode=\"stream\" by default\nstream = reader.stream() # lazy: nothing is decoded yet\n```\n\nEvery parameter after `path` is keyword-only. One reader handles one file — for\nseveral cameras, build one reader per file and `LazyChunkStream.merge(...)` them\n(see below).\n\n## Two modes\n\n| `mode` | emits | when |\n| -------------------- | ------------------------------------------------------- | ----------------------------------------------------------------- |\n| `\"stream\"` (default) | a codec chunk, per-GOP sample chunks, a keyframe marker | almost always — every frame is time-indexed and queryable |\n| `\"asset\"` | the whole file as one blob, plus a frame index | the codec cannot be a `VideoStream`, or the source starts mid-GOP |\n\nReach for stream mode; the next section covers exactly what it emits. Asset mode\ncopies the entire file into the recording as one blob, so nothing downstream can\nlook at a single frame without the whole asset. It is the fallback for codecs a\n`VideoStream` cannot carry (`mp4v`, image-sequence mp4), and it is what `rerun\nvideo.mp4` and the built-in file importer use.\n\n## What stream mode emits\n\nFor `tests/assets/video/Big_Buck_Bunny_1080_1s_h264_nobframes.mp4` (30 frames,\none GOP) with `entity_path=\"/camera/front\"`, the whole output is three chunks:\n\n```\n[0] entity=/camera/front static=True rows=1 timelines=[] cols=['VideoStream:codec']\n[1] entity=/camera/front static=False rows=30 timelines=['video'] cols=['VideoStream:sample']\n[2] entity=/camera/front static=False rows=1 timelines=['video'] cols=['VideoStream:is_keyframe']\n```\n\n- **One static chunk, one row**, holding `VideoStream:codec`.\n It carries **no timeline** — code that walks the stream must\n handle that (`if chunk.is_static: continue`).\n- **One temporal chunk per GOP**: a keyframe plus every sample that depends on\n it, up to (not including) the next keyframe. Samples only — the keyframe flags\n are not a column here.\n- **One trailing `is_keyframe` marker chunk**, holding a *sparse* `True` row at\n each keyframe's time and nothing else. Keeping it out of the sample chunks is\n what lets a keyframe-only query skip the sample payload, and what\n `collect(optimize=…)` accepts as canonical. It is list-per-row, so a row reads\n back as `[True]`, not `True`.\n- **Timeline `video`**, `duration[ns]`, values are the mp4 PTS from the start of\n the video.\n- **Every chunk on the same entity.** `entity_path=None` (the default) derives\n it from the file's *absolute* path, so `foo/video.mp4` run from `/data`\n becomes `/data/foo/video.mp4`. Pass `entity_path` explicitly in any real\n pipeline.\n\nA short clip is often a single GOP, so \"three chunks total\" is the normal shape —\nnot a sign that something was dropped. `chunk_by_gop=False` emits one Rerun\nchunk per sample instead (the marker chunk is unaffected). That is a debugging\nshape — one chunk per frame is a poor storage layout, and it also makes a\nfollowing optimize pass **6× more expensive** (measured below) — so leave the\ndefault alone unless you are inspecting individual samples.\n\nDistinguish the three by their columns, not by position or `is_static`: the codec\nchunk is the static one, sample chunks carry `VideoStream:sample`, and the marker\nchunk carries only `VideoStream:is_keyframe`.\n\nSupported stream-mode codecs are exactly the five `VideoCodec` values: H264,\nH265, AV1, VP8, VP9.\n\n## Transcoding through FFmpeg\n\n`VideoStream` cannot yet model DTS != PTS, so an H.264/H.265 source with\ncontainer-level B-frame reordering **cannot be emitted directly**. The reader\nhandles that itself: it re-encodes through FFmpeg with `-bf 0`, streams the\nresult back as a fragmented mp4, and turns each fragment into one GOP chunk, so\nonly one GOP is resident at a time. This is automatic and invisible in the API —\nthe output has the same shape as a clean source. It does need an `ffmpeg`\nexecutable. Asset mode is unaffected by B-frames.\n\n`Mp4TranscodeOptions` (stream mode only) additionally *requests* a transcode:\n\n| field | effect |\n| ----------------- | -------------------------------------------------------------------------- |\n| `output_codec` | re-encode to another `VideoCodec`; the emitted `VideoStream:codec` follows |\n| `gop_size` | force a keyframe every N frames — the knob for seek cost in the viewer |\n| `try_gpu` | best-effort hardware encode, NVENC / VideoToolbox only |\n| `ffmpeg_override` | use this `ffmpeg` instead of the one on `PATH` |\n\nRequesting the `output_codec` the source already uses is a **no-op**: it stays on\nthe direct, no-FFmpeg path. With `chunk_by_gop=True`, `gop_size=N` makes every\nGOP chunk but the last hold exactly N samples — a 30-frame clip at `gop_size=10`\ngives 10, 10, 10. `try_gpu` realistically covers H264/H265 plus AV1 on newer\nNVIDIA; VP8/VP9 always fall back to software, and it does nothing unless a\ntranscode is already happening.\n\n```python\nMp4Reader(\n video_path,\n entity_path=\"/camera/front\",\n # ~1s GOPs on a 60fps source, so seeking in the viewer stays snappy.\n transcode=Mp4TranscodeOptions(gop_size=64),\n).stream()\n```\n\n## Timelines: mp4 PTS is not your recording's clock\n\nThe emitted times are always PTS — elapsed nanoseconds from the start of *that\nvideo*. In a multi-sensor recording, that is almost never the timeline you want\nto align on.\n\n- `timeline_name=\"real_time\"` renames the timeline.\n- `timeline_type=\"timestamp\"` only **retypes** the same PTS values as\n nanoseconds since the Unix epoch. The reader does not shift them, so on its\n own it renders the video near 1970. It is meaningful only paired with a retag\n step.\n- **Retag with `stream.map(...)`.** Samples arrive in presentation order\n (B-frames are already stripped), so a running cursor maps sample `i` to\n `capture_times_ns[i]` — but the cursor must skip the keyframe marker chunk,\n which holds no samples. Map that one through the PTS the samples were already\n assigned:\n\n```python\nSAMPLE_COL = \"VideoStream:sample\"\n\n\ndef _reindex_to_capture_times(stream, capture_times_ns, timeline_name):\n cursor = 0\n pts_to_time = {}\n\n def _retag(chunk):\n nonlocal cursor\n if chunk.is_static: # the codec chunk carries no timeline\n return chunk\n batch = chunk.to_record_batch()\n col_index = batch.schema.get_field_index(timeline_name)\n old_field = batch.schema.field(col_index)\n old_pts = np.asarray(batch.column(col_index).cast(pa.int64()))\n\n if SAMPLE_COL in batch.schema.names:\n # Clamp in case the decoder yields a slightly different frame count.\n indices = np.clip(np.arange(cursor, cursor + chunk.num_rows), 0, len(capture_times_ns) - 1)\n cursor += chunk.num_rows\n new_times_ns = capture_times_ns[indices]\n pts_to_time.update(zip(old_pts.tolist(), new_times_ns.tolist()))\n else:\n # The sparse keyframe marker, emitted after every sample chunk.\n new_times_ns = np.array([pts_to_time[pts] for pts in old_pts.tolist()], dtype=np.int64)\n\n times = pa.array(new_times_ns.astype(\"datetime64[ns]\"))\n # `metadata=` is load-bearing — see gotcha 3.\n new_field = pa.field(old_field.name, times.type, nullable=old_field.nullable, metadata=old_field.metadata)\n return Chunk.from_record_batch(batch.set_column(col_index, new_field, times))[0]\n\n return stream.map(_retag)\n```\n\nThe cursor makes this order-dependent, so keep the retag on the single reader's\nstream, before any merge. This is the DROID loader's pattern — see the\nreferences.\n\nAny `map`/`flat_map` that assumes \"every non-static chunk is samples, in order\"\nhas this same bug: it advances over the marker chunk and stamps it with whatever\ntime comes next, silently moving the keyframe markers off their samples.\n\n## Several cameras into one recording\n\nOne reader per file, distinct entity paths, then merge:\n\n```python\nstreams = [\n Mp4Reader(path, entity_path=f\"/camera/{name}\", timeline_name=\"real_time\", timeline_type=\"timestamp\").stream()\n for name, path in cameras.items()\n]\n(\n LazyChunkStream\n .merge(*streams)\n .collect(optimize=OptimizationProfile.OBJECT_STORE)\n .write_rrd(out_path, application_id=app_id, recording_id=segment_id)\n)\n```\n\n`OBJECT_STORE` GOP-rebatches the video and preserves the reader's keyframe\nmarker as-is; no `fix_keyframe=True` is needed. If you *do* see\n`skipping GoP rebatching … is_keyframe data is incorrect`, something upstream\nrewrote the marker (see gotcha 4) — `fix_keyframe=True` re-derives it from the\nencoded samples as an escape hatch.\n\n## How the reader's chunks relate to optimize's\n\nThe reader and `collect(optimize=…)` agree on **where GOPs start** but not on\n**how many GOPs share a chunk**, and that is by design:\n\n- The reader emits the **finest GOP-aligned partition**: exactly one chunk per\n GOP. It cannot do better, because it does not know which profile the data is\n headed for.\n- Optimize then applies the size policy, **merging consecutive GOPs** up to the\n profile's `max_bytes` / `max_rows`. It never splits a GOP across chunks, so\n every boundary it keeps is one the reader already produced.\n\nSo optimize's partition is a pure *coarsening* of the reader's: same samples in\nthe same order, every optimized chunk a run of whole reader GOPs, keyframe marker\nuntouched. Where each GOP already sits near the profile's budget the two come out\nidentical (a 30-frame clip at `gop_size=10` is 3 chunks either way). Where GOPs\nare small they diverge sharply — a 12-GOP clip becomes **1** chunk under\n`OBJECT_STORE` and **4** under `LIVE`.\n\nThe practical consequence: **do not pre-merge or re-chunk the reader's output**\nto \"help\" optimize. Hand it the per-GOP stream and let the profile decide. And if\nyou skip optimize entirely (writing straight through `send_chunks`, as the DROID\nloader does), you are storing the finest partition — correct, but more chunks\nthan object storage wants.\n\n### The optimize pass is cheap on GOP-chunked input\n\nRunning optimize over the reader's output is not redundant work worth avoiding.\nOn an 89 MB / 1800-frame / 60-GOP H.264 file:\n\n| | `collect()` | `collect(optimize=OBJECT_STORE)` | delta |\n| ----------------------------- | ----------- | -------------------------------- | ------------------- |\n| `chunk_by_gop=True` (default) | 11.1 ms | 16.3 ms | **+5.2 ms** (1.5×) |\n| `chunk_by_gop=False` | 14.2 ms | 48.6 ms | **+34.4 ms** (3.4×) |\n\nNeither half of the pass is expensive on GOP-aligned input:\n\n- **Detecting GOP starts is header-only.** `build_sample_index` runs\n `detect_gop_start` on every sample, but that reads a few bytes of each sample's\n header — it never decodes. The whole optimize delta above (5.2 ms) is smaller\n than a Python loop calling the same detector over the same 1800 samples\n (6.0 ms), which is mostly FFI overhead.\n- **Rebuilding the chunks copies nothing.** `chunk_from_gop` calls\n `taken(0..n)`, and `re_arrow_util::take_array` returns the array **uncopied**\n when the indices are consecutive from zero over the whole array. One chunk per\n GOP means that is always the case, so the multi-MB sample buffers are never\n duplicated — hence 89 MB \"rebuilt\" in 5 ms.\n\nThat fast path is exactly what `chunk_by_gop=False` gives up: each GOP then spans\n30 source chunks, so `chunk_from_gop` has to `concat_and_sort` them and the sample\nbytes really are copied. `LIVE` also costs more than `OBJECT_STORE` (+13.2 ms vs\n+5.2 ms on the same file), because its much smaller chunk budget makes it attempt\ncompaction it cannot complete — a GOP is never split.\n\nRetag each camera *before* the merge — the retag above walks chunks with a\ncursor, so it must see one video's chunks in order.\n\n## Gotchas\n\n1. **Errors surface on the first pull, not at construction.** The constructor\n only checks that the file exists and validates its arguments; `stream()`\n builds a lazy pipeline and also succeeds. Codec support, keyframe layout, and\n FFmpeg availability are checked when the stream runs, so wrap the\n *consumption* (`to_chunks()`, `send_chunks`, iteration), not the\n `Mp4Reader(…)` call.\n2. **A codec outside the five `VideoCodec` values cannot be a `VideoStream`.**\n `mp4v` (MPEG-4 Part 2) is the one you will actually meet in robot datasets;\n it raises `RuntimeError: MP4 error: MP4 demux: Video track uses unsupported\ncodec \"mp4v\"`. Asset mode does accept the file, but only partly: the blob\n chunk is emitted and the `VideoFrameReference` index chunk is **skipped with\n a warning**, because the frame timestamps cannot be read either — so you get\n the bytes and no timeline. Skipping that camera and recording the fact as a\n recording property (what the DROID loader does) is usually better than a\n timeline-less blob, and either beats failing a whole episode over one camera.\n3. **A `map`/`flat_map` that rewrites the time column must carry the field\n metadata over.** `pa.field(...)` without `metadata=old_field.metadata` drops\n `rerun:kind: 'index'`, and the rebuilt chunk silently becomes **static** —\n no error, no timeline, and the samples land outside time entirely.\n4. **The trailing marker chunk is temporal but holds no samples**, so any\n per-chunk logic keyed on `not chunk.is_static` will process it as if it were\n samples. Key on the `VideoStream:sample` column instead. See the retag above.\n5. **Asset mode is capped at ~2 GiB** by Arrow's i32 offsets, and duplicates the\n file's bytes into the RRD.\n6. `mode=\"asset\"` rejects both `chunk_by_gop=False` and `transcode=` with\n `ValueError` — those are stream-mode-only knobs.\n7. `timeline_type=\"timestamp\"` on its own shifts nothing. Without a retag, the\n video sits at the epoch.\n8. **The reader emits compressed samples, never pixels.** Thumbnails, CLIP\n embeddings, or anything else needing RGB have to decode the file separately\n (OpenCV, PyAV); the `VideoStream` chunks cannot supply them.\n9. Every `stream()` call re-decodes the file from scratch, and every terminal\n call re-runs the pipeline. `collect()` once when you need more than one pass.\n10. Handle the static codec chunk explicitly in any `map`/`flat_map` — it has no\n timeline and no samples, and blind indexing into a time column will fail on\n it.\n11. **Samples before the first keyframe** are rejected in stream mode (a decoder\n cannot start mid-GOP); use asset mode for such a file.\n\n## References\n\n- Canonical worked examples: `rerun_py/tests/integration/test_mp4_reader.py`\n (both modes, `chunk_by_gop`, entity paths, `timeline_type`, transcode\n transforms, and the error cases).\n- Rust core: `crates/data_flow/re_mp4_reader/` (`stream.rs` for the GOP/transcode\n path, `asset.rs` for the blob+index path), with\n `crates/data_flow/re_mp4_reader/tests/stream.rs` covering codec pairs and GOP\n spacing.\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-data-model` (where\n video, calibration, and thumbnails belong in the recording).\n","skills/rerun-data-model/SKILL.md":"---\nname: rerun-data-model\ndescription: \"How raw multimodal robot data maps onto the Rerun data model. Read FIRST, before modeling or converting a dataset — and whenever you are about to convert/ingest/preprocess robot data into an .rrd or build a Rerun recording, even if not asked for the data model. Resolves the entity-vs-component, property-vs-component-vs-layer, and static-vs-temporal decisions and routes to the mechanism (do it with readers and lenses, not hand-built chunks or per-message rr.log): rerun-chunk-processing and the importer skills rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun data model\n\nThe hard part of ingesting a dataset is the **modeling decision**, not the API call.\nGet the model right and any mechanism works; get it wrong and queries, views, and training all break.\nThis skill is just the decisions.\nFor mechanism details see `rerun-chunk-processing` (pipeline mechanics) and the importer skills it routes to: `rerun-mcap`,`rerun-urdf`,`rerun-parquet`, `rerun-mp4`, `rerun-lerobot`.\nFor exact signatures, the docs at `rerun.io/docs/concepts/logging-and-ingestion`.\n\n**Before writing conversion code, fill in the mapping table below.** It is the design, and a human can review it in seconds.\n\n## Pick the mechanism before you model the bytes\n\nModeling decides _what_ each datum becomes; this decides _how_ it gets there — and the default is **a reader + lenses, not hand-built chunks**:\n\n- **Does a reader exist for this source?** MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`, LeRobot dir→`log_file_from_path`. **Yes →** `reader.stream()` + lenses; the reader produces the chunks, you do not.\n- **No, and it is genuine external metadata** (JSON calibration, offsets) or a specific custom use case that can't be covered by a generic reader**→** `Chunk.from_columns`.\n- **Otherwise**: consider if you are about to hand-build something a reader or lens should produce and ask for clarification.\n\nFor MCAP specifically: a Foxglove- or ROS-decodable file emits archetypes like `Transform3D`, `Pinhole`, and `VideoStream` for certain supported message schemas ready-made — pass those through, never re-derive them; only custom protobuf signal topics need lenses. The full decision tree and the anti-pattern list are in `rerun-chunk-processing`.\n\n## The model\n\n```\nDataset → Segment → Layer → Recording → Entity → Component → Chunk\n```\n\n- **Entity** = a thing, named by a path (`/robot/arm/camera`).\n **Component** = one typed field on it (positions, image, a scalar).\n **Archetype** = a standard bundle of components that log and render together (`Points3D`, `Image`, `Transform3D`).\n- Every value you log sits on two axes: **where** (entity path + component) and **when** (which timeline, or _static_ = all time).\n- **Segment** = one episode; on registration its `recording_id` becomes the `segment_id`. **Layer** = an extra `.rrd` on top of a segment. It attaches by matching the segment's `recording_id` and nothing else: that shared id is the whole layering mechanism.\n\n## The decisions\n\n**Entity vs component on an existing entity**\nNew entity if it has its own spatial frame (`Transform3D`/`Pinhole`), its own annotation context, or should be shown/cleared/shared independently (each robot link, each camera, each sensor). Same entity + extra component for auxiliary data at the same instances (per-point confidence).\nUse `AnyValues` for non-standard fields.\n\n**Property vs component vs layer** (the most common confusion)\n\n- **Component**: per-timestamp signal on the timeline (joint angle, image).\n- **Segment property**: one-per-episode metadata for catalog filter/search (operator, robot, site, task, date). Not per timestamp.\n- **Layer**: a whole derived `.rrd` over a base segment (FK transforms, point clouds, gripper state, labels, quality scores). Queryable as if part of base.\n\n**Static vs temporal**\nStatic belongs to all timelines and shadows any temporal value of the same component on the same entity. Use it for invariants (calibration, coordinate frames, robot meshes, annotation context, a video asset). Never make per-frame data static.\n\n**Which timeline**\nA `timestamp` timeline (ns since epoch) for cross-sensor clock alignment, a `sequence` timeline for frame/ordinal alignment; stamp on both when useful.\n**Do not resample to a common rate.** Latest-at reconciles multi-rate streams at query time by holding each component's last sample (no interpolation).\n\n**Base vs layer**\nBase = faithful conversion of the raw streams, nothing computed.\nLayer = anything derived (FK from URDF + joint states, clouds from depth + intrinsics).\nKeep them separate `.rrd`s.\n\n## The mapping table (produce before coding)\n\n| Source (topic/column/key) | Entity path | Archetype | Component(s) | Timeline | Static/temporal | Base/layer | Property? |\n| ------------------------- | -------------------- | ------------------------------------- | -------------------- | ------------- | --------------------------- | ---------- | --------- |\n| mcap `/joint_states` | `/robot/` | `Scalars` | `scalars` | `sensor_time` | temporal | base | no |\n| `cam0/color.mp4` | `/camera/cam0/video` | `VideoStream` | codec+sample | `video_time` | codec static, samples temporal | base | no |\n| `calibration.json` | `/camera/cam0` | `Pinhole` (+`Transform3D` extrinsics) | `image_from_camera` | — | static | base | no |\n| URDF + joints (computed) | `/robot/` on the CLI) at the dataset _directory_ and it ingests episodes, camera videos, and state/action tables with no conversion code.\n- **Converting?** `rr.experimental.LeRobotReader` reads the dataset one episode at a time as a lazy chunk stream, which is what you want for per-episode RRDs and for lenses.\n\nThe download step needs `huggingface_hub`.\n\n## Step 0: pick a dataset and check the HF token\n\nBoth scripts below live in `scripts/` next to this file, and run standalone through `uv` — they declare `huggingface_hub` inline, so there is no environment to set up.\n\nTwo facts decide what a dataset costs to fetch, and the Hub search results show neither.\nGet both, for every candidate, in one call:\n\n```bash\nuv run scripts/find_lerobot_dataset.py --robot-type --want 5\n```\n\n**`codebase_version` decides whether a partial download is possible at all.**\n\n- **v2.1 and earlier** store one parquet and one mp4 per episode (`video_path` ends in `episode_{episode_index:06d}.mp4`), so \"just the first N episodes\" downloads only those episodes.\n- **v3.0** concatenates every episode into a few large files (`video_path` ends in `file-{file_index:03d}.mp4`), so any subset costs the whole download.\n If the user asked for a few episodes, prefer a v2.1 dataset — for the _download_ only.\n `LeRobotReader` reads a single episode out of either version once the files are local.\n\n**How camera frames are stored decides the size**, and it varies by two orders of magnitude.\nThe `frames` and `MB/ep` columns report it:\n\n- `mp4`: a `video`-dtype feature, one file per episode beside the parquet.\n- `inline`: an `image`-dtype feature, raw frames inside the episode parquet, which is far larger for the same footage.\n\nBoth scripts report the actual size for the dataset in hand, so use that number rather than guessing.\nTell the user the size before you start the download, and prefer `mp4` unless they asked for something only the other set has.\n\n`HF_TOKEN` is not required for public datasets, but without it the Hub rate-limits and throttles downloads.\nIf it is unset, say so and point the user at to create a read token, then `export HF_TOKEN=hf_…` (or `hf auth login`).\nDo not block on it — carry on unauthenticated and let the user decide.\n\n```bash\n[ -n \"$HF_TOKEN\" ] || echo \"HF_TOKEN unset — downloads will be throttled\"\n```\n\n## Step 1: dataset -> one combined RRD\n\nOnly needed if you want every episode in one file — to hand someone a single RRD, or to reprocess data you cannot re-read from the dataset directory.\nFor per-episode segments, skip to Step 2 and let `LeRobotReader` read the dataset directly.\n\n```python\nfrom huggingface_hub import snapshot_download\nimport rerun as rr\n\ndataset_dir = snapshot_download(repo_id=repo_id, repo_type=\"dataset\", local_dir=dest)\n\nwith rr.RecordingStream(\"rerun_example_lerobot\") as rec:\n rec.save(str(combined_rrd))\n rec.log_file_from_path(str(dataset_dir)) # the built-in importer\n```\n\nThe importer emits one recording per episode (recording ids like `episode_1`), plus a metadata-only root recording, all into the single RRD.\n\n`rr.RecordingStream` + `log_file_from_path` here is the **importer bootstrap** — the one place `RecordingStream` is correct in an ingestion pipeline (it drives the built-in importer, not per-message logging). Do not generalize it to `rr.log`-per-message loops; for everything after import, reprocess with a reader + lenses (see `rerun-chunk-processing`: Chunk API vs logging API).\n\n### Downloading only the first few episodes\n\n```bash\nuv run scripts/fetch_lerobot_episodes.py --episodes 5 # prints the dataset directory\n```\n\nIt reports the download size before starting, downloads `meta/*` plus the wanted `episode_%06d` files, trims `meta/` to match, verifies that exactly the requested episodes landed, and prints what it got.\nIts last line is the dataset directory; do not follow it with `find` or `du`.\nDo the equivalent by hand and three things bite:\n\n- The importer iterates the episode list in `meta/`, so an untrimmed `meta/` warns once per absent episode.\n- `snapshot_download` writes into an existing directory, so episodes left by an earlier run with a different N survive and inflate the result.\n- Both failures are invisible until you count the recordings in the viewer, which is why the script asserts instead of leaving you to look.\n\n### Just looking at it in the viewer?\n\nSkip the RRD entirely — the viewer imports the dataset directory itself.\nPass the plain path (`rerun /path/to/dataset`, or `open_url` over MCP).\nEach episode arrives as its own recording, with `recording_id` `episode_0`, `episode_1`, ….\n\n## Step 2: one RRD per episode\n\nCatalog segments are one-recording-per-file, and `recording_id` becomes the segment id on registration.\n\n`LeRobotReader` goes straight there, without building the combined RRD of Step 1 first:\n\n```python\nreader = rr.experimental.LeRobotReader(dataset_dir) # v2 or v3\nfor episode in reader.episodes():\n episode_id = f\"episode_{episode:05d}\" # zero-padded; see below\n reader.stream(episode).write_rrd(\n rrd_dir / f\"{episode_id}.rrd\",\n application_id=\"my_dataset\",\n recording_id=episode_id,\n )\n```\n\nStreaming is lazy end to end, so memory is bounded by chunk size rather than by episode or dataset size, and video is cut to the episode's time window.\nB-frame-free video (AV1, LeRobot's default codec) streams directly; a stream that must be re-encoded — H.264 with B-frames, or a window starting mid-GOP — needs `ffmpeg` on the `PATH`.\n`stream()` also takes `entity_path_prefix`, `timeline`, and `video_mode=\"skip\"` to drop video entirely.\n\nTo add lenses, drop topics, or fix data, put them between the stream and the write — see `rerun-chunk-processing`.\n\n### Splitting an RRD you already have\n\nIf the combined RRD of Step 1 already exists — or the data came from somewhere other than the dataset directory — split it with `RrdReader` instead:\n\n```python\nreader = rr.chunk.RrdReader(str(combined_rrd))\nfor entry in reader.recordings():\n store = reader.store(store=entry)\n if not store.schema().entity_paths(): # skip the metadata-only root recording\n continue\n episode_id = zero_pad(entry.recording_id) # episode_1 -> episode_00001\n with rr.RecordingStream(\"rerun_example_lerobot\", recording_id=episode_id, send_properties=False) as rec:\n rec.save(str(rrd_dir / f\"{episode_id}.rrd\"))\n rec.send_chunks(store)\n```\n\nTwo non-obvious moves:\n\n- **Zero-pad the episode id.** `episode_10` sorts before `episode_2`\n lexicographically; segment tables and viewers sort lexicographically. Pad to\n a fixed width when re-assigning `recording_id`.\n- **`send_properties=False`** on the new stream, so the copy doesn't inject\n fresh recording properties on top of the copied chunks.\n\n`send_chunks` does not preserve the source store's identity; the new stream's\n`recording_id` wins, which is exactly what makes the rename work.\n\nIf episodes need cleanup (drop topics, fix data, add derived components), run the store through lenses between read and write: `reader.stream(store=entry).drop(...).lenses(...)` then `collect().write_rrd(..., recording_id=episode_id)` (see `rerun-chunk-processing`).\n\nComputed layers and per-episode properties then follow the standard patterns in `rerun-data-model` (layer `recording_id` must equal the episode segment id).\n\n## Step 3: add the robot model (URDF)\n\nA LeRobot dataset ships joint values, never a robot.\nWithout a URDF you get camera views and joint plots but no 3D arm, which is usually not what the user pictured when they asked to look at the data.\nAdding one is a forward-kinematics layer — `rerun-urdf` owns that; this section is only the LeRobot-specific wiring.\n\n- **The URDF.** `meta/info.json` `robot_type` names it (`so100`, `so101`, `aloha`, …); `rerun-urdf` has the source table.\n- **The joint names.** `meta/info.json` `features.observation.state.names` lists them in column order.\n Map each source name explicitly to the corresponding URDF joint name; do not assume that removing a prefix produces a valid URDF name.\n- **The values.** `observation.state` is the measured pose and the one to run FK on; `action` is the commanded pose and belongs on the plots, not the robot.\n- **The units and calibration.** Confirm the units, sign, and offset against the dataset and robot documentation.\n Run `rerun-urdf`'s `check_joint_mapping.py` against the episode parquets before trusting a pose.\n\nThe FK source is the per-episode parquet, read directly rather than out of the imported RRD:\n\n```python\nParquetReader(str(episode_parquet)).stream(\n column_grouping=\"individual\",\n index_columns=[IndexColumn.sequence(\"frame_index\")],\n).drop(content=\"/__properties/**\")\n```\n\n`frame_index` is the index the importer uses too, so the FK transforms line up with the video and plots.\nMerge the URDF model stream plus the FK stream into each episode's RRD, keeping the episode's `recording_id` (see `rerun-urdf`'s \"Minimal shape\").\nFor a whole dataset, the model belongs in a shared asset rather than in every segment — also `rerun-urdf`.\n\n## Gotchas\n\n1. `log_file_from_path` must target the dataset **root directory**, not a file inside it. So does `LeRobotReader`.\n2. Unpadded episode ids sort incorrectly downstream; pad before registering.\n3. The combined RRD contains a metadata-only root recording; skip stores with no entity paths or you register an empty segment.\n4. A partially-downloaded dataset imports fine, but warns once per episode listed in `meta/` and missing on disk.\n5. Downloading a subset into a directory an earlier run already populated leaves the older episodes in place; start from an empty directory.\n6. `image`-dtype (inline) datasets can be ~100x larger per episode than `mp4` ones; check before downloading, not after.\n7. Check signs and units - e.g. an `observation.state` in degrees could produce wrong FK when fed into a robot model that expects radians.\n\n## References\n\n- `rerun-chunk-processing` (`LazyChunkStream`, lenses, `RrdReader`)\n- `rerun-urdf` (adding the robot model and the joint-value mapping)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/dataloader` `prepare_dataset.py` (download → import → split → register, complete and runnable) and `train.py` (training-side consumption via `rerun.experimental.dataloader`)\n","skills/rerun-mcap/SKILL.md":"---\nname: rerun-mcap\ndescription: Ingest MCAP files into Rerun chunk streams with rerun.chunk.McapReader. Read when converting an MCAP recording, selecting topics or decoders, decoding custom protobuf messages, or when an MCAP-derived stream comes out empty. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (what the topics should become).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun MCAP ingestion\n\n`McapReader` turns an MCAP file into a lazy chunk stream: one entity per topic\nat the topic's path, message payloads decoded by pluggable decoders. This\nskill covers the reader's options, what each topic becomes, and the failure\nmodes that yield an empty stream with no error. Stream mechanics (filter, drop,\nlenses, merge, write) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.chunk import McapReader\n\nreader = McapReader(mcap_path) # see help(McapReader) for the full option set\nstream = reader.stream()\n```\n\nA URDF embedded in the MCAP can be ingested as well (then see `rerun-urdf`).\n\n## What a topic becomes\n\nWith the default decoders (`decoders=None`), the message **schema name** decides what a topic becomes — **pass archetypes through, lens only the raw `:message` topics**:\n\n| MCAP schema name | decodes to | what to do |\n| --- | --- | --- |\n| `foxglove.FrameTransforms` | `Transform3D` | pass through; do **not** hand-build |\n| `foxglove.CameraCalibration` | `Pinhole` | pass through |\n| `foxglove.CompressedVideo` | `VideoStream` (real sample bytes) + `CoordinateFrame` | pass through |\n| other `foxglove.*` well-known types | the matching archetype | pass through |\n| ros2 well-known types (`ros2msg` / `ros2_reflection`) | archetype | pass through |\n| your own `schemas.proto.*` / custom protobuf | one `:message` struct | attach semantics with a `DeriveLens` + `Selector` |\n\nSo a camera topic already arrives as `Pinhole`, its video as `VideoStream`, and a `frame_transforms` topic as `Transform3D` — only custom messages (e.g. a custom joint states schema, a custom gripper status enum) come through reflection or raw only and need lenses.\n\nThe `foxglove` decoder does the schema→archetype mapping; because foxglove messages are protobuf-_encoded_ it rides on the `protobuf` decoder, so keep `decoders=None` (verified: `decoders=[\"protobuf\"]` alone leaves `foxglove.CameraCalibration` a raw `:message`; adding `foxglove` makes it a `Pinhole`). Confirm on your file: `McapReader(path).stream()`, then read `McapSchema:name` and a few `Chunk.format()` before deciding anything is missing or needs rebuilding.\n\n- Entity path = topic name (`/sensors/joint_states` stays `/sensors/joint_states`).\n Filter early: `McapReader(path).stream().filter(content=\"/sensors/**\")`.\n- A reflection-decoded message lands as one struct component named `:message`.\n Navigate it with `Selector` (`Selector(\".joint_positions\")`) inside lenses; this is how custom messages\n get Rerun semantics attached (see the DeriveLens patterns in `rerun-chunk-processing`).\n- Topic regexes use RE2 syntax and are **not anchored**: `cam` matches\n `/external/cam_low` and `/camera_info`.\n Anchor explicitly (`^/external/cam`) when it matters. Prefer reader-level topic filtering over `.filter(...)`\n when you can, so excluded topics are never decoded at all.\n\n## When to use the low-level `mcap` package instead\n\n`McapReader` keeps payloads in columnar chunk streams; that is almost always what you want.\nDrop to `mcap.reader.make_reader` only when you need raw record metadata without payloads, or when you need to rewrite the container itself (re-registering schemas, channels, and messages).\n\n## Gotchas\n\n1. Empty stream, no error: a topic regex that matched nothing, or a channel\n whose decoder produced no rows. Check `Chunk.format()` on a few chunks of\n `reader.stream().to_chunks()` against a tiny test file, or compare topic\n names with the `mcap` CLI / package first.\n2. Topic regexes are unanchored RE2; excludes run after includes.\n3. `timeline_type=\"timestamp\"` interprets MCAP log times as wall-clock ns\n since epoch. If the recording's clock is wrong, fix it at the reader with\n `timestamp_offset_ns` rather than mutating timestamps downstream.\n4. Decoder subsets silently skip topics no decoder claims; when a topic is\n missing, retry with `decoders=None` to rule out decoder selection.\n5. Example fix-lenses are dataset-specific. Before copying a `MutateLens` like\n the `Pinhole:resolution` swap from the `robot_data_preprocessing` example,\n read the raw component from `McapReader(path).stream()` and confirm the defect\n exists in _your_ data — applied blindly it corrupts correct calibration (a\n correct 648×480 flipped to 480×648).\n6. `foxglove` derives both the camera's `Pinhole:child_frame` and the video's\n `CoordinateFrame:frame` from each message's `.frame_id` (plus an image-plane\n suffix), so they **match** when the calibration and video topics share a\n `frame_id`. Only when those topics carry _different_ `frame_id`s does the\n video frame diverge and orphan the video from its image plane — re-home it\n then with a per-camera `MutateLens` on `CoordinateFrame:frame`.\n\n## References\n\n- End-to-end MCAP pipeline: `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-urdf` (FK from joint-state topics), `rerun-data-model` (modeling decisions)\n","skills/rerun-catalog-queries/SKILL.md":"---\nname: rerun-catalog-queries\ndescription: Performance patterns and gotchas for querying a Rerun catalog from Python. Reach for this when exploring an unfamiliar dataset, when a CatalogClient/dataset query is unexpectedly slow, or when shaping a per-segment / per-episode pipeline that hits the catalog from many places.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash\n---\n\n# Rerun catalog queries\n\nPractical performance patterns for querying a Rerun catalog\nfrom Python (`rerun.catalog.CatalogClient` →\n`dataset.reader(...)` → DataFusion `DataFrame`). The DataFusion side of\nthe stack is covered by the **`datafusion-python`** skill — load that\nfor `DataFrame` / `SessionContext` / expression-API references. This\nskill focuses on catalog-specific behaviors and the round-trip costs\nthat catch teams off guard.\n\n---\n\n## Connect with a `rerun://` URL\n\n```python\nfrom rerun.catalog import CatalogClient\n\nclient = CatalogClient(\"rerun+https://api.example.cloud.rerun.io:443\")\ndataset = client.get_dataset(id=dataset_id)\n```\n\n`CatalogClient` takes a `rerun://` or `rerun+https://` URL, or\n`rerun+http://` for a local server. Do not pass a plain `https://` URL.\nA segment URL such as\n`rerun://host:443/dataset/?segment_id=` gives\nthe server, the dataset id and the segment id.\n\nThe client reads the token from `REDAP_TOKEN`, or from the stored\ncredentials of `rerun auth login`, so `token=` is usually not needed.\n\n---\n\n## The query cost model in one sentence\n\nEvery materialization of a `dataset.reader(...)` DataFrame is **one\ncloud round-trip**, with most of the cost being the network/decode\npair, not the compute. Plan for round-trip count and payload bytes, in\nthat order.\n\nA typical catalog round-trip is **a few seconds** even for a tiny\nresult. So:\n- 30 segments × 1 query each ≈ 90s. (Naive per-segment loop.)\n- 30 segments × 4 queries each ≈ 6 minutes. (A splitter that runs\n `count` + `collect_column` for both starts and stops.)\n- 1 query covering all 30 segments ≈ 3s.\n\nThe same fan-out happens along the **entity** axis: 10 entities ×\n1 `filter_contents([one_entity]).reader()` each ≈ 30s, vs one\n`filter_contents([all_entities]).reader()` ≈ 3s. Push as much as you\ncan into one round-trip — across segments and across entities.\n\n---\n\n## Always apply `filter_contents` and time-window filters before `.reader(...)`\n\nThe single biggest lever:\n\n- `filter_contents([entity_globs])` restricts which entity-path columns\n the reader produces. Without it, every entity in the dataset is\n read.\n- Time-window filters push down to storage and dramatically reduce\n bytes scanned when they compare the bare index column with a literal:\n `df.filter(col(index) >= lit(start)).filter(col(index) <= lit(end))`.\n Integer, timestamp and duration literals of any unit work. A `cast`\n or arithmetic on the index column is not pushed down.\n\nCombined: `dataset.filter_segments(seg).filter_contents(entities).reader(...)\n.filter(in_window).select(...)`.\n\n---\n\n## `df.cache()` is your friend for repeated probes\n\nWhen the same materialization gets used by multiple downstream\nfilter/count/collect calls, materialize once with `DataFrame.cache()`\nand operate on the cached frame:\n\n```python\ncached = (\n dataset\n .filter_segments(seg)\n .filter_contents([entity])\n .reader(index=index_col)\n .select(col(index_col).cast(pa.int64()).alias(index_col), value.alias(\"v\"))\n .cache() # one network round-trip, materializes into in-memory batches\n)\nstarts = cached.filter(col(\"v\") == start_val).collect_column(index_col)\nstops = cached.filter(stop_pred(col(\"v\"))).collect_column(index_col)\n```\n\nWithout `cache()`, each `count()` / `collect_column()` re-executes the\nwhole reader chain.\n\n**When NOT to cache.** `cache()` forces materialization into Arrow\nbatches, breaking laziness. If downstream code keeps composing more\nDataFusion ops on top (joins, windows, further filters) and only\nmaterializes once at the end, caching mid-pipeline turns one execution\ninto two and pre-empts whatever physical-plan optimizations the engine\ncould have done across the boundary. Reach for `cache()` when the\nconsumers are terminal (`count()`, `collect_column()`, `to_arrow_table()`),\nnot when they're another lazy `DataFrame`.\n\n---\n\n## Cross-segment batching: drop `filter_segments`, group by `rerun_segment_id`\n\nFor pipelines that need the same query on many segments, omit\n`filter_segments(...)` entirely and pull a single cross-segment table.\nEvery reader row carries a `rerun_segment_id` column — group locally:\n\n```python\ndf = dataset.filter_contents(entities).reader(index=index_col)\ncached = df.select(\n \"rerun_segment_id\",\n col(index_col).cast(pa.int64()).alias(index_col),\n value.alias(\"v\"),\n).cache()\n\n# Now N filter/aggregate calls are local, not network.\nstarts = cached.filter(col(\"v\") == start_val).select(\"rerun_segment_id\", index_col).to_arrow_table()\n```\n\nTrigger / event columns are tiny enough that pulling all segments at\nonce dominates per-segment looping by an order of magnitude.\n\n---\n\n## Per-entity fan-out within a segment\n\nSymmetric to cross-segment batching, along the entity axis. If you\nneed data from N entities of a single segment, **don't** loop:\n\n```python\n# Anti-pattern: N reader setups, N round-trips.\nfor entity in entities:\n df = dataset.filter_segments(seg).filter_contents([entity]).reader(index=ix)\n ...\n```\n\nInstead pull all N at once and project per entity locally. Per \"Reader\nrow layout\" below, every row carries data for one entity and NULLs\nfor the others, so `col(\"::\").is_not_null()`\nis the per-entity filter:\n\n```python\nshared = (\n dataset\n .filter_segments(seg)\n .filter_contents(sorted(set(entities)))\n .reader(index=ix)\n .filter(col(ix).cast(pa.int64()).between(start_ns, end_ns))\n)\n# Each downstream consumer narrows to its entity's rows lazily.\nsrc_a = shared.filter(col(f\"{ent_a}:{comp_a}\").is_not_null()).select(ix, f\"{ent_a}:{comp_a}\")\nsrc_b = shared.filter(col(f\"{ent_b}:{comp_b}\").is_not_null()).select(ix, f\"{ent_b}:{comp_b}\")\n```\n\nDataFusion can share the underlying scan across the per-entity\nprojections when it builds the physical plan, so this stays a single\ncatalog round-trip even though there are N logical consumers. Works\ninside generators that build per-source DataFusion plans (resampling,\nbracket lookup, nearest-in-time joins) — collapse the network fan-out\nwithout changing the per-source logic.\n\nA trap when refactoring: if a downstream query uses a reader column's\nfully-qualified name (`col(f\"{entity}:{archetype}:{component}\")`),\nyou don't need to alias the column in `shared`. The shared reader's\noutput schema preserves native column names, so existing per-entity\nprojection helpers keep working unchanged.\n\n---\n\n## `count()` is *not* free\n\nCounter-intuitive: `df.count()` and `df.aggregate([], [F.count(col)])`\ndo not always push down. Aggregate plans can force the engine to\nmaterialize the underlying column data server-side, then count on the\nclient. `F.count(col)` over wide entity-columns can ship full struct or\nblob payloads to count nullity.\n\nAlternatives, in order of preference for \"is anything here\":\n\n| Need | Use |\n|---|---|\n| \"any row in this filter?\" | `df.select(col(index)).limit(1).to_arrow_table().num_rows > 0` — server short-circuits on first match |\n| \"count rows in a tiny window\" | `df.filter(window).select(col(index)).count()` after the time filter |\n| \"count *each* entity in a wide query\" | per-entity `limit(1)` probes, threaded — *not* one big `count(col)` aggregate |\n\nA trap that fooled us: assuming `bool_or(col.is_not_null())` would only\nneed nullity buffers. It does not — the operator still touches payload\ndata on most plans.\n\n---\n\n## `using_index_values` + `fill_latest_at` is great for resampling, not for presence\n\n```python\n.reader(index=index_col, using_index_values=targets, fill_latest_at=True)\n```\n\n- Returns one row per target timestamp, each entity column carrying its\n latest non-null value at-or-before the target.\n- Excellent for nearest-prior resampling (no DataFusion required).\n- **Don't** use it as a presence check. Two reasons:\n 1. Semantics are \"ever emitted before T\", not \"emitted in [start, T]\".\n 2. The server still ships the full struct/blob payload for every\n entity to compute the latest-known value; a downstream\n `is_not_null()` projection runs post-transfer and doesn't reduce\n wire bytes.\n\nFor a strict in-window presence check, prefer per-entity `limit(1)`\nover the time-filtered reader, run concurrently.\n\n---\n\n## Explore an unfamiliar dataset with `dataset.info()`\n\n```python\ninfo = dataset.info()\nprint(info) # compact text overview\n```\n\nIt reports the segment count and size, each layer with how many\nsegments have it, each timeline with its kind, value range and segment\ncoverage, and the property column names. It reads only the segment\ntable and the schema, no recording data, so it stays cheap on large\ndatasets.\n\nUse it to choose `reader(index=…)` and to bound time-window filters\nbefore the first real query. `TimelineInfo.min` / `.max` are numpy\nvalues that work directly as literals:\n\n```python\nts = next(t for t in info.timelines if t.name == \"capture_time\")\ndf = dataset.reader(index=ts.name).filter(col(ts.name) >= lit(ts.min))\n```\n\nProperties found here can be filtered via `dataset.segment_table()`,\nand layer names tell you what `register(…, layer_name=…)` already\nattached.\n\n---\n\n## Schema introspection is cheap; use it before probing\n\n```python\nschema = dataset.filter_segments(seg).schema()\navailable = {(c.entity_path, c.component) for c in schema.component_columns()}\n```\n\nThis is a single round-trip and tells you which `(entity, component)`\npairs the segment registered. If a column isn't in the schema, you can\ndrop it from your manifest without any further cloud queries. This is\noften a complete substitute for \"does this entity have events\"\nprobing.\n\nCaveat: schema presence ≠ events. The MCAP records the topic schema\neven for unused topics. If your pipeline cares about distinguishing\n\"registered but never emitted\" vs \"registered with events\", you have\nto probe — see \"is anything here\" patterns above.\n\n---\n\n## Reader row layout: entities are columns, not rows\n\nEvery row from `dataset..reader(...)` corresponds to a single\nevent on a single entity. Other entities' columns are null on that\nrow. Implications:\n\n- `select(\"rerun_segment_id\", \"::\")`\n works — quote the entity column when using SQL.\n- There is **no** `rerun_entity_path` row attribute. To attribute rows\n to entities you either filter to one entity at a time, or pick a\n per-entity column (e.g. `:McapChannel:id`) whose non-null pattern\n identifies the source.\n- `df.count()` returns *total events across all entities*, not\n per-entity counts.\n\n---\n\n## Field access on a null struct returns `0.0`, not null\n\nA DataFusion gotcha that bites pipelines reading struct messages:\n\n```python\ncol(\"/some/entity:msg.MyType:message\")[0][\"sub\"][\"x\"]\n# When the parent struct is null on a row, this evaluates to 0.0\n# (and \"\" for strings), not null.\n```\n\nWrap struct-walk projections with a null guard:\n\n```python\nparent = col(\"/some/entity:msg.MyType:message\")\nleaf = parent[0][\"sub\"][\"x\"]\nguard = parent.is_null() | parent[0].is_null() | parent[0][\"sub\"].is_null()\nexpr = F.when(guard, lit(None)).otherwise(leaf)\n```\n\n**Only apply this to struct sources.** For scalar / blob columns\n(`Scalars:scalars`, `EncodedImage:blob`, etc.) the wrap is a no-op at\nbest and at worst rewrites the plan in ways that change downstream\njoin behavior. Gate the guard on whether the source actually walks\nthrough a struct boundary.\n\n---\n\n## Common debug recipe\n\nWhen a query stage is slower than expected:\n\n1. **Count round-trips.** Wrap each `to_arrow_table()` /\n `collect_column()` / `count()` with `time.perf_counter()`. Each is\n a round-trip. If you see N segment queries, that's N × few-seconds\n minimum.\n2. **Split build vs materialize timing.** Time the lazy DataFrame\n construction *separately* from the terminal `to_arrow_table()` call.\n If \"build\" takes seconds, something inside is materializing eagerly\n (an Arrow round-trip in a join helper, a `cache()` in a generator,\n a `.collect()` hidden in a chained-join utility). A correctly lazy\n plan should build in ~milliseconds regardless of result size.\n3. **Measure bytes.** `tbl.nbytes` after `to_arrow_table()` reveals\n when \"I projected `is_not_null()`\" actually shipped megabytes. If\n bytes are large despite a small projection, the operator didn't\n push down.\n4. **Cross-segment first, then cross-entity.** If the per-segment query\n is fundamentally the same (just scoped by id), drop `filter_segments`\n and group by `rerun_segment_id` locally. If you also have a\n per-entity loop within a segment, collapse it the same way (one\n `filter_contents([all])` reader, per-entity `is_not_null()` filters\n downstream).\n5. **Cache before terminal re-use.** If two `count()` / `collect_column()`\n calls share the same reader, `df.cache()` between them. Don't cache\n if the consumers are themselves lazy DataFrames being composed\n further — caching breaks plan-wide optimization.\n6. **Window first.** Always push the time filter before any projection\n or aggregate that touches payload columns.\n\n---\n\n## See also\n\n- `datafusion-python` skill — DataFrame API, SQL parity, expression\n building, common pitfalls (boolean operators, immutability, etc.).\n Not installed? Ask the user to install it globally:\n `npx skills add apache/datafusion-python`.\n","ARCHITECTURE.md":"# Rerun architecture\n\nThis document describes the technical architecture of Rerun.\n\n## See also\n\n- [`BUILD.md`](BUILD.md)\n- [`TESTING.md`](TESTING.md)\n- [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK\n- [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)\n- [`CODE_STYLE.md`](CODE_STYLE.md)\n- [`CONTRIBUTING.md`](CONTRIBUTING.md)\n- [`RELEASES.md`](RELEASES.md)\n\n## The major components\n\n### Logging APIs\n\nIt all starts with logging. You can log rich data (point clouds, images, etc.) with either our Python SDK or our Rust SDK.\n\nThe logging SDK:s encodes the data using Apache Arrow (see more below).\n\nThe logging data can be written to disk as `.rrd` files, or transmitted over gRPC to either a Rerun Viewer or a Rerun Server.\n\n### Rerun Viewer\n\nThe Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser.\n\n#### Native Viewer\n\nThe easiest way to launch the Viewer is directly from the logging API with `rr.init(\"rerun_example_app\", spawn=True)`. However, the standalone Viewer can also be run from the command line, for example to view an `.rrd` file: `rerun mydata.rrd`.\n\n#### Web viewer\n\nYou can try running the Viewer in a browser using `rr.serve()` in Python, or using `rerun --web-viewer mydata.rrd`.\n\nThe web viewer consists of just a few small files - a thin `.html`, a `.wasm` blob, and an auto-generated `.js` bridge for the wasm. These files are served using the [`re_web_viewer_server`](https://github.com/rerun-io/rerun/tree/latest/crates/top/re_web_viewer_server) crate.\n\nThe web viewer can load `.rrd` files (just drag-drop them into the browser), or read logging data streamed over gRPC.\n\n### `.rrd` files\n\n`.rrd` (\"**R**e**r**un **D**ata\") is just a bunch of log messages appended one after the other to a file.\n\nNOTE: `.rrd` files do not yet offer full backwards or forwards compatibility. However, the current version of Rerun will always be able to open `.rrd` files generated by the *previous* version. This is not guaranteed to work for files generated with older versions.\n\n## Technologies we use\n\n### Apache Arrow\n\n[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_chunk_store`](crates/store/re_chunk_store/README.md).\n\nIn Rust, we use the [`arrow` crate](https://crates.io/crates/arrow).\n\n### `wgpu`\n\nThe Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web.\n\nOn web builds, we use WebGPU when available on the Web, but automatically fall back to a WebGL based emulation layer (with a more limited feature set).\n\nWe have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/viewer_support/re_renderer/README.md).\n\n### `egui`\n\nThe GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode).\n\nWe use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web.\n\n### Wasm\n\nWasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.\nThe Rerun Viewer can be compiled to Wasm and run in a browser.\n\nThreading support in Wasm is nascent, so care must be taken that we don't spawn any threads when compiling for `wasm32`.\n\nWasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:\n\n- Access files\n- Read environment variables\n- Get the current time (use [`instant`](https://crates.io/crates/instant) instead)\n- Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)\n- etc\n\n## Immediate mode\n\nThe Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch.\n\nIn fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.\n\nThe advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.\n\nImmediate mode is also a forcing function, forcing us to relentlessly optimize our code.\nThis leads to a very responsive GUI, where there is no \"hiccups\" when switching data source or doing time scrubbing.\n\nOf course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.\n\n## Crates\n\nHere is an overview of the crates included in the project, and how they depend on each other:\n\n\n\nEach band is one folder under `crates/`, with `rerun_py` and `rerun_c` folded into the top band.\nEvery arrow points downwards, because the folders are layers: a crate may only depend on its own folder or one below it.\n[`scripts/check_crate_layers.py`](scripts/check_crate_layers.py) enforces that in CI.\nA `dev-dependency` is exempt, since a test may reach anywhere.\n\nSee [`crates/README.md`](crates/README.md) for the layers as a list, and for which one a new crate belongs in.\nThe tables below give a one-line description of every crate, one table per folder.\n\nOnly the arrows *inside* a band are drawn.\n\nThe diagram and the crate tables below are both generated from `cargo metadata`, so neither has to be written by hand.\nA crate's description is its `description` field in its own `Cargo.toml`, so there is only one place to write it.\nRegenerate both after adding, removing, or renaming a crate:\n\n```sh\npixi run crate-graph\n```\n\n**Run that command yourself whenever you add or remove a crate.**\nNothing checks it for you: `pixi run crate-graph-check` is disabled in CI, because graphviz lays the same graph out slightly differently on macOS and on Linux, so whoever regenerated the diagram last decided whether the check passed.\n\nPass `--edges all` to `scripts/generate_crate_graph.py` for a version with every dependency drawn, cross-band ones included.\n\n\n\n### Test support\n\n[`crates/tests`](./crates/tests) — only tests depend on these, so they sit at the top of the diagram.\n\n| Crate | Description |\n| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |\n| re_redap_tests | Official test suite for the Rerun Data Protocol |\n| re_renderer_tests | Snapshot tests for `re_renderer`'s rendering primitives. |\n| re_test_context | A common context used for tests. |\n| re_test_mocks | In-process server doubles (`MockOtlpCollector`, `MockPostHog`) used by tests that need to capture outbound OTel/PostHog traffic. |\n| re_test_viewport | A viewport used for tests |\n| re_web_tests | Discovers and runs Rerun web tests |\n\n### SDK / CLI / Wasm\n\n[`crates/top`](./crates/top) — the entry points for our users, and the viewer app itself.\n\n| Crate | Description |\n| -------------------- | ---------------------------------------------------------------------------------- |\n| re_sdk | Rerun logging SDK |\n| re_viewer | The Rerun viewer |\n| re_viewer_mcp | MCP server that allows llm agents to use the Rerun Viewer. |\n| re_web_viewer_server | Serves the Rerun web viewer (Wasm and HTML) over HTTP |\n| rerun | The Rerun Rust SDK: log images, point clouds, etc, and visualize them effortlessly |\n| rerun-cli | The Rerun command-line tool: the `rerun` binary, including the native Viewer |\n| rerun_c | Rerun C SDK |\n| rerun_py | The Rerun Python SDK |\n\n### Views\n\n[`crates/views`](./crates/views) — the visualizations a user can put in the viewport.\nA sibling of `crates/panels`: neither depends on the other.\n\n| Crate | Description |\n| ---------------------- | --------------------------------------------------------------------------- |\n| re_view_audio | A view that shows the waveform of an audio asset and plays it back. |\n| re_view_bar_chart | A view that shows a single bar chart. |\n| re_view_dataframe | A view that shows the data contained in entities in a table. |\n| re_view_graph | A view that shows a graph (node-link diagram). |\n| re_view_map | A view that shows a map. |\n| re_view_spatial | Views that show entities in a 2D or 3D spatial relationship. |\n| re_view_state_timeline | A view that shows state transitions as horizontal lanes over time. |\n| re_view_tensor | A view dedicated to visualizing tensors with arbitrary dimensionality. |\n| re_view_text_document | A simple view that shows a single text box. |\n| re_view_text_log | A view that shows text entries in a table and scrolls with the active time. |\n| re_view_time_series | A view that shows plots over Rerun timelines. |\n\n### Panels\n\n[`crates/panels`](./crates/panels) — the panels the app is assembled from, and the widgets they are built out of.\nA sibling of `crates/views`: neither depends on the other.\n\n| Crate | Description |\n| ------------------ | ----------------------------------------------------------------------------------------- |\n| re_agent_ui | Chat UI for driving a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_blueprint_tree | The UI for the blueprint tree in the left panel. |\n| re_chunk_store_ui | Display the contents of chunk stores. |\n| re_memory_view | Flamegraph visualization for memory usage trees |\n| re_recording_panel | The UI for the recording panel. |\n| re_redap_browser | The UI and communication to implement the in-viewer redap server browser. |\n| re_selection_panel | The UI for the selection panel. |\n| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |\n| re_viewport | The central viewport panel of the Rerun viewer. |\n\n### Viewer support\n\n[`crates/viewer_support`](./crates/viewer_support) — the UI and rendering machinery the views are built on.\n\n| Crate | Description |\n| ---------------------- | ---------------------------------------------------------------------------------------------------------- |\n| re_arrow_ui | Show arrow data in a tree of rerun list_items and format arrow with syntax highlighting. |\n| re_component_fallbacks | Handles registering various component fallbacks to `FallbackProviderRegistry`. |\n| re_component_ui | Provides ui editors for Rerun component data for registration with the Rerun Viewer component ui registry. |\n| re_context_menu | Support crate for context menu and actions. |\n| re_data_ui | Provides ui elements for Rerun component data for the Rerun Viewer. |\n| re_dataframe_ui | Rich table widget over DataFusion. |\n| re_gamepad | Gamepad input handling for the Rerun viewer. |\n| re_plot | Shared utilities for plot-based views. |\n| re_renderer | A wgpu based renderer for all your visualization needs. |\n| re_renderer_examples | Examples for the re_renderer crate. |\n| re_time_ruler | Time ruler widget shared by the time panel and time-aware views. |\n| re_ui | Rerun GUI theme and helpers, built around egui |\n| re_view | Types & utilities for defining view classes and communicating with the viewport. |\n| re_viewer_context | Rerun viewer state that is shared with the viewer's code components. |\n| re_viewport_blueprint | The data model describing the layout of the viewport. |\n\n### Application-level store\n\n[`crates/store_app`](./crates/store_app) — the queryable state a viewer or a server works with.\n\n| Crate | Description |\n| ------------- | --------------------------------------- |\n| re_dataframe | High-level query APIs |\n| re_datafusion | High-level query APIs |\n| re_entity_db | In-memory storage of Rerun entities |\n| re_query | High-level query APIs |\n| re_server | An ephemeral local Rerun catalog server |\n| re_tf | Rerun spatial transform processing |\n\n### Data flow\n\n[`crates/data_flow`](./crates/data_flow) — getting data in and out: clients, servers, and file importers.\n\n| Crate | Description |\n| ----------------- | --------------------------------------------------------------------------------- |\n| re_catalog_client | High-level client for Rerun catalogs |\n| re_data_source | Handles loading of Rerun data |\n| re_grpc_client | Client for the legacy StoreHub API (`re_grpc_server`). |\n| re_grpc_server | Server for the legacy StoreHub API |\n| re_hdf5 | Core HDF5-to-chunk loading logic for Rerun |\n| re_importer | Handles importing of Rerun data from file using importer plugins |\n| re_lenses | Lenses are an API for extracting, transforming, and restructuring component data. |\n| re_lerobot | Core LeRobot-dataset-to-chunk loading logic for Rerun |\n| re_mcap | Convert MCAP into Rerun-compatible data |\n| re_mp4_reader | Core mp4-to-chunk loading logic for Rerun |\n| re_parquet | Core parquet-to-chunk loading logic for Rerun |\n| re_redap_client | Official gRPC client for the Rerun Data Protocol |\n\n### Data model & chunk store\n\n[`crates/store`](./crates/store) — the data model, and the in-memory store that holds it.\n\n| Crate | Description |\n| ------------------ | --------------------------------------------------------------------------------------------- |\n| re_arrow_json | Rerun data as JSON, and back |\n| re_chunk | A chunk of Rerun data, encoded using Arrow. Used for logging, transport, storage and compute. |\n| re_chunk_index | Indexes of Rerun chunks: RRD manifests and the chunk providers that serve them |\n| re_chunk_optimizer | Analysis and memory-bounded optimization of Rerun chunk layouts |\n| re_chunk_store | A storage engine for Rerun's Chunks |\n| re_lenses_core | Core lens types and composable Arrow array transformations. |\n| re_log_channel | An in-memory channel of Rerun data messages. |\n| re_log_encoding | Encode/decode RRD streams: the physical byte layout of .rrd files |\n| re_log_msg | The messages that carry Rerun data between the SDK, `.rrd` files and the viewer. |\n| re_log_types | The basic building blocks of the Rerun log format: entity paths, timelines, and store ids. |\n| re_protos | Rerun remote gRPC/protobuf API types |\n| re_sdk_types | The standard Rerun data types, component types, and archetypes. |\n| re_sorbet | Rerun arrow metadata definitions |\n| re_types_core | The core traits and types that power Rerun's data model. |\n| re_uri | Parsing and constructing Rerun URIs |\n\n### Build support\n\n[`crates/build`](./crates/build) — crates that run at build time.\n\n| Crate | Description |\n| ------------------------ | --------------------------------------------------------------------------------------- |\n| re_build_tools | build.rs helpers for generating build info |\n| re_dev_tools | Various tools for rerun development |\n| re_protos_builder | Code generation for Rerun's Protobuf and gRPC definitions. |\n| re_type_definitions | Rerun's type definitions: a subset of Rust that generates the Rust, Python and C++ SDKs |\n| re_types_builder | Generates code for Rerun's SDKs from the type definitions. |\n| re_types_builder_macros | The `#[rerun_type]` attribute macro used by Rerun's IDL definitions. |\n| re_types_builder_prelude | The vocabulary that Rerun's IDL definitions are written against. |\n\n### Utilities\n\n[`crates/utils`](./crates/utils) — small crates that depend on nothing outside `crates/utils`.\n\n| Crate | Description |\n| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| re_agent | Talk to a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_analytics | Rerun's analytics SDK |\n| re_arrow_util | Helpers for working with arrow. |\n| re_async | Async runtime abstractions for native and WebAssembly targets. |\n| re_audio | Rerun audio decoding and playback utilities. |\n| re_auth | Authentication helpers for Rerun |\n| re_backoff | Utilities for retrying operations with backoff. |\n| re_build_info | Information about the build. Use together with re_build_tools |\n| re_byte_size | Calculate the heap-allocated size of values at runtime. |\n| re_byte_size_derive | Derive macro for the `SizeBytes` trait from `re_byte_size`. |\n| re_capabilities | Capability tokens for the Rerun code base. |\n| re_case | Case conversions, the way Rerun likes them |\n| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |\n| re_dump_puffin | Convert puffin profiler recordings (.puffin files) into JSON for analysis |\n| re_error | Helpers for handling errors. |\n| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |\n| re_grpc_headers | Rerun gRPC header conventions: well-known header names, the `RerunVersionInterceptor`, and the tower `Layer` machinery that propagates them across requests and responses. |\n| re_int | Small numeric helper traits: saturating integer casts and unsigned absolute value |\n| re_log | Helpers for setting up and doing text logging in the Rerun crates. |\n| re_memory | Run-time memory tracking and profiling. |\n| re_mutex | A wrapper around `parking_lot::Mutex` useful for debugging deadlocks. |\n| re_perf_telemetry | In and out of process performance profiling utilities for Rerun & Redap |\n| re_quota_channel | A channel that applies backpressure based on byte size. |\n| re_ros_msg | Parse and decode ROS messages |\n| re_rvl | Support for compressed depth data using the RVL format. |\n| re_span | An integer range that always has a non-negative length |\n| re_string_interner | Yet another string interning library |\n| re_tracing | Helpers for tracing/spans/flamegraphs and such. |\n| re_tuid | 128-bit Time-based Unique Identifier |\n| re_video | Rerun video processing utilities. |\n| re_web | Utilities for interacting with Web APIs |\n| re_workspace_hack | Unifies third-party dependency features across the workspace to avoid recompiles. Managed by cargo-hakari. |\n\n\n\n### Dependencies and docs\n\nIn order to get a dependency graph for our in-house crates and their docs, we recommend you run:\n\n```\ncargo install cargo-depgraph\ncargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > /tmp/rerun-deps.png\nopen /tmp/rerun-deps.png\n```\n\nand:\n\n```\ncargo doc --no-deps --open\n```\n\nand then browse through the `re_*` crates.\n","AGENTS.md":"# CLAUDE.md\n\nGuidance for LLMs working in this repo.\n\n## Project overview\n\nRerun is a time-aware multimodal data stack and visualizations tool used in robotics, spatial AI, computer vision, and similar domains. It provides SDKs (Python, Rust, C++) for logging rich data (images, point clouds, tensors, etc.) and a Viewer for visualization.\n\n## Build system\n\nWe use `pixi` for task management and dependency installation. Check `pixi.toml` for a full list of tasks.\n\n### Essential commands\n\n**Building:**\n- `pixi run py-build` - Build Python SDK into local .venv (uses uv)\n- `pixi run rerun-build` - Build native viewer (without web viewer)\n- `pixi run rerun-build-web` - Build web viewer (wasm)\n- `pixi run cpp-build-all` - Build all C++ artifacts\n\n**Running:**\n- `pixi run rerun` - Run the viewer\n- `pixi run uvpy script.py` - Run Python scripts with rerun SDK\n- `cargo run -p ` - Run specific Rust example (e.g., `cargo run -p dna`)\n\n**Code generation:**\n- `pixi run codegen` - Generate Rust/Python/C++ code from the `re_type_definitions` crate\n\n**Formatting:**\n- `pixi run rs-fmt` - Format all Rust files. Always run this after making changes.\n- `pixi run py-fmt` - Format Python files\n- `pixi run cpp-fmt` - Format C++ files\n- `pixi run toml-fmt` - Format TOML files\n\n**Testing:**\n- `cargo clippy -p ` - Run rust checks before building\n- `cargo nextest run --all-features --no-fail-fast -p ` - Run tests for specific crate\n - Example: `cargo nextest run --all-features --no-fail-fast -p re_view_spatial`\n- Use `cargo nextest` (not `cargo test`) for better output + parallelism\n- Always use `--all-features` unless specific reason not to\n- Use `--no-fail-fast` to gather all failures in single run\n\n**Snapshots:**\n- **`insta` snapshots**: Text-based, run with regular Rust tests. On failure: `cargo insta review` (install: `cargo install cargo-insta`)\n- **Image comparison tests**: Render image vs checked-in reference. Uses `egui_kittest`'s `Harness::snapshot` + `TestContext` for mocking viewer.\n - Results saved to `tests/snapshots/`, failures produce `diff.png`\n - Update refs: `UPDATE_SNAPSHOTS=1`\n - Update from failed CI run: `./scripts/update_snapshots_from_ci.sh`\n - Best practices: see [egui_kittest README](https://github.com/emilk/egui/tree/master/crates/egui_kittest#snapshot-testing)\n\n### Long-running commands (for agents)\n\nBuilds and test runs here take a long time, so run them in the background.\nA background command notifies you when it exits, so never start a second command to watch the first one.\nNo `sleep`-and-`tail` pollers, no `until` loops on the log.\nStart the build, do something else, and read its output once the notification arrives.\n\n### Driving a running egui UI (for agents)\n\nTwo MCP servers let an agent see and click a running UI instead of guessing from code:\n\n- `rerun viewer-mcp` drives a running Rerun Viewer over gRPC. See `docs/content/reference/viewer/mcp.md`.\n- `egui-mcp` drives *any* egui app started with `EGUI_INSPECTION=1`, including our example apps and headless `egui_kittest` harnesses.\n It exposes `attach`, `widget_tree`, `click`, `type_text`, `screenshot`, `wait_for`, and friends.\n Both are configured in the repo-root `.mcp.json`; install with `cargo install egui_mcp`.\n\nTypical loop: start the app with `EGUI_INSPECTION=1`, call `attach` (host `127.0.0.1`, port `5719`), `widget_tree` to find widgets, `click`/`type_text`, then `screenshot` with a `save_path` and look at the image.\n`egui-mcp` needs the app to paint frames: windowed apps must not be occluded on macOS, so prefer a headless harness where one exists.\nA headless `egui_kittest` harness can opt in with `egui_inspection::attach_from_env(&harness.ctx, label)`.\nExample: `EGUI_INSPECTION=1 cargo run -p re_agent_ui --example agent_app -- --headless`.\n\n## Code generation system\n\n**Critical: Never edit generated files directly.** All generated files are marked \"DO NOT EDIT\" at the top.\n\n### Type definition flow\n\n```\nre_type_definitions → pixi run codegen → Generated code (Rust/Python/C++) + docs (docs/content/reference/types/)\n```\n\n- Type definitions live in `crates/build/re_type_definitions/rerun/`\n - `encodings/*.def.rs` - Low-level types (Vec3D, Mat4x4, etc.)\n - `components/*.def.rs` - Component types (Position3D, Color, etc.)\n - `archetypes/*.def.rs` - Archetypes (Points3D, Image, etc.)\n - `blueprint/*.def.rs` - Blueprint system types\n- Codegen implementation is in `crates/build/re_types_builder/`\n- After modifying a definition, run `pixi run codegen` to regenerate code\n\n### Extension pattern\n\nTo add custom functionality to generated types, create `_ext` files:\n- Rust: `filename_ext.rs` (automatically imported by codegen)\n- Python: `filename_ext.py` (mixed in with generated class)\n- C++: `filename_ext.cpp` (compiled and included automatically, parts of it may be marked for copy into the header by codegen)\n\n## Code conventions\n\n### General\n\n- use `…` instead of `...` \n- validate various custom conventions via `pixi run lint-rerun ` (not passing any file will check everything)\n- Use `format!(\"{x}\")` over `format!(\"{}, x)` (same in log calls etc)\n- Don't write trivial comments that add nothing new\n- In error and log messages, put the error first and any file path at the end (e.g. `Failed to import: {err}\\nFile path: {path}`), never in the middle.\n Paths can be long or sensitive, so trailing placement makes them easy to strip when copy-pasting.\n- Prose style (em vs en dash, sentence endings, casing) — see [`DESIGN.md`](DESIGN.md). In short: spaced em dash ` — `, never unspaced `word—word`, and don't use `–` as a sentence dash (it's for numeric ranges only) \n- One sentence per line in markdown files.\n Markdown joins consecutive lines into a paragraph, so rendering is unchanged — but diffs become much easier to review.\n\n## Architecture overview\n\n### Crate organization\n\n```\ncrates/\n├── build/ # Code generation (re_types_builder)\n├── store/ # Data types, storage, querying\n├── top/ # User-facing SDKs and CLI\n└── viewer/ # Viewer UI and rendering\n```\n\nFor more details about the architecture see `ARCHITECTURE.md`.\n\n**When adding, removing, or renaming a crate**, run `pixi run crate-graph`.\nIt regenerates both the crate dependency diagram (`crate_graph.svg`) and the crate tables in `ARCHITECTURE.md` from `cargo metadata`.\nA crate's one-line description comes from the `description` field of its `Cargo.toml`.\n\n### Type system hierarchy\n\nThe type system has three levels (generated from `re_type_definitions`):\n\n1. **Encodings** (`rerun.encodings.*`) - Basic types like Vec3D, Color\n2. **Components** (`rerun.components.*`) - Named semantic wrappers (Position3D, Radius)\n3. **Archetypes** (`rerun.archetypes.*`) - Collections of components (Points3D, Image)\n\nEach archetype specifies:\n- Required components (must be provided)\n- Recommended components (have good defaults)\n- Optional components (purely optional)\n\nExample: `Points3D` archetype requires `positions`, recommends `colors` and `radii`, allows optional `labels`.\n\n### Data flow\n\n```\nSDK (log archetype)\n ↓ encode to Apache Arrow\nLogMsg (encoded data)\n ↓ transport (gRPC/file/memory)\nre_chunk_store (indexed time series DB)\n ↓ query\nViewer (immediate mode rendering)\n```\n\n### Blueprint system\n\nThe blueprint is the viewer's configuration layer:\n- Stored as a separate store (`re_entity_db`) with \"blueprint\" timeline\n- Defines: view layout, visibility, per-entity overrides, view properties\n- Uses the same type system as logged data\n- Basic blueprint path hierarchy: `/viewport/`, `/view/{uuid}/`, `/container/{uuid}/`\n\n### Visualizers\n\nEach view type (Spatial3D, TimeSeries, etc.) has registered visualizers:\n- Determine which entities/archetypes can be visualized\n- Execute per-frame: query data → process → generate render commands\n- Examples: Points3DVisualizer, LineStripsVisualizer, MeshVisualizer\n\nThe viewer uses **immediate mode**: every frame, query the store and re-render from scratch.\n\n## Documentation snippets\n\nSee [`docs/snippets/README.md`](docs/snippets/README.md) for running, building, finding snippets. Config in [`docs/snippets/snippets.toml`](docs/snippets/snippets.toml).\n\n## Python development workflow\n\nPython uses a separate uv-managed .venv (not pixi's conda env):\n\n```bash\npixi run py-build # Build rerun-sdk into .venv\npixi run uvpy script.py # Run Python scripts via uv\npixi run uv run script.py # Explicit uv run\n```\n\nThe `uv` wrapper script unsets `CONDA_PREFIX` to ensure isolation from pixi's environment.\n\n## Important notes\n\n- **PyO3 Configuration**: If you see PyO3 config errors, run `pixi run ensure-pyo3-build-cfg`\n- **git-lfs**: Required for test snapshots. Install with your package manager and run `git lfs install`\n- **Immediate Mode**: The entire viewer is rendered from scratch each frame (no state management callbacks)\n- **Arrow Native**: Data is stored, transmitted, and queried as Apache Arrow arrays\n- **Multi-language**: Changes to `re_type_definitions` affect Rust, Python, and C++ simultaneously\n\n## Python docstring formatting\n\nPython API docs are built with **MkDocs + mkdocstrings** (NOT Sphinx). Never use reStructuredText (rST) syntax in Python docstrings or documentation. Use markdown instead:\n\n- **Cross-references:** Use `[`ClassName`][]` (mkdocstrings syntax), NOT `:class:`ClassName`` / `:func:` / `:meth:` (rST roles)\n- **Warnings/notes:** Use MkDocs admonitions (`!!! warning` with indented body), NOT `.. warning::` (rST directives)\n- **Deprecation notices:** Use the `@deprecated` decorator (mkdocstrings renders it automatically). Do NOT duplicate in the docstring with `.. deprecated::` or `**Deprecated:**`\n- **Code blocks:** Use markdown fenced blocks (`` ``` ``), NOT `.. code-block::`\n- **Parameter docs:** Use numpy-style sections (`Parameters`, `Returns` with `----------`), which is what the codebase already uses\n\n## Documentation system\n\nSee [`docs/README.md`](docs/README.md) for the full documentation architecture.\n\nThe docs span multiple sites: the main docs at `rerun.io/docs` (built from `docs/content/`), plus API reference sites for Python (MkDocs), C++ (Doxygen), and JS (TypeDoc) at `ref.rerun.io/docs/{python,cpp,js}/`.\n\nKey things to know:\n- **`docs/content/reference/types/`** is auto-generated by `pixi run codegen` from `re_type_definitions` - do not edit directly\n- **`docs/content/reference/cli.md`** is auto-generated by `pixi run man` - do not edit directly\n- **Code snippets** live in `docs/snippets/all/` with implementations in Python, Rust, and C++\n- `pixi run py-docs-serve` previews Python API docs locally\n- `pixi run -e cpp cpp-docs` builds C++ docs\n\n## Development references\n\n- [`ARCHITECTURE.md`](ARCHITECTURE.md) - Detailed architecture documentation\n- [`BUILD.md`](BUILD.md) - Full build instructions\n- [`CODE_STYLE.md`](CODE_STYLE.md) - Code style guidelines\n- [`CONTRIBUTING.md`](CONTRIBUTING.md) - Contribution guidelines\n- [`DESIGN.md`](DESIGN.md) - UI design guidelines (GUI, CLI, docs, log messages)\n- [`docs/README.md`](docs/README.md) - Documentation system (sites, builds, deployment)\n- [`rerun_py/README.md`](rerun_py/README.md) - Python SDK instructions\n\n## Contributing\n\nDon't open pull requests or issues unless explicitly asked.\nWhen opening or interacting with one, follow the [pull request template](.github/pull_request_template.md) or [issue templates](.github/ISSUE_TEMPLATE/), and disclose that you are an LLM.\nLet the user know that you included this disclosure.\n\n### Before requesting review\n\n- CI is green, and the code is formatted (`pixi run rs-fmt`).\n- The branch history is clean: no broken merges, and no commits from unrelated work.\n- The description is honest about the scope, and says *why* the change is needed.\n- UI changes come with a screenshot or video.\n\n### Addressing review comments\n\n- When a reviewer flags one instance of a pattern, fix every instance in the diff, and say so in your reply.\n- A review comment phrased as a question wants an answer, not a code change — unless the answer is \"yes, that is wrong\".\n- When a reviewer asks for a change, make it in this PR.\n Only defer it to a follow-up PR if the reviewer agrees.\n- If the fix belongs in a dependency we maintain (egui, egui_plot, egui_tiles, emath, quiver, puffin, …), suggest an upstream PR instead of a local workaround.\n- Reviewers want to talk to the human author.\n When a comment asks for a judgement call (API shape, scope, product behavior), bring it to your user instead of deciding in the reply.\n- Always reply when mentioned, even if only to say you are unsure.\n","CLAUDE.md":"# Claude Agent Guidelines & System Prompt\n\nPath: `CLAUDE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/CLAUDE.md)","DESIGN.md":"# UI Design System & Style Tokens\n\nPath: `DESIGN.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/DESIGN.md)","rerun_py/ARCHITECTURE.md":"# rerun_py - System Architecture & Module Boundaries\n\nPath: `rerun_py/ARCHITECTURE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/rerun_py/ARCHITECTURE.md)","crates/build/re_dev_tools/README.md":"# re_dev_tools Documentation\n\nPath: `crates/build/re_dev_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/README.md)","crates/build/re_build_tools/README.md":"# re_build_tools Documentation\n\nPath: `crates/build/re_build_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_build_tools/README.md)","crates/build/re_dev_tools/src/build_web_viewer/README.md":"# build_web_viewer Documentation\n\nPath: `crates/build/re_dev_tools/src/build_web_viewer/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_web_viewer/README.md)","crates/build/re_dev_tools/src/build_search_index/README.md":"# build_search_index Documentation\n\nPath: `crates/build/re_dev_tools/src/build_search_index/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_search_index/README.md)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# CLAUDE.md\n\nGuidance for LLMs working in this repo.\n\n## Project overview\n\nRerun is a time-aware multimodal data stack and visualizations tool used in robotics, spatial AI, computer vision, and similar domains. It provides SDKs (Python, Rust, C++) for logging rich data (images, point clouds, tensors, etc.) and a Viewer for visualization.\n\n## Build system\n\nWe use `pixi` for task management and dependency installation. Check `pixi.toml` for a full list of tasks.\n\n### Essential commands\n\n**Building:**\n- `pixi run py-build` - Build Python SDK into local .venv (uses uv)\n- `pixi run rerun-build` - Build native viewer (without web viewer)\n- `pixi run rerun-build-web` - Build web viewer (wasm)\n- `pixi run cpp-build-all` - Build all C++ artifacts\n\n**Running:**\n- `pixi run rerun` - Run the viewer\n- `pixi run uvpy script.py` - Run Python scripts with rerun SDK\n- `cargo run -p ` - Run specific Rust example (e.g., `cargo run -p dna`)\n\n**Code generation:**\n- `pixi run codegen` - Generate Rust/Python/C++ code from the `re_type_definitions` crate\n\n**Formatting:**\n- `pixi run rs-fmt` - Format all Rust files. Always run this after making changes.\n- `pixi run py-fmt` - Format Python files\n- `pixi run cpp-fmt` - Format C++ files\n- `pixi run toml-fmt` - Format TOML files\n\n**Testing:**\n- `cargo clippy -p ` - Run rust checks before building\n- `cargo nextest run --all-features --no-fail-fast -p ` - Run tests for specific crate\n - Example: `cargo nextest run --all-features --no-fail-fast -p re_view_spatial`\n- Use `cargo nextest` (not `cargo test`) for better output + parallelism\n- Always use `--all-features` unless specific reason not to\n- Use `--no-fail-fast` to gather all failures in single run\n\n**Snapshots:**\n- **`insta` snapshots**: Text-based, run with regular Rust tests. On failure: `cargo insta review` (install: `cargo install cargo-insta`)\n- **Image comparison tests**: Render image vs checked-in reference. Uses `egui_kittest`'s `Harness::snapshot` + `TestContext` for mocking viewer.\n - Results saved to `tests/snapshots/`, failures produce `diff.png`\n - Update refs: `UPDATE_SNAPSHOTS=1`\n - Update from failed CI run: `./scripts/update_snapshots_from_ci.sh`\n - Best practices: see [egui_kittest README](https://github.com/emilk/egui/tree/master/crates/egui_kittest#snapshot-testing)\n\n### Long-running commands (for agents)\n\nBuilds and test runs here take a long time, so run them in the background.\nA background command notifies you when it exits, so never start a second command to watch the first one.\nNo `sleep`-and-`tail` pollers, no `until` loops on the log.\nStart the build, do something else, and read its output once the notification arrives.\n\n### Driving a running egui UI (for agents)\n\nTwo MCP servers let an agent see and click a running UI instead of guessing from code:\n\n- `rerun viewer-mcp` drives a running Rerun Viewer over gRPC. See `docs/content/reference/viewer/mcp.md`.\n- `egui-mcp` drives *any* egui app started with `EGUI_INSPECTION=1`, including our example apps and headless `egui_kittest` harnesses.\n It exposes `attach`, `widget_tree`, `click`, `type_text`, `screenshot`, `wait_for`, and friends.\n Both are configured in the repo-root `.mcp.json`; install with `cargo install egui_mcp`.\n\nTypical loop: start the app with `EGUI_INSPECTION=1`, call `attach` (host `127.0.0.1`, port `5719`), `widget_tree` to find widgets, `click`/`type_text`, then `screenshot` with a `save_path` and look at the image.\n`egui-mcp` needs the app to paint frames: windowed apps must not be occluded on macOS, so prefer a headless harness where one exists.\nA headless `egui_kittest` harness can opt in with `egui_inspection::attach_from_env(&harness.ctx, label)`.\nExample: `EGUI_INSPECTION=1 cargo run -p re_agent_ui --example agent_app -- --headless`.\n\n## Code generation system\n\n**Critical: Never edit generated files directly.** All generated files are marked \"DO NOT EDIT\" at the top.\n\n### Type definition flow\n\n```\nre_type_definitions → pixi run codegen → Generated code (Rust/Python/C++) + docs (docs/content/reference/types/)\n```\n\n- Type definitions live in `crates/build/re_type_definitions/rerun/`\n - `encodings/*.def.rs` - Low-level types (Vec3D, Mat4x4, etc.)\n - `components/*.def.rs` - Component types (Position3D, Color, etc.)\n - `archetypes/*.def.rs` - Archetypes (Points3D, Image, etc.)\n - `blueprint/*.def.rs` - Blueprint system types\n- Codegen implementation is in `crates/build/re_types_builder/`\n- After modifying a definition, run `pixi run codegen` to regenerate code\n\n### Extension pattern\n\nTo add custom functionality to generated types, create `_ext` files:\n- Rust: `filename_ext.rs` (automatically imported by codegen)\n- Python: `filename_ext.py` (mixed in with generated class)\n- C++: `filename_ext.cpp` (compiled and included automatically, parts of it may be marked for copy into the header by codegen)\n\n## Code conventions\n\n### General\n\n- use `…` instead of `...` \n- validate various custom conventions via `pixi run lint-rerun ` (not passing any file will check everything)\n- Use `format!(\"{x}\")` over `format!(\"{}, x)` (same in log calls etc)\n- Don't write trivial comments that add nothing new\n- In error and log messages, put the error first and any file path at the end (e.g. `Failed to import: {err}\\nFile path: {path}`), never in the middle.\n Paths can be long or sensitive, so trailing placement makes them easy to strip when copy-pasting.\n- Prose style (em vs en dash, sentence endings, casing) — see [`DESIGN.md`](DESIGN.md). In short: spaced em dash ` — `, never unspaced `word—word`, and don't use `–` as a sentence dash (it's for numeric ranges only) \n- One sentence per line in markdown files.\n Markdown joins consecutive lines into a paragraph, so rendering is unchanged — but diffs become much easier to review.\n\n## Architecture overview\n\n### Crate organization\n\n```\ncrates/\n├── build/ # Code generation (re_types_builder)\n├── store/ # Data types, storage, querying\n├── top/ # User-facing SDKs and CLI\n└── viewer/ # Viewer UI and rendering\n```\n\nFor more details about the architecture see `ARCHITECTURE.md`.\n\n**When adding, removing, or renaming a crate**, run `pixi run crate-graph`.\nIt regenerates both the crate dependency diagram (`crate_graph.svg`) and the crate tables in `ARCHITECTURE.md` from `cargo metadata`.\nA crate's one-line description comes from the `description` field of its `Cargo.toml`.\n\n### Type system hierarchy\n\nThe type system has three levels (generated from `re_type_definitions`):\n\n1. **Encodings** (`rerun.encodings.*`) - Basic types like Vec3D, Color\n2. **Components** (`rerun.components.*`) - Named semantic wrappers (Position3D, Radius)\n3. **Archetypes** (`rerun.archetypes.*`) - Collections of components (Points3D, Image)\n\nEach archetype specifies:\n- Required components (must be provided)\n- Recommended components (have good defaults)\n- Optional components (purely optional)\n\nExample: `Points3D` archetype requires `positions`, recommends `colors` and `radii`, allows optional `labels`.\n\n### Data flow\n\n```\nSDK (log archetype)\n ↓ encode to Apache Arrow\nLogMsg (encoded data)\n ↓ transport (gRPC/file/memory)\nre_chunk_store (indexed time series DB)\n ↓ query\nViewer (immediate mode rendering)\n```\n\n### Blueprint system\n\nThe blueprint is the viewer's configuration layer:\n- Stored as a separate store (`re_entity_db`) with \"blueprint\" timeline\n- Defines: view layout, visibility, per-entity overrides, view properties\n- Uses the same type system as logged data\n- Basic blueprint path hierarchy: `/viewport/`, `/view/{uuid}/`, `/container/{uuid}/`\n\n### Visualizers\n\nEach view type (Spatial3D, TimeSeries, etc.) has registered visualizers:\n- Determine which entities/archetypes can be visualized\n- Execute per-frame: query data → process → generate render commands\n- Examples: Points3DVisualizer, LineStripsVisualizer, MeshVisualizer\n\nThe viewer uses **immediate mode**: every frame, query the store and re-render from scratch.\n\n## Documentation snippets\n\nSee [`docs/snippets/README.md`](docs/snippets/README.md) for running, building, finding snippets. Config in [`docs/snippets/snippets.toml`](docs/snippets/snippets.toml).\n\n## Python development workflow\n\nPython uses a separate uv-managed .venv (not pixi's conda env):\n\n```bash\npixi run py-build # Build rerun-sdk into .venv\npixi run uvpy script.py # Run Python scripts via uv\npixi run uv run script.py # Explicit uv run\n```\n\nThe `uv` wrapper script unsets `CONDA_PREFIX` to ensure isolation from pixi's environment.\n\n## Important notes\n\n- **PyO3 Configuration**: If you see PyO3 config errors, run `pixi run ensure-pyo3-build-cfg`\n- **git-lfs**: Required for test snapshots. Install with your package manager and run `git lfs install`\n- **Immediate Mode**: The entire viewer is rendered from scratch each frame (no state management callbacks)\n- **Arrow Native**: Data is stored, transmitted, and queried as Apache Arrow arrays\n- **Multi-language**: Changes to `re_type_definitions` affect Rust, Python, and C++ simultaneously\n\n## Python docstring formatting\n\nPython API docs are built with **MkDocs + mkdocstrings** (NOT Sphinx). Never use reStructuredText (rST) syntax in Python docstrings or documentation. Use markdown instead:\n\n- **Cross-references:** Use `[`ClassName`][]` (mkdocstrings syntax), NOT `:class:`ClassName`` / `:func:` / `:meth:` (rST roles)\n- **Warnings/notes:** Use MkDocs admonitions (`!!! warning` with indented body), NOT `.. warning::` (rST directives)\n- **Deprecation notices:** Use the `@deprecated` decorator (mkdocstrings renders it automatically). Do NOT duplicate in the docstring with `.. deprecated::` or `**Deprecated:**`\n- **Code blocks:** Use markdown fenced blocks (`` ``` ``), NOT `.. code-block::`\n- **Parameter docs:** Use numpy-style sections (`Parameters`, `Returns` with `----------`), which is what the codebase already uses\n\n## Documentation system\n\nSee [`docs/README.md`](docs/README.md) for the full documentation architecture.\n\nThe docs span multiple sites: the main docs at `rerun.io/docs` (built from `docs/content/`), plus API reference sites for Python (MkDocs), C++ (Doxygen), and JS (TypeDoc) at `ref.rerun.io/docs/{python,cpp,js}/`.\n\nKey things to know:\n- **`docs/content/reference/types/`** is auto-generated by `pixi run codegen` from `re_type_definitions` - do not edit directly\n- **`docs/content/reference/cli.md`** is auto-generated by `pixi run man` - do not edit directly\n- **Code snippets** live in `docs/snippets/all/` with implementations in Python, Rust, and C++\n- `pixi run py-docs-serve` previews Python API docs locally\n- `pixi run -e cpp cpp-docs` builds C++ docs\n\n## Development references\n\n- [`ARCHITECTURE.md`](ARCHITECTURE.md) - Detailed architecture documentation\n- [`BUILD.md`](BUILD.md) - Full build instructions\n- [`CODE_STYLE.md`](CODE_STYLE.md) - Code style guidelines\n- [`CONTRIBUTING.md`](CONTRIBUTING.md) - Contribution guidelines\n- [`DESIGN.md`](DESIGN.md) - UI design guidelines (GUI, CLI, docs, log messages)\n- [`docs/README.md`](docs/README.md) - Documentation system (sites, builds, deployment)\n- [`rerun_py/README.md`](rerun_py/README.md) - Python SDK instructions\n\n## Contributing\n\nDon't open pull requests or issues unless explicitly asked.\nWhen opening or interacting with one, follow the [pull request template](.github/pull_request_template.md) or [issue templates](.github/ISSUE_TEMPLATE/), and disclose that you are an LLM.\nLet the user know that you included this disclosure.\n\n### Before requesting review\n\n- CI is green, and the code is formatted (`pixi run rs-fmt`).\n- The branch history is clean: no broken merges, and no commits from unrelated work.\n- The description is honest about the scope, and says *why* the change is needed.\n- UI changes come with a screenshot or video.\n\n### Addressing review comments\n\n- When a reviewer flags one instance of a pattern, fix every instance in the diff, and say so in your reply.\n- A review comment phrased as a question wants an answer, not a code change — unless the answer is \"yes, that is wrong\".\n- When a reviewer asks for a change, make it in this PR.\n Only defer it to a follow-up PR if the reviewer agrees.\n- If the fix belongs in a dependency we maintain (egui, egui_plot, egui_tiles, emath, quiver, puffin, …), suggest an upstream PR instead of a local workaround.\n- Reviewers want to talk to the human author.\n When a comment asks for a judgement call (API shape, scope, product behavior), bring it to your user instead of deciding in the reply.\n- Always reply when mentioned, even if only to say you are unsure.\n","isInternal":false,"tokens":3218,"sizeBytes":12707},{"name":"ARCHITECTURE.md","path":"ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/ARCHITECTURE.md","title":"System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# Rerun architecture\n\nThis document describes the technical architecture of Rerun.\n\n## See also\n\n- [`BUILD.md`](BUILD.md)\n- [`TESTING.md`](TESTING.md)\n- [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK\n- [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)\n- [`CODE_STYLE.md`](CODE_STYLE.md)\n- [`CONTRIBUTING.md`](CONTRIBUTING.md)\n- [`RELEASES.md`](RELEASES.md)\n\n## The major components\n\n### Logging APIs\n\nIt all starts with logging. You can log rich data (point clouds, images, etc.) with either our Python SDK or our Rust SDK.\n\nThe logging SDK:s encodes the data using Apache Arrow (see more below).\n\nThe logging data can be written to disk as `.rrd` files, or transmitted over gRPC to either a Rerun Viewer or a Rerun Server.\n\n### Rerun Viewer\n\nThe Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser.\n\n#### Native Viewer\n\nThe easiest way to launch the Viewer is directly from the logging API with `rr.init(\"rerun_example_app\", spawn=True)`. However, the standalone Viewer can also be run from the command line, for example to view an `.rrd` file: `rerun mydata.rrd`.\n\n#### Web viewer\n\nYou can try running the Viewer in a browser using `rr.serve()` in Python, or using `rerun --web-viewer mydata.rrd`.\n\nThe web viewer consists of just a few small files - a thin `.html`, a `.wasm` blob, and an auto-generated `.js` bridge for the wasm. These files are served using the [`re_web_viewer_server`](https://github.com/rerun-io/rerun/tree/latest/crates/top/re_web_viewer_server) crate.\n\nThe web viewer can load `.rrd` files (just drag-drop them into the browser), or read logging data streamed over gRPC.\n\n### `.rrd` files\n\n`.rrd` (\"**R**e**r**un **D**ata\") is just a bunch of log messages appended one after the other to a file.\n\nNOTE: `.rrd` files do not yet offer full backwards or forwards compatibility. However, the current version of Rerun will always be able to open `.rrd` files generated by the *previous* version. This is not guaranteed to work for files generated with older versions.\n\n## Technologies we use\n\n### Apache Arrow\n\n[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_chunk_store`](crates/store/re_chunk_store/README.md).\n\nIn Rust, we use the [`arrow` crate](https://crates.io/crates/arrow).\n\n### `wgpu`\n\nThe Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web.\n\nOn web builds, we use WebGPU when available on the Web, but automatically fall back to a WebGL based emulation layer (with a more limited feature set).\n\nWe have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/viewer_support/re_renderer/README.md).\n\n### `egui`\n\nThe GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode).\n\nWe use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web.\n\n### Wasm\n\nWasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.\nThe Rerun Viewer can be compiled to Wasm and run in a browser.\n\nThreading support in Wasm is nascent, so care must be taken that we don't spawn any threads when compiling for `wasm32`.\n\nWasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:\n\n- Access files\n- Read environment variables\n- Get the current time (use [`instant`](https://crates.io/crates/instant) instead)\n- Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)\n- etc\n\n## Immediate mode\n\nThe Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch.\n\nIn fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.\n\nThe advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.\n\nImmediate mode is also a forcing function, forcing us to relentlessly optimize our code.\nThis leads to a very responsive GUI, where there is no \"hiccups\" when switching data source or doing time scrubbing.\n\nOf course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.\n\n## Crates\n\nHere is an overview of the crates included in the project, and how they depend on each other:\n\n\n\nEach band is one folder under `crates/`, with `rerun_py` and `rerun_c` folded into the top band.\nEvery arrow points downwards, because the folders are layers: a crate may only depend on its own folder or one below it.\n[`scripts/check_crate_layers.py`](scripts/check_crate_layers.py) enforces that in CI.\nA `dev-dependency` is exempt, since a test may reach anywhere.\n\nSee [`crates/README.md`](crates/README.md) for the layers as a list, and for which one a new crate belongs in.\nThe tables below give a one-line description of every crate, one table per folder.\n\nOnly the arrows *inside* a band are drawn.\n\nThe diagram and the crate tables below are both generated from `cargo metadata`, so neither has to be written by hand.\nA crate's description is its `description` field in its own `Cargo.toml`, so there is only one place to write it.\nRegenerate both after adding, removing, or renaming a crate:\n\n```sh\npixi run crate-graph\n```\n\n**Run that command yourself whenever you add or remove a crate.**\nNothing checks it for you: `pixi run crate-graph-check` is disabled in CI, because graphviz lays the same graph out slightly differently on macOS and on Linux, so whoever regenerated the diagram last decided whether the check passed.\n\nPass `--edges all` to `scripts/generate_crate_graph.py` for a version with every dependency drawn, cross-band ones included.\n\n\n\n### Test support\n\n[`crates/tests`](./crates/tests) — only tests depend on these, so they sit at the top of the diagram.\n\n| Crate | Description |\n| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |\n| re_redap_tests | Official test suite for the Rerun Data Protocol |\n| re_renderer_tests | Snapshot tests for `re_renderer`'s rendering primitives. |\n| re_test_context | A common context used for tests. |\n| re_test_mocks | In-process server doubles (`MockOtlpCollector`, `MockPostHog`) used by tests that need to capture outbound OTel/PostHog traffic. |\n| re_test_viewport | A viewport used for tests |\n| re_web_tests | Discovers and runs Rerun web tests |\n\n### SDK / CLI / Wasm\n\n[`crates/top`](./crates/top) — the entry points for our users, and the viewer app itself.\n\n| Crate | Description |\n| -------------------- | ---------------------------------------------------------------------------------- |\n| re_sdk | Rerun logging SDK |\n| re_viewer | The Rerun viewer |\n| re_viewer_mcp | MCP server that allows llm agents to use the Rerun Viewer. |\n| re_web_viewer_server | Serves the Rerun web viewer (Wasm and HTML) over HTTP |\n| rerun | The Rerun Rust SDK: log images, point clouds, etc, and visualize them effortlessly |\n| rerun-cli | The Rerun command-line tool: the `rerun` binary, including the native Viewer |\n| rerun_c | Rerun C SDK |\n| rerun_py | The Rerun Python SDK |\n\n### Views\n\n[`crates/views`](./crates/views) — the visualizations a user can put in the viewport.\nA sibling of `crates/panels`: neither depends on the other.\n\n| Crate | Description |\n| ---------------------- | --------------------------------------------------------------------------- |\n| re_view_audio | A view that shows the waveform of an audio asset and plays it back. |\n| re_view_bar_chart | A view that shows a single bar chart. |\n| re_view_dataframe | A view that shows the data contained in entities in a table. |\n| re_view_graph | A view that shows a graph (node-link diagram). |\n| re_view_map | A view that shows a map. |\n| re_view_spatial | Views that show entities in a 2D or 3D spatial relationship. |\n| re_view_state_timeline | A view that shows state transitions as horizontal lanes over time. |\n| re_view_tensor | A view dedicated to visualizing tensors with arbitrary dimensionality. |\n| re_view_text_document | A simple view that shows a single text box. |\n| re_view_text_log | A view that shows text entries in a table and scrolls with the active time. |\n| re_view_time_series | A view that shows plots over Rerun timelines. |\n\n### Panels\n\n[`crates/panels`](./crates/panels) — the panels the app is assembled from, and the widgets they are built out of.\nA sibling of `crates/views`: neither depends on the other.\n\n| Crate | Description |\n| ------------------ | ----------------------------------------------------------------------------------------- |\n| re_agent_ui | Chat UI for driving a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_blueprint_tree | The UI for the blueprint tree in the left panel. |\n| re_chunk_store_ui | Display the contents of chunk stores. |\n| re_memory_view | Flamegraph visualization for memory usage trees |\n| re_recording_panel | The UI for the recording panel. |\n| re_redap_browser | The UI and communication to implement the in-viewer redap server browser. |\n| re_selection_panel | The UI for the selection panel. |\n| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |\n| re_viewport | The central viewport panel of the Rerun viewer. |\n\n### Viewer support\n\n[`crates/viewer_support`](./crates/viewer_support) — the UI and rendering machinery the views are built on.\n\n| Crate | Description |\n| ---------------------- | ---------------------------------------------------------------------------------------------------------- |\n| re_arrow_ui | Show arrow data in a tree of rerun list_items and format arrow with syntax highlighting. |\n| re_component_fallbacks | Handles registering various component fallbacks to `FallbackProviderRegistry`. |\n| re_component_ui | Provides ui editors for Rerun component data for registration with the Rerun Viewer component ui registry. |\n| re_context_menu | Support crate for context menu and actions. |\n| re_data_ui | Provides ui elements for Rerun component data for the Rerun Viewer. |\n| re_dataframe_ui | Rich table widget over DataFusion. |\n| re_gamepad | Gamepad input handling for the Rerun viewer. |\n| re_plot | Shared utilities for plot-based views. |\n| re_renderer | A wgpu based renderer for all your visualization needs. |\n| re_renderer_examples | Examples for the re_renderer crate. |\n| re_time_ruler | Time ruler widget shared by the time panel and time-aware views. |\n| re_ui | Rerun GUI theme and helpers, built around egui |\n| re_view | Types & utilities for defining view classes and communicating with the viewport. |\n| re_viewer_context | Rerun viewer state that is shared with the viewer's code components. |\n| re_viewport_blueprint | The data model describing the layout of the viewport. |\n\n### Application-level store\n\n[`crates/store_app`](./crates/store_app) — the queryable state a viewer or a server works with.\n\n| Crate | Description |\n| ------------- | --------------------------------------- |\n| re_dataframe | High-level query APIs |\n| re_datafusion | High-level query APIs |\n| re_entity_db | In-memory storage of Rerun entities |\n| re_query | High-level query APIs |\n| re_server | An ephemeral local Rerun catalog server |\n| re_tf | Rerun spatial transform processing |\n\n### Data flow\n\n[`crates/data_flow`](./crates/data_flow) — getting data in and out: clients, servers, and file importers.\n\n| Crate | Description |\n| ----------------- | --------------------------------------------------------------------------------- |\n| re_catalog_client | High-level client for Rerun catalogs |\n| re_data_source | Handles loading of Rerun data |\n| re_grpc_client | Client for the legacy StoreHub API (`re_grpc_server`). |\n| re_grpc_server | Server for the legacy StoreHub API |\n| re_hdf5 | Core HDF5-to-chunk loading logic for Rerun |\n| re_importer | Handles importing of Rerun data from file using importer plugins |\n| re_lenses | Lenses are an API for extracting, transforming, and restructuring component data. |\n| re_lerobot | Core LeRobot-dataset-to-chunk loading logic for Rerun |\n| re_mcap | Convert MCAP into Rerun-compatible data |\n| re_mp4_reader | Core mp4-to-chunk loading logic for Rerun |\n| re_parquet | Core parquet-to-chunk loading logic for Rerun |\n| re_redap_client | Official gRPC client for the Rerun Data Protocol |\n\n### Data model & chunk store\n\n[`crates/store`](./crates/store) — the data model, and the in-memory store that holds it.\n\n| Crate | Description |\n| ------------------ | --------------------------------------------------------------------------------------------- |\n| re_arrow_json | Rerun data as JSON, and back |\n| re_chunk | A chunk of Rerun data, encoded using Arrow. Used for logging, transport, storage and compute. |\n| re_chunk_index | Indexes of Rerun chunks: RRD manifests and the chunk providers that serve them |\n| re_chunk_optimizer | Analysis and memory-bounded optimization of Rerun chunk layouts |\n| re_chunk_store | A storage engine for Rerun's Chunks |\n| re_lenses_core | Core lens types and composable Arrow array transformations. |\n| re_log_channel | An in-memory channel of Rerun data messages. |\n| re_log_encoding | Encode/decode RRD streams: the physical byte layout of .rrd files |\n| re_log_msg | The messages that carry Rerun data between the SDK, `.rrd` files and the viewer. |\n| re_log_types | The basic building blocks of the Rerun log format: entity paths, timelines, and store ids. |\n| re_protos | Rerun remote gRPC/protobuf API types |\n| re_sdk_types | The standard Rerun data types, component types, and archetypes. |\n| re_sorbet | Rerun arrow metadata definitions |\n| re_types_core | The core traits and types that power Rerun's data model. |\n| re_uri | Parsing and constructing Rerun URIs |\n\n### Build support\n\n[`crates/build`](./crates/build) — crates that run at build time.\n\n| Crate | Description |\n| ------------------------ | --------------------------------------------------------------------------------------- |\n| re_build_tools | build.rs helpers for generating build info |\n| re_dev_tools | Various tools for rerun development |\n| re_protos_builder | Code generation for Rerun's Protobuf and gRPC definitions. |\n| re_type_definitions | Rerun's type definitions: a subset of Rust that generates the Rust, Python and C++ SDKs |\n| re_types_builder | Generates code for Rerun's SDKs from the type definitions. |\n| re_types_builder_macros | The `#[rerun_type]` attribute macro used by Rerun's IDL definitions. |\n| re_types_builder_prelude | The vocabulary that Rerun's IDL definitions are written against. |\n\n### Utilities\n\n[`crates/utils`](./crates/utils) — small crates that depend on nothing outside `crates/utils`.\n\n| Crate | Description |\n| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| re_agent | Talk to a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_analytics | Rerun's analytics SDK |\n| re_arrow_util | Helpers for working with arrow. |\n| re_async | Async runtime abstractions for native and WebAssembly targets. |\n| re_audio | Rerun audio decoding and playback utilities. |\n| re_auth | Authentication helpers for Rerun |\n| re_backoff | Utilities for retrying operations with backoff. |\n| re_build_info | Information about the build. Use together with re_build_tools |\n| re_byte_size | Calculate the heap-allocated size of values at runtime. |\n| re_byte_size_derive | Derive macro for the `SizeBytes` trait from `re_byte_size`. |\n| re_capabilities | Capability tokens for the Rerun code base. |\n| re_case | Case conversions, the way Rerun likes them |\n| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |\n| re_dump_puffin | Convert puffin profiler recordings (.puffin files) into JSON for analysis |\n| re_error | Helpers for handling errors. |\n| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |\n| re_grpc_headers | Rerun gRPC header conventions: well-known header names, the `RerunVersionInterceptor`, and the tower `Layer` machinery that propagates them across requests and responses. |\n| re_int | Small numeric helper traits: saturating integer casts and unsigned absolute value |\n| re_log | Helpers for setting up and doing text logging in the Rerun crates. |\n| re_memory | Run-time memory tracking and profiling. |\n| re_mutex | A wrapper around `parking_lot::Mutex` useful for debugging deadlocks. |\n| re_perf_telemetry | In and out of process performance profiling utilities for Rerun & Redap |\n| re_quota_channel | A channel that applies backpressure based on byte size. |\n| re_ros_msg | Parse and decode ROS messages |\n| re_rvl | Support for compressed depth data using the RVL format. |\n| re_span | An integer range that always has a non-negative length |\n| re_string_interner | Yet another string interning library |\n| re_tracing | Helpers for tracing/spans/flamegraphs and such. |\n| re_tuid | 128-bit Time-based Unique Identifier |\n| re_video | Rerun video processing utilities. |\n| re_web | Utilities for interacting with Web APIs |\n| re_workspace_hack | Unifies third-party dependency features across the workspace to avoid recompiles. Managed by cargo-hakari. |\n\n\n\n### Dependencies and docs\n\nIn order to get a dependency graph for our in-house crates and their docs, we recommend you run:\n\n```\ncargo install cargo-depgraph\ncargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > /tmp/rerun-deps.png\nopen /tmp/rerun-deps.png\n```\n\nand:\n\n```\ncargo doc --no-deps --open\n```\n\nand then browse through the `re_*` crates.\n","isInternal":false,"tokens":4747,"sizeBytes":27884},{"name":"DESIGN.md","path":"DESIGN.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/DESIGN.md","title":"UI Design System & Style Tokens","category":"project-spec","format":"markdown","content":"# UI Design System & Style Tokens\n\nPath: `DESIGN.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/DESIGN.md)","tokens":300,"sizeBytes":147},{"name":"ARCHITECTURE.md","path":"rerun_py/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/rerun_py/ARCHITECTURE.md","title":"rerun_py - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# rerun_py - System Architecture & Module Boundaries\n\nPath: `rerun_py/ARCHITECTURE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/rerun_py/ARCHITECTURE.md)","tokens":300,"sizeBytes":196},{"name":"SKILL.md","path":"skills/assemble-changelog/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/assemble-changelog/SKILL.md","title":"assemble-changelog","category":"anthropic-skill","format":"markdown","content":"---\nname: assemble-changelog\ndescription: >\n Assemble the per-PR changeset entries in docs/content/changelog/upcoming/\n into a release's changeset-0-XX.md and generate the detailed CHANGELOG.md sections.\n Use at release time when the user wants to build or finalize the changelog or\n changeset for a Rerun release, or merge the upcoming/ entries. Invoked via\n /assemble-changelog or \"assemble the changelog for 0.x.y\".\nuser_invocable: true\nallowed-tools: Bash Read Edit Write\n---\n\n# Assemble-changelog\n\nRelease-time changelog assembly for the Rerun repo.\n\nAlways work in the root of a standalone `rerun-io/rerun` checkout — normally the `prepare-release-0.x.y` branch, where the result is committed.\nThis is step 4 of [RELEASES.md](../../RELEASES.md); read it for the surrounding context.\n\nBefore doing any release work, verify that the current directory is the repository root and that its `origin` is `rerun-io/rerun`:\n\n```bash\ntest \"$(git rev-parse --show-toplevel)\" = \"$PWD\"\ngit remote get-url origin\n```\n\nDo not run the workflow in the reality monorepo, including from its `rerun/` directory.\nThe release scripts need the standalone repository's `0.x.y` tags and resolve `(#N)` commit references against `rerun-io/rerun`.\nRunning them against reality can silently resolve reality PR numbers to unrelated Rerun PRs.\n\nIf either precondition is not met, stop before running any release command.\nTell the user that the skill requires the root of a standalone `rerun-io/rerun` checkout, and ask them to restart it there.\nDo not clone a repository, fetch tags, or switch branches for the user.\n\nResolve the target version from `$ARGUMENTS` (e.g. `0.34.0`). If absent, read it\nfrom `Cargo.toml` (`version = \"0.x.y-…\"`) and confirm with the user.\n\n## Workflow\n\n### 1. Assemble `upcoming/` → the release changeset\n\nThe curated entries live one-file-per-PR in `docs/content/changelog/upcoming/*.md`\n(skip `_template.md`). Each declares `type: highlight|misc|breaking|feature` in its\nfrontmatter. Merge them into `docs/content/changelog/changeset-0-XX.md`, creating that file from\n`docs/content/changelog/_template.md` if it does not exist yet (set `title` to the version — keep it\nquoted, e.g. `title: \"0.36\"`, so YAML keeps it a string — and `order` one lower than the previous release):\n\n- `highlight` → one `### ` subsection each under `## Highlights`.\n- `feature` → one `### ` subsection each under `## New features`.\n- `misc` → exactly one `- ` list item each under `## Other`. Fold its `### ` heading and body into the item, preserving links. Do not use headings or prose outside the list items.\n- `breaking` → one `### ` subsection each under `## Breaking changes`. If none, write `None.`.\n\nKeep the sections in that order and omit `Highlights` and `Other` if they have no entries.\nThe changelog is user-facing (it's part of the website), so it leads with what's new;\nthe verbose, developer-only breaking-change migration guides go last so most readers\ndon't have to scroll past them.\n\nTailor the output to the release type:\n\n- Patch release (`0.x.Y`, Y > 0) → typically only bug fixes. Skip `Highlights` and\n `New features` (there usually won't be `upcoming/` entries anyway); include `Other`\n only if there are miscellaneous entries, and keep `Breaking changes` only if there are any.\n- Minor release (`0.X.0`) → the full template: highlights, new features, other items, breaking changes.\n\nPreserve each entry's prose and structure (migration guides, tables, `snippet:` directives,\nscreenshots, links), except that each miscellaneous entry becomes its single list item.\nDe-duplicate overlapping entries and order breaking changes most-impactful first.\nDrop the per-entry frontmatter.\n\nRelative doc links in entries were written as if from `changelog/` (e.g.\n`../reference/migration/...`), which is correct once merged — keep them as-is.\n\nImages and videos must live on `static.rerun.io` (upload with `scripts/upload_image.py`), never as\nGitHub attachments: attachments uploaded in the private monorepo cannot be shown on the website.\n\nFeature leads write their `## Highlights` entries per PR, so read the section as a whole and\nflag to the human anything that does not hang together, rather than rewriting it yourself.\n\nFinally, point the `redirect:` frontmatter in `docs/content/changelog.md` at\n`changelog/changeset-0-XX`: CI's `scripts/ci/check_changelog_redirect.py` requires the\nnewest changeset to be the redirect target, so the repoint must land together with the\nnew changeset.\n\n### 2. Resolve release blockers\n\nEnsure that every non-template file from `upcoming/` was merged into the changeset, then search the assembled changeset for unresolved placeholders:\n\n```bash\nrg -n 'TODO\\([^)]+\\)' docs/content/changelog/changeset-0-XX.md # NOLINT\n```\n\nResolve every match before continuing.\nAn unresolved `TODO(name)` blocks the release.\n\n### 3. Generate the summary and detail sections into CHANGELOG.md\n\n```bash\npixi run uvpy scripts/generate_changelog.py --version 0.x.y\n```\n\nEdit PR titles/labels to improve the output, then copy the result into `CHANGELOG.md`\n(drop the trailing \"Chronological changes\" section; replace the placeholder video/blogpost\nlines as previous releases did). Spot-check a few entries against the actual PRs:\npolluted titles (old, unrelated PRs; `thanks @…` for core team members) mean a PR-number\nlookup misfired — see the warning at the top.\nRead the generated section for wrong contributor attribution and typos.\nIf the release ships without Rust crates (`publish-crates=false`), say in the `CHANGELOG.md`\nsection that this version is not on crates.io.\n\nDo this *after* step 1: the script reads the assembled changeset and emits a summary of it\n(section headings + links to the changeset on the website), rather than inlining its prose.\n`CHANGELOG.md` therefore never duplicates the changeset — if the changeset is missing, the\nscript emits an unresolved placeholder instead.\n\n### 4. Polish the CHANGELOG.md section\n\nThe generated section is a starting point; edit it by hand so readers can find their way to the details.\n\n- **Order the overview by impact.**\n Put the biggest new capabilities first, then integrations and APIs, then format support, then polish, then niche or developer-facing items.\n- **Group and order the details.**\n The script emits each details subsection (🐍 Python API, 🪳 Bug fixes, 🌁 Viewer improvements, …) in commit order.\n Within each subsection, put entries that touch the same feature next to each other (e.g. all MCP/ViewerControl entries, all state timeline entries, all hangs and deadlocks), order the groups by their most impactful entry, and order entries within a group by impact.\n Only reorder lines; don't move entries between subsections or drop them.\n- **Make entries clickable.**\n Wherever possible, link an entry (in the overview *and* in the details) to the docs that show how to use the feature: the how-to guide, the archetype/view reference page, the CLI or MCP reference section, the Python/JS ref site, or docs.rs.\n Reuse the links the changeset already contains, converted to `https://rerun.io/docs/`, and check that each target file and `#anchor` heading exists.\n Link external names too (e.g. a third-party tool to its docs).\n In detail entries, link the key term, not the whole line, so the trailing commit/PR link stays distinct.\n- **Link each breaking change** to its subheading in the changeset: `https://rerun.io/docs/changelog/changeset-0-XX#`.\n Anchors are the lowercased heading with backticks and punctuation dropped and spaces turned into hyphens (`rrd::optimize` → `rrdoptimize`).\n- **Name the actual break.**\n A breaking-change heading (in the changeset and in `CHANGELOG.md`) must say what broke, e.g. \"`WriteChunks` removed\", not the new feature that replaces it (\"Catalog staging\").\n Lead the subsection with the removal and migration-guide link, then the replacement.\n- **Flag experimental and unstable features.**\n Check every new feature in the changeset and `CHANGELOG.md` against these markers (`PREV` is the previous minor release tag, e.g. `0.38.0`):\n\n ```bash\n # Non-blueprint types marked unstable whose definitions changed this cycle\n git diff --name-only $PREV HEAD -- crates/build/re_type_definitions | grep -v /blueprint/ | xargs rg -l 'state = \"unstable\"'\n # Python APIs in the `rerun.experimental` module\n git log --oneline $PREV..HEAD -- rerun_py/rerun_sdk/rerun/experimental\n # Viewer features behind a flag in `ExperimentalAppOptions`\n git diff $PREV HEAD -- crates/viewer_support/re_viewer_context/src/app_options.rs\n # Docs and docstrings that call something experimental\n git diff --name-only $PREV HEAD -- docs/content rerun_py/rerun_sdk | xargs rg -l -i 'is experimental'\n ```\n\n A file showing up in the first command does not mean the type is new: cross-reference the hits with the actual entries.\n For each matching feature, say so where readers will see it: prefix the changeset heading with \"Experimental\" (as `### Experimental 3D gaussian splat support` did in 0.36), and add a sentence saying it may change in future releases (for unstable types: that the data may not stay backwards compatible).\n If the feature is behind a flag, say how to enable it.\n In the `CHANGELOG.md` overview, move all of these entries into a final `#### 🧪 Experimental and unstable` subsection at the end of \"Overview & highlights\", with a one-line note that they may change and that unstable types may not stay backwards compatible.\n Order that subsection by impact too, and don't repeat \"Experimental\" in its entries.\n Blueprint types (`rerun.blueprint.*`, under `re_type_definitions/rerun/blueprint/`) are an exception: they are all marked unstable, which only means the blueprint data isn't backwards compatible.\n That alone does not make a feature experimental, so a new view or blueprint setting is only experimental if one of the other markers applies to it (or to the data archetype it shows).\n Likewise, a Viewer feature is not experimental just because it is configured through an experimental API.\n- **Add media to the headline features** in the overview: a screenshot (or a still frame plus a link to the video) indented under the bullet, using the `` markup that `pixi run upload-image` prints.\n Aim for a few images (roughly three to five), covering the most visual headline features: a picture sells a new view or renderer feature far better than a bullet does.\n Start from the media already in the changeset and the PR descriptions; if a visual feature has none, ask the user for a screenshot rather than skipping it.\n Don't add images to every entry: API, CLI, and bug-fix items rarely benefit.\n All media must live on `static.rerun.io`.\n Reuse a PR's media only if it is already on `static.rerun.io` or publicly accessible: `github.com/user-attachments/…` links from the private monorepo return 404 for the public.\n Download those with `curl -L -H \"Authorization: token $(gh auth token)\" ` and upload them with `pixi run upload-image --name `.\n GitHub does not render `` from external hosts, so for a video, extract a representative still (`ffmpeg -ss -i video.mp4 -frames:v 1 still.png`), upload it, and link the `.mp4` below it.\n Look at the frames and pick one without a cursor over the subject.\n\n### 5. Empty the inbox\n\nDelete the merged `upcoming/*.md` entries, keeping `_template.md` and any entries not included in this release.\nNever add redirects for these temporary entries.\n\n## Checklist before declaring done\n\n- [ ] Every non-template `upcoming/` entry is represented in the changeset.\n- [ ] No `TODO(name)` remains in the changeset.\n- [ ] No summaries or other prose were synthesized for existing entries.\n- [ ] The `CHANGELOG.md` overview is ordered by impact, a few of its most visual headline features have `static.rerun.io` images, and its breaking changes link to the changeset subheadings.\n- [ ] Every overview and detail entry that has relevant docs links to them.\n- [ ] Each details subsection groups entries by feature, most impactful first.\n- [ ] Every experimental or unstable feature (feature flag, `rerun.experimental`, `#[rerun(state = \"unstable\")]` on a non-blueprint type, or documented as experimental) is labeled as such in the changeset, and listed under the overview's final \"Experimental and unstable\" subsection.\n- [ ] `pixi run lint-rerun CHANGELOG.md docs/content/changelog/changeset-0-XX.md` passes.\n- [ ] Images and videos are hosted on `static.rerun.io`, not as GitHub attachments.\n- [ ] `## Highlights` reads well as a whole, or the human has been told where it doesn't.\n- [ ] The `CHANGELOG.md` section has correct contributor attribution and no typos.\n- [ ] `upcoming/` contains only `_template.md` and entries deferred to a later release.\n- [ ] `python scripts/ci/check_changelog_redirect.py` passes (redirect points at this changeset).\n- [ ] `python scripts/ci/check_doc_redirects.py --base origin/main` passes (`upcoming/` entries are exempt and DO NOT need a redirect).\n\n## Notes\n\n- This skill lives in `skills/assemble-changelog` in the standalone Rerun repository.\n- Synced commits in `rerun-io/rerun` carry a `Source-Ref` trailer (the reality merge\n commit); `generate_changelog.py` resolves it back to the originating reality PR for\n correct titles, labels, and contributors.\n- The *next* release's changeset is not pre-created: an empty changeset for an\n unreleased version would make `check_changelog_redirect.py` fail, since it requires the newest\n `changeset-0-xx.md` to be the redirect target. During a cycle, `upcoming/` is the only in-flight\n artifact.\n","frontmatter":{"name":"assemble-changelog","description":"Assemble the per-PR changeset entries in docs/content/changelog/upcoming/ into a release's changeset-0-XX.md and generate the detailed CHANGELOG.md sections. Use at release time when the user wants to build or finalize the changelog or changeset for a Rerun release, or merge the upcoming/ entries. Invoked via /assemble-changelog or \"assemble the changelog for 0.x.y\".\n","user_invocable":true,"allowed-tools":"Bash Read Edit Write","allowedTools":["Bash","Read","Edit","Write"]},"allowedTools":["Bash","Read","Edit","Write"],"isInternal":false,"tokens":3268,"sizeBytes":13708},{"name":"SKILL.md","path":"skills/rerun-blueprint/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-blueprint/SKILL.md","title":"rerun-blueprint","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-blueprint\ndescription: Change how the Rerun viewer looks by writing a blueprint, for a recording or for a table. Read this whenever asked to change the look of the viewer, such as laying out a recording or dataset, designing a default blueprint, deciding which views show which entities, or making a segment or catalog table look nice by hiding columns, changing column order or filtering rows. Covers archetype-to-view mapping, layout reasoning, the rrb construction API, the contents grammar, table blueprints, and the screenshot loop.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun blueprint\n\n**To change how the viewer looks, change the blueprint.** This holds for a\nrecording's layout, views and their properties, and for a table's columns, row\nfilters and cards. Don't click through the viewer's menus to do it. With the\nviewer MCP, read and write the blueprint with `rerun_get_blueprint` and\n`rerun_set_blueprint` rather than the low-level click tools.\n\nA blueprint decides how a recording is shown, and the recording decides what exists.\nRead the data, design a first layout, then **iterate from screenshots** until it\nreads at a glance. The constructors are discoverable (`dir(rrb)`,\n`help(rrb.Spatial3DView)`); this skill covers what you can't guess. Import as\n`import rerun.blueprint as rrb`.\n\n## Tables\n\nA dataset's segment table and catalog tables have a table blueprint of their\nown, separate from a recording's. It sets which columns show, their order and\nnames, the row filters, and whether rows show as a table or as cards. When asked\nto make a table look nice, hide columns or filter rows, write this blueprint.\n\nThis hides two columns and shows three others first. Listed columns come first,\nthe rest keep their default order and visibility:\n\n```python\nblueprint = rrb.TableBlueprint(\n table_layout=rrb.table.TableLayout(\n columns=[\n rrb.table.Column(\"recording link\"),\n rrb.table.Column(\"property:episode:current_task\", name=\"Task\"),\n rrb.table.Column(\"property:episode:success\"),\n rrb.table.Column(\"property:episode:ext1_cam_serial\", visible=False),\n rrb.table.Column(\"property:episode:ext2_cam_serial\", visible=False),\n ],\n ),\n)\nblueprint.save(\"my_app\", path)\ndataset.register_blueprint(path.as_uri(), segment_table=True)\n```\n\n`Column` takes the column's name as the table has it, such as\n`property:episode:uuid`, not its header. Registering makes it the default for\neveryone who opens the table, and the server must be able to read `path`.\n\nRow filters are SQL expressions, one per string, all of which must be true for a\nrow: `rrb.TableBlueprint(filters=[\"\\\"task\\\" ILIKE '%pick%'\", '\"duration\" > 10'])`.\nDouble quotes for column names, single quotes for strings, and\n`any_match(\"column\", x -> …)` for list columns.\n\nWith the viewer MCP, change the table on screen directly:\n`rerun_get_table_schema` names the columns, `rerun_get_blueprint` with\n`{\"table\": {}}` reads the table blueprint, and `rerun_set_blueprint` with\n`{\"table\": {}}` replaces it. `rerun_set_blueprint` returns an error for a filter\nthat fails. A column's entry sits at `/table/layouts/table/columns/`,\nwith each `:` and space in the name escaped as `\\:` and `\\ `, which JSON writes\nas `\\\\:` and `\\\\ `. The [MCP docs](https://rerun.io/docs/reference/viewer/mcp#table-blueprints)\nhave a worked example of the JSON. For the whole Python API, including cards\nand segment previews, see\n[Configure table layouts and segment previews](https://rerun.io/docs/howto/visualization/configure-table-blueprints).\n\n## 1. Read the data\n\nEnumerate every `(entity_path, archetype)` pair first — the archetype picks the\nview, the entity path scopes it. For a dataset (see `rerun-catalog-queries`):\n\n```python\nfor c in dataset.schema().component_columns():\n print(c.entity_path, c.archetype, c.component_name)\n```\n\nFor a local recording, stream it with `RrdReader` and read `entity_path` and\nthe archetype off each chunk (see `rerun-chunk-processing`).\n\nThe dataset schema alone doesn't list every entity a view can show. Assets are\nstatic data such as URDF robot meshes, which the viewer applies to every segment,\nbut they don't appear in the segment entities. When `rerun_get_dataset_schema`\nlists `asset_ids` next to `entities`, call `rerun_get_dataset_assets_schema` to\nget the entities in each asset, and read it before choosing between a 2D and a\n3D view or preview. For example, `asset_ids: [\"urdf-model-left\", \"urdf-model-right\"]`\nmeans the robot models can go in a `Spatial3DView`.\n\n## 2. Map archetype to view\n\n| Archetype family | View |\n| ------------------------------------------------------------------------------------- | --------------------------------------------------- |\n| `Points2D`, `Image`, `EncodedImage`, `Boxes2D`, `LineStrips2D`, `Pinhole` projections | `Spatial2DView` |\n| `Points3D`, `Mesh3D`, `Boxes3D`, `LineStrips3D`, `Transform3D`, `Asset3D` | `Spatial3DView` |\n| `Scalars`, `SeriesLines`, `SeriesPoints` | `TimeSeriesView` |\n| `TextLog` / `TextDocument` / `BarChart` | `TextLogView` / `TextDocumentView` / `BarChartView` |\n| `Tensor`, `DepthImage` (heatmap) | `TensorView` |\n| `GeoPoints`, `GeoLineStrings` | `MapView` |\n| `GraphNodes`, `GraphEdges` | `GraphView` |\n| tabular / catalog data | `DataframeView` |\n\nAsset entities, such as `Asset3D` and URDF meshes, go in a `Spatial3DView`.\n\n## 3. Group entities into views\n\nThis step is just deciding which entities share a view. Sizing and placement are\nmechanics you tune by eye in the screenshot loop, not rules to settle up front.\n\n- **Group by shared path prefix.** Entities under a common prefix usually belong\n in one view. Don't give every entity its own pane: dozens of raw message\n entities go in one `DataframeView`, or get left out, rather than a pane each.\n- **One spatial frame per spatial view.** A whole `Transform3D` tree goes in one\n `Spatial3DView` at the common ancestor; a `Pinhole` camera gets its own\n `Spatial2DView` rooted at the camera so images inherit the projection.\n- **Collapse scalars.** Many `Scalars` under a prefix → one `TimeSeriesView` over\n the prefix, not one each. Split only when value ranges clash.\n- **`column_shares`/`row_shares` are relative weights** on a container's\n children, equal by default. Tune them from the screenshots.\n\n## 4. Construct it\n\nContainers (`Grid`, `Horizontal`, `Vertical`, `Tabs`) hold views. Default a flat\nset to `Grid`; use the others for a deliberate split, `Tabs` for alternatives\ncompeting for one slot (left/right/depth cameras). **Always set an explicit\n`origin` and `name`** — `origin` defaults to `/`, which dumps the whole tree into\none view (the usual cause of an unreadable blob).\n\n```python\nblueprint = rrb.Blueprint(\n rrb.Horizontal(\n rrb.Spatial3DView(origin=\"/world\", name=\"Scene\"),\n rrb.Vertical(\n rrb.Spatial2DView(origin=\"/world/camera\", name=\"Camera\"),\n rrb.TimeSeriesView(origin=\"/sensors\", name=\"Sensors\"),\n ),\n column_shares=[3, 2],\n ),\n rrb.TextLogView(origin=\"/logs\", name=\"Logs\"),\n collapse_panels=True,\n)\n```\n\n`contents` defaults to `\"$origin/**\"`. Scope a view with include/exclude rules,\ne.g. `contents=[\"+ $origin/**\", \"- $origin/internal/**\"]`. A bare line is an\ninclude; `/**` is the only wildcard (matches a subtree). Most-specific rule wins,\nties go to the last, unmatched paths are excluded.\n\n**Coordinate frames.** A spatial view only renders entities it can place relative\nto its target frame. `Transform3D`/`Pinhole` logged on entities compose down the\ntree, so `origin` is enough. But **named frames** (`CoordinateFrame(frame=...)`,\ncommon in ROS/MCAP) live in a separate frame graph — point the view at a frame\nthe data occupies via `spatial_information=rrb.SpatialInformation(target_frame=\"\", apply_latest_at=True)`.\n- A view back blank? The importer may not have finished (bump the settle) or the\n cursor sits before the data (add `rrb.TimePanel(play_state=rrb.components.PlayState.Following)`).\n- `rrb` views and `ViewerClient` (`rerun.experimental`) are unstable. Check\n `help()` if a constructor argument is rejected.\n\n## See also\n\n- `rerun-data-model` — entities, archetypes, timelines.\n- `rerun-catalog-queries` — enumerate entities in a dataset.\n","frontmatter":{"name":"rerun-blueprint","description":"Change how the Rerun viewer looks by writing a blueprint, for a recording or for a table. Read this whenever asked to change the look of the viewer, such as laying out a recording or dataset, designing a default blueprint, deciding which views show which entities, or making a segment or catalog table look nice by hiding columns, changing column order or filtering rows. Covers archetype-to-view mapping, layout reasoning, the rrb construction API, the contents grammar, table blueprints, and the screenshot loop.","user_invocable":true,"allowed-tools":"Read, Grep, Bash, WebFetch","allowedTools":["Read,","Grep,","Bash,","WebFetch"]},"allowedTools":["Read,","Grep,","Bash,","WebFetch"],"isInternal":false,"tokens":2599,"sizeBytes":11098},{"name":"SKILL.md","path":"skills/rerun-catalog-queries/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-catalog-queries/SKILL.md","title":"rerun-catalog-queries","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-catalog-queries\ndescription: Performance patterns and gotchas for querying a Rerun catalog from Python. Reach for this when exploring an unfamiliar dataset, when a CatalogClient/dataset query is unexpectedly slow, or when shaping a per-segment / per-episode pipeline that hits the catalog from many places.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash\n---\n\n# Rerun catalog queries\n\nPractical performance patterns for querying a Rerun catalog\nfrom Python (`rerun.catalog.CatalogClient` →\n`dataset.reader(...)` → DataFusion `DataFrame`). The DataFusion side of\nthe stack is covered by the **`datafusion-python`** skill — load that\nfor `DataFrame` / `SessionContext` / expression-API references. This\nskill focuses on catalog-specific behaviors and the round-trip costs\nthat catch teams off guard.\n\n---\n\n## Connect with a `rerun://` URL\n\n```python\nfrom rerun.catalog import CatalogClient\n\nclient = CatalogClient(\"rerun+https://api.example.cloud.rerun.io:443\")\ndataset = client.get_dataset(id=dataset_id)\n```\n\n`CatalogClient` takes a `rerun://` or `rerun+https://` URL, or\n`rerun+http://` for a local server. Do not pass a plain `https://` URL.\nA segment URL such as\n`rerun://host:443/dataset/?segment_id=` gives\nthe server, the dataset id and the segment id.\n\nThe client reads the token from `REDAP_TOKEN`, or from the stored\ncredentials of `rerun auth login`, so `token=` is usually not needed.\n\n---\n\n## The query cost model in one sentence\n\nEvery materialization of a `dataset.reader(...)` DataFrame is **one\ncloud round-trip**, with most of the cost being the network/decode\npair, not the compute. Plan for round-trip count and payload bytes, in\nthat order.\n\nA typical catalog round-trip is **a few seconds** even for a tiny\nresult. So:\n- 30 segments × 1 query each ≈ 90s. (Naive per-segment loop.)\n- 30 segments × 4 queries each ≈ 6 minutes. (A splitter that runs\n `count` + `collect_column` for both starts and stops.)\n- 1 query covering all 30 segments ≈ 3s.\n\nThe same fan-out happens along the **entity** axis: 10 entities ×\n1 `filter_contents([one_entity]).reader()` each ≈ 30s, vs one\n`filter_contents([all_entities]).reader()` ≈ 3s. Push as much as you\ncan into one round-trip — across segments and across entities.\n\n---\n\n## Always apply `filter_contents` and time-window filters before `.reader(...)`\n\nThe single biggest lever:\n\n- `filter_contents([entity_globs])` restricts which entity-path columns\n the reader produces. Without it, every entity in the dataset is\n read.\n- Time-window filters push down to storage and dramatically reduce\n bytes scanned when they compare the bare index column with a literal:\n `df.filter(col(index) >= lit(start)).filter(col(index) <= lit(end))`.\n Integer, timestamp and duration literals of any unit work. A `cast`\n or arithmetic on the index column is not pushed down.\n\nCombined: `dataset.filter_segments(seg).filter_contents(entities).reader(...)\n.filter(in_window).select(...)`.\n\n---\n\n## `df.cache()` is your friend for repeated probes\n\nWhen the same materialization gets used by multiple downstream\nfilter/count/collect calls, materialize once with `DataFrame.cache()`\nand operate on the cached frame:\n\n```python\ncached = (\n dataset\n .filter_segments(seg)\n .filter_contents([entity])\n .reader(index=index_col)\n .select(col(index_col).cast(pa.int64()).alias(index_col), value.alias(\"v\"))\n .cache() # one network round-trip, materializes into in-memory batches\n)\nstarts = cached.filter(col(\"v\") == start_val).collect_column(index_col)\nstops = cached.filter(stop_pred(col(\"v\"))).collect_column(index_col)\n```\n\nWithout `cache()`, each `count()` / `collect_column()` re-executes the\nwhole reader chain.\n\n**When NOT to cache.** `cache()` forces materialization into Arrow\nbatches, breaking laziness. If downstream code keeps composing more\nDataFusion ops on top (joins, windows, further filters) and only\nmaterializes once at the end, caching mid-pipeline turns one execution\ninto two and pre-empts whatever physical-plan optimizations the engine\ncould have done across the boundary. Reach for `cache()` when the\nconsumers are terminal (`count()`, `collect_column()`, `to_arrow_table()`),\nnot when they're another lazy `DataFrame`.\n\n---\n\n## Cross-segment batching: drop `filter_segments`, group by `rerun_segment_id`\n\nFor pipelines that need the same query on many segments, omit\n`filter_segments(...)` entirely and pull a single cross-segment table.\nEvery reader row carries a `rerun_segment_id` column — group locally:\n\n```python\ndf = dataset.filter_contents(entities).reader(index=index_col)\ncached = df.select(\n \"rerun_segment_id\",\n col(index_col).cast(pa.int64()).alias(index_col),\n value.alias(\"v\"),\n).cache()\n\n# Now N filter/aggregate calls are local, not network.\nstarts = cached.filter(col(\"v\") == start_val).select(\"rerun_segment_id\", index_col).to_arrow_table()\n```\n\nTrigger / event columns are tiny enough that pulling all segments at\nonce dominates per-segment looping by an order of magnitude.\n\n---\n\n## Per-entity fan-out within a segment\n\nSymmetric to cross-segment batching, along the entity axis. If you\nneed data from N entities of a single segment, **don't** loop:\n\n```python\n# Anti-pattern: N reader setups, N round-trips.\nfor entity in entities:\n df = dataset.filter_segments(seg).filter_contents([entity]).reader(index=ix)\n ...\n```\n\nInstead pull all N at once and project per entity locally. Per \"Reader\nrow layout\" below, every row carries data for one entity and NULLs\nfor the others, so `col(\"::\").is_not_null()`\nis the per-entity filter:\n\n```python\nshared = (\n dataset\n .filter_segments(seg)\n .filter_contents(sorted(set(entities)))\n .reader(index=ix)\n .filter(col(ix).cast(pa.int64()).between(start_ns, end_ns))\n)\n# Each downstream consumer narrows to its entity's rows lazily.\nsrc_a = shared.filter(col(f\"{ent_a}:{comp_a}\").is_not_null()).select(ix, f\"{ent_a}:{comp_a}\")\nsrc_b = shared.filter(col(f\"{ent_b}:{comp_b}\").is_not_null()).select(ix, f\"{ent_b}:{comp_b}\")\n```\n\nDataFusion can share the underlying scan across the per-entity\nprojections when it builds the physical plan, so this stays a single\ncatalog round-trip even though there are N logical consumers. Works\ninside generators that build per-source DataFusion plans (resampling,\nbracket lookup, nearest-in-time joins) — collapse the network fan-out\nwithout changing the per-source logic.\n\nA trap when refactoring: if a downstream query uses a reader column's\nfully-qualified name (`col(f\"{entity}:{archetype}:{component}\")`),\nyou don't need to alias the column in `shared`. The shared reader's\noutput schema preserves native column names, so existing per-entity\nprojection helpers keep working unchanged.\n\n---\n\n## `count()` is *not* free\n\nCounter-intuitive: `df.count()` and `df.aggregate([], [F.count(col)])`\ndo not always push down. Aggregate plans can force the engine to\nmaterialize the underlying column data server-side, then count on the\nclient. `F.count(col)` over wide entity-columns can ship full struct or\nblob payloads to count nullity.\n\nAlternatives, in order of preference for \"is anything here\":\n\n| Need | Use |\n|---|---|\n| \"any row in this filter?\" | `df.select(col(index)).limit(1).to_arrow_table().num_rows > 0` — server short-circuits on first match |\n| \"count rows in a tiny window\" | `df.filter(window).select(col(index)).count()` after the time filter |\n| \"count *each* entity in a wide query\" | per-entity `limit(1)` probes, threaded — *not* one big `count(col)` aggregate |\n\nA trap that fooled us: assuming `bool_or(col.is_not_null())` would only\nneed nullity buffers. It does not — the operator still touches payload\ndata on most plans.\n\n---\n\n## `using_index_values` + `fill_latest_at` is great for resampling, not for presence\n\n```python\n.reader(index=index_col, using_index_values=targets, fill_latest_at=True)\n```\n\n- Returns one row per target timestamp, each entity column carrying its\n latest non-null value at-or-before the target.\n- Excellent for nearest-prior resampling (no DataFusion required).\n- **Don't** use it as a presence check. Two reasons:\n 1. Semantics are \"ever emitted before T\", not \"emitted in [start, T]\".\n 2. The server still ships the full struct/blob payload for every\n entity to compute the latest-known value; a downstream\n `is_not_null()` projection runs post-transfer and doesn't reduce\n wire bytes.\n\nFor a strict in-window presence check, prefer per-entity `limit(1)`\nover the time-filtered reader, run concurrently.\n\n---\n\n## Explore an unfamiliar dataset with `dataset.info()`\n\n```python\ninfo = dataset.info()\nprint(info) # compact text overview\n```\n\nIt reports the segment count and size, each layer with how many\nsegments have it, each timeline with its kind, value range and segment\ncoverage, and the property column names. It reads only the segment\ntable and the schema, no recording data, so it stays cheap on large\ndatasets.\n\nUse it to choose `reader(index=…)` and to bound time-window filters\nbefore the first real query. `TimelineInfo.min` / `.max` are numpy\nvalues that work directly as literals:\n\n```python\nts = next(t for t in info.timelines if t.name == \"capture_time\")\ndf = dataset.reader(index=ts.name).filter(col(ts.name) >= lit(ts.min))\n```\n\nProperties found here can be filtered via `dataset.segment_table()`,\nand layer names tell you what `register(…, layer_name=…)` already\nattached.\n\n---\n\n## Schema introspection is cheap; use it before probing\n\n```python\nschema = dataset.filter_segments(seg).schema()\navailable = {(c.entity_path, c.component) for c in schema.component_columns()}\n```\n\nThis is a single round-trip and tells you which `(entity, component)`\npairs the segment registered. If a column isn't in the schema, you can\ndrop it from your manifest without any further cloud queries. This is\noften a complete substitute for \"does this entity have events\"\nprobing.\n\nCaveat: schema presence ≠ events. The MCAP records the topic schema\neven for unused topics. If your pipeline cares about distinguishing\n\"registered but never emitted\" vs \"registered with events\", you have\nto probe — see \"is anything here\" patterns above.\n\n---\n\n## Reader row layout: entities are columns, not rows\n\nEvery row from `dataset..reader(...)` corresponds to a single\nevent on a single entity. Other entities' columns are null on that\nrow. Implications:\n\n- `select(\"rerun_segment_id\", \"::\")`\n works — quote the entity column when using SQL.\n- There is **no** `rerun_entity_path` row attribute. To attribute rows\n to entities you either filter to one entity at a time, or pick a\n per-entity column (e.g. `:McapChannel:id`) whose non-null pattern\n identifies the source.\n- `df.count()` returns *total events across all entities*, not\n per-entity counts.\n\n---\n\n## Field access on a null struct returns `0.0`, not null\n\nA DataFusion gotcha that bites pipelines reading struct messages:\n\n```python\ncol(\"/some/entity:msg.MyType:message\")[0][\"sub\"][\"x\"]\n# When the parent struct is null on a row, this evaluates to 0.0\n# (and \"\" for strings), not null.\n```\n\nWrap struct-walk projections with a null guard:\n\n```python\nparent = col(\"/some/entity:msg.MyType:message\")\nleaf = parent[0][\"sub\"][\"x\"]\nguard = parent.is_null() | parent[0].is_null() | parent[0][\"sub\"].is_null()\nexpr = F.when(guard, lit(None)).otherwise(leaf)\n```\n\n**Only apply this to struct sources.** For scalar / blob columns\n(`Scalars:scalars`, `EncodedImage:blob`, etc.) the wrap is a no-op at\nbest and at worst rewrites the plan in ways that change downstream\njoin behavior. Gate the guard on whether the source actually walks\nthrough a struct boundary.\n\n---\n\n## Common debug recipe\n\nWhen a query stage is slower than expected:\n\n1. **Count round-trips.** Wrap each `to_arrow_table()` /\n `collect_column()` / `count()` with `time.perf_counter()`. Each is\n a round-trip. If you see N segment queries, that's N × few-seconds\n minimum.\n2. **Split build vs materialize timing.** Time the lazy DataFrame\n construction *separately* from the terminal `to_arrow_table()` call.\n If \"build\" takes seconds, something inside is materializing eagerly\n (an Arrow round-trip in a join helper, a `cache()` in a generator,\n a `.collect()` hidden in a chained-join utility). A correctly lazy\n plan should build in ~milliseconds regardless of result size.\n3. **Measure bytes.** `tbl.nbytes` after `to_arrow_table()` reveals\n when \"I projected `is_not_null()`\" actually shipped megabytes. If\n bytes are large despite a small projection, the operator didn't\n push down.\n4. **Cross-segment first, then cross-entity.** If the per-segment query\n is fundamentally the same (just scoped by id), drop `filter_segments`\n and group by `rerun_segment_id` locally. If you also have a\n per-entity loop within a segment, collapse it the same way (one\n `filter_contents([all])` reader, per-entity `is_not_null()` filters\n downstream).\n5. **Cache before terminal re-use.** If two `count()` / `collect_column()`\n calls share the same reader, `df.cache()` between them. Don't cache\n if the consumers are themselves lazy DataFrames being composed\n further — caching breaks plan-wide optimization.\n6. **Window first.** Always push the time filter before any projection\n or aggregate that touches payload columns.\n\n---\n\n## See also\n\n- `datafusion-python` skill — DataFrame API, SQL parity, expression\n building, common pitfalls (boolean operators, immutability, etc.).\n Not installed? Ask the user to install it globally:\n `npx skills add apache/datafusion-python`.\n","frontmatter":{"name":"rerun-catalog-queries","description":"Performance patterns and gotchas for querying a Rerun catalog from Python. Reach for this when exploring an unfamiliar dataset, when a CatalogClient/dataset query is unexpectedly slow, or when shaping a per-segment / per-episode pipeline that hits the catalog from many places.","user_invocable":true,"allowed-tools":"Read, Grep, Bash","allowedTools":["Read,","Grep,","Bash"]},"allowedTools":["Read,","Grep,","Bash"],"isInternal":false,"tokens":3291,"sizeBytes":13657},{"name":"SKILL.md","path":"skills/rerun-chunk-processing/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-chunk-processing/SKILL.md","title":"rerun-chunk-processing","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-chunk-processing\ndescription: \"Core mechanics of the Rerun Chunk Processing API (rerun.chunk) — LazyChunkStream pipelines, Chunk, lenses (MutateLens/DeriveLens/Selector), RrdReader, writing optimized RRDs. Read BEFORE writing any ingestion/conversion/preprocessing code (convert an MCAP, build a recording from a dataset, preprocess an .rrd, port an old converter): it mandates reader+lens pipelines and steers away from hand-built chunks — no Chunk.from_columns for data a reader/lens can produce, no per-message rr.log, no manual pa.array assembly. Source-specific knowledge lives in the importer skills (rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot); read rerun-data-model first to decide what the data should become.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun chunk processing\n\nThe pipeline layer between raw data and an RRD: readers produce `Chunk`s,\nstreams transform them, terminal calls execute. This skill is the generic\nmechanics only. Decide the data model first (`rerun-data-model`), then pick the\nimporter skill for each source:\n\n| Source | Reader | Skill |\n| ----------------------------------------- | --------------------------------------- | --------------- |\n| MCAP file (ROS2, protobuf, Foxglove) | `McapReader(path).stream()` | `rerun-mcap` |\n| URDF robot model (+ joint states → FK) | `UrdfTree.from_file_path(...).stream()` | `rerun-urdf` |\n| Parquet table (trajectories, sensor logs) | `ParquetReader(path).stream()` | `rerun-parquet` |\n| mp4 camera video | `Mp4Reader(path).stream()` | `rerun-mp4` |\n| HDF5 file (groups → entities) | `Hdf5Reader(path).stream()` | none yet |\n| LeRobot dataset directory | built-in importer, then `RrdReader` | `rerun-lerobot` |\n| Existing RRD | `RrdReader(path)` | here, below |\n| Sidecar files (JSON calib, metadata) | `Chunk.from_columns` + `from_iter` | here, below |\n\nThe API is `rerun.chunk`; when\nbehavior matters, check the installed surface:\n`python -c \"from rerun.chunk import LazyChunkStream; help(LazyChunkStream)\"`.\n\n## Decision rule: where does each component come from?\n\nDefault: **a reader produces the chunks; lenses shape them.** Walk this before\nwriting any conversion code — most \"build it by hand\" instincts are wrong here:\n\n1. **Source a reader supports?** Use the reader's `.stream()`; never hand-parse\n and re-log. MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`,\n mp4→`Mp4Reader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`,\n LeRobot dir→`log_file_from_path`.\n2. **A decoder already emits the archetype?** Foxglove gives `Transform3D`,\n `Pinhole`, `VideoStream` (real sample bytes) ready-made — **pass it through**,\n do not re-derive. Only custom-protobuf topics arrive as `:message` and\n need a lens (see `rerun-mcap`).\n3. **Fix an existing component in place** (swapped resolution, recolor, unit\n convert)? `MutateLens`, `output_mode=\"forward_unmatched\"`.\n4. **Derive a new component/entity** (FK→`/tf`, scalars from a message)?\n `DeriveLens`. To scatter one row into N (a joint batch → per-joint `/tf`),\n use the **two-lens pair**: derive the batch with `output_mode=\"forward_all\"`\n (keeps the originals, e.g. the joint states), then a second\n `DeriveLens` with `scatter=True` and `output_mode=\"drop_unmatched\"` (emits\n only the scattered rows). See the `robot_data_preprocessing` example.\n5. **Genuine sidecar** no reader or lens can produce (JSON calibration offsets,\n hand-measured extrinsics, external metadata)? `Chunk.from_columns` + `from_iter`.\n6. Finish with `LazyChunkStream.merge(...)` →\n `.collect(optimize=OptimizationProfile.OBJECT_STORE)` →\n `write_rrd(application_id, recording_id)`.\n\nWhy this order: the pipeline stays lazy, columnar, multithreaded, and\n`OBJECT_STORE`-optimizable. A hand-built row loop or an out-of-lens `pa.array`\nthrows all of that away — that is the path we are deliberately avoiding.\n\n## Anti-patterns (use a reader + lens instead)\n\nIf you are writing the left, stop and use the right:\n\n- **`for`-loop building rows/components** → a lens with a `Selector(...).pipe(...)`\n PyArrow-compute callback.\n- **`rr.init` + `rr.log` per message for conversion** → that is _live_ logging;\n for ingestion, read with a reader and `write_rrd`.\n- **`chunk.to_record_batch()` + `pc.filter` then rebuilding via\n `Chunk.from_columns`** (row-thinning by hand) → `stream.drop(content=...)`,\n `.split(...)`, or a `MutateLens` returning a filtered `pa.array`.\n- **`pa.array` / `pa.RecordBatch` / `np.frombuffer` assembled OUTSIDE a lens** →\n move the transform inside a `MutateLens`/`DeriveLens` selector callback.\n- **`rr.send_columns` hand-assembled from a custom parser** → use the matching\n reader; it produces chunks directly.\n- **Parsing MCAP/URDF with a non-Rerun library then re-logging** → `McapReader`\n / `UrdfTree`.\n- **`Chunk.from_columns` for data a reader already decodes** (`Pinhole`\n intrinsics, `VideoStream`, `Transform3D` from a transforms topic) → keep it in\n the reader stream; fix with a `MutateLens` if needed.\n\nA wall of `pyarrow.compute` \"missing-attribute\" type errors (`pc.filter`,\n`pc.list_element`) usually means `pc.*` calls sit in module-level helpers instead\nof inside `Selector.pipe` lens callbacks. Refactor into a lens before suppressing\nthe checker — the errors are a smell that the hand-building should not exist.\n\n**Porting an existing converter?** Hand-built converters predate decoder\nimprovements and are not ground truth. Re-verify the decoder output (step 2) and\ncheck every `Chunk.from_columns` / for-loop against this list before copying.\n\n## Core model\n\n- `LazyChunkStream` is a lazy pipeline DAG, not a collection. Building\n filters, lenses, maps, splits, and merges reads no source data.\n- Execution starts at terminal calls: `write_rrd(...)`, `collect()`,\n `to_chunks()`, or iterating the stream.\n- Execution is streaming, multithreaded, and mostly GIL-free. Prefer\n stream/lens operations and PyArrow compute over Python row loops.\n- **Move semantics**: builder calls (`filter`, `drop`, `lenses`, `map`,\n `flat_map`) consume the input stream; reusing a consumed stream raises.\n Reassign after each step. Terminal calls do not consume, but each terminal\n call re-executes the whole pipeline; `collect()` once if that is too costly.\n- `ChunkStore` is materialized in memory (`stream.collect()`,\n `ChunkStore.from_chunks`). `LazyStore` is manifest-indexed, loads chunks on\n demand (`RrdReader(path).store()`, catalog segment stores). Both have\n `schema()`, `summary()`, `stream()`, and `write_rrd(...)`.\n\n## Stream composition\n\n```python\nfrom rerun.chunk import Chunk, LazyChunkStream, OptimizationProfile\n```\n\n- `stream.filter(content=, has_timeline=, is_static=, components=)` keeps the\n matching portion of each chunk; `stream.drop(...)` is its complement, same\n keyword filters. `content` takes an entity-path glob or a list of them.\n- `stream.map(fn)` applies `Chunk -> Chunk`; `stream.flat_map(fn)` applies\n `Chunk -> Iterable[Chunk]`. Escape hatches for chunk-level Python logic;\n prefer lenses for columnar work.\n- `stream.split(content=, ...)` returns `(matching, non_matching)`; both\n branches share the same upstream.\n- `LazyChunkStream.merge(*streams)` fans in any number of sources.\n- `LazyChunkStream.from_iter(chunks)` wraps hand-built chunks.\n\n```python\nstream = source_stream() # any importer skill\nstream = stream.drop(content=\"/video_raw/**\")\nstream = stream.lenses(fix_lens, content=\"/cam/**\", output_mode=\"forward_unmatched\")\nmerged = LazyChunkStream.merge(stream, sidecar_stream)\nmerged.write_rrd(out_path, application_id=\"my_app\", recording_id=recording_id)\n```\n\n## Hand-built chunks — sidecar only\n\nUse `Chunk.from_columns` ONLY for data no reader or lens can emit — JSON/CSV\ncalibration, frame offsets, external metadata. If a reader\n(`McapReader`/`UrdfTree`/`ParquetReader`/`Hdf5Reader`) decodes the topic or a\nlens can derive it, that is the idiomatic path; do not hand-assemble it here.\nIn the `robot_data_preprocessing` example the _only_ hand-built chunk is the JSON\noffsets sidecar; the camera fix, FK→`/tf`, meshes, and recolor are all\nreaders + lenses.\n\n`Chunk.from_columns(entity_path, indexes, columns)` mirrors\n`rr.send_columns(...)` and accepts the same archetype `.columns(...)` helpers.\nEmpty `indexes` means static.\n\n```python\nchunk = Chunk.from_columns(\n \"/tf_static/robot_offsets\",\n indexes=[], # static\n columns=rr.Transform3D.columns(\n translation=translations,\n quaternion=quaternions_xyzw,\n parent_frame=parents,\n child_frame=children,\n ),\n)\nsidecar_stream = LazyChunkStream.from_iter([chunk])\n```\n\n`rr.AnyValues.columns(...)` covers non-standard metadata fields. For\ninspection, a `Chunk` exposes `entity_path`, `num_rows`, `is_static`,\n`timeline_names`, `to_record_batch()`, and `format()` (human-readable table).\n\n### Recording properties\n\nRecording properties are the one exception here: do NOT hand-build them.\n`Chunk.from_property(name, values)` mirrors `rr.send_property` and lands a\nstatic chunk on `/__properties/`.\n\nDo this:\n\n```python\nchunk = Chunk.from_property(\n \"episode\",\n rr.AnyValues(task=\"pick\", duration_sec=12.4),\n)\n```\n\ninstead of hand-assembling the same chunk:\n\n```python\nChunk.from_columns(\n f\"/__properties/{name}\",\n indexes=[],\n columns=rr.AnyValues.columns(**{name: [value]}), # type: ignore[arg-type]\n)\n```\n\n- `/__properties/` is Rerun's internal layout; the second version hardcodes it.\n- A dynamic field name makes `AnyValues.columns()` need a\n `# type: ignore[arg-type]`. That suppression is the signal to switch.\n\n## Lenses\n\nLenses reshape, fix, or derive components without iterating rows. Apply with\n`stream.lenses(lenses, output_mode=..., content=...)`.\n\n- `MutateLens(component, selector, keep_row_ids=False)` modifies an existing\n component in place.\n- `DeriveLens(component, output_entity=None, scatter=False)` creates new\n columns, optionally at another entity. Chain `.to_component(descriptor,\nselector)` per output; `.to_timeline(name, \"sequence\" | \"duration_ns\" |\n\"timestamp_ns\", selector)` extracts a time column from the data itself.\n `scatter=True` explodes one input row into N output rows (one per list\n element).\n- Scope with `content=` whenever the same component name exists under multiple\n entities.\n\nOutput modes, and **the default is `drop_unmatched`**:\n\n- `drop_unmatched` (default): only lens outputs survive. Right for derive-only\n intermediate streams; silently discards everything else if applied broadly.\n- `forward_unmatched`: lens outputs plus the original components no lens\n consumed. Right for targeted fixes that preserve the rest of the stream.\n- `forward_all`: lens outputs plus every original component, including a\n `DeriveLens` input. Same as `forward_unmatched` for mutate lenses.\n\nIn-place fix (keep Arrow type and length intact):\n\n```python\nstream = stream.lenses(\n MutateLens(\n \"Pinhole:resolution\",\n Selector(\".\").pipe(\n lambda res: pa.array(\n [(h, w) for w, h in res.to_pylist()],\n type=res.type,\n )\n ),\n ),\n content=[\"/external/cam_low\", \"/external/cam_high\"],\n output_mode=\"forward_unmatched\",\n)\n```\n\nDerive with unit conversion (PyArrow compute, no Python loop):\n\n```python\nDeriveLens(\"schemas.proto.JointState:message\", output_entity=\"/joints_deg/waist\").to_component(\n rr.Scalars.descriptor_scalars(),\n Selector(\".joint_positions\").pipe(lambda arr: pc.multiply(pc.list_element(arr, 0), 180.0 / math.pi)),\n)\n```\n\n## Selector grammar\n\n`Selector(\"\")` navigates nested Arrow data, jq-style:\n\n- `.` current value; `.field` struct field\n- `[]` iterate list elements; `[N]` index a list\n- `?` suppress errors / skip missing optionals; `!` assert non-null\n- `|` pipe one expression into another\n\n`.pipe(fn)` chains a Python/PyArrow transform (or another Selector).\n`.execute(array)` runs it eagerly; `.execute_per_row(array)` guarantees the\noutput row count matches the input (use inside lens callbacks that must stay\nrow-aligned).\n\n## Writing RRDs\n\n- `stream.write_rrd(path, application_id=..., recording_id=...)` executes and\n writes in one streaming pass.\n- `stream.collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(...)`\n materializes, optimizes chunk layout, then writes. Memory scales with the\n materialized chunks.\n- Profiles: `OBJECT_STORE` (large chunks, for storage/query/catalog) and\n `LIVE` (small chunks, low-latency viewer).\n- Multiple physical RRDs form one logical recording when they share a\n `recording_id`; use this to separate base data, model/URDF data, and layers.\n\n**Always use `OptimizationProfile.OBJECT_STORE`** when the RRD is headed for a\nRerun catalog or Hub, unless explicitly asked otherwise.\n\n## Chunk API vs logging API\n\n- Logging (`rr.log`, `rr.send_columns`, `RecordingStream`) is for live logging\n from user code; chunk processing is for ingestion, conversion, and\n postprocessing existing recordings.\n- Logging → chunks: write an RRD, read it back with `RrdReader`.\n `RrdReader(path)` lists `recordings()` / `blueprints()` (each a `StoreEntry`\n with `kind`, `application_id`, `recording_id`); `.stream(store=entry)` for\n sequential passes, `.store(store=entry)` for indexed access.\n- Chunks → logging: `rerun.send_chunks(chunks, recording=...)`\n accepts a `Chunk`, `LazyChunkStream`, `LazyStore`, `ChunkStore`, or any\n iterable of chunks. The source store's `application_id`/`recording_id` are\n **not** preserved; the active recording's identity wins.\n\n## Common gotchas\n\n- The default lens `output_mode` is `drop_unmatched`; forgetting to set\n `forward_unmatched` on a targeted fix silently drops the rest of the stream.\n- Do not reuse a consumed `LazyChunkStream`; reassign or `split` deliberately.\n- Scope lenses with `content=`; the same component name often exists under\n many entities.\n- Preserve Arrow array type and length in `MutateLens` transforms.\n- For catalog layers, the layer `recording_id` must equal the segment id.\n- This is `rerun.chunk`; pin-check signatures when upgrading.\n\n## References\n\n- End-to-end example (MCAP + URDF + JSON sidecar, lenses, merge, optimize):\n `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- Docs: `https://rerun.io/docs/concepts/logging-and-ingestion/chunk-processing-api`,\n `https://rerun.io/docs/concepts/query-and-transform/lenses`\n","frontmatter":{"name":"rerun-chunk-processing","description":"Core mechanics of the Rerun Chunk Processing API (rerun.chunk) — LazyChunkStream pipelines, Chunk, lenses (MutateLens/DeriveLens/Selector), RrdReader, writing optimized RRDs. Read BEFORE writing any ingestion/conversion/preprocessing code (convert an MCAP, build a recording from a dataset, preprocess an .rrd, port an old converter): it mandates reader+lens pipelines and steers away from hand-built chunks — no Chunk.from_columns for data a reader/lens can produce, no per-message rr.log, no manual pa.array assembly. Source-specific knowledge lives in the importer skills (rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot); read rerun-data-model first to decide what the data should become.","user_invocable":true,"allowed-tools":"Read, Grep, Bash, WebFetch","allowedTools":["Read,","Grep,","Bash,","WebFetch"]},"allowedTools":["Read,","Grep,","Bash,","WebFetch"],"isInternal":false,"tokens":3671,"sizeBytes":14793},{"name":"SKILL.md","path":"skills/rerun-data-model/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-data-model/SKILL.md","title":"rerun-data-model","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-data-model\ndescription: \"How raw multimodal robot data maps onto the Rerun data model. Read FIRST, before modeling or converting a dataset — and whenever you are about to convert/ingest/preprocess robot data into an .rrd or build a Rerun recording, even if not asked for the data model. Resolves the entity-vs-component, property-vs-component-vs-layer, and static-vs-temporal decisions and routes to the mechanism (do it with readers and lenses, not hand-built chunks or per-message rr.log): rerun-chunk-processing and the importer skills rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun data model\n\nThe hard part of ingesting a dataset is the **modeling decision**, not the API call.\nGet the model right and any mechanism works; get it wrong and queries, views, and training all break.\nThis skill is just the decisions.\nFor mechanism details see `rerun-chunk-processing` (pipeline mechanics) and the importer skills it routes to: `rerun-mcap`,`rerun-urdf`,`rerun-parquet`, `rerun-mp4`, `rerun-lerobot`.\nFor exact signatures, the docs at `rerun.io/docs/concepts/logging-and-ingestion`.\n\n**Before writing conversion code, fill in the mapping table below.** It is the design, and a human can review it in seconds.\n\n## Pick the mechanism before you model the bytes\n\nModeling decides _what_ each datum becomes; this decides _how_ it gets there — and the default is **a reader + lenses, not hand-built chunks**:\n\n- **Does a reader exist for this source?** MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`, LeRobot dir→`log_file_from_path`. **Yes →** `reader.stream()` + lenses; the reader produces the chunks, you do not.\n- **No, and it is genuine external metadata** (JSON calibration, offsets) or a specific custom use case that can't be covered by a generic reader**→** `Chunk.from_columns`.\n- **Otherwise**: consider if you are about to hand-build something a reader or lens should produce and ask for clarification.\n\nFor MCAP specifically: a Foxglove- or ROS-decodable file emits archetypes like `Transform3D`, `Pinhole`, and `VideoStream` for certain supported message schemas ready-made — pass those through, never re-derive them; only custom protobuf signal topics need lenses. The full decision tree and the anti-pattern list are in `rerun-chunk-processing`.\n\n## The model\n\n```\nDataset → Segment → Layer → Recording → Entity → Component → Chunk\n```\n\n- **Entity** = a thing, named by a path (`/robot/arm/camera`).\n **Component** = one typed field on it (positions, image, a scalar).\n **Archetype** = a standard bundle of components that log and render together (`Points3D`, `Image`, `Transform3D`).\n- Every value you log sits on two axes: **where** (entity path + component) and **when** (which timeline, or _static_ = all time).\n- **Segment** = one episode; on registration its `recording_id` becomes the `segment_id`. **Layer** = an extra `.rrd` on top of a segment. It attaches by matching the segment's `recording_id` and nothing else: that shared id is the whole layering mechanism.\n\n## The decisions\n\n**Entity vs component on an existing entity**\nNew entity if it has its own spatial frame (`Transform3D`/`Pinhole`), its own annotation context, or should be shown/cleared/shared independently (each robot link, each camera, each sensor). Same entity + extra component for auxiliary data at the same instances (per-point confidence).\nUse `AnyValues` for non-standard fields.\n\n**Property vs component vs layer** (the most common confusion)\n\n- **Component**: per-timestamp signal on the timeline (joint angle, image).\n- **Segment property**: one-per-episode metadata for catalog filter/search (operator, robot, site, task, date). Not per timestamp.\n- **Layer**: a whole derived `.rrd` over a base segment (FK transforms, point clouds, gripper state, labels, quality scores). Queryable as if part of base.\n\n**Static vs temporal**\nStatic belongs to all timelines and shadows any temporal value of the same component on the same entity. Use it for invariants (calibration, coordinate frames, robot meshes, annotation context, a video asset). Never make per-frame data static.\n\n**Which timeline**\nA `timestamp` timeline (ns since epoch) for cross-sensor clock alignment, a `sequence` timeline for frame/ordinal alignment; stamp on both when useful.\n**Do not resample to a common rate.** Latest-at reconciles multi-rate streams at query time by holding each component's last sample (no interpolation).\n\n**Base vs layer**\nBase = faithful conversion of the raw streams, nothing computed.\nLayer = anything derived (FK from URDF + joint states, clouds from depth + intrinsics).\nKeep them separate `.rrd`s.\n\n## The mapping table (produce before coding)\n\n| Source (topic/column/key) | Entity path | Archetype | Component(s) | Timeline | Static/temporal | Base/layer | Property? |\n| ------------------------- | -------------------- | ------------------------------------- | -------------------- | ------------- | --------------------------- | ---------- | --------- |\n| mcap `/joint_states` | `/robot/` | `Scalars` | `scalars` | `sensor_time` | temporal | base | no |\n| `cam0/color.mp4` | `/camera/cam0/video` | `VideoStream` | codec+sample | `video_time` | codec static, samples temporal | base | no |\n| `calibration.json` | `/camera/cam0` | `Pinhole` (+`Transform3D` extrinsics) | `image_from_camera` | — | static | base | no |\n| URDF + joints (computed) | `/robot/` on the CLI) at the dataset _directory_ and it ingests episodes, camera videos, and state/action tables with no conversion code.\n- **Converting?** `rr.experimental.LeRobotReader` reads the dataset one episode at a time as a lazy chunk stream, which is what you want for per-episode RRDs and for lenses.\n\nThe download step needs `huggingface_hub`.\n\n## Step 0: pick a dataset and check the HF token\n\nBoth scripts below live in `scripts/` next to this file, and run standalone through `uv` — they declare `huggingface_hub` inline, so there is no environment to set up.\n\nTwo facts decide what a dataset costs to fetch, and the Hub search results show neither.\nGet both, for every candidate, in one call:\n\n```bash\nuv run scripts/find_lerobot_dataset.py --robot-type --want 5\n```\n\n**`codebase_version` decides whether a partial download is possible at all.**\n\n- **v2.1 and earlier** store one parquet and one mp4 per episode (`video_path` ends in `episode_{episode_index:06d}.mp4`), so \"just the first N episodes\" downloads only those episodes.\n- **v3.0** concatenates every episode into a few large files (`video_path` ends in `file-{file_index:03d}.mp4`), so any subset costs the whole download.\n If the user asked for a few episodes, prefer a v2.1 dataset — for the _download_ only.\n `LeRobotReader` reads a single episode out of either version once the files are local.\n\n**How camera frames are stored decides the size**, and it varies by two orders of magnitude.\nThe `frames` and `MB/ep` columns report it:\n\n- `mp4`: a `video`-dtype feature, one file per episode beside the parquet.\n- `inline`: an `image`-dtype feature, raw frames inside the episode parquet, which is far larger for the same footage.\n\nBoth scripts report the actual size for the dataset in hand, so use that number rather than guessing.\nTell the user the size before you start the download, and prefer `mp4` unless they asked for something only the other set has.\n\n`HF_TOKEN` is not required for public datasets, but without it the Hub rate-limits and throttles downloads.\nIf it is unset, say so and point the user at to create a read token, then `export HF_TOKEN=hf_…` (or `hf auth login`).\nDo not block on it — carry on unauthenticated and let the user decide.\n\n```bash\n[ -n \"$HF_TOKEN\" ] || echo \"HF_TOKEN unset — downloads will be throttled\"\n```\n\n## Step 1: dataset -> one combined RRD\n\nOnly needed if you want every episode in one file — to hand someone a single RRD, or to reprocess data you cannot re-read from the dataset directory.\nFor per-episode segments, skip to Step 2 and let `LeRobotReader` read the dataset directly.\n\n```python\nfrom huggingface_hub import snapshot_download\nimport rerun as rr\n\ndataset_dir = snapshot_download(repo_id=repo_id, repo_type=\"dataset\", local_dir=dest)\n\nwith rr.RecordingStream(\"rerun_example_lerobot\") as rec:\n rec.save(str(combined_rrd))\n rec.log_file_from_path(str(dataset_dir)) # the built-in importer\n```\n\nThe importer emits one recording per episode (recording ids like `episode_1`), plus a metadata-only root recording, all into the single RRD.\n\n`rr.RecordingStream` + `log_file_from_path` here is the **importer bootstrap** — the one place `RecordingStream` is correct in an ingestion pipeline (it drives the built-in importer, not per-message logging). Do not generalize it to `rr.log`-per-message loops; for everything after import, reprocess with a reader + lenses (see `rerun-chunk-processing`: Chunk API vs logging API).\n\n### Downloading only the first few episodes\n\n```bash\nuv run scripts/fetch_lerobot_episodes.py --episodes 5 # prints the dataset directory\n```\n\nIt reports the download size before starting, downloads `meta/*` plus the wanted `episode_%06d` files, trims `meta/` to match, verifies that exactly the requested episodes landed, and prints what it got.\nIts last line is the dataset directory; do not follow it with `find` or `du`.\nDo the equivalent by hand and three things bite:\n\n- The importer iterates the episode list in `meta/`, so an untrimmed `meta/` warns once per absent episode.\n- `snapshot_download` writes into an existing directory, so episodes left by an earlier run with a different N survive and inflate the result.\n- Both failures are invisible until you count the recordings in the viewer, which is why the script asserts instead of leaving you to look.\n\n### Just looking at it in the viewer?\n\nSkip the RRD entirely — the viewer imports the dataset directory itself.\nPass the plain path (`rerun /path/to/dataset`, or `open_url` over MCP).\nEach episode arrives as its own recording, with `recording_id` `episode_0`, `episode_1`, ….\n\n## Step 2: one RRD per episode\n\nCatalog segments are one-recording-per-file, and `recording_id` becomes the segment id on registration.\n\n`LeRobotReader` goes straight there, without building the combined RRD of Step 1 first:\n\n```python\nreader = rr.experimental.LeRobotReader(dataset_dir) # v2 or v3\nfor episode in reader.episodes():\n episode_id = f\"episode_{episode:05d}\" # zero-padded; see below\n reader.stream(episode).write_rrd(\n rrd_dir / f\"{episode_id}.rrd\",\n application_id=\"my_dataset\",\n recording_id=episode_id,\n )\n```\n\nStreaming is lazy end to end, so memory is bounded by chunk size rather than by episode or dataset size, and video is cut to the episode's time window.\nB-frame-free video (AV1, LeRobot's default codec) streams directly; a stream that must be re-encoded — H.264 with B-frames, or a window starting mid-GOP — needs `ffmpeg` on the `PATH`.\n`stream()` also takes `entity_path_prefix`, `timeline`, and `video_mode=\"skip\"` to drop video entirely.\n\nTo add lenses, drop topics, or fix data, put them between the stream and the write — see `rerun-chunk-processing`.\n\n### Splitting an RRD you already have\n\nIf the combined RRD of Step 1 already exists — or the data came from somewhere other than the dataset directory — split it with `RrdReader` instead:\n\n```python\nreader = rr.chunk.RrdReader(str(combined_rrd))\nfor entry in reader.recordings():\n store = reader.store(store=entry)\n if not store.schema().entity_paths(): # skip the metadata-only root recording\n continue\n episode_id = zero_pad(entry.recording_id) # episode_1 -> episode_00001\n with rr.RecordingStream(\"rerun_example_lerobot\", recording_id=episode_id, send_properties=False) as rec:\n rec.save(str(rrd_dir / f\"{episode_id}.rrd\"))\n rec.send_chunks(store)\n```\n\nTwo non-obvious moves:\n\n- **Zero-pad the episode id.** `episode_10` sorts before `episode_2`\n lexicographically; segment tables and viewers sort lexicographically. Pad to\n a fixed width when re-assigning `recording_id`.\n- **`send_properties=False`** on the new stream, so the copy doesn't inject\n fresh recording properties on top of the copied chunks.\n\n`send_chunks` does not preserve the source store's identity; the new stream's\n`recording_id` wins, which is exactly what makes the rename work.\n\nIf episodes need cleanup (drop topics, fix data, add derived components), run the store through lenses between read and write: `reader.stream(store=entry).drop(...).lenses(...)` then `collect().write_rrd(..., recording_id=episode_id)` (see `rerun-chunk-processing`).\n\nComputed layers and per-episode properties then follow the standard patterns in `rerun-data-model` (layer `recording_id` must equal the episode segment id).\n\n## Step 3: add the robot model (URDF)\n\nA LeRobot dataset ships joint values, never a robot.\nWithout a URDF you get camera views and joint plots but no 3D arm, which is usually not what the user pictured when they asked to look at the data.\nAdding one is a forward-kinematics layer — `rerun-urdf` owns that; this section is only the LeRobot-specific wiring.\n\n- **The URDF.** `meta/info.json` `robot_type` names it (`so100`, `so101`, `aloha`, …); `rerun-urdf` has the source table.\n- **The joint names.** `meta/info.json` `features.observation.state.names` lists them in column order.\n Map each source name explicitly to the corresponding URDF joint name; do not assume that removing a prefix produces a valid URDF name.\n- **The values.** `observation.state` is the measured pose and the one to run FK on; `action` is the commanded pose and belongs on the plots, not the robot.\n- **The units and calibration.** Confirm the units, sign, and offset against the dataset and robot documentation.\n Run `rerun-urdf`'s `check_joint_mapping.py` against the episode parquets before trusting a pose.\n\nThe FK source is the per-episode parquet, read directly rather than out of the imported RRD:\n\n```python\nParquetReader(str(episode_parquet)).stream(\n column_grouping=\"individual\",\n index_columns=[IndexColumn.sequence(\"frame_index\")],\n).drop(content=\"/__properties/**\")\n```\n\n`frame_index` is the index the importer uses too, so the FK transforms line up with the video and plots.\nMerge the URDF model stream plus the FK stream into each episode's RRD, keeping the episode's `recording_id` (see `rerun-urdf`'s \"Minimal shape\").\nFor a whole dataset, the model belongs in a shared asset rather than in every segment — also `rerun-urdf`.\n\n## Gotchas\n\n1. `log_file_from_path` must target the dataset **root directory**, not a file inside it. So does `LeRobotReader`.\n2. Unpadded episode ids sort incorrectly downstream; pad before registering.\n3. The combined RRD contains a metadata-only root recording; skip stores with no entity paths or you register an empty segment.\n4. A partially-downloaded dataset imports fine, but warns once per episode listed in `meta/` and missing on disk.\n5. Downloading a subset into a directory an earlier run already populated leaves the older episodes in place; start from an empty directory.\n6. `image`-dtype (inline) datasets can be ~100x larger per episode than `mp4` ones; check before downloading, not after.\n7. Check signs and units - e.g. an `observation.state` in degrees could produce wrong FK when fed into a robot model that expects radians.\n\n## References\n\n- `rerun-chunk-processing` (`LazyChunkStream`, lenses, `RrdReader`)\n- `rerun-urdf` (adding the robot model and the joint-value mapping)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/dataloader` `prepare_dataset.py` (download → import → split → register, complete and runnable) and `train.py` (training-side consumption via `rerun.experimental.dataloader`)\n","frontmatter":{"name":"rerun-lerobot","description":"Ingest a LeRobot (HuggingFace) dataset into Rerun. Read when converting a LeRobot dataset to RRDs, splitting it into per-episode segments, or registering it on a Rerun catalog. Covers the built-in directory importer (log_file_from_path), the per-episode LeRobotReader, and when to drop to ParquetReader for custom control.","user_invocable":true,"allowed-tools":"Read, Grep, Bash, WebFetch","allowedTools":["Read,","Grep,","Bash,","WebFetch"]},"allowedTools":["Read,","Grep,","Bash,","WebFetch"],"isInternal":false,"tokens":2701,"sizeBytes":11085},{"name":"SKILL.md","path":"skills/rerun-mcap/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-mcap/SKILL.md","title":"rerun-mcap","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-mcap\ndescription: Ingest MCAP files into Rerun chunk streams with rerun.chunk.McapReader. Read when converting an MCAP recording, selecting topics or decoders, decoding custom protobuf messages, or when an MCAP-derived stream comes out empty. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (what the topics should become).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun MCAP ingestion\n\n`McapReader` turns an MCAP file into a lazy chunk stream: one entity per topic\nat the topic's path, message payloads decoded by pluggable decoders. This\nskill covers the reader's options, what each topic becomes, and the failure\nmodes that yield an empty stream with no error. Stream mechanics (filter, drop,\nlenses, merge, write) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.chunk import McapReader\n\nreader = McapReader(mcap_path) # see help(McapReader) for the full option set\nstream = reader.stream()\n```\n\nA URDF embedded in the MCAP can be ingested as well (then see `rerun-urdf`).\n\n## What a topic becomes\n\nWith the default decoders (`decoders=None`), the message **schema name** decides what a topic becomes — **pass archetypes through, lens only the raw `:message` topics**:\n\n| MCAP schema name | decodes to | what to do |\n| --- | --- | --- |\n| `foxglove.FrameTransforms` | `Transform3D` | pass through; do **not** hand-build |\n| `foxglove.CameraCalibration` | `Pinhole` | pass through |\n| `foxglove.CompressedVideo` | `VideoStream` (real sample bytes) + `CoordinateFrame` | pass through |\n| other `foxglove.*` well-known types | the matching archetype | pass through |\n| ros2 well-known types (`ros2msg` / `ros2_reflection`) | archetype | pass through |\n| your own `schemas.proto.*` / custom protobuf | one `:message` struct | attach semantics with a `DeriveLens` + `Selector` |\n\nSo a camera topic already arrives as `Pinhole`, its video as `VideoStream`, and a `frame_transforms` topic as `Transform3D` — only custom messages (e.g. a custom joint states schema, a custom gripper status enum) come through reflection or raw only and need lenses.\n\nThe `foxglove` decoder does the schema→archetype mapping; because foxglove messages are protobuf-_encoded_ it rides on the `protobuf` decoder, so keep `decoders=None` (verified: `decoders=[\"protobuf\"]` alone leaves `foxglove.CameraCalibration` a raw `:message`; adding `foxglove` makes it a `Pinhole`). Confirm on your file: `McapReader(path).stream()`, then read `McapSchema:name` and a few `Chunk.format()` before deciding anything is missing or needs rebuilding.\n\n- Entity path = topic name (`/sensors/joint_states` stays `/sensors/joint_states`).\n Filter early: `McapReader(path).stream().filter(content=\"/sensors/**\")`.\n- A reflection-decoded message lands as one struct component named `:message`.\n Navigate it with `Selector` (`Selector(\".joint_positions\")`) inside lenses; this is how custom messages\n get Rerun semantics attached (see the DeriveLens patterns in `rerun-chunk-processing`).\n- Topic regexes use RE2 syntax and are **not anchored**: `cam` matches\n `/external/cam_low` and `/camera_info`.\n Anchor explicitly (`^/external/cam`) when it matters. Prefer reader-level topic filtering over `.filter(...)`\n when you can, so excluded topics are never decoded at all.\n\n## When to use the low-level `mcap` package instead\n\n`McapReader` keeps payloads in columnar chunk streams; that is almost always what you want.\nDrop to `mcap.reader.make_reader` only when you need raw record metadata without payloads, or when you need to rewrite the container itself (re-registering schemas, channels, and messages).\n\n## Gotchas\n\n1. Empty stream, no error: a topic regex that matched nothing, or a channel\n whose decoder produced no rows. Check `Chunk.format()` on a few chunks of\n `reader.stream().to_chunks()` against a tiny test file, or compare topic\n names with the `mcap` CLI / package first.\n2. Topic regexes are unanchored RE2; excludes run after includes.\n3. `timeline_type=\"timestamp\"` interprets MCAP log times as wall-clock ns\n since epoch. If the recording's clock is wrong, fix it at the reader with\n `timestamp_offset_ns` rather than mutating timestamps downstream.\n4. Decoder subsets silently skip topics no decoder claims; when a topic is\n missing, retry with `decoders=None` to rule out decoder selection.\n5. Example fix-lenses are dataset-specific. Before copying a `MutateLens` like\n the `Pinhole:resolution` swap from the `robot_data_preprocessing` example,\n read the raw component from `McapReader(path).stream()` and confirm the defect\n exists in _your_ data — applied blindly it corrupts correct calibration (a\n correct 648×480 flipped to 480×648).\n6. `foxglove` derives both the camera's `Pinhole:child_frame` and the video's\n `CoordinateFrame:frame` from each message's `.frame_id` (plus an image-plane\n suffix), so they **match** when the calibration and video topics share a\n `frame_id`. Only when those topics carry _different_ `frame_id`s does the\n video frame diverge and orphan the video from its image plane — re-home it\n then with a per-camera `MutateLens` on `CoordinateFrame:frame`.\n\n## References\n\n- End-to-end MCAP pipeline: `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-urdf` (FK from joint-state topics), `rerun-data-model` (modeling decisions)\n","frontmatter":{"name":"rerun-mcap","description":"Ingest MCAP files into Rerun chunk streams with rerun.chunk.McapReader. Read when converting an MCAP recording, selecting topics or decoders, decoding custom protobuf messages, or when an MCAP-derived stream comes out empty. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (what the topics should become).","user_invocable":true,"allowed-tools":"Read, Grep, Bash, WebFetch","allowedTools":["Read,","Grep,","Bash,","WebFetch"]},"allowedTools":["Read,","Grep,","Bash,","WebFetch"],"isInternal":false,"tokens":1369,"sizeBytes":5510},{"name":"SKILL.md","path":"skills/rerun-mp4/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-mp4/SKILL.md","title":"rerun-mp4","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-mp4\ndescription: Ingest .mp4 video into Rerun chunk streams with rerun.experimental.Mp4Reader. Read when converting video into a VideoStream, choosing stream vs asset mode, transcoding (B-frames, output codec, GOP size) through FFmpeg, or aligning video PTS onto a recording's wall-clock timeline. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (where the video belongs in the recording).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun mp4 ingestion\n\n`Mp4Reader` turns one `.mp4` file into a lazy chunk stream on **one entity**:\ncompressed video samples, no decode to pixels, nothing re-encoded unless it has\nto be. Everything is configured on the constructor; `stream()` takes no\narguments. Stream mechanics after `.stream()` (filter, map, merge, collect,\nwrite) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.experimental import Mp4Reader, Mp4TranscodeOptions\n\nreader = Mp4Reader(video_path, entity_path=\"/camera/front\") # mode=\"stream\" by default\nstream = reader.stream() # lazy: nothing is decoded yet\n```\n\nEvery parameter after `path` is keyword-only. One reader handles one file — for\nseveral cameras, build one reader per file and `LazyChunkStream.merge(...)` them\n(see below).\n\n## Two modes\n\n| `mode` | emits | when |\n| -------------------- | ------------------------------------------------------- | ----------------------------------------------------------------- |\n| `\"stream\"` (default) | a codec chunk, per-GOP sample chunks, a keyframe marker | almost always — every frame is time-indexed and queryable |\n| `\"asset\"` | the whole file as one blob, plus a frame index | the codec cannot be a `VideoStream`, or the source starts mid-GOP |\n\nReach for stream mode; the next section covers exactly what it emits. Asset mode\ncopies the entire file into the recording as one blob, so nothing downstream can\nlook at a single frame without the whole asset. It is the fallback for codecs a\n`VideoStream` cannot carry (`mp4v`, image-sequence mp4), and it is what `rerun\nvideo.mp4` and the built-in file importer use.\n\n## What stream mode emits\n\nFor `tests/assets/video/Big_Buck_Bunny_1080_1s_h264_nobframes.mp4` (30 frames,\none GOP) with `entity_path=\"/camera/front\"`, the whole output is three chunks:\n\n```\n[0] entity=/camera/front static=True rows=1 timelines=[] cols=['VideoStream:codec']\n[1] entity=/camera/front static=False rows=30 timelines=['video'] cols=['VideoStream:sample']\n[2] entity=/camera/front static=False rows=1 timelines=['video'] cols=['VideoStream:is_keyframe']\n```\n\n- **One static chunk, one row**, holding `VideoStream:codec`.\n It carries **no timeline** — code that walks the stream must\n handle that (`if chunk.is_static: continue`).\n- **One temporal chunk per GOP**: a keyframe plus every sample that depends on\n it, up to (not including) the next keyframe. Samples only — the keyframe flags\n are not a column here.\n- **One trailing `is_keyframe` marker chunk**, holding a *sparse* `True` row at\n each keyframe's time and nothing else. Keeping it out of the sample chunks is\n what lets a keyframe-only query skip the sample payload, and what\n `collect(optimize=…)` accepts as canonical. It is list-per-row, so a row reads\n back as `[True]`, not `True`.\n- **Timeline `video`**, `duration[ns]`, values are the mp4 PTS from the start of\n the video.\n- **Every chunk on the same entity.** `entity_path=None` (the default) derives\n it from the file's *absolute* path, so `foo/video.mp4` run from `/data`\n becomes `/data/foo/video.mp4`. Pass `entity_path` explicitly in any real\n pipeline.\n\nA short clip is often a single GOP, so \"three chunks total\" is the normal shape —\nnot a sign that something was dropped. `chunk_by_gop=False` emits one Rerun\nchunk per sample instead (the marker chunk is unaffected). That is a debugging\nshape — one chunk per frame is a poor storage layout, and it also makes a\nfollowing optimize pass **6× more expensive** (measured below) — so leave the\ndefault alone unless you are inspecting individual samples.\n\nDistinguish the three by their columns, not by position or `is_static`: the codec\nchunk is the static one, sample chunks carry `VideoStream:sample`, and the marker\nchunk carries only `VideoStream:is_keyframe`.\n\nSupported stream-mode codecs are exactly the five `VideoCodec` values: H264,\nH265, AV1, VP8, VP9.\n\n## Transcoding through FFmpeg\n\n`VideoStream` cannot yet model DTS != PTS, so an H.264/H.265 source with\ncontainer-level B-frame reordering **cannot be emitted directly**. The reader\nhandles that itself: it re-encodes through FFmpeg with `-bf 0`, streams the\nresult back as a fragmented mp4, and turns each fragment into one GOP chunk, so\nonly one GOP is resident at a time. This is automatic and invisible in the API —\nthe output has the same shape as a clean source. It does need an `ffmpeg`\nexecutable. Asset mode is unaffected by B-frames.\n\n`Mp4TranscodeOptions` (stream mode only) additionally *requests* a transcode:\n\n| field | effect |\n| ----------------- | -------------------------------------------------------------------------- |\n| `output_codec` | re-encode to another `VideoCodec`; the emitted `VideoStream:codec` follows |\n| `gop_size` | force a keyframe every N frames — the knob for seek cost in the viewer |\n| `try_gpu` | best-effort hardware encode, NVENC / VideoToolbox only |\n| `ffmpeg_override` | use this `ffmpeg` instead of the one on `PATH` |\n\nRequesting the `output_codec` the source already uses is a **no-op**: it stays on\nthe direct, no-FFmpeg path. With `chunk_by_gop=True`, `gop_size=N` makes every\nGOP chunk but the last hold exactly N samples — a 30-frame clip at `gop_size=10`\ngives 10, 10, 10. `try_gpu` realistically covers H264/H265 plus AV1 on newer\nNVIDIA; VP8/VP9 always fall back to software, and it does nothing unless a\ntranscode is already happening.\n\n```python\nMp4Reader(\n video_path,\n entity_path=\"/camera/front\",\n # ~1s GOPs on a 60fps source, so seeking in the viewer stays snappy.\n transcode=Mp4TranscodeOptions(gop_size=64),\n).stream()\n```\n\n## Timelines: mp4 PTS is not your recording's clock\n\nThe emitted times are always PTS — elapsed nanoseconds from the start of *that\nvideo*. In a multi-sensor recording, that is almost never the timeline you want\nto align on.\n\n- `timeline_name=\"real_time\"` renames the timeline.\n- `timeline_type=\"timestamp\"` only **retypes** the same PTS values as\n nanoseconds since the Unix epoch. The reader does not shift them, so on its\n own it renders the video near 1970. It is meaningful only paired with a retag\n step.\n- **Retag with `stream.map(...)`.** Samples arrive in presentation order\n (B-frames are already stripped), so a running cursor maps sample `i` to\n `capture_times_ns[i]` — but the cursor must skip the keyframe marker chunk,\n which holds no samples. Map that one through the PTS the samples were already\n assigned:\n\n```python\nSAMPLE_COL = \"VideoStream:sample\"\n\n\ndef _reindex_to_capture_times(stream, capture_times_ns, timeline_name):\n cursor = 0\n pts_to_time = {}\n\n def _retag(chunk):\n nonlocal cursor\n if chunk.is_static: # the codec chunk carries no timeline\n return chunk\n batch = chunk.to_record_batch()\n col_index = batch.schema.get_field_index(timeline_name)\n old_field = batch.schema.field(col_index)\n old_pts = np.asarray(batch.column(col_index).cast(pa.int64()))\n\n if SAMPLE_COL in batch.schema.names:\n # Clamp in case the decoder yields a slightly different frame count.\n indices = np.clip(np.arange(cursor, cursor + chunk.num_rows), 0, len(capture_times_ns) - 1)\n cursor += chunk.num_rows\n new_times_ns = capture_times_ns[indices]\n pts_to_time.update(zip(old_pts.tolist(), new_times_ns.tolist()))\n else:\n # The sparse keyframe marker, emitted after every sample chunk.\n new_times_ns = np.array([pts_to_time[pts] for pts in old_pts.tolist()], dtype=np.int64)\n\n times = pa.array(new_times_ns.astype(\"datetime64[ns]\"))\n # `metadata=` is load-bearing — see gotcha 3.\n new_field = pa.field(old_field.name, times.type, nullable=old_field.nullable, metadata=old_field.metadata)\n return Chunk.from_record_batch(batch.set_column(col_index, new_field, times))[0]\n\n return stream.map(_retag)\n```\n\nThe cursor makes this order-dependent, so keep the retag on the single reader's\nstream, before any merge. This is the DROID loader's pattern — see the\nreferences.\n\nAny `map`/`flat_map` that assumes \"every non-static chunk is samples, in order\"\nhas this same bug: it advances over the marker chunk and stamps it with whatever\ntime comes next, silently moving the keyframe markers off their samples.\n\n## Several cameras into one recording\n\nOne reader per file, distinct entity paths, then merge:\n\n```python\nstreams = [\n Mp4Reader(path, entity_path=f\"/camera/{name}\", timeline_name=\"real_time\", timeline_type=\"timestamp\").stream()\n for name, path in cameras.items()\n]\n(\n LazyChunkStream\n .merge(*streams)\n .collect(optimize=OptimizationProfile.OBJECT_STORE)\n .write_rrd(out_path, application_id=app_id, recording_id=segment_id)\n)\n```\n\n`OBJECT_STORE` GOP-rebatches the video and preserves the reader's keyframe\nmarker as-is; no `fix_keyframe=True` is needed. If you *do* see\n`skipping GoP rebatching … is_keyframe data is incorrect`, something upstream\nrewrote the marker (see gotcha 4) — `fix_keyframe=True` re-derives it from the\nencoded samples as an escape hatch.\n\n## How the reader's chunks relate to optimize's\n\nThe reader and `collect(optimize=…)` agree on **where GOPs start** but not on\n**how many GOPs share a chunk**, and that is by design:\n\n- The reader emits the **finest GOP-aligned partition**: exactly one chunk per\n GOP. It cannot do better, because it does not know which profile the data is\n headed for.\n- Optimize then applies the size policy, **merging consecutive GOPs** up to the\n profile's `max_bytes` / `max_rows`. It never splits a GOP across chunks, so\n every boundary it keeps is one the reader already produced.\n\nSo optimize's partition is a pure *coarsening* of the reader's: same samples in\nthe same order, every optimized chunk a run of whole reader GOPs, keyframe marker\nuntouched. Where each GOP already sits near the profile's budget the two come out\nidentical (a 30-frame clip at `gop_size=10` is 3 chunks either way). Where GOPs\nare small they diverge sharply — a 12-GOP clip becomes **1** chunk under\n`OBJECT_STORE` and **4** under `LIVE`.\n\nThe practical consequence: **do not pre-merge or re-chunk the reader's output**\nto \"help\" optimize. Hand it the per-GOP stream and let the profile decide. And if\nyou skip optimize entirely (writing straight through `send_chunks`, as the DROID\nloader does), you are storing the finest partition — correct, but more chunks\nthan object storage wants.\n\n### The optimize pass is cheap on GOP-chunked input\n\nRunning optimize over the reader's output is not redundant work worth avoiding.\nOn an 89 MB / 1800-frame / 60-GOP H.264 file:\n\n| | `collect()` | `collect(optimize=OBJECT_STORE)` | delta |\n| ----------------------------- | ----------- | -------------------------------- | ------------------- |\n| `chunk_by_gop=True` (default) | 11.1 ms | 16.3 ms | **+5.2 ms** (1.5×) |\n| `chunk_by_gop=False` | 14.2 ms | 48.6 ms | **+34.4 ms** (3.4×) |\n\nNeither half of the pass is expensive on GOP-aligned input:\n\n- **Detecting GOP starts is header-only.** `build_sample_index` runs\n `detect_gop_start` on every sample, but that reads a few bytes of each sample's\n header — it never decodes. The whole optimize delta above (5.2 ms) is smaller\n than a Python loop calling the same detector over the same 1800 samples\n (6.0 ms), which is mostly FFI overhead.\n- **Rebuilding the chunks copies nothing.** `chunk_from_gop` calls\n `taken(0..n)`, and `re_arrow_util::take_array` returns the array **uncopied**\n when the indices are consecutive from zero over the whole array. One chunk per\n GOP means that is always the case, so the multi-MB sample buffers are never\n duplicated — hence 89 MB \"rebuilt\" in 5 ms.\n\nThat fast path is exactly what `chunk_by_gop=False` gives up: each GOP then spans\n30 source chunks, so `chunk_from_gop` has to `concat_and_sort` them and the sample\nbytes really are copied. `LIVE` also costs more than `OBJECT_STORE` (+13.2 ms vs\n+5.2 ms on the same file), because its much smaller chunk budget makes it attempt\ncompaction it cannot complete — a GOP is never split.\n\nRetag each camera *before* the merge — the retag above walks chunks with a\ncursor, so it must see one video's chunks in order.\n\n## Gotchas\n\n1. **Errors surface on the first pull, not at construction.** The constructor\n only checks that the file exists and validates its arguments; `stream()`\n builds a lazy pipeline and also succeeds. Codec support, keyframe layout, and\n FFmpeg availability are checked when the stream runs, so wrap the\n *consumption* (`to_chunks()`, `send_chunks`, iteration), not the\n `Mp4Reader(…)` call.\n2. **A codec outside the five `VideoCodec` values cannot be a `VideoStream`.**\n `mp4v` (MPEG-4 Part 2) is the one you will actually meet in robot datasets;\n it raises `RuntimeError: MP4 error: MP4 demux: Video track uses unsupported\ncodec \"mp4v\"`. Asset mode does accept the file, but only partly: the blob\n chunk is emitted and the `VideoFrameReference` index chunk is **skipped with\n a warning**, because the frame timestamps cannot be read either — so you get\n the bytes and no timeline. Skipping that camera and recording the fact as a\n recording property (what the DROID loader does) is usually better than a\n timeline-less blob, and either beats failing a whole episode over one camera.\n3. **A `map`/`flat_map` that rewrites the time column must carry the field\n metadata over.** `pa.field(...)` without `metadata=old_field.metadata` drops\n `rerun:kind: 'index'`, and the rebuilt chunk silently becomes **static** —\n no error, no timeline, and the samples land outside time entirely.\n4. **The trailing marker chunk is temporal but holds no samples**, so any\n per-chunk logic keyed on `not chunk.is_static` will process it as if it were\n samples. Key on the `VideoStream:sample` column instead. See the retag above.\n5. **Asset mode is capped at ~2 GiB** by Arrow's i32 offsets, and duplicates the\n file's bytes into the RRD.\n6. `mode=\"asset\"` rejects both `chunk_by_gop=False` and `transcode=` with\n `ValueError` — those are stream-mode-only knobs.\n7. `timeline_type=\"timestamp\"` on its own shifts nothing. Without a retag, the\n video sits at the epoch.\n8. **The reader emits compressed samples, never pixels.** Thumbnails, CLIP\n embeddings, or anything else needing RGB have to decode the file separately\n (OpenCV, PyAV); the `VideoStream` chunks cannot supply them.\n9. Every `stream()` call re-decodes the file from scratch, and every terminal\n call re-runs the pipeline. `collect()` once when you need more than one pass.\n10. Handle the static codec chunk explicitly in any `map`/`flat_map` — it has no\n timeline and no samples, and blind indexing into a time column will fail on\n it.\n11. **Samples before the first keyframe** are rejected in stream mode (a decoder\n cannot start mid-GOP); use asset mode for such a file.\n\n## References\n\n- Canonical worked examples: `rerun_py/tests/integration/test_mp4_reader.py`\n (both modes, `chunk_by_gop`, entity paths, `timeline_type`, transcode\n transforms, and the error cases).\n- Rust core: `crates/data_flow/re_mp4_reader/` (`stream.rs` for the GOP/transcode\n path, `asset.rs` for the blob+index path), with\n `crates/data_flow/re_mp4_reader/tests/stream.rs` covering codec pairs and GOP\n spacing.\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-data-model` (where\n video, calibration, and thumbnails belong in the recording).\n","frontmatter":{"name":"rerun-mp4","description":"Ingest .mp4 video into Rerun chunk streams with rerun.experimental.Mp4Reader. Read when converting video into a VideoStream, choosing stream vs asset mode, transcoding (B-frames, output codec, GOP size) through FFmpeg, or aligning video PTS onto a recording's wall-clock timeline. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (where the video belongs in the recording).","user_invocable":true,"allowed-tools":"Read, Grep, Bash, WebFetch","allowedTools":["Read,","Grep,","Bash,","WebFetch"]},"allowedTools":["Read,","Grep,","Bash,","WebFetch"],"isInternal":false,"tokens":4017,"sizeBytes":16410},{"name":"SKILL.md","path":"skills/rerun-parquet/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-parquet/SKILL.md","title":"rerun-parquet","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-parquet\ndescription: Ingest tabular Parquet files into Rerun chunk streams with rerun.experimental.ParquetReader. Read when converting trajectory or sensor tables (LeRobot-style parquet, exported logs) into entities and components — column grouping, timeline/index columns, static columns, and lenses (DeriveLens) that assemble the typed components (Transform3D, Scalars) from the reader's grouped struct/scalar output. Builds on rerun-chunk-processing and rerun-data-model.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun parquet ingestion\n\n`ParquetReader` is a **pure reader**: it maps a flat table onto the Rerun\nmodel by turning raw columns into grouped, time-indexed chunks of struct and\nscalar components. Column-name prefixes become entities, grouped columns\nbecome a single struct component, designated columns become timelines. The\nreader does **not** assemble archetypes anymore — mapping struct fields into\ntyped Rerun components (Transform3D, Scalars, Points3D) is done with lenses on\nthe reader's `.stream()`. The whole reader job is configuration; fill in the\n`rerun-data-model` mapping table first, then express it through `stream()`.\nStream mechanics after `.stream()` are in `rerun-chunk-processing`.\n\n**The whole table is configuration, not code.** If you find yourself building\n`Chunk.from_columns` from a parquet, or munging it in pandas first, stop —\n`ParquetReader` plus a lens almost certainly expresses it. Anything the reader\ncannot express (per-row entity routing, derived values, unit conversion)\nbelongs in lenses downstream, not in pre-pandas munging; keep the pipeline\ncolumnar.\n\n## The API\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nreader = ParquetReader(table_path) # a lightweight handle over the file\nstream = reader.stream(\n entity_path_prefix=\"/world\", # prepended to every entity path\n column_grouping=\"prefix\", # \"prefix\" | \"individual\" | \"explicit_prefixes\"\n delimiter=\"_\", # split for column_grouping=\"prefix\"\n prefixes=None, # required for \"explicit_prefixes\"\n use_structs=True, # pack grouped columns into one struct component\n static_columns=[\"robot_type\"], # constant-per-file values, logged static\n index_columns=[IndexColumn.timestamp(\"timestamp\", input_unit=\"us\"), IndexColumn.sequence(\"frame_index\")],\n)\n```\n\nThe constructor takes only `path`; all loading options live on `stream()`, and every one is keyword-only.\nEach `stream()` call is independent, so one reader can drive several differently-configured streams over the same file.\nThe config is validated against the file's schema at `stream()` (missing index/static columns raise `ValueError` there, not mid-iteration).\nThere is no `column_rules` kwarg — typed-component assembly moved to lenses (below).\n\n## What the reader emits\n\nThe reader turns the table into chunks, one chunk per group, then leaves the\ndata as generic struct/scalar components for lenses to map. The naming is the\nkey thing the rest of the pipeline keys off:\n\n- **A grouped multi-column prefix `X`** → entity `/X`, with a single struct\n component named **`data`**. The struct's fields are the column names with the\n prefix (and delimiter, for `\"prefix\"`) stripped. So `A_pos_x`, `A_quat_w`\n under prefix `A` land as struct `data` with fields `pos_x`, `quat_w` on\n entity `/A`.\n- **A lone column with no group** → its own entity named after the column, and\n a raw component named after the column — *not* a `data` struct. So a `speed`\n column becomes entity `/speed`, component `speed`.\n- **A `/__properties` metadata chunk** built from the parquet file's schema\n metadata. You typically drop it right after `.stream()`:\n\n ```python\n stream = reader.stream().drop(content=\"/__properties/**\")\n ```\n\n## Column grouping: which columns share an entity\n\n- `\"prefix\"` (default): split each column name on `delimiter`, group by the\n first segment. `gripper_pos_x`, `gripper_pos_y` → entity `/gripper`, struct\n `data{pos_x, pos_y}`.\n- `\"explicit_prefixes\"`: group by the exact strings in `prefixes`, tried\n longest-first; the prefix is stripped from each struct field name (a raw\n string match, no delimiter — `foo` + `a` → field `a`). Columns matching no\n prefix become individual groups. Use this when names contain the delimiter\n ambiguously (`observation.state` vs `observation.images.top`: pass the full\n prefixes).\n- `\"individual\"`: every column is its own chunk/entity with a raw component\n named after the column — no struct packing at all, even for columns sharing a\n prefix. `use_structs` is ignored here. Rarely the model you want; reach for\n it only as a debugging baseline.\n\n`use_structs=True` (default) packs a group's columns into a single Arrow\nstruct component (the `data` field) for `\"prefix\"`/`\"explicit_prefixes\"`;\n`False` emits one component per column (the pre-struct flat layout, what\nqueries see as separate columns).\n\n## Timelines: `index_columns`\n\nEach entry is built with `IndexColumn`: `IndexColumn.timestamp(name, input_unit=...)` (since epoch), `IndexColumn.duration(name, input_unit=...)` (elapsed), or `IndexColumn.sequence(name)` (ordinal int).\n\n- `input_unit` describes what the raw integers in the column *are* (`\"ns\"` default,\n `\"us\"`, `\"ms\"`, `\"s\"`); Rerun rescales to ns internally. Sequences take no unit.\n\n**If omitted, a synthetic `row_index` sequence timeline is generated.** That\nis almost never the timeline you want to query or align against; always name\nthe real time columns. Stamp both a timestamp and a sequence timeline when the\ntable has both (multi-rate alignment, see `rerun-data-model`).\n\n## Static columns: `static_columns`\n\nListed columns are constant across all rows; they are emitted once as a single\nstatic (timeless) chunk, separate from the temporal data. A listed column that\nactually varies raises an error when the stream runs — that error is a\ndata-quality signal, not a reason to drop the static declaration.\n\n## Typed components via lenses\n\nThe reader's grouped output is generic struct (`data`) and scalar data. A\n`DeriveLens` reads that struct's fields, packs and casts them into real Rerun\ncomponents, and writes them to an output entity — this is what the old\n`column_rules` API used to do, now done downstream on the stream.\n\nConstruct a lens against the reader's struct component (`\"data\"` for grouped\nprefixes, or the column name for a lone/individual column), then add one or\nmore `.to_*` builder methods. Each builder returns a fresh lens, so they chain.\n\n| Builder | Produces | Argument order |\n|---|---|---|\n| `to_translation(x, y, z)` | `Transform3D:translation` | x, y, z |\n| `to_quaternion(x, y, z, w)` | `Transform3D:quaternion` | x, y, z, w (xyzw) |\n| `to_scale(x, y, z)` | `Transform3D:scale` | x, y, z |\n| `to_rotation_axis_angle(axis_x, axis_y, axis_z, angle)` | `Transform3D:rotation_axis_angle` | axis_x, axis_y, axis_z, angle (radians) |\n| `to_scalars(*fields)` | `Scalars:scalars` | one or more field names |\n| `to_packed_component(component, *fields)` | the given component | descriptor, then field names |\n| `to_component(component, selector)` | the given component | descriptor, then a `Selector` |\n| `to_timeline(name, type, selector)` | a timeline (not a component) | name, `\"sequence\"`/`\"duration_ns\"`/`\"timestamp_ns\"`, selector |\n\n`to_packed_component` packs the named struct fields (in order, at least one\nrequired) into the fixed-size list the component expects, and by default\n**auto-casts `f64`→`f32`** to match component types. The `to_translation`,\n`to_quaternion`, `to_scale` helpers are convenience wrappers over it, so they\nall auto-cast. `to_rotation_axis_angle` builds a `Struct{axis, angle}` and\nhard-casts axis and angle to `f32` internally. `to_scalars` with a single field\nemits a plain scalar per row (not a 1-element list); with multiple fields it\nemits one scalar series per field at the same entity.\n\nApply lenses with `.stream().lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")`:\n\n- `content` is a pre-filter on the *source* entity path — it scopes which\n chunks the lens may touch. Out-of-scope chunks pass through unchanged. Set it\n to the reader's grouped entity (e.g. `\"/A\"`).\n- `output_mode` decides the fate of in-scope-but-unmatched chunks:\n `\"drop_unmatched\"` (default, keep only lens output), `\"forward_unmatched\"`\n (output replaces matched, other originals survive), or `\"forward_all\"`\n (output plus all originals).\n- The lens's own `output_entity=` sets the *destination* entity — independent\n of `content`, which gates the input side.\n\nEnd-to-end Transform3D example. The reader groups `A_*` columns into a `data`\nstruct at `/A`; the lens reads the prefix-stripped field names (`pos_x`,\n`quat_w`), packs and casts them, and writes a full `Transform3D` to `/pose`:\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nlens = (\n DeriveLens(\"data\", output_entity=\"/pose\")\n .to_translation(\"pos_x\", \"pos_y\", \"pos_z\")\n .to_quaternion(\"quat_x\", \"quat_y\", \"quat_z\", \"quat_w\")\n)\n\nchunks = (\n ParquetReader(table_path)\n .stream(index_columns=[IndexColumn.sequence(\"frame_index\")])\n .lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")\n .to_chunks()\n)\n```\n\nChaining several `.to_*` on **one lens with a shared `output_entity`**\naccumulates multiple component columns into the same archetype at that entity —\nabove, both `Transform3D:translation` and `Transform3D:quaternion` land on\n`/pose`, forming a complete `Transform3D`. For a generic fixed-size-list\ncomponent, pass the descriptor to `to_packed_component`:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens\nfrom rerun.experimental import ParquetReader\n\nlens = DeriveLens(\"data\", output_entity=\"/points\").to_packed_component(\n rr.Points3D.descriptor_positions(), \"x\", \"y\", \"z\"\n)\n```\n\n## Selectors\n\nLens field paths use `Selector`, a jq-like grammar over Arrow columns\n(`.field` to access a struct field, `[]` to iterate a list, `[N]` to index, `?`\nto suppress errors on absent fields, `!` to assert non-null, `|` to pipe, and\n`pack(.x, .y, .z)` to zip paths into a fixed-size list). The `to_*` helpers\nbuild these selectors for you; reach for `to_component(component, Selector(\".x\"))`\nwhen you need a custom field path. Field paths reference the\n**prefix-stripped** struct field names — the lens sees `pos_x`, not `A_pos_x`.\n\n## Gotchas\n\n1. No `index_columns` → synthetic `row_index` timeline only. Queries that\n expect a timestamp timeline find nothing.\n2. The `unit` is the raw column's unit, not a desired output unit; a\n microsecond column declared `\"ns\"` lands 1000x in the past.\n3. `static_columns` raises if a listed column actually varies; that error is a\n data-quality signal, not a reason to drop the static declaration.\n Uniformity is data-dependent, so it is raised lazily when the stream runs;\n a listed column that doesn't *exist* raises eagerly at `stream()`.\n4. A grouped prefix's struct component is named **`data`** — that is the\n `input_component` string a `DeriveLens` matches against. A lone or\n `\"individual\"` column is instead a raw component named after the column.\n5. Selector field paths reference the **prefix-stripped** struct field names\n (`pos_x`, not `gripper_pos_x`).\n6. Drop the `/__properties` metadata chunk the reader emits from parquet schema\n metadata: `.stream().drop(content=\"/__properties/**\")`.\n7. Quaternion column order is x, y, z, w in `to_quaternion`; check the source's\n convention before wiring fields.\n8. `to_packed_component` (and the transform helpers built on it) auto-casts\n `f64`→`f32` to match component types; this is usually what you want for\n parquet's double columns.\n9. Anything the reader cannot express (per-row entity routing, derived values,\n unit conversion) belongs in lenses downstream, not in pre-pandas munging;\n keep the pipeline columnar (`rerun-chunk-processing`).\n\n## References\n\n- Lens builder source with full docstrings: `rerun/chunk/_lens.py` in\n the installed `rerun-sdk` package (`to_translation`, `to_quaternion`,\n `to_scale`, `to_rotation_axis_angle`, `to_scalars`, `to_packed_component`,\n `to_component`, `to_timeline`).\n- Reader source: `rerun/experimental/_parquet_reader.py`, or\n `python -c \"from rerun.experimental import ParquetReader; help(ParquetReader)\"`\n- Canonical worked examples: the integration tests\n `rerun_py/tests/integration/test_parquet_reader.py` (grouping, index/static\n columns, and the Transform3D / Points3D / Scalars lens flows) and\n `rerun_py/tests/integration/test_lazy_chunk_stream.py` (lens application,\n `content`/`output_mode`, selectors).\n- `rerun-lerobot` — LeRobot datasets store episodes as parquet; that skill\n covers the built-in importer route vs reading the parquet directly with\n this reader.\n- `rerun-data-model` (mapping decisions), `rerun-chunk-processing` (stream\n mechanics after `.stream()`)\n","frontmatter":{"name":"rerun-parquet","description":"Ingest tabular Parquet files into Rerun chunk streams with rerun.experimental.ParquetReader. Read when converting trajectory or sensor tables (LeRobot-style parquet, exported logs) into entities and components — column grouping, timeline/index columns, static columns, and lenses (DeriveLens) that assemble the typed components (Transform3D, Scalars) from the reader's grouped struct/scalar output. Builds on rerun-chunk-processing and rerun-data-model.","user_invocable":true,"allowed-tools":"Read, Grep, Bash, WebFetch","allowedTools":["Read,","Grep,","Bash,","WebFetch"]},"allowedTools":["Read,","Grep,","Bash,","WebFetch"],"isInternal":false,"tokens":3265,"sizeBytes":13014},{"name":"SKILL.md","path":"skills/rerun-urdf/SKILL.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/skills/rerun-urdf/SKILL.md","title":"rerun-urdf","category":"anthropic-skill","format":"markdown","content":"---\nname: rerun-urdf\ndescription: Drive the Rerun URDF API (rerun.urdf.UrdfTree) to ingest a URDF as a Transform3D layer on a robot recording. Read when logging a robot model, running forward kinematics from joint states, composing a fixed chain for sensor extrinsics, or when the transform tree will not connect from the data alone. Builds on rerun-chunk-processing (stream/lens mechanics) and rerun-data-model (entity paths, timeline, base-vs-layer).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun URDF ingestion\n\nA URDF gives you a robot's geometry and its kinematic tree.\nIngesting it means two API calls on one `rerun.urdf.UrdfTree`: stream the static model, then drive it with forward kinematics from joint states. The transforms it produces are derived, so they are a **layer**, never base (the `URDF + joints (computed)` row of the `rerun-data-model` table).\n\nThis skill is the `UrdfTree` API and the two judgment calls it cannot make for\nyou: how your joint values map to URDF joints, and how the disconnected frames\nin the scene connect to one root. The stream/lens plumbing (`LazyChunkStream`,\n`DeriveLens`, `Selector`, writing and optimizing RRDs) is in\n`rerun-chunk-processing`; reach for it, do not re-derive it. Nothing below is\ntied to a data format: where your joint names, joint values, and calibration\ncome from is yours to wire in.\n\n## Getting the URDF\n\nDatasets rarely ship one, and the robot's own repo is not always the fastest route.\nIn order:\n\n- [`robot_descriptions.py`](https://github.com/robot-descriptions/robot_descriptions.py) — `pip install robot_descriptions`, then `from robot_descriptions import ur5e_description; ur5e_description.URDF_PATH`.\n It downloads and caches the upstream description and hands you a plain path with the meshes already next to it.\n Check its index first; it covers most of the table below.\n- The vendor repo, when the description is newer than the package's pin or is not in it.\n- [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) — MJCF, not URDF, so it needs conversion. Worth it only when nothing else exists.\n\n| Robot | Source | Notes |\n| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |\n| SO-100 / SO-101 (LeRobot) | [TheRobotStudio/SO-ARM100](https://github.com/TheRobotStudio/SO-ARM100) | `Simulation/SO100/so100.urdf`, `Simulation/SO101/so101_new_calib.urdf`; plain URDF, relative mesh paths |\n| ALOHA (ViperX 300, WidowX) | [Interbotix/interbotix_ros_manipulators](https://github.com/Interbotix/interbotix_ros_manipulators) | xacro |\n| Franka Panda / FR3 | [frankaemika/franka_description](https://github.com/frankaemika/franka_description) | xacro |\n| Universal Robots UR3 to UR20 | [UniversalRobots/Universal_Robots_ROS2_Description](https://github.com/UniversalRobots/Universal_Robots_ROS2_Description) | xacro, arm parameters in YAML |\n| Kinova Gen3 | [Kinovarobotics/ros2_kortex](https://github.com/Kinovarobotics/ros2_kortex) | xacro |\n| UFactory xArm / Lite 6 | [xArm-Developer/xarm_ros2](https://github.com/xArm-Developer/xarm_ros2) | xacro |\n| Unitree Z1, G1, H1, Go2 | [unitreerobotics/unitree_ros](https://github.com/unitreerobotics/unitree_ros) | plain URDF per robot |\n\nTwo things bite before FK ever runs:\n\n- **xacro is not URDF.** First run `xacro arm.urdf.xacro > arm.urdf` (`pip install xacro`), passing whatever arguments the file requires (gripper variant, arm size).\n A xacro file handed to `UrdfTree.from_file_path` fails to parse.\n- **`package://` mesh URIs.** The importer resolves them by scanning `ROS_PACKAGE_PATH` (ROS 1) and `AMENT_PREFIX_PATH` (ROS 2) for the package, and otherwise treats the rest of the URI as relative to the URDF's own directory.\n Outside a sourced ROS workspace, either export one of those variables or lay the meshes out relative to the URDF.\n Meshes that fail to resolve give you a kinematically correct but invisible robot.\n\n## The API\n\n```python\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(\n urdf_path,\n entity_path_prefix=\"robot\", # links log under /robot/)`. The `recording_id` must equal the\nbase segment id or the layer never attaches; `application_id` is discarded on\nregistration.\n\n## Mapping joint state to URDF joints (you supply this; the data will not)\n\n`compute_joint_transform_batches` is only as right as the `names`/`values` you\nhand it, and the mapping is not in the URDF. Three things go wrong silently:\n\n- **Order.** Build an explicit `names` array aligned to the `values` you read.\n Never assume your message's field order matches the URDF's `` order.\n- **Count.** The URDF's non-`fixed` joint count rarely equals your reported value\n count. A gripper sent as one value is often two prismatic joints in the URDF;\n a mimic joint may be omitted from the message. Reconcile explicitly, and use\n the API to do it: iterate `urdf.joints()`, partition by `joint_type` and\n `mimic`. A joint with `mimic` set derives its value from the driver joint as\n `driver * multiplier + offset`; feed it that, not a message field. Confirm\n the count against `urdf.joints()`, not the message length.\n- **Units.** URDF joints are radians (revolute) and meters (prismatic). Convert\n if your source differs.\n\nGet any of these wrong and FK runs and writes a confident, wrong pose.\n\n### Verify joint mapping and limits\n\nJoint values in the source data aren't guaranteed to be within URDF joint limits.\nThis can happen with any format (MCAP, Parquet, etc).\nIt is therefore useful to check whether joint values in the data match the URDF model's limits.\nOut-of-range values can indicate issues like wrong indexing or model mismatch and should be clarified before calculating the forward kinematics.\n\nTo get limits of a URDF model:\n\n```python\nlimits = {joint.name: (joint.limit_lower, joint.limit_upper) for joint in urdf.joints() if joint.joint_type != \"fixed\"}\n# then compare each column's min/max against its joint's range\n```\n\n`scripts/check_joint_mapping.py` is that check worked out for one source shape — a LeRobot-style parquet column of per-frame joint values.\nRead it as the example to copy and adapt for your own format, and run it directly if your data is already a LeRobot-style parquet file:\n\n```bash\nuv run scripts/check_joint_mapping.py --column observation.state --degrees\n```\n\n**Important:** this is a validation meant to be done before full processing — don't run this for _every_ file in a larger dataset.\nSelect a single representative file or a small subset for this verification.\n\nFor each joint it tries the usual candidate corrections (identity, negate, `% 360`, ±90°/±180° offsets, and their combinations) and reports which ones land inside that joint's limits.\nIt reports the fit, not the answer: a joint with two surviving candidates is **ambiguous**, and no amount of range-checking will separate them.\nValidate ambiguous mappings against known poses or geometry for the dataset.\n\n**Read the failures by shape.**\nA whole column outside its range means a wrong sign, unit, or offset — a bug, and the pose will be confidently wrong.\nA few frames a few degrees past a limit is calibration slack — expected, and `clamp=True` is the fix, not a workaround.\n`clamp=True` warns once per offending row, so run it while working out the mapping: a flood of warnings is a mapping smell, and a thin trickle at the extremes of one joint is a healthy recording.\n\n## Make the joint states readable first\n\nWhere the joint values come from is not this skill's problem, but a dead joint\nsource produces an empty FK layer **with no error**: a source path that\nmatched nothing, a decoder that yielded zero rows, or a reader that dropped the\nmessage silently. Whatever the source, confirm the joint-state stream yields\nrows before debugging FK. The importer skill for your source format covers its\nown empty-stream failure modes.\n\n## How transforms compose (reason about this before logging anything)\n\nRerun resolves a pose by chaining transforms from a frame up to a root. There\nare two ways an edge in that chain gets defined, and a URDF ingest mixes both:\n\n- **By entity-path hierarchy.** A `Transform3D` on `/a/b` with no frame names is\n the transform of `/a/b` relative to its parent path `/a`. Composition follows\n the entity tree.\n- **By explicit frame graph.** A `Transform3D` that carries `parent_frame` and\n `child_frame` defines an edge between two **named frames**, independent of\n where in the entity tree it is logged. URDF FK uses this: every joint\n transform names its parent and child link frames.\n\nSo a URDF ingest is a graph of named frames. An edge exists only if some\n`Transform3D` names that exact `parent_frame -> child_frame` pair. A frame with\nno incoming edge is a root. The viewer renders every root at the world origin,\nwhich is why two unconnected robots silently overlap instead of erroring.\n\n## Resolving the transform forest (the part the data cannot always give you)\n\nA URDF is **one tree rooted at its base link**. FK and fixed joints supply every\nedge inside that tree. A real scene is a **forest** of roots the URDF never\nconnects: a world or scene frame, each robot's base, every camera or sensor\nframe. The edges that join those roots come from calibration (extrinsics in a\nsidecar, a TF static publisher, a hand-eye result), not from the URDF, and some\nare simply absent. A single connected tree is not always solvable. Resolve it\ndeliberately:\n\n1. **Enumerate every frame.** Iterate `urdf.joints()` and collect the\n `parent_link`/`child_link` pairs (the in-tree edges, frame-prefixed);\n `urdf.root_link()` is that URDF's root. Add every sensor/world frame the\n scene needs (from the `rerun-data-model` table). Decide the one intended\n root.\n2. **Classify each edge by source.** In-URDF edges (FK joints, `fixed` joints)\n come from the URDF plus joint values. Inter-root edges (root to each robot\n base, root to each fixed sensor, an arm link to a wrist-mounted camera) come\n from calibration and you must log them yourself.\n3. **Compose fixed chains from the URDF** when you need the transform between two\n links joined only by `fixed` joints (a camera bracket, a tool mount): walk\n parent links across `fixed` joints via `urdf.joints()`, turning each joint's\n `origin_xyz`/`origin_rpy` into a homogeneous matrix and multiplying along\n the chain. If the walk cannot reach\n the target link, **stop and say so** (\"no fixed chain from A to B; stuck at\n C\"). A broken chain is a wrong pose, not a missing one.\n4. **Build the edge set and find the roots.** Collect every `parent_frame ->\nchild_frame` pair you will log (URDF + calibration). Walk parents from each\n frame; any frame that does not reach the intended root is an unconnected root\n and names a **missing edge**. This is a pure graph check you can run before\n writing the RRD.\n5. **Resolve every missing edge, or fail loudly.** For each one:\n - If calibration supplies the transform, log it (next section).\n - If the data does not, the tree is unsolvable. Do **not** leave the frame\n disconnected (it collapses onto the origin and reads as one merged scene).\n Either abort and name the missing edge, or log identity and emit a loud\n warning naming the assumed edge. State which you did.\n6. **Match frame names across sources.** FK-derived `parent_frame`/`child_frame`\n must equal the names `urdf.stream()` emits for the static geometry, and your\n calibration edges must use those same names, or links float off the mesh. The\n `frame_prefix` is what keeps them identical; reuse it everywhere.\n\nA correct ingest has exactly one root, and a path from every frame to it.\n\n## Logging a connection correctly\n\nA connecting edge is a **static** `Transform3D` carrying the bridging frame\nnames. Static (no time index) so it holds for the whole recording; the frame\nnames, not the entity path, are what create the graph edge. This\n`Chunk.from_columns` is the rare **sidecar exception** — a calibration transform\nno reader or FK lens can produce; do not generalize it to transforms a reader\nemits (a `frame_transforms` topic → `Transform3D`) or that FK derives:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import Chunk, LazyChunkStream\n\n# world -> this robot's base, from your calibration (translation + xyzw quaternion)\nedge = Chunk.from_columns(\n \"/world/robot_base\", # any sensible bridging entity path\n indexes=[], # no index == static\n columns=rr.Transform3D.columns(\n translation=[translation],\n quaternion=[quaternion_xyzw],\n parent_frame=[\"world\"], # must match the root frame name\n child_frame=[\"arm_base_link\"], # must match the URDF root frame (prefixed)\n ),\n)\nedges = LazyChunkStream.from_iter([edge]) # merge alongside model + FK streams\n```\n\nLog a fixed-chain result (step 3) the same way, with `parent_frame` and\n`child_frame` set to the two link frames the chain spans. Merge all edge chunks\ninto the same recording as the model and FK streams so they share the graph.\n\n## Minimal shape (one robot, generic joint source)\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens, LazyChunkStream, OptimizationProfile, Selector\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(urdf_path, entity_path_prefix=\"robot\", static_transform_entity_path=\"robot/tf_static\")\nmodel = urdf.stream(include_joint_transforms=True).drop(content=\"/robot/**/collision_geometries/**\")\n\njoints = source_joint_state_stream() # your reader; one message column of names+values\nfk = (\n joints\n .lenses(\n DeriveLens(JOINT_MSG_COMPONENT, output_entity=\"/tmp/batches\").to_component(\n \"rerun.urdf.JointTransformBatch\",\n Selector(\".\").pipe(lambda msgs: urdf.compute_joint_transform_batches(read_names(msgs), read_values(msgs))),\n ),\n content=JOINT_SOURCE_PATH,\n output_mode=\"forward_all\",\n )\n .lenses(\n DeriveLens(\"rerun.urdf.JointTransformBatch\", output_entity=\"/robot/transforms\", scatter=True)\n .to_component(rr.Transform3D.descriptor_translation(), Selector(\"[].translation\"))\n .to_component(rr.Transform3D.descriptor_quaternion(), Selector(\"[].quaternion\"))\n .to_component(rr.Transform3D.descriptor_parent_frame(), Selector(\"[].parent_frame\"))\n .to_component(rr.Transform3D.descriptor_child_frame(), Selector(\"[].child_frame\")),\n content=\"/tmp/batches\",\n output_mode=\"drop_unmatched\",\n )\n .filter(content=\"/robot/transforms\")\n)\n\nLazyChunkStream.merge(model, fk).collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n out_path,\n application_id=\"urdf\",\n recording_id=segment_id,\n)\n```\n\n`read_names`, `read_values`, `JOINT_MSG_COMPONENT`, and `JOINT_SOURCE_PATH` are\nthe only data-specific pieces, and the mapping section above is what makes them\ncorrect.\n\n## One URDF for a whole dataset (assets)\n\n`urdf.stream()` emits only static chunks — meshes and fixed-joint transforms, with no time index.\nThe model does not belong in a per-episode layer, where every segment pays for the same meshes.\nRegister it once as a dataset **asset**:\n\n```python\nurdf.stream().collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n model_rrd, application_id=\"urdf\", recording_id=\"so100_model\"\n)\nasset_id = dataset.register_asset(\"s3://bucket/so100_model.rrd\") # uri must be readable by the server\n```\n\nThe split is along static vs temporal: the **model** is an asset, and the **FK transforms** stay a per-segment layer keyed by `recording_id`.\nKeeping `include_joint_transforms=True` in the asset costs nothing and gives the rest pose for segments or time ranges the FK layer does not cover.\n\n- The viewer loads a dataset's assets alongside any segment it opens, and caches them across segments.\n- `dataset.segment_store(segment_id)` includes assets; pass `include_assets=False` to skip the extra manifest request per asset.\n- Dataframe queries never see assets — they stay on the dataset's own segments.\n\nWithout a catalog, the same split still pays: write the model RRD once and open it next to whichever episode you are looking at, rather than copying it into each one.\n\n## Gotchas that cause real failures\n\n1. Empty layer, no error: dead joint-state source (decoded to zero rows; see the importer skill for your format) or wrong `JOINT_SOURCE_PATH`/component name.\n2. Confident wrong pose: joint count, order, or units wrong.\n3. Layer writes but never attaches: `recording_id != segment_id`.\n4. Frames collide: two robots sharing a `frame_prefix`.\n5. Scene looks merged at the origin: unconnected roots logged as identity without a calibration edge.\n6. Catalog ingest rejects or misorders chunks: `OBJECT_STORE` optimization skipped.\n7. Layer opened in a running viewer never merges: opening an `.rrd` **by path** registers it as its own dataset and replaces its `application_id`, so it lands beside the segment instead of on it.\n To eyeball a layer locally, either merge model + FK + base into one RRD, or send both through a `RecordingStream(application_id, recording_id=segment_id)` pointed at the viewer with `rec.connect_grpc(url=\"rerun+http://127.0.0.1:9876/proxy\")`, which keeps the IDs and merges the stores.\n8. Robot invisible but transforms correct: meshes failed to resolve (see \"Getting the URDF\").\n9. No 3D view after adding a URDF to an existing recording: the auto-layout does not necessarily rebuild for the new entities. Send an explicit blueprint with `rrb.Spatial3DView(origin=\"/robot\")` (see `rerun-blueprint`).\n\n## References\n\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing` (FK two-lens pattern, two robots + scene URDFs, prefixes, recoloring, calibration offsets)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/animated_urdf` (classic logging API: `log_urdf_to_recording`, per-joint `compute_transform`)\n- [Catalog object model: assets](https://rerun.io/docs/concepts/query-and-transform/catalog-object-model#assets) (`register_asset`, the limits, and how assets are read back)\n- `rerun-data-model` (the mapping table this skill consumes)\n- the importer skill for your joint-state source format (making the source readable)\n- `rerun-chunk-processing` (lens/stream, write/optimize mechanics)\n","frontmatter":{"name":"rerun-urdf","description":"Drive the Rerun URDF API (rerun.urdf.UrdfTree) to ingest a URDF as a Transform3D layer on a robot recording. Read when logging a robot model, running forward kinematics from joint states, composing a fixed chain for sensor extrinsics, or when the transform tree will not connect from the data alone. Builds on rerun-chunk-processing (stream/lens mechanics) and rerun-data-model (entity paths, timeline, base-vs-layer).","user_invocable":true,"allowed-tools":"Read, Grep, Bash, WebFetch","allowedTools":["Read,","Grep,","Bash,","WebFetch"]},"allowedTools":["Read,","Grep,","Bash,","WebFetch"],"isInternal":false,"tokens":5098,"sizeBytes":22190},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# Claude Agent Guidelines & System Prompt\n\nPath: `CLAUDE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/CLAUDE.md)","tokens":300,"sizeBytes":155},{"name":"README.md","path":"crates/build/re_build_tools/README.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_build_tools/README.md","title":"re_build_tools Documentation","category":"plugin-manifest","format":"markdown","content":"# re_build_tools Documentation\n\nPath: `crates/build/re_build_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_build_tools/README.md)","tokens":300,"sizeBytes":200},{"name":"README.md","path":"crates/build/re_dev_tools/README.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/README.md","title":"re_dev_tools Documentation","category":"plugin-manifest","format":"markdown","content":"# re_dev_tools Documentation\n\nPath: `crates/build/re_dev_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/README.md)","tokens":300,"sizeBytes":194},{"name":"README.md","path":"crates/build/re_dev_tools/src/build_search_index/README.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_search_index/README.md","title":"build_search_index Documentation","category":"plugin-manifest","format":"markdown","content":"# build_search_index Documentation\n\nPath: `crates/build/re_dev_tools/src/build_search_index/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_search_index/README.md)","tokens":300,"sizeBytes":246},{"name":"README.md","path":"crates/build/re_dev_tools/src/build_web_viewer/README.md","rawUrl":"https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_web_viewer/README.md","title":"build_web_viewer Documentation","category":"plugin-manifest","format":"markdown","content":"# build_web_viewer Documentation\n\nPath: `crates/build/re_dev_tools/src/build_web_viewer/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_web_viewer/README.md)","tokens":300,"sizeBytes":240}],"systemPromptSnippet":"\n\n# CLAUDE.md\n\nGuidance for LLMs working in this repo.\n\n## Project overview\n\nRerun is a time-aware multimodal data stack and visualizations tool used in robotics, spatial AI, computer vision, and similar domains. It provides SDKs (Python, Rust, C++) for logging rich data (images, point clouds, tensors, etc.) and a Viewer for visualization.\n\n## Build system\n\nWe use `pixi` for task management and dependency installation. Check `pixi.toml` for a full list of tasks.\n\n### Essential commands\n\n**Building:**\n- `pixi run py-build` - Build Python SDK into local .venv (uses uv)\n- `pixi run rerun-build` - Build native viewer (without web viewer)\n- `pixi run rerun-build-web` - Build web viewer (wasm)\n- `pixi run cpp-build-all` - Build all C++ artifacts\n\n**Running:**\n- `pixi run rerun` - Run the viewer\n- `pixi run uvpy script.py` - Run Python scripts with rerun SDK\n- `cargo run -p ` - Run specific Rust example (e.g., `cargo run -p dna`)\n\n**Code generation:**\n- `pixi run codegen` - Generate Rust/Python/C++ code from the `re_type_definitions` crate\n\n**Formatting:**\n- `pixi run rs-fmt` - Format all Rust files. Always run this after making changes.\n- `pixi run py-fmt` - Format Python files\n- `pixi run cpp-fmt` - Format C++ files\n- `pixi run toml-fmt` - Format TOML files\n\n**Testing:**\n- `cargo clippy -p ` - Run rust checks before building\n- `cargo nextest run --all-features --no-fail-fast -p ` - Run tests for specific crate\n - Example: `cargo nextest run --all-features --no-fail-fast -p re_view_spatial`\n- Use `cargo nextest` (not `cargo test`) for better output + parallelism\n- Always use `--all-features` unless specific reason not to\n- Use `--no-fail-fast` to gather all failures in single run\n\n**Snapshots:**\n- **`insta` snapshots**: Text-based, run with regular Rust tests. On failure: `cargo insta review` (install: `cargo install cargo-insta`)\n- **Image comparison tests**: Render image vs checked-in reference. Uses `egui_kittest`'s `Harness::snapshot` + `TestContext` for mocking viewer.\n - Results saved to `tests/snapshots/`, failures produce `diff.png`\n - Update refs: `UPDATE_SNAPSHOTS=1`\n - Update from failed CI run: `./scripts/update_snapshots_from_ci.sh`\n - Best practices: see [egui_kittest README](https://github.com/emilk/egui/tree/master/crates/egui_kittest#snapshot-testing)\n\n### Long-running commands (for agents)\n\nBuilds and test runs here take a long time, so run them in the background.\nA background command notifies you when it exits, so never start a second command to watch the first one.\nNo `sleep`-and-`tail` pollers, no `until` loops on the log.\nStart the build, do something else, and read its output once the notification arrives.\n\n### Driving a running egui UI (for agents)\n\nTwo MCP servers let an agent see and click a running UI instead of guessing from code:\n\n- `rerun viewer-mcp` drives a running Rerun Viewer over gRPC. See `docs/content/reference/viewer/mcp.md`.\n- `egui-mcp` drives *any* egui app started with `EGUI_INSPECTION=1`, including our example apps and headless `egui_kittest` harnesses.\n It exposes `attach`, `widget_tree`, `click`, `type_text`, `screenshot`, `wait_for`, and friends.\n Both are configured in the repo-root `.mcp.json`; install with `cargo install egui_mcp`.\n\nTypical loop: start the app with `EGUI_INSPECTION=1`, call `attach` (host `127.0.0.1`, port `5719`), `widget_tree` to find widgets, `click`/`type_text`, then `screenshot` with a `save_path` and look at the image.\n`egui-mcp` needs the app to paint frames: windowed apps must not be occluded on macOS, so prefer a headless harness where one exists.\nA headless `egui_kittest` harness can opt in with `egui_inspection::attach_from_env(&harness.ctx, label)`.\nExample: `EGUI_INSPECTION=1 cargo run -p re_agent_ui --example agent_app -- --headless`.\n\n## Code generation system\n\n**Critical: Never edit generated files directly.** All generated files are marked \"DO NOT EDIT\" at the top.\n\n### Type definition flow\n\n```\nre_type_definitions → pixi run codegen → Generated code (Rust/Python/C++) + docs (docs/content/reference/types/)\n```\n\n- Type definitions live in `crates/build/re_type_definitions/rerun/`\n - `encodings/*.def.rs` - Low-level types (Vec3D, Mat4x4, etc.)\n - `components/*.def.rs` - Component types (Position3D, Color, etc.)\n - `archetypes/*.def.rs` - Archetypes (Points3D, Image, etc.)\n - `blueprint/*.def.rs` - Blueprint system types\n- Codegen implementation is in `crates/build/re_types_builder/`\n- After modifying a definition, run `pixi run codegen` to regenerate code\n\n### Extension pattern\n\nTo add custom functionality to generated types, create `_ext` files:\n- Rust: `filename_ext.rs` (automatically imported by codegen)\n- Python: `filename_ext.py` (mixed in with generated class)\n- C++: `filename_ext.cpp` (compiled and included automatically, parts of it may be marked for copy into the header by codegen)\n\n## Code conventions\n\n### General\n\n- use `…` instead of `...` \n- validate various custom conventions via `pixi run lint-rerun ` (not passing any file will check everything)\n- Use `format!(\"{x}\")` over `format!(\"{}, x)` (same in log calls etc)\n- Don't write trivial comments that add nothing new\n- In error and log messages, put the error first and any file path at the end (e.g. `Failed to import: {err}\\nFile path: {path}`), never in the middle.\n Paths can be long or sensitive, so trailing placement makes them easy to strip when copy-pasting.\n- Prose style (em vs en dash, sentence endings, casing) — see [`DESIGN.md`](DESIGN.md). In short: spaced em dash ` — `, never unspaced `word—word`, and don't use `–` as a sentence dash (it's for numeric ranges only) \n- One sentence per line in markdown files.\n Markdown joins consecutive lines into a paragraph, so rendering is unchanged — but diffs become much easier to review.\n\n## Architecture overview\n\n### Crate organization\n\n```\ncrates/\n├── build/ # Code generation (re_types_builder)\n├── store/ # Data types, storage, querying\n├── top/ # User-facing SDKs and CLI\n└── viewer/ # Viewer UI and rendering\n```\n\nFor more details about the architecture see `ARCHITECTURE.md`.\n\n**When adding, removing, or renaming a crate**, run `pixi run crate-graph`.\nIt regenerates both the crate dependency diagram (`crate_graph.svg`) and the crate tables in `ARCHITECTURE.md` from `cargo metadata`.\nA crate's one-line description comes from the `description` field of its `Cargo.toml`.\n\n### Type system hierarchy\n\nThe type system has three levels (generated from `re_type_definitions`):\n\n1. **Encodings** (`rerun.encodings.*`) - Basic types like Vec3D, Color\n2. **Components** (`rerun.components.*`) - Named semantic wrappers (Position3D, Radius)\n3. **Archetypes** (`rerun.archetypes.*`) - Collections of components (Points3D, Image)\n\nEach archetype specifies:\n- Required components (must be provided)\n- Recommended components (have good defaults)\n- Optional components (purely optional)\n\nExample: `Points3D` archetype requires `positions`, recommends `colors` and `radii`, allows optional `labels`.\n\n### Data flow\n\n```\nSDK (log archetype)\n ↓ encode to Apache Arrow\nLogMsg (encoded data)\n ↓ transport (gRPC/file/memory)\nre_chunk_store (indexed time series DB)\n ↓ query\nViewer (immediate mode rendering)\n```\n\n### Blueprint system\n\nThe blueprint is the viewer's configuration layer:\n- Stored as a separate store (`re_entity_db`) with \"blueprint\" timeline\n- Defines: view layout, visibility, per-entity overrides, view properties\n- Uses the same type system as logged data\n- Basic blueprint path hierarchy: `/viewport/`, `/view/{uuid}/`, `/container/{uuid}/`\n\n### Visualizers\n\nEach view type (Spatial3D, TimeSeries, etc.) has registered visualizers:\n- Determine which entities/archetypes can be visualized\n- Execute per-frame: query data → process → generate render commands\n- Examples: Points3DVisualizer, LineStripsVisualizer, MeshVisualizer\n\nThe viewer uses **immediate mode**: every frame, query the store and re-render from scratch.\n\n## Documentation snippets\n\nSee [`docs/snippets/README.md`](docs/snippets/README.md) for running, building, finding snippets. Config in [`docs/snippets/snippets.toml`](docs/snippets/snippets.toml).\n\n## Python development workflow\n\nPython uses a separate uv-managed .venv (not pixi's conda env):\n\n```bash\npixi run py-build # Build rerun-sdk into .venv\npixi run uvpy script.py # Run Python scripts via uv\npixi run uv run script.py # Explicit uv run\n```\n\nThe `uv` wrapper script unsets `CONDA_PREFIX` to ensure isolation from pixi's environment.\n\n## Important notes\n\n- **PyO3 Configuration**: If you see PyO3 config errors, run `pixi run ensure-pyo3-build-cfg`\n- **git-lfs**: Required for test snapshots. Install with your package manager and run `git lfs install`\n- **Immediate Mode**: The entire viewer is rendered from scratch each frame (no state management callbacks)\n- **Arrow Native**: Data is stored, transmitted, and queried as Apache Arrow arrays\n- **Multi-language**: Changes to `re_type_definitions` affect Rust, Python, and C++ simultaneously\n\n## Python docstring formatting\n\nPython API docs are built with **MkDocs + mkdocstrings** (NOT Sphinx). Never use reStructuredText (rST) syntax in Python docstrings or documentation. Use markdown instead:\n\n- **Cross-references:** Use `[`ClassName`][]` (mkdocstrings syntax), NOT `:class:`ClassName`` / `:func:` / `:meth:` (rST roles)\n- **Warnings/notes:** Use MkDocs admonitions (`!!! warning` with indented body), NOT `.. warning::` (rST directives)\n- **Deprecation notices:** Use the `@deprecated` decorator (mkdocstrings renders it automatically). Do NOT duplicate in the docstring with `.. deprecated::` or `**Deprecated:**`\n- **Code blocks:** Use markdown fenced blocks (`` ``` ``), NOT `.. code-block::`\n- **Parameter docs:** Use numpy-style sections (`Parameters`, `Returns` with `----------`), which is what the codebase already uses\n\n## Documentation system\n\nSee [`docs/README.md`](docs/README.md) for the full documentation architecture.\n\nThe docs span multiple sites: the main docs at `rerun.io/docs` (built from `docs/content/`), plus API reference sites for Python (MkDocs), C++ (Doxygen), and JS (TypeDoc) at `ref.rerun.io/docs/{python,cpp,js}/`.\n\nKey things to know:\n- **`docs/content/reference/types/`** is auto-generated by `pixi run codegen` from `re_type_definitions` - do not edit directly\n- **`docs/content/reference/cli.md`** is auto-generated by `pixi run man` - do not edit directly\n- **Code snippets** live in `docs/snippets/all/` with implementations in Python, Rust, and C++\n- `pixi run py-docs-serve` previews Python API docs locally\n- `pixi run -e cpp cpp-docs` builds C++ docs\n\n## Development references\n\n- [`ARCHITECTURE.md`](ARCHITECTURE.md) - Detailed architecture documentation\n- [`BUILD.md`](BUILD.md) - Full build instructions\n- [`CODE_STYLE.md`](CODE_STYLE.md) - Code style guidelines\n- [`CONTRIBUTING.md`](CONTRIBUTING.md) - Contribution guidelines\n- [`DESIGN.md`](DESIGN.md) - UI design guidelines (GUI, CLI, docs, log messages)\n- [`docs/README.md`](docs/README.md) - Documentation system (sites, builds, deployment)\n- [`rerun_py/README.md`](rerun_py/README.md) - Python SDK instructions\n\n## Contributing\n\nDon't open pull requests or issues unless explicitly asked.\nWhen opening or interacting with one, follow the [pull request template](.github/pull_request_template.md) or [issue templates](.github/ISSUE_TEMPLATE/), and disclose that you are an LLM.\nLet the user know that you included this disclosure.\n\n### Before requesting review\n\n- CI is green, and the code is formatted (`pixi run rs-fmt`).\n- The branch history is clean: no broken merges, and no commits from unrelated work.\n- The description is honest about the scope, and says *why* the change is needed.\n- UI changes come with a screenshot or video.\n\n### Addressing review comments\n\n- When a reviewer flags one instance of a pattern, fix every instance in the diff, and say so in your reply.\n- A review comment phrased as a question wants an answer, not a code change — unless the answer is \"yes, that is wrong\".\n- When a reviewer asks for a change, make it in this PR.\n Only defer it to a follow-up PR if the reviewer agrees.\n- If the fix belongs in a dependency we maintain (egui, egui_plot, egui_tiles, emath, quiver, puffin, …), suggest an upstream PR instead of a local workaround.\n- Reviewers want to talk to the human author.\n When a comment asks for a judgement call (API shape, scope, product behavior), bring it to your user instead of deciding in the reply.\n- Always reply when mentioned, even if only to say you are unsure.\n\n\n# Rerun architecture\n\nThis document describes the technical architecture of Rerun.\n\n## See also\n\n- [`BUILD.md`](BUILD.md)\n- [`TESTING.md`](TESTING.md)\n- [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK\n- [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)\n- [`CODE_STYLE.md`](CODE_STYLE.md)\n- [`CONTRIBUTING.md`](CONTRIBUTING.md)\n- [`RELEASES.md`](RELEASES.md)\n\n## The major components\n\n### Logging APIs\n\nIt all starts with logging. You can log rich data (point clouds, images, etc.) with either our Python SDK or our Rust SDK.\n\nThe logging SDK:s encodes the data using Apache Arrow (see more below).\n\nThe logging data can be written to disk as `.rrd` files, or transmitted over gRPC to either a Rerun Viewer or a Rerun Server.\n\n### Rerun Viewer\n\nThe Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser.\n\n#### Native Viewer\n\nThe easiest way to launch the Viewer is directly from the logging API with `rr.init(\"rerun_example_app\", spawn=True)`. However, the standalone Viewer can also be run from the command line, for example to view an `.rrd` file: `rerun mydata.rrd`.\n\n#### Web viewer\n\nYou can try running the Viewer in a browser using `rr.serve()` in Python, or using `rerun --web-viewer mydata.rrd`.\n\nThe web viewer consists of just a few small files - a thin `.html`, a `.wasm` blob, and an auto-generated `.js` bridge for the wasm. These files are served using the [`re_web_viewer_server`](https://github.com/rerun-io/rerun/tree/latest/crates/top/re_web_viewer_server) crate.\n\nThe web viewer can load `.rrd` files (just drag-drop them into the browser), or read logging data streamed over gRPC.\n\n### `.rrd` files\n\n`.rrd` (\"**R**e**r**un **D**ata\") is just a bunch of log messages appended one after the other to a file.\n\nNOTE: `.rrd` files do not yet offer full backwards or forwards compatibility. However, the current version of Rerun will always be able to open `.rrd` files generated by the *previous* version. This is not guaranteed to work for files generated with older versions.\n\n## Technologies we use\n\n### Apache Arrow\n\n[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_chunk_store`](crates/store/re_chunk_store/README.md).\n\nIn Rust, we use the [`arrow` crate](https://crates.io/crates/arrow).\n\n### `wgpu`\n\nThe Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web.\n\nOn web builds, we use WebGPU when available on the Web, but automatically fall back to a WebGL based emulation layer (with a more limited feature set).\n\nWe have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/viewer_support/re_renderer/README.md).\n\n### `egui`\n\nThe GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode).\n\nWe use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web.\n\n### Wasm\n\nWasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.\nThe Rerun Viewer can be compiled to Wasm and run in a browser.\n\nThreading support in Wasm is nascent, so care must be taken that we don't spawn any threads when compiling for `wasm32`.\n\nWasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:\n\n- Access files\n- Read environment variables\n- Get the current time (use [`instant`](https://crates.io/crates/instant) instead)\n- Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)\n- etc\n\n## Immediate mode\n\nThe Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch.\n\nIn fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.\n\nThe advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.\n\nImmediate mode is also a forcing function, forcing us to relentlessly optimize our code.\nThis leads to a very responsive GUI, where there is no \"hiccups\" when switching data source or doing time scrubbing.\n\nOf course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.\n\n## Crates\n\nHere is an overview of the crates included in the project, and how they depend on each other:\n\n\n\nEach band is one folder under `crates/`, with `rerun_py` and `rerun_c` folded into the top band.\nEvery arrow points downwards, because the folders are layers: a crate may only depend on its own folder or one below it.\n[`scripts/check_crate_layers.py`](scripts/check_crate_layers.py) enforces that in CI.\nA `dev-dependency` is exempt, since a test may reach anywhere.\n\nSee [`crates/README.md`](crates/README.md) for the layers as a list, and for which one a new crate belongs in.\nThe tables below give a one-line description of every crate, one table per folder.\n\nOnly the arrows *inside* a band are drawn.\n\nThe diagram and the crate tables below are both generated from `cargo metadata`, so neither has to be written by hand.\nA crate's description is its `description` field in its own `Cargo.toml`, so there is only one place to write it.\nRegenerate both after adding, removing, or renaming a crate:\n\n```sh\npixi run crate-graph\n```\n\n**Run that command yourself whenever you add or remove a crate.**\nNothing checks it for you: `pixi run crate-graph-check` is disabled in CI, because graphviz lays the same graph out slightly differently on macOS and on Linux, so whoever regenerated the diagram last decided whether the check passed.\n\nPass `--edges all` to `scripts/generate_crate_graph.py` for a version with every dependency drawn, cross-band ones included.\n\n\n\n### Test support\n\n[`crates/tests`](./crates/tests) — only tests depend on these, so they sit at the top of the diagram.\n\n| Crate | Description |\n| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |\n| re_redap_tests | Official test suite for the Rerun Data Protocol |\n| re_renderer_tests | Snapshot tests for `re_renderer`'s rendering primitives. |\n| re_test_context | A common context used for tests. |\n| re_test_mocks | In-process server doubles (`MockOtlpCollector`, `MockPostHog`) used by tests that need to capture outbound OTel/PostHog traffic. |\n| re_test_viewport | A viewport used for tests |\n| re_web_tests | Discovers and runs Rerun web tests |\n\n### SDK / CLI / Wasm\n\n[`crates/top`](./crates/top) — the entry points for our users, and the viewer app itself.\n\n| Crate | Description |\n| -------------------- | ---------------------------------------------------------------------------------- |\n| re_sdk | Rerun logging SDK |\n| re_viewer | The Rerun viewer |\n| re_viewer_mcp | MCP server that allows llm agents to use the Rerun Viewer. |\n| re_web_viewer_server | Serves the Rerun web viewer (Wasm and HTML) over HTTP |\n| rerun | The Rerun Rust SDK: log images, point clouds, etc, and visualize them effortlessly |\n| rerun-cli | The Rerun command-line tool: the `rerun` binary, including the native Viewer |\n| rerun_c | Rerun C SDK |\n| rerun_py | The Rerun Python SDK |\n\n### Views\n\n[`crates/views`](./crates/views) — the visualizations a user can put in the viewport.\nA sibling of `crates/panels`: neither depends on the other.\n\n| Crate | Description |\n| ---------------------- | --------------------------------------------------------------------------- |\n| re_view_audio | A view that shows the waveform of an audio asset and plays it back. |\n| re_view_bar_chart | A view that shows a single bar chart. |\n| re_view_dataframe | A view that shows the data contained in entities in a table. |\n| re_view_graph | A view that shows a graph (node-link diagram). |\n| re_view_map | A view that shows a map. |\n| re_view_spatial | Views that show entities in a 2D or 3D spatial relationship. |\n| re_view_state_timeline | A view that shows state transitions as horizontal lanes over time. |\n| re_view_tensor | A view dedicated to visualizing tensors with arbitrary dimensionality. |\n| re_view_text_document | A simple view that shows a single text box. |\n| re_view_text_log | A view that shows text entries in a table and scrolls with the active time. |\n| re_view_time_series | A view that shows plots over Rerun timelines. |\n\n### Panels\n\n[`crates/panels`](./crates/panels) — the panels the app is assembled from, and the widgets they are built out of.\nA sibling of `crates/views`: neither depends on the other.\n\n| Crate | Description |\n| ------------------ | ----------------------------------------------------------------------------------------- |\n| re_agent_ui | Chat UI for driving a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_blueprint_tree | The UI for the blueprint tree in the left panel. |\n| re_chunk_store_ui | Display the contents of chunk stores. |\n| re_memory_view | Flamegraph visualization for memory usage trees |\n| re_recording_panel | The UI for the recording panel. |\n| re_redap_browser | The UI and communication to implement the in-viewer redap server browser. |\n| re_selection_panel | The UI for the selection panel. |\n| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |\n| re_viewport | The central viewport panel of the Rerun viewer. |\n\n### Viewer support\n\n[`crates/viewer_support`](./crates/viewer_support) — the UI and rendering machinery the views are built on.\n\n| Crate | Description |\n| ---------------------- | ---------------------------------------------------------------------------------------------------------- |\n| re_arrow_ui | Show arrow data in a tree of rerun list_items and format arrow with syntax highlighting. |\n| re_component_fallbacks | Handles registering various component fallbacks to `FallbackProviderRegistry`. |\n| re_component_ui | Provides ui editors for Rerun component data for registration with the Rerun Viewer component ui registry. |\n| re_context_menu | Support crate for context menu and actions. |\n| re_data_ui | Provides ui elements for Rerun component data for the Rerun Viewer. |\n| re_dataframe_ui | Rich table widget over DataFusion. |\n| re_gamepad | Gamepad input handling for the Rerun viewer. |\n| re_plot | Shared utilities for plot-based views. |\n| re_renderer | A wgpu based renderer for all your visualization needs. |\n| re_renderer_examples | Examples for the re_renderer crate. |\n| re_time_ruler | Time ruler widget shared by the time panel and time-aware views. |\n| re_ui | Rerun GUI theme and helpers, built around egui |\n| re_view | Types & utilities for defining view classes and communicating with the viewport. |\n| re_viewer_context | Rerun viewer state that is shared with the viewer's code components. |\n| re_viewport_blueprint | The data model describing the layout of the viewport. |\n\n### Application-level store\n\n[`crates/store_app`](./crates/store_app) — the queryable state a viewer or a server works with.\n\n| Crate | Description |\n| ------------- | --------------------------------------- |\n| re_dataframe | High-level query APIs |\n| re_datafusion | High-level query APIs |\n| re_entity_db | In-memory storage of Rerun entities |\n| re_query | High-level query APIs |\n| re_server | An ephemeral local Rerun catalog server |\n| re_tf | Rerun spatial transform processing |\n\n### Data flow\n\n[`crates/data_flow`](./crates/data_flow) — getting data in and out: clients, servers, and file importers.\n\n| Crate | Description |\n| ----------------- | --------------------------------------------------------------------------------- |\n| re_catalog_client | High-level client for Rerun catalogs |\n| re_data_source | Handles loading of Rerun data |\n| re_grpc_client | Client for the legacy StoreHub API (`re_grpc_server`). |\n| re_grpc_server | Server for the legacy StoreHub API |\n| re_hdf5 | Core HDF5-to-chunk loading logic for Rerun |\n| re_importer | Handles importing of Rerun data from file using importer plugins |\n| re_lenses | Lenses are an API for extracting, transforming, and restructuring component data. |\n| re_lerobot | Core LeRobot-dataset-to-chunk loading logic for Rerun |\n| re_mcap | Convert MCAP into Rerun-compatible data |\n| re_mp4_reader | Core mp4-to-chunk loading logic for Rerun |\n| re_parquet | Core parquet-to-chunk loading logic for Rerun |\n| re_redap_client | Official gRPC client for the Rerun Data Protocol |\n\n### Data model & chunk store\n\n[`crates/store`](./crates/store) — the data model, and the in-memory store that holds it.\n\n| Crate | Description |\n| ------------------ | --------------------------------------------------------------------------------------------- |\n| re_arrow_json | Rerun data as JSON, and back |\n| re_chunk | A chunk of Rerun data, encoded using Arrow. Used for logging, transport, storage and compute. |\n| re_chunk_index | Indexes of Rerun chunks: RRD manifests and the chunk providers that serve them |\n| re_chunk_optimizer | Analysis and memory-bounded optimization of Rerun chunk layouts |\n| re_chunk_store | A storage engine for Rerun's Chunks |\n| re_lenses_core | Core lens types and composable Arrow array transformations. |\n| re_log_channel | An in-memory channel of Rerun data messages. |\n| re_log_encoding | Encode/decode RRD streams: the physical byte layout of .rrd files |\n| re_log_msg | The messages that carry Rerun data between the SDK, `.rrd` files and the viewer. |\n| re_log_types | The basic building blocks of the Rerun log format: entity paths, timelines, and store ids. |\n| re_protos | Rerun remote gRPC/protobuf API types |\n| re_sdk_types | The standard Rerun data types, component types, and archetypes. |\n| re_sorbet | Rerun arrow metadata definitions |\n| re_types_core | The core traits and types that power Rerun's data model. |\n| re_uri | Parsing and constructing Rerun URIs |\n\n### Build support\n\n[`crates/build`](./crates/build) — crates that run at build time.\n\n| Crate | Description |\n| ------------------------ | --------------------------------------------------------------------------------------- |\n| re_build_tools | build.rs helpers for generating build info |\n| re_dev_tools | Various tools for rerun development |\n| re_protos_builder | Code generation for Rerun's Protobuf and gRPC definitions. |\n| re_type_definitions | Rerun's type definitions: a subset of Rust that generates the Rust, Python and C++ SDKs |\n| re_types_builder | Generates code for Rerun's SDKs from the type definitions. |\n| re_types_builder_macros | The `#[rerun_type]` attribute macro used by Rerun's IDL definitions. |\n| re_types_builder_prelude | The vocabulary that Rerun's IDL definitions are written against. |\n\n### Utilities\n\n[`crates/utils`](./crates/utils) — small crates that depend on nothing outside `crates/utils`.\n\n| Crate | Description |\n| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| re_agent | Talk to a coding agent (Claude Code, Codex, …) over the Agent Client Protocol |\n| re_analytics | Rerun's analytics SDK |\n| re_arrow_util | Helpers for working with arrow. |\n| re_async | Async runtime abstractions for native and WebAssembly targets. |\n| re_audio | Rerun audio decoding and playback utilities. |\n| re_auth | Authentication helpers for Rerun |\n| re_backoff | Utilities for retrying operations with backoff. |\n| re_build_info | Information about the build. Use together with re_build_tools |\n| re_byte_size | Calculate the heap-allocated size of values at runtime. |\n| re_byte_size_derive | Derive macro for the `SizeBytes` trait from `re_byte_size`. |\n| re_capabilities | Capability tokens for the Rerun code base. |\n| re_case | Case conversions, the way Rerun likes them |\n| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |\n| re_dump_puffin | Convert puffin profiler recordings (.puffin files) into JSON for analysis |\n| re_error | Helpers for handling errors. |\n| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |\n| re_grpc_headers | Rerun gRPC header conventions: well-known header names, the `RerunVersionInterceptor`, and the tower `Layer` machinery that propagates them across requests and responses. |\n| re_int | Small numeric helper traits: saturating integer casts and unsigned absolute value |\n| re_log | Helpers for setting up and doing text logging in the Rerun crates. |\n| re_memory | Run-time memory tracking and profiling. |\n| re_mutex | A wrapper around `parking_lot::Mutex` useful for debugging deadlocks. |\n| re_perf_telemetry | In and out of process performance profiling utilities for Rerun & Redap |\n| re_quota_channel | A channel that applies backpressure based on byte size. |\n| re_ros_msg | Parse and decode ROS messages |\n| re_rvl | Support for compressed depth data using the RVL format. |\n| re_span | An integer range that always has a non-negative length |\n| re_string_interner | Yet another string interning library |\n| re_tracing | Helpers for tracing/spans/flamegraphs and such. |\n| re_tuid | 128-bit Time-based Unique Identifier |\n| re_video | Rerun video processing utilities. |\n| re_web | Utilities for interacting with Web APIs |\n| re_workspace_hack | Unifies third-party dependency features across the workspace to avoid recompiles. Managed by cargo-hakari. |\n\n\n\n### Dependencies and docs\n\nIn order to get a dependency graph for our in-house crates and their docs, we recommend you run:\n\n```\ncargo install cargo-depgraph\ncargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > /tmp/rerun-deps.png\nopen /tmp/rerun-deps.png\n```\n\nand:\n\n```\ncargo doc --no-deps --open\n```\n\nand then browse through the `re_*` crates.\n\n\n# UI Design System & Style Tokens\n\nPath: `DESIGN.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/DESIGN.md)\n\n\n# rerun_py - System Architecture & Module Boundaries\n\nPath: `rerun_py/ARCHITECTURE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/rerun_py/ARCHITECTURE.md)\n\n\n---\nname: assemble-changelog\ndescription: >\n Assemble the per-PR changeset entries in docs/content/changelog/upcoming/\n into a release's changeset-0-XX.md and generate the detailed CHANGELOG.md sections.\n Use at release time when the user wants to build or finalize the changelog or\n changeset for a Rerun release, or merge the upcoming/ entries. Invoked via\n /assemble-changelog or \"assemble the changelog for 0.x.y\".\nuser_invocable: true\nallowed-tools: Bash Read Edit Write\n---\n\n# Assemble-changelog\n\nRelease-time changelog assembly for the Rerun repo.\n\nAlways work in the root of a standalone `rerun-io/rerun` checkout — normally the `prepare-release-0.x.y` branch, where the result is committed.\nThis is step 4 of [RELEASES.md](../../RELEASES.md); read it for the surrounding context.\n\nBefore doing any release work, verify that the current directory is the repository root and that its `origin` is `rerun-io/rerun`:\n\n```bash\ntest \"$(git rev-parse --show-toplevel)\" = \"$PWD\"\ngit remote get-url origin\n```\n\nDo not run the workflow in the reality monorepo, including from its `rerun/` directory.\nThe release scripts need the standalone repository's `0.x.y` tags and resolve `(#N)` commit references against `rerun-io/rerun`.\nRunning them against reality can silently resolve reality PR numbers to unrelated Rerun PRs.\n\nIf either precondition is not met, stop before running any release command.\nTell the user that the skill requires the root of a standalone `rerun-io/rerun` checkout, and ask them to restart it there.\nDo not clone a repository, fetch tags, or switch branches for the user.\n\nResolve the target version from `$ARGUMENTS` (e.g. `0.34.0`). If absent, read it\nfrom `Cargo.toml` (`version = \"0.x.y-…\"`) and confirm with the user.\n\n## Workflow\n\n### 1. Assemble `upcoming/` → the release changeset\n\nThe curated entries live one-file-per-PR in `docs/content/changelog/upcoming/*.md`\n(skip `_template.md`). Each declares `type: highlight|misc|breaking|feature` in its\nfrontmatter. Merge them into `docs/content/changelog/changeset-0-XX.md`, creating that file from\n`docs/content/changelog/_template.md` if it does not exist yet (set `title` to the version — keep it\nquoted, e.g. `title: \"0.36\"`, so YAML keeps it a string — and `order` one lower than the previous release):\n\n- `highlight` → one `### ` subsection each under `## Highlights`.\n- `feature` → one `### ` subsection each under `## New features`.\n- `misc` → exactly one `- ` list item each under `## Other`. Fold its `### ` heading and body into the item, preserving links. Do not use headings or prose outside the list items.\n- `breaking` → one `### ` subsection each under `## Breaking changes`. If none, write `None.`.\n\nKeep the sections in that order and omit `Highlights` and `Other` if they have no entries.\nThe changelog is user-facing (it's part of the website), so it leads with what's new;\nthe verbose, developer-only breaking-change migration guides go last so most readers\ndon't have to scroll past them.\n\nTailor the output to the release type:\n\n- Patch release (`0.x.Y`, Y > 0) → typically only bug fixes. Skip `Highlights` and\n `New features` (there usually won't be `upcoming/` entries anyway); include `Other`\n only if there are miscellaneous entries, and keep `Breaking changes` only if there are any.\n- Minor release (`0.X.0`) → the full template: highlights, new features, other items, breaking changes.\n\nPreserve each entry's prose and structure (migration guides, tables, `snippet:` directives,\nscreenshots, links), except that each miscellaneous entry becomes its single list item.\nDe-duplicate overlapping entries and order breaking changes most-impactful first.\nDrop the per-entry frontmatter.\n\nRelative doc links in entries were written as if from `changelog/` (e.g.\n`../reference/migration/...`), which is correct once merged — keep them as-is.\n\nImages and videos must live on `static.rerun.io` (upload with `scripts/upload_image.py`), never as\nGitHub attachments: attachments uploaded in the private monorepo cannot be shown on the website.\n\nFeature leads write their `## Highlights` entries per PR, so read the section as a whole and\nflag to the human anything that does not hang together, rather than rewriting it yourself.\n\nFinally, point the `redirect:` frontmatter in `docs/content/changelog.md` at\n`changelog/changeset-0-XX`: CI's `scripts/ci/check_changelog_redirect.py` requires the\nnewest changeset to be the redirect target, so the repoint must land together with the\nnew changeset.\n\n### 2. Resolve release blockers\n\nEnsure that every non-template file from `upcoming/` was merged into the changeset, then search the assembled changeset for unresolved placeholders:\n\n```bash\nrg -n 'TODO\\([^)]+\\)' docs/content/changelog/changeset-0-XX.md # NOLINT\n```\n\nResolve every match before continuing.\nAn unresolved `TODO(name)` blocks the release.\n\n### 3. Generate the summary and detail sections into CHANGELOG.md\n\n```bash\npixi run uvpy scripts/generate_changelog.py --version 0.x.y\n```\n\nEdit PR titles/labels to improve the output, then copy the result into `CHANGELOG.md`\n(drop the trailing \"Chronological changes\" section; replace the placeholder video/blogpost\nlines as previous releases did). Spot-check a few entries against the actual PRs:\npolluted titles (old, unrelated PRs; `thanks @…` for core team members) mean a PR-number\nlookup misfired — see the warning at the top.\nRead the generated section for wrong contributor attribution and typos.\nIf the release ships without Rust crates (`publish-crates=false`), say in the `CHANGELOG.md`\nsection that this version is not on crates.io.\n\nDo this *after* step 1: the script reads the assembled changeset and emits a summary of it\n(section headings + links to the changeset on the website), rather than inlining its prose.\n`CHANGELOG.md` therefore never duplicates the changeset — if the changeset is missing, the\nscript emits an unresolved placeholder instead.\n\n### 4. Polish the CHANGELOG.md section\n\nThe generated section is a starting point; edit it by hand so readers can find their way to the details.\n\n- **Order the overview by impact.**\n Put the biggest new capabilities first, then integrations and APIs, then format support, then polish, then niche or developer-facing items.\n- **Group and order the details.**\n The script emits each details subsection (🐍 Python API, 🪳 Bug fixes, 🌁 Viewer improvements, …) in commit order.\n Within each subsection, put entries that touch the same feature next to each other (e.g. all MCP/ViewerControl entries, all state timeline entries, all hangs and deadlocks), order the groups by their most impactful entry, and order entries within a group by impact.\n Only reorder lines; don't move entries between subsections or drop them.\n- **Make entries clickable.**\n Wherever possible, link an entry (in the overview *and* in the details) to the docs that show how to use the feature: the how-to guide, the archetype/view reference page, the CLI or MCP reference section, the Python/JS ref site, or docs.rs.\n Reuse the links the changeset already contains, converted to `https://rerun.io/docs/`, and check that each target file and `#anchor` heading exists.\n Link external names too (e.g. a third-party tool to its docs).\n In detail entries, link the key term, not the whole line, so the trailing commit/PR link stays distinct.\n- **Link each breaking change** to its subheading in the changeset: `https://rerun.io/docs/changelog/changeset-0-XX#`.\n Anchors are the lowercased heading with backticks and punctuation dropped and spaces turned into hyphens (`rrd::optimize` → `rrdoptimize`).\n- **Name the actual break.**\n A breaking-change heading (in the changeset and in `CHANGELOG.md`) must say what broke, e.g. \"`WriteChunks` removed\", not the new feature that replaces it (\"Catalog staging\").\n Lead the subsection with the removal and migration-guide link, then the replacement.\n- **Flag experimental and unstable features.**\n Check every new feature in the changeset and `CHANGELOG.md` against these markers (`PREV` is the previous minor release tag, e.g. `0.38.0`):\n\n ```bash\n # Non-blueprint types marked unstable whose definitions changed this cycle\n git diff --name-only $PREV HEAD -- crates/build/re_type_definitions | grep -v /blueprint/ | xargs rg -l 'state = \"unstable\"'\n # Python APIs in the `rerun.experimental` module\n git log --oneline $PREV..HEAD -- rerun_py/rerun_sdk/rerun/experimental\n # Viewer features behind a flag in `ExperimentalAppOptions`\n git diff $PREV HEAD -- crates/viewer_support/re_viewer_context/src/app_options.rs\n # Docs and docstrings that call something experimental\n git diff --name-only $PREV HEAD -- docs/content rerun_py/rerun_sdk | xargs rg -l -i 'is experimental'\n ```\n\n A file showing up in the first command does not mean the type is new: cross-reference the hits with the actual entries.\n For each matching feature, say so where readers will see it: prefix the changeset heading with \"Experimental\" (as `### Experimental 3D gaussian splat support` did in 0.36), and add a sentence saying it may change in future releases (for unstable types: that the data may not stay backwards compatible).\n If the feature is behind a flag, say how to enable it.\n In the `CHANGELOG.md` overview, move all of these entries into a final `#### 🧪 Experimental and unstable` subsection at the end of \"Overview & highlights\", with a one-line note that they may change and that unstable types may not stay backwards compatible.\n Order that subsection by impact too, and don't repeat \"Experimental\" in its entries.\n Blueprint types (`rerun.blueprint.*`, under `re_type_definitions/rerun/blueprint/`) are an exception: they are all marked unstable, which only means the blueprint data isn't backwards compatible.\n That alone does not make a feature experimental, so a new view or blueprint setting is only experimental if one of the other markers applies to it (or to the data archetype it shows).\n Likewise, a Viewer feature is not experimental just because it is configured through an experimental API.\n- **Add media to the headline features** in the overview: a screenshot (or a still frame plus a link to the video) indented under the bullet, using the `` markup that `pixi run upload-image` prints.\n Aim for a few images (roughly three to five), covering the most visual headline features: a picture sells a new view or renderer feature far better than a bullet does.\n Start from the media already in the changeset and the PR descriptions; if a visual feature has none, ask the user for a screenshot rather than skipping it.\n Don't add images to every entry: API, CLI, and bug-fix items rarely benefit.\n All media must live on `static.rerun.io`.\n Reuse a PR's media only if it is already on `static.rerun.io` or publicly accessible: `github.com/user-attachments/…` links from the private monorepo return 404 for the public.\n Download those with `curl -L -H \"Authorization: token $(gh auth token)\" ` and upload them with `pixi run upload-image --name `.\n GitHub does not render `` from external hosts, so for a video, extract a representative still (`ffmpeg -ss -i video.mp4 -frames:v 1 still.png`), upload it, and link the `.mp4` below it.\n Look at the frames and pick one without a cursor over the subject.\n\n### 5. Empty the inbox\n\nDelete the merged `upcoming/*.md` entries, keeping `_template.md` and any entries not included in this release.\nNever add redirects for these temporary entries.\n\n## Checklist before declaring done\n\n- [ ] Every non-template `upcoming/` entry is represented in the changeset.\n- [ ] No `TODO(name)` remains in the changeset.\n- [ ] No summaries or other prose were synthesized for existing entries.\n- [ ] The `CHANGELOG.md` overview is ordered by impact, a few of its most visual headline features have `static.rerun.io` images, and its breaking changes link to the changeset subheadings.\n- [ ] Every overview and detail entry that has relevant docs links to them.\n- [ ] Each details subsection groups entries by feature, most impactful first.\n- [ ] Every experimental or unstable feature (feature flag, `rerun.experimental`, `#[rerun(state = \"unstable\")]` on a non-blueprint type, or documented as experimental) is labeled as such in the changeset, and listed under the overview's final \"Experimental and unstable\" subsection.\n- [ ] `pixi run lint-rerun CHANGELOG.md docs/content/changelog/changeset-0-XX.md` passes.\n- [ ] Images and videos are hosted on `static.rerun.io`, not as GitHub attachments.\n- [ ] `## Highlights` reads well as a whole, or the human has been told where it doesn't.\n- [ ] The `CHANGELOG.md` section has correct contributor attribution and no typos.\n- [ ] `upcoming/` contains only `_template.md` and entries deferred to a later release.\n- [ ] `python scripts/ci/check_changelog_redirect.py` passes (redirect points at this changeset).\n- [ ] `python scripts/ci/check_doc_redirects.py --base origin/main` passes (`upcoming/` entries are exempt and DO NOT need a redirect).\n\n## Notes\n\n- This skill lives in `skills/assemble-changelog` in the standalone Rerun repository.\n- Synced commits in `rerun-io/rerun` carry a `Source-Ref` trailer (the reality merge\n commit); `generate_changelog.py` resolves it back to the originating reality PR for\n correct titles, labels, and contributors.\n- The *next* release's changeset is not pre-created: an empty changeset for an\n unreleased version would make `check_changelog_redirect.py` fail, since it requires the newest\n `changeset-0-xx.md` to be the redirect target. During a cycle, `upcoming/` is the only in-flight\n artifact.\n\n\n---\nname: rerun-blueprint\ndescription: Change how the Rerun viewer looks by writing a blueprint, for a recording or for a table. Read this whenever asked to change the look of the viewer, such as laying out a recording or dataset, designing a default blueprint, deciding which views show which entities, or making a segment or catalog table look nice by hiding columns, changing column order or filtering rows. Covers archetype-to-view mapping, layout reasoning, the rrb construction API, the contents grammar, table blueprints, and the screenshot loop.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun blueprint\n\n**To change how the viewer looks, change the blueprint.** This holds for a\nrecording's layout, views and their properties, and for a table's columns, row\nfilters and cards. Don't click through the viewer's menus to do it. With the\nviewer MCP, read and write the blueprint with `rerun_get_blueprint` and\n`rerun_set_blueprint` rather than the low-level click tools.\n\nA blueprint decides how a recording is shown, and the recording decides what exists.\nRead the data, design a first layout, then **iterate from screenshots** until it\nreads at a glance. The constructors are discoverable (`dir(rrb)`,\n`help(rrb.Spatial3DView)`); this skill covers what you can't guess. Import as\n`import rerun.blueprint as rrb`.\n\n## Tables\n\nA dataset's segment table and catalog tables have a table blueprint of their\nown, separate from a recording's. It sets which columns show, their order and\nnames, the row filters, and whether rows show as a table or as cards. When asked\nto make a table look nice, hide columns or filter rows, write this blueprint.\n\nThis hides two columns and shows three others first. Listed columns come first,\nthe rest keep their default order and visibility:\n\n```python\nblueprint = rrb.TableBlueprint(\n table_layout=rrb.table.TableLayout(\n columns=[\n rrb.table.Column(\"recording link\"),\n rrb.table.Column(\"property:episode:current_task\", name=\"Task\"),\n rrb.table.Column(\"property:episode:success\"),\n rrb.table.Column(\"property:episode:ext1_cam_serial\", visible=False),\n rrb.table.Column(\"property:episode:ext2_cam_serial\", visible=False),\n ],\n ),\n)\nblueprint.save(\"my_app\", path)\ndataset.register_blueprint(path.as_uri(), segment_table=True)\n```\n\n`Column` takes the column's name as the table has it, such as\n`property:episode:uuid`, not its header. Registering makes it the default for\neveryone who opens the table, and the server must be able to read `path`.\n\nRow filters are SQL expressions, one per string, all of which must be true for a\nrow: `rrb.TableBlueprint(filters=[\"\\\"task\\\" ILIKE '%pick%'\", '\"duration\" > 10'])`.\nDouble quotes for column names, single quotes for strings, and\n`any_match(\"column\", x -> …)` for list columns.\n\nWith the viewer MCP, change the table on screen directly:\n`rerun_get_table_schema` names the columns, `rerun_get_blueprint` with\n`{\"table\": {}}` reads the table blueprint, and `rerun_set_blueprint` with\n`{\"table\": {}}` replaces it. `rerun_set_blueprint` returns an error for a filter\nthat fails. A column's entry sits at `/table/layouts/table/columns/`,\nwith each `:` and space in the name escaped as `\\:` and `\\ `, which JSON writes\nas `\\\\:` and `\\\\ `. The [MCP docs](https://rerun.io/docs/reference/viewer/mcp#table-blueprints)\nhave a worked example of the JSON. For the whole Python API, including cards\nand segment previews, see\n[Configure table layouts and segment previews](https://rerun.io/docs/howto/visualization/configure-table-blueprints).\n\n## 1. Read the data\n\nEnumerate every `(entity_path, archetype)` pair first — the archetype picks the\nview, the entity path scopes it. For a dataset (see `rerun-catalog-queries`):\n\n```python\nfor c in dataset.schema().component_columns():\n print(c.entity_path, c.archetype, c.component_name)\n```\n\nFor a local recording, stream it with `RrdReader` and read `entity_path` and\nthe archetype off each chunk (see `rerun-chunk-processing`).\n\nThe dataset schema alone doesn't list every entity a view can show. Assets are\nstatic data such as URDF robot meshes, which the viewer applies to every segment,\nbut they don't appear in the segment entities. When `rerun_get_dataset_schema`\nlists `asset_ids` next to `entities`, call `rerun_get_dataset_assets_schema` to\nget the entities in each asset, and read it before choosing between a 2D and a\n3D view or preview. For example, `asset_ids: [\"urdf-model-left\", \"urdf-model-right\"]`\nmeans the robot models can go in a `Spatial3DView`.\n\n## 2. Map archetype to view\n\n| Archetype family | View |\n| ------------------------------------------------------------------------------------- | --------------------------------------------------- |\n| `Points2D`, `Image`, `EncodedImage`, `Boxes2D`, `LineStrips2D`, `Pinhole` projections | `Spatial2DView` |\n| `Points3D`, `Mesh3D`, `Boxes3D`, `LineStrips3D`, `Transform3D`, `Asset3D` | `Spatial3DView` |\n| `Scalars`, `SeriesLines`, `SeriesPoints` | `TimeSeriesView` |\n| `TextLog` / `TextDocument` / `BarChart` | `TextLogView` / `TextDocumentView` / `BarChartView` |\n| `Tensor`, `DepthImage` (heatmap) | `TensorView` |\n| `GeoPoints`, `GeoLineStrings` | `MapView` |\n| `GraphNodes`, `GraphEdges` | `GraphView` |\n| tabular / catalog data | `DataframeView` |\n\nAsset entities, such as `Asset3D` and URDF meshes, go in a `Spatial3DView`.\n\n## 3. Group entities into views\n\nThis step is just deciding which entities share a view. Sizing and placement are\nmechanics you tune by eye in the screenshot loop, not rules to settle up front.\n\n- **Group by shared path prefix.** Entities under a common prefix usually belong\n in one view. Don't give every entity its own pane: dozens of raw message\n entities go in one `DataframeView`, or get left out, rather than a pane each.\n- **One spatial frame per spatial view.** A whole `Transform3D` tree goes in one\n `Spatial3DView` at the common ancestor; a `Pinhole` camera gets its own\n `Spatial2DView` rooted at the camera so images inherit the projection.\n- **Collapse scalars.** Many `Scalars` under a prefix → one `TimeSeriesView` over\n the prefix, not one each. Split only when value ranges clash.\n- **`column_shares`/`row_shares` are relative weights** on a container's\n children, equal by default. Tune them from the screenshots.\n\n## 4. Construct it\n\nContainers (`Grid`, `Horizontal`, `Vertical`, `Tabs`) hold views. Default a flat\nset to `Grid`; use the others for a deliberate split, `Tabs` for alternatives\ncompeting for one slot (left/right/depth cameras). **Always set an explicit\n`origin` and `name`** — `origin` defaults to `/`, which dumps the whole tree into\none view (the usual cause of an unreadable blob).\n\n```python\nblueprint = rrb.Blueprint(\n rrb.Horizontal(\n rrb.Spatial3DView(origin=\"/world\", name=\"Scene\"),\n rrb.Vertical(\n rrb.Spatial2DView(origin=\"/world/camera\", name=\"Camera\"),\n rrb.TimeSeriesView(origin=\"/sensors\", name=\"Sensors\"),\n ),\n column_shares=[3, 2],\n ),\n rrb.TextLogView(origin=\"/logs\", name=\"Logs\"),\n collapse_panels=True,\n)\n```\n\n`contents` defaults to `\"$origin/**\"`. Scope a view with include/exclude rules,\ne.g. `contents=[\"+ $origin/**\", \"- $origin/internal/**\"]`. A bare line is an\ninclude; `/**` is the only wildcard (matches a subtree). Most-specific rule wins,\nties go to the last, unmatched paths are excluded.\n\n**Coordinate frames.** A spatial view only renders entities it can place relative\nto its target frame. `Transform3D`/`Pinhole` logged on entities compose down the\ntree, so `origin` is enough. But **named frames** (`CoordinateFrame(frame=...)`,\ncommon in ROS/MCAP) live in a separate frame graph — point the view at a frame\nthe data occupies via `spatial_information=rrb.SpatialInformation(target_frame=\"\", apply_latest_at=True)`.\n- A view back blank? The importer may not have finished (bump the settle) or the\n cursor sits before the data (add `rrb.TimePanel(play_state=rrb.components.PlayState.Following)`).\n- `rrb` views and `ViewerClient` (`rerun.experimental`) are unstable. Check\n `help()` if a constructor argument is rejected.\n\n## See also\n\n- `rerun-data-model` — entities, archetypes, timelines.\n- `rerun-catalog-queries` — enumerate entities in a dataset.\n\n\n---\nname: rerun-catalog-queries\ndescription: Performance patterns and gotchas for querying a Rerun catalog from Python. Reach for this when exploring an unfamiliar dataset, when a CatalogClient/dataset query is unexpectedly slow, or when shaping a per-segment / per-episode pipeline that hits the catalog from many places.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash\n---\n\n# Rerun catalog queries\n\nPractical performance patterns for querying a Rerun catalog\nfrom Python (`rerun.catalog.CatalogClient` →\n`dataset.reader(...)` → DataFusion `DataFrame`). The DataFusion side of\nthe stack is covered by the **`datafusion-python`** skill — load that\nfor `DataFrame` / `SessionContext` / expression-API references. This\nskill focuses on catalog-specific behaviors and the round-trip costs\nthat catch teams off guard.\n\n---\n\n## Connect with a `rerun://` URL\n\n```python\nfrom rerun.catalog import CatalogClient\n\nclient = CatalogClient(\"rerun+https://api.example.cloud.rerun.io:443\")\ndataset = client.get_dataset(id=dataset_id)\n```\n\n`CatalogClient` takes a `rerun://` or `rerun+https://` URL, or\n`rerun+http://` for a local server. Do not pass a plain `https://` URL.\nA segment URL such as\n`rerun://host:443/dataset/?segment_id=` gives\nthe server, the dataset id and the segment id.\n\nThe client reads the token from `REDAP_TOKEN`, or from the stored\ncredentials of `rerun auth login`, so `token=` is usually not needed.\n\n---\n\n## The query cost model in one sentence\n\nEvery materialization of a `dataset.reader(...)` DataFrame is **one\ncloud round-trip**, with most of the cost being the network/decode\npair, not the compute. Plan for round-trip count and payload bytes, in\nthat order.\n\nA typical catalog round-trip is **a few seconds** even for a tiny\nresult. So:\n- 30 segments × 1 query each ≈ 90s. (Naive per-segment loop.)\n- 30 segments × 4 queries each ≈ 6 minutes. (A splitter that runs\n `count` + `collect_column` for both starts and stops.)\n- 1 query covering all 30 segments ≈ 3s.\n\nThe same fan-out happens along the **entity** axis: 10 entities ×\n1 `filter_contents([one_entity]).reader()` each ≈ 30s, vs one\n`filter_contents([all_entities]).reader()` ≈ 3s. Push as much as you\ncan into one round-trip — across segments and across entities.\n\n---\n\n## Always apply `filter_contents` and time-window filters before `.reader(...)`\n\nThe single biggest lever:\n\n- `filter_contents([entity_globs])` restricts which entity-path columns\n the reader produces. Without it, every entity in the dataset is\n read.\n- Time-window filters push down to storage and dramatically reduce\n bytes scanned when they compare the bare index column with a literal:\n `df.filter(col(index) >= lit(start)).filter(col(index) <= lit(end))`.\n Integer, timestamp and duration literals of any unit work. A `cast`\n or arithmetic on the index column is not pushed down.\n\nCombined: `dataset.filter_segments(seg).filter_contents(entities).reader(...)\n.filter(in_window).select(...)`.\n\n---\n\n## `df.cache()` is your friend for repeated probes\n\nWhen the same materialization gets used by multiple downstream\nfilter/count/collect calls, materialize once with `DataFrame.cache()`\nand operate on the cached frame:\n\n```python\ncached = (\n dataset\n .filter_segments(seg)\n .filter_contents([entity])\n .reader(index=index_col)\n .select(col(index_col).cast(pa.int64()).alias(index_col), value.alias(\"v\"))\n .cache() # one network round-trip, materializes into in-memory batches\n)\nstarts = cached.filter(col(\"v\") == start_val).collect_column(index_col)\nstops = cached.filter(stop_pred(col(\"v\"))).collect_column(index_col)\n```\n\nWithout `cache()`, each `count()` / `collect_column()` re-executes the\nwhole reader chain.\n\n**When NOT to cache.** `cache()` forces materialization into Arrow\nbatches, breaking laziness. If downstream code keeps composing more\nDataFusion ops on top (joins, windows, further filters) and only\nmaterializes once at the end, caching mid-pipeline turns one execution\ninto two and pre-empts whatever physical-plan optimizations the engine\ncould have done across the boundary. Reach for `cache()` when the\nconsumers are terminal (`count()`, `collect_column()`, `to_arrow_table()`),\nnot when they're another lazy `DataFrame`.\n\n---\n\n## Cross-segment batching: drop `filter_segments`, group by `rerun_segment_id`\n\nFor pipelines that need the same query on many segments, omit\n`filter_segments(...)` entirely and pull a single cross-segment table.\nEvery reader row carries a `rerun_segment_id` column — group locally:\n\n```python\ndf = dataset.filter_contents(entities).reader(index=index_col)\ncached = df.select(\n \"rerun_segment_id\",\n col(index_col).cast(pa.int64()).alias(index_col),\n value.alias(\"v\"),\n).cache()\n\n# Now N filter/aggregate calls are local, not network.\nstarts = cached.filter(col(\"v\") == start_val).select(\"rerun_segment_id\", index_col).to_arrow_table()\n```\n\nTrigger / event columns are tiny enough that pulling all segments at\nonce dominates per-segment looping by an order of magnitude.\n\n---\n\n## Per-entity fan-out within a segment\n\nSymmetric to cross-segment batching, along the entity axis. If you\nneed data from N entities of a single segment, **don't** loop:\n\n```python\n# Anti-pattern: N reader setups, N round-trips.\nfor entity in entities:\n df = dataset.filter_segments(seg).filter_contents([entity]).reader(index=ix)\n ...\n```\n\nInstead pull all N at once and project per entity locally. Per \"Reader\nrow layout\" below, every row carries data for one entity and NULLs\nfor the others, so `col(\"::\").is_not_null()`\nis the per-entity filter:\n\n```python\nshared = (\n dataset\n .filter_segments(seg)\n .filter_contents(sorted(set(entities)))\n .reader(index=ix)\n .filter(col(ix).cast(pa.int64()).between(start_ns, end_ns))\n)\n# Each downstream consumer narrows to its entity's rows lazily.\nsrc_a = shared.filter(col(f\"{ent_a}:{comp_a}\").is_not_null()).select(ix, f\"{ent_a}:{comp_a}\")\nsrc_b = shared.filter(col(f\"{ent_b}:{comp_b}\").is_not_null()).select(ix, f\"{ent_b}:{comp_b}\")\n```\n\nDataFusion can share the underlying scan across the per-entity\nprojections when it builds the physical plan, so this stays a single\ncatalog round-trip even though there are N logical consumers. Works\ninside generators that build per-source DataFusion plans (resampling,\nbracket lookup, nearest-in-time joins) — collapse the network fan-out\nwithout changing the per-source logic.\n\nA trap when refactoring: if a downstream query uses a reader column's\nfully-qualified name (`col(f\"{entity}:{archetype}:{component}\")`),\nyou don't need to alias the column in `shared`. The shared reader's\noutput schema preserves native column names, so existing per-entity\nprojection helpers keep working unchanged.\n\n---\n\n## `count()` is *not* free\n\nCounter-intuitive: `df.count()` and `df.aggregate([], [F.count(col)])`\ndo not always push down. Aggregate plans can force the engine to\nmaterialize the underlying column data server-side, then count on the\nclient. `F.count(col)` over wide entity-columns can ship full struct or\nblob payloads to count nullity.\n\nAlternatives, in order of preference for \"is anything here\":\n\n| Need | Use |\n|---|---|\n| \"any row in this filter?\" | `df.select(col(index)).limit(1).to_arrow_table().num_rows > 0` — server short-circuits on first match |\n| \"count rows in a tiny window\" | `df.filter(window).select(col(index)).count()` after the time filter |\n| \"count *each* entity in a wide query\" | per-entity `limit(1)` probes, threaded — *not* one big `count(col)` aggregate |\n\nA trap that fooled us: assuming `bool_or(col.is_not_null())` would only\nneed nullity buffers. It does not — the operator still touches payload\ndata on most plans.\n\n---\n\n## `using_index_values` + `fill_latest_at` is great for resampling, not for presence\n\n```python\n.reader(index=index_col, using_index_values=targets, fill_latest_at=True)\n```\n\n- Returns one row per target timestamp, each entity column carrying its\n latest non-null value at-or-before the target.\n- Excellent for nearest-prior resampling (no DataFusion required).\n- **Don't** use it as a presence check. Two reasons:\n 1. Semantics are \"ever emitted before T\", not \"emitted in [start, T]\".\n 2. The server still ships the full struct/blob payload for every\n entity to compute the latest-known value; a downstream\n `is_not_null()` projection runs post-transfer and doesn't reduce\n wire bytes.\n\nFor a strict in-window presence check, prefer per-entity `limit(1)`\nover the time-filtered reader, run concurrently.\n\n---\n\n## Explore an unfamiliar dataset with `dataset.info()`\n\n```python\ninfo = dataset.info()\nprint(info) # compact text overview\n```\n\nIt reports the segment count and size, each layer with how many\nsegments have it, each timeline with its kind, value range and segment\ncoverage, and the property column names. It reads only the segment\ntable and the schema, no recording data, so it stays cheap on large\ndatasets.\n\nUse it to choose `reader(index=…)` and to bound time-window filters\nbefore the first real query. `TimelineInfo.min` / `.max` are numpy\nvalues that work directly as literals:\n\n```python\nts = next(t for t in info.timelines if t.name == \"capture_time\")\ndf = dataset.reader(index=ts.name).filter(col(ts.name) >= lit(ts.min))\n```\n\nProperties found here can be filtered via `dataset.segment_table()`,\nand layer names tell you what `register(…, layer_name=…)` already\nattached.\n\n---\n\n## Schema introspection is cheap; use it before probing\n\n```python\nschema = dataset.filter_segments(seg).schema()\navailable = {(c.entity_path, c.component) for c in schema.component_columns()}\n```\n\nThis is a single round-trip and tells you which `(entity, component)`\npairs the segment registered. If a column isn't in the schema, you can\ndrop it from your manifest without any further cloud queries. This is\noften a complete substitute for \"does this entity have events\"\nprobing.\n\nCaveat: schema presence ≠ events. The MCAP records the topic schema\neven for unused topics. If your pipeline cares about distinguishing\n\"registered but never emitted\" vs \"registered with events\", you have\nto probe — see \"is anything here\" patterns above.\n\n---\n\n## Reader row layout: entities are columns, not rows\n\nEvery row from `dataset..reader(...)` corresponds to a single\nevent on a single entity. Other entities' columns are null on that\nrow. Implications:\n\n- `select(\"rerun_segment_id\", \"::\")`\n works — quote the entity column when using SQL.\n- There is **no** `rerun_entity_path` row attribute. To attribute rows\n to entities you either filter to one entity at a time, or pick a\n per-entity column (e.g. `:McapChannel:id`) whose non-null pattern\n identifies the source.\n- `df.count()` returns *total events across all entities*, not\n per-entity counts.\n\n---\n\n## Field access on a null struct returns `0.0`, not null\n\nA DataFusion gotcha that bites pipelines reading struct messages:\n\n```python\ncol(\"/some/entity:msg.MyType:message\")[0][\"sub\"][\"x\"]\n# When the parent struct is null on a row, this evaluates to 0.0\n# (and \"\" for strings), not null.\n```\n\nWrap struct-walk projections with a null guard:\n\n```python\nparent = col(\"/some/entity:msg.MyType:message\")\nleaf = parent[0][\"sub\"][\"x\"]\nguard = parent.is_null() | parent[0].is_null() | parent[0][\"sub\"].is_null()\nexpr = F.when(guard, lit(None)).otherwise(leaf)\n```\n\n**Only apply this to struct sources.** For scalar / blob columns\n(`Scalars:scalars`, `EncodedImage:blob`, etc.) the wrap is a no-op at\nbest and at worst rewrites the plan in ways that change downstream\njoin behavior. Gate the guard on whether the source actually walks\nthrough a struct boundary.\n\n---\n\n## Common debug recipe\n\nWhen a query stage is slower than expected:\n\n1. **Count round-trips.** Wrap each `to_arrow_table()` /\n `collect_column()` / `count()` with `time.perf_counter()`. Each is\n a round-trip. If you see N segment queries, that's N × few-seconds\n minimum.\n2. **Split build vs materialize timing.** Time the lazy DataFrame\n construction *separately* from the terminal `to_arrow_table()` call.\n If \"build\" takes seconds, something inside is materializing eagerly\n (an Arrow round-trip in a join helper, a `cache()` in a generator,\n a `.collect()` hidden in a chained-join utility). A correctly lazy\n plan should build in ~milliseconds regardless of result size.\n3. **Measure bytes.** `tbl.nbytes` after `to_arrow_table()` reveals\n when \"I projected `is_not_null()`\" actually shipped megabytes. If\n bytes are large despite a small projection, the operator didn't\n push down.\n4. **Cross-segment first, then cross-entity.** If the per-segment query\n is fundamentally the same (just scoped by id), drop `filter_segments`\n and group by `rerun_segment_id` locally. If you also have a\n per-entity loop within a segment, collapse it the same way (one\n `filter_contents([all])` reader, per-entity `is_not_null()` filters\n downstream).\n5. **Cache before terminal re-use.** If two `count()` / `collect_column()`\n calls share the same reader, `df.cache()` between them. Don't cache\n if the consumers are themselves lazy DataFrames being composed\n further — caching breaks plan-wide optimization.\n6. **Window first.** Always push the time filter before any projection\n or aggregate that touches payload columns.\n\n---\n\n## See also\n\n- `datafusion-python` skill — DataFrame API, SQL parity, expression\n building, common pitfalls (boolean operators, immutability, etc.).\n Not installed? Ask the user to install it globally:\n `npx skills add apache/datafusion-python`.\n\n\n---\nname: rerun-chunk-processing\ndescription: \"Core mechanics of the Rerun Chunk Processing API (rerun.chunk) — LazyChunkStream pipelines, Chunk, lenses (MutateLens/DeriveLens/Selector), RrdReader, writing optimized RRDs. Read BEFORE writing any ingestion/conversion/preprocessing code (convert an MCAP, build a recording from a dataset, preprocess an .rrd, port an old converter): it mandates reader+lens pipelines and steers away from hand-built chunks — no Chunk.from_columns for data a reader/lens can produce, no per-message rr.log, no manual pa.array assembly. Source-specific knowledge lives in the importer skills (rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot); read rerun-data-model first to decide what the data should become.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun chunk processing\n\nThe pipeline layer between raw data and an RRD: readers produce `Chunk`s,\nstreams transform them, terminal calls execute. This skill is the generic\nmechanics only. Decide the data model first (`rerun-data-model`), then pick the\nimporter skill for each source:\n\n| Source | Reader | Skill |\n| ----------------------------------------- | --------------------------------------- | --------------- |\n| MCAP file (ROS2, protobuf, Foxglove) | `McapReader(path).stream()` | `rerun-mcap` |\n| URDF robot model (+ joint states → FK) | `UrdfTree.from_file_path(...).stream()` | `rerun-urdf` |\n| Parquet table (trajectories, sensor logs) | `ParquetReader(path).stream()` | `rerun-parquet` |\n| mp4 camera video | `Mp4Reader(path).stream()` | `rerun-mp4` |\n| HDF5 file (groups → entities) | `Hdf5Reader(path).stream()` | none yet |\n| LeRobot dataset directory | built-in importer, then `RrdReader` | `rerun-lerobot` |\n| Existing RRD | `RrdReader(path)` | here, below |\n| Sidecar files (JSON calib, metadata) | `Chunk.from_columns` + `from_iter` | here, below |\n\nThe API is `rerun.chunk`; when\nbehavior matters, check the installed surface:\n`python -c \"from rerun.chunk import LazyChunkStream; help(LazyChunkStream)\"`.\n\n## Decision rule: where does each component come from?\n\nDefault: **a reader produces the chunks; lenses shape them.** Walk this before\nwriting any conversion code — most \"build it by hand\" instincts are wrong here:\n\n1. **Source a reader supports?** Use the reader's `.stream()`; never hand-parse\n and re-log. MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`,\n mp4→`Mp4Reader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`,\n LeRobot dir→`log_file_from_path`.\n2. **A decoder already emits the archetype?** Foxglove gives `Transform3D`,\n `Pinhole`, `VideoStream` (real sample bytes) ready-made — **pass it through**,\n do not re-derive. Only custom-protobuf topics arrive as `:message` and\n need a lens (see `rerun-mcap`).\n3. **Fix an existing component in place** (swapped resolution, recolor, unit\n convert)? `MutateLens`, `output_mode=\"forward_unmatched\"`.\n4. **Derive a new component/entity** (FK→`/tf`, scalars from a message)?\n `DeriveLens`. To scatter one row into N (a joint batch → per-joint `/tf`),\n use the **two-lens pair**: derive the batch with `output_mode=\"forward_all\"`\n (keeps the originals, e.g. the joint states), then a second\n `DeriveLens` with `scatter=True` and `output_mode=\"drop_unmatched\"` (emits\n only the scattered rows). See the `robot_data_preprocessing` example.\n5. **Genuine sidecar** no reader or lens can produce (JSON calibration offsets,\n hand-measured extrinsics, external metadata)? `Chunk.from_columns` + `from_iter`.\n6. Finish with `LazyChunkStream.merge(...)` →\n `.collect(optimize=OptimizationProfile.OBJECT_STORE)` →\n `write_rrd(application_id, recording_id)`.\n\nWhy this order: the pipeline stays lazy, columnar, multithreaded, and\n`OBJECT_STORE`-optimizable. A hand-built row loop or an out-of-lens `pa.array`\nthrows all of that away — that is the path we are deliberately avoiding.\n\n## Anti-patterns (use a reader + lens instead)\n\nIf you are writing the left, stop and use the right:\n\n- **`for`-loop building rows/components** → a lens with a `Selector(...).pipe(...)`\n PyArrow-compute callback.\n- **`rr.init` + `rr.log` per message for conversion** → that is _live_ logging;\n for ingestion, read with a reader and `write_rrd`.\n- **`chunk.to_record_batch()` + `pc.filter` then rebuilding via\n `Chunk.from_columns`** (row-thinning by hand) → `stream.drop(content=...)`,\n `.split(...)`, or a `MutateLens` returning a filtered `pa.array`.\n- **`pa.array` / `pa.RecordBatch` / `np.frombuffer` assembled OUTSIDE a lens** →\n move the transform inside a `MutateLens`/`DeriveLens` selector callback.\n- **`rr.send_columns` hand-assembled from a custom parser** → use the matching\n reader; it produces chunks directly.\n- **Parsing MCAP/URDF with a non-Rerun library then re-logging** → `McapReader`\n / `UrdfTree`.\n- **`Chunk.from_columns` for data a reader already decodes** (`Pinhole`\n intrinsics, `VideoStream`, `Transform3D` from a transforms topic) → keep it in\n the reader stream; fix with a `MutateLens` if needed.\n\nA wall of `pyarrow.compute` \"missing-attribute\" type errors (`pc.filter`,\n`pc.list_element`) usually means `pc.*` calls sit in module-level helpers instead\nof inside `Selector.pipe` lens callbacks. Refactor into a lens before suppressing\nthe checker — the errors are a smell that the hand-building should not exist.\n\n**Porting an existing converter?** Hand-built converters predate decoder\nimprovements and are not ground truth. Re-verify the decoder output (step 2) and\ncheck every `Chunk.from_columns` / for-loop against this list before copying.\n\n## Core model\n\n- `LazyChunkStream` is a lazy pipeline DAG, not a collection. Building\n filters, lenses, maps, splits, and merges reads no source data.\n- Execution starts at terminal calls: `write_rrd(...)`, `collect()`,\n `to_chunks()`, or iterating the stream.\n- Execution is streaming, multithreaded, and mostly GIL-free. Prefer\n stream/lens operations and PyArrow compute over Python row loops.\n- **Move semantics**: builder calls (`filter`, `drop`, `lenses`, `map`,\n `flat_map`) consume the input stream; reusing a consumed stream raises.\n Reassign after each step. Terminal calls do not consume, but each terminal\n call re-executes the whole pipeline; `collect()` once if that is too costly.\n- `ChunkStore` is materialized in memory (`stream.collect()`,\n `ChunkStore.from_chunks`). `LazyStore` is manifest-indexed, loads chunks on\n demand (`RrdReader(path).store()`, catalog segment stores). Both have\n `schema()`, `summary()`, `stream()`, and `write_rrd(...)`.\n\n## Stream composition\n\n```python\nfrom rerun.chunk import Chunk, LazyChunkStream, OptimizationProfile\n```\n\n- `stream.filter(content=, has_timeline=, is_static=, components=)` keeps the\n matching portion of each chunk; `stream.drop(...)` is its complement, same\n keyword filters. `content` takes an entity-path glob or a list of them.\n- `stream.map(fn)` applies `Chunk -> Chunk`; `stream.flat_map(fn)` applies\n `Chunk -> Iterable[Chunk]`. Escape hatches for chunk-level Python logic;\n prefer lenses for columnar work.\n- `stream.split(content=, ...)` returns `(matching, non_matching)`; both\n branches share the same upstream.\n- `LazyChunkStream.merge(*streams)` fans in any number of sources.\n- `LazyChunkStream.from_iter(chunks)` wraps hand-built chunks.\n\n```python\nstream = source_stream() # any importer skill\nstream = stream.drop(content=\"/video_raw/**\")\nstream = stream.lenses(fix_lens, content=\"/cam/**\", output_mode=\"forward_unmatched\")\nmerged = LazyChunkStream.merge(stream, sidecar_stream)\nmerged.write_rrd(out_path, application_id=\"my_app\", recording_id=recording_id)\n```\n\n## Hand-built chunks — sidecar only\n\nUse `Chunk.from_columns` ONLY for data no reader or lens can emit — JSON/CSV\ncalibration, frame offsets, external metadata. If a reader\n(`McapReader`/`UrdfTree`/`ParquetReader`/`Hdf5Reader`) decodes the topic or a\nlens can derive it, that is the idiomatic path; do not hand-assemble it here.\nIn the `robot_data_preprocessing` example the _only_ hand-built chunk is the JSON\noffsets sidecar; the camera fix, FK→`/tf`, meshes, and recolor are all\nreaders + lenses.\n\n`Chunk.from_columns(entity_path, indexes, columns)` mirrors\n`rr.send_columns(...)` and accepts the same archetype `.columns(...)` helpers.\nEmpty `indexes` means static.\n\n```python\nchunk = Chunk.from_columns(\n \"/tf_static/robot_offsets\",\n indexes=[], # static\n columns=rr.Transform3D.columns(\n translation=translations,\n quaternion=quaternions_xyzw,\n parent_frame=parents,\n child_frame=children,\n ),\n)\nsidecar_stream = LazyChunkStream.from_iter([chunk])\n```\n\n`rr.AnyValues.columns(...)` covers non-standard metadata fields. For\ninspection, a `Chunk` exposes `entity_path`, `num_rows`, `is_static`,\n`timeline_names`, `to_record_batch()`, and `format()` (human-readable table).\n\n### Recording properties\n\nRecording properties are the one exception here: do NOT hand-build them.\n`Chunk.from_property(name, values)` mirrors `rr.send_property` and lands a\nstatic chunk on `/__properties/`.\n\nDo this:\n\n```python\nchunk = Chunk.from_property(\n \"episode\",\n rr.AnyValues(task=\"pick\", duration_sec=12.4),\n)\n```\n\ninstead of hand-assembling the same chunk:\n\n```python\nChunk.from_columns(\n f\"/__properties/{name}\",\n indexes=[],\n columns=rr.AnyValues.columns(**{name: [value]}), # type: ignore[arg-type]\n)\n```\n\n- `/__properties/` is Rerun's internal layout; the second version hardcodes it.\n- A dynamic field name makes `AnyValues.columns()` need a\n `# type: ignore[arg-type]`. That suppression is the signal to switch.\n\n## Lenses\n\nLenses reshape, fix, or derive components without iterating rows. Apply with\n`stream.lenses(lenses, output_mode=..., content=...)`.\n\n- `MutateLens(component, selector, keep_row_ids=False)` modifies an existing\n component in place.\n- `DeriveLens(component, output_entity=None, scatter=False)` creates new\n columns, optionally at another entity. Chain `.to_component(descriptor,\nselector)` per output; `.to_timeline(name, \"sequence\" | \"duration_ns\" |\n\"timestamp_ns\", selector)` extracts a time column from the data itself.\n `scatter=True` explodes one input row into N output rows (one per list\n element).\n- Scope with `content=` whenever the same component name exists under multiple\n entities.\n\nOutput modes, and **the default is `drop_unmatched`**:\n\n- `drop_unmatched` (default): only lens outputs survive. Right for derive-only\n intermediate streams; silently discards everything else if applied broadly.\n- `forward_unmatched`: lens outputs plus the original components no lens\n consumed. Right for targeted fixes that preserve the rest of the stream.\n- `forward_all`: lens outputs plus every original component, including a\n `DeriveLens` input. Same as `forward_unmatched` for mutate lenses.\n\nIn-place fix (keep Arrow type and length intact):\n\n```python\nstream = stream.lenses(\n MutateLens(\n \"Pinhole:resolution\",\n Selector(\".\").pipe(\n lambda res: pa.array(\n [(h, w) for w, h in res.to_pylist()],\n type=res.type,\n )\n ),\n ),\n content=[\"/external/cam_low\", \"/external/cam_high\"],\n output_mode=\"forward_unmatched\",\n)\n```\n\nDerive with unit conversion (PyArrow compute, no Python loop):\n\n```python\nDeriveLens(\"schemas.proto.JointState:message\", output_entity=\"/joints_deg/waist\").to_component(\n rr.Scalars.descriptor_scalars(),\n Selector(\".joint_positions\").pipe(lambda arr: pc.multiply(pc.list_element(arr, 0), 180.0 / math.pi)),\n)\n```\n\n## Selector grammar\n\n`Selector(\"\")` navigates nested Arrow data, jq-style:\n\n- `.` current value; `.field` struct field\n- `[]` iterate list elements; `[N]` index a list\n- `?` suppress errors / skip missing optionals; `!` assert non-null\n- `|` pipe one expression into another\n\n`.pipe(fn)` chains a Python/PyArrow transform (or another Selector).\n`.execute(array)` runs it eagerly; `.execute_per_row(array)` guarantees the\noutput row count matches the input (use inside lens callbacks that must stay\nrow-aligned).\n\n## Writing RRDs\n\n- `stream.write_rrd(path, application_id=..., recording_id=...)` executes and\n writes in one streaming pass.\n- `stream.collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(...)`\n materializes, optimizes chunk layout, then writes. Memory scales with the\n materialized chunks.\n- Profiles: `OBJECT_STORE` (large chunks, for storage/query/catalog) and\n `LIVE` (small chunks, low-latency viewer).\n- Multiple physical RRDs form one logical recording when they share a\n `recording_id`; use this to separate base data, model/URDF data, and layers.\n\n**Always use `OptimizationProfile.OBJECT_STORE`** when the RRD is headed for a\nRerun catalog or Hub, unless explicitly asked otherwise.\n\n## Chunk API vs logging API\n\n- Logging (`rr.log`, `rr.send_columns`, `RecordingStream`) is for live logging\n from user code; chunk processing is for ingestion, conversion, and\n postprocessing existing recordings.\n- Logging → chunks: write an RRD, read it back with `RrdReader`.\n `RrdReader(path)` lists `recordings()` / `blueprints()` (each a `StoreEntry`\n with `kind`, `application_id`, `recording_id`); `.stream(store=entry)` for\n sequential passes, `.store(store=entry)` for indexed access.\n- Chunks → logging: `rerun.send_chunks(chunks, recording=...)`\n accepts a `Chunk`, `LazyChunkStream`, `LazyStore`, `ChunkStore`, or any\n iterable of chunks. The source store's `application_id`/`recording_id` are\n **not** preserved; the active recording's identity wins.\n\n## Common gotchas\n\n- The default lens `output_mode` is `drop_unmatched`; forgetting to set\n `forward_unmatched` on a targeted fix silently drops the rest of the stream.\n- Do not reuse a consumed `LazyChunkStream`; reassign or `split` deliberately.\n- Scope lenses with `content=`; the same component name often exists under\n many entities.\n- Preserve Arrow array type and length in `MutateLens` transforms.\n- For catalog layers, the layer `recording_id` must equal the segment id.\n- This is `rerun.chunk`; pin-check signatures when upgrading.\n\n## References\n\n- End-to-end example (MCAP + URDF + JSON sidecar, lenses, merge, optimize):\n `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- Docs: `https://rerun.io/docs/concepts/logging-and-ingestion/chunk-processing-api`,\n `https://rerun.io/docs/concepts/query-and-transform/lenses`\n\n\n---\nname: rerun-data-model\ndescription: \"How raw multimodal robot data maps onto the Rerun data model. Read FIRST, before modeling or converting a dataset — and whenever you are about to convert/ingest/preprocess robot data into an .rrd or build a Rerun recording, even if not asked for the data model. Resolves the entity-vs-component, property-vs-component-vs-layer, and static-vs-temporal decisions and routes to the mechanism (do it with readers and lenses, not hand-built chunks or per-message rr.log): rerun-chunk-processing and the importer skills rerun-mcap, rerun-urdf, rerun-parquet, rerun-mp4, rerun-lerobot.\"\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun data model\n\nThe hard part of ingesting a dataset is the **modeling decision**, not the API call.\nGet the model right and any mechanism works; get it wrong and queries, views, and training all break.\nThis skill is just the decisions.\nFor mechanism details see `rerun-chunk-processing` (pipeline mechanics) and the importer skills it routes to: `rerun-mcap`,`rerun-urdf`,`rerun-parquet`, `rerun-mp4`, `rerun-lerobot`.\nFor exact signatures, the docs at `rerun.io/docs/concepts/logging-and-ingestion`.\n\n**Before writing conversion code, fill in the mapping table below.** It is the design, and a human can review it in seconds.\n\n## Pick the mechanism before you model the bytes\n\nModeling decides _what_ each datum becomes; this decides _how_ it gets there — and the default is **a reader + lenses, not hand-built chunks**:\n\n- **Does a reader exist for this source?** MCAP→`McapReader`, URDF→`UrdfTree`, parquet→`ParquetReader`, HDF5→`Hdf5Reader`, RRD→`RrdReader`, LeRobot dir→`log_file_from_path`. **Yes →** `reader.stream()` + lenses; the reader produces the chunks, you do not.\n- **No, and it is genuine external metadata** (JSON calibration, offsets) or a specific custom use case that can't be covered by a generic reader**→** `Chunk.from_columns`.\n- **Otherwise**: consider if you are about to hand-build something a reader or lens should produce and ask for clarification.\n\nFor MCAP specifically: a Foxglove- or ROS-decodable file emits archetypes like `Transform3D`, `Pinhole`, and `VideoStream` for certain supported message schemas ready-made — pass those through, never re-derive them; only custom protobuf signal topics need lenses. The full decision tree and the anti-pattern list are in `rerun-chunk-processing`.\n\n## The model\n\n```\nDataset → Segment → Layer → Recording → Entity → Component → Chunk\n```\n\n- **Entity** = a thing, named by a path (`/robot/arm/camera`).\n **Component** = one typed field on it (positions, image, a scalar).\n **Archetype** = a standard bundle of components that log and render together (`Points3D`, `Image`, `Transform3D`).\n- Every value you log sits on two axes: **where** (entity path + component) and **when** (which timeline, or _static_ = all time).\n- **Segment** = one episode; on registration its `recording_id` becomes the `segment_id`. **Layer** = an extra `.rrd` on top of a segment. It attaches by matching the segment's `recording_id` and nothing else: that shared id is the whole layering mechanism.\n\n## The decisions\n\n**Entity vs component on an existing entity**\nNew entity if it has its own spatial frame (`Transform3D`/`Pinhole`), its own annotation context, or should be shown/cleared/shared independently (each robot link, each camera, each sensor). Same entity + extra component for auxiliary data at the same instances (per-point confidence).\nUse `AnyValues` for non-standard fields.\n\n**Property vs component vs layer** (the most common confusion)\n\n- **Component**: per-timestamp signal on the timeline (joint angle, image).\n- **Segment property**: one-per-episode metadata for catalog filter/search (operator, robot, site, task, date). Not per timestamp.\n- **Layer**: a whole derived `.rrd` over a base segment (FK transforms, point clouds, gripper state, labels, quality scores). Queryable as if part of base.\n\n**Static vs temporal**\nStatic belongs to all timelines and shadows any temporal value of the same component on the same entity. Use it for invariants (calibration, coordinate frames, robot meshes, annotation context, a video asset). Never make per-frame data static.\n\n**Which timeline**\nA `timestamp` timeline (ns since epoch) for cross-sensor clock alignment, a `sequence` timeline for frame/ordinal alignment; stamp on both when useful.\n**Do not resample to a common rate.** Latest-at reconciles multi-rate streams at query time by holding each component's last sample (no interpolation).\n\n**Base vs layer**\nBase = faithful conversion of the raw streams, nothing computed.\nLayer = anything derived (FK from URDF + joint states, clouds from depth + intrinsics).\nKeep them separate `.rrd`s.\n\n## The mapping table (produce before coding)\n\n| Source (topic/column/key) | Entity path | Archetype | Component(s) | Timeline | Static/temporal | Base/layer | Property? |\n| ------------------------- | -------------------- | ------------------------------------- | -------------------- | ------------- | --------------------------- | ---------- | --------- |\n| mcap `/joint_states` | `/robot/` | `Scalars` | `scalars` | `sensor_time` | temporal | base | no |\n| `cam0/color.mp4` | `/camera/cam0/video` | `VideoStream` | codec+sample | `video_time` | codec static, samples temporal | base | no |\n| `calibration.json` | `/camera/cam0` | `Pinhole` (+`Transform3D` extrinsics) | `image_from_camera` | — | static | base | no |\n| URDF + joints (computed) | `/robot/` on the CLI) at the dataset _directory_ and it ingests episodes, camera videos, and state/action tables with no conversion code.\n- **Converting?** `rr.experimental.LeRobotReader` reads the dataset one episode at a time as a lazy chunk stream, which is what you want for per-episode RRDs and for lenses.\n\nThe download step needs `huggingface_hub`.\n\n## Step 0: pick a dataset and check the HF token\n\nBoth scripts below live in `scripts/` next to this file, and run standalone through `uv` — they declare `huggingface_hub` inline, so there is no environment to set up.\n\nTwo facts decide what a dataset costs to fetch, and the Hub search results show neither.\nGet both, for every candidate, in one call:\n\n```bash\nuv run scripts/find_lerobot_dataset.py --robot-type --want 5\n```\n\n**`codebase_version` decides whether a partial download is possible at all.**\n\n- **v2.1 and earlier** store one parquet and one mp4 per episode (`video_path` ends in `episode_{episode_index:06d}.mp4`), so \"just the first N episodes\" downloads only those episodes.\n- **v3.0** concatenates every episode into a few large files (`video_path` ends in `file-{file_index:03d}.mp4`), so any subset costs the whole download.\n If the user asked for a few episodes, prefer a v2.1 dataset — for the _download_ only.\n `LeRobotReader` reads a single episode out of either version once the files are local.\n\n**How camera frames are stored decides the size**, and it varies by two orders of magnitude.\nThe `frames` and `MB/ep` columns report it:\n\n- `mp4`: a `video`-dtype feature, one file per episode beside the parquet.\n- `inline`: an `image`-dtype feature, raw frames inside the episode parquet, which is far larger for the same footage.\n\nBoth scripts report the actual size for the dataset in hand, so use that number rather than guessing.\nTell the user the size before you start the download, and prefer `mp4` unless they asked for something only the other set has.\n\n`HF_TOKEN` is not required for public datasets, but without it the Hub rate-limits and throttles downloads.\nIf it is unset, say so and point the user at to create a read token, then `export HF_TOKEN=hf_…` (or `hf auth login`).\nDo not block on it — carry on unauthenticated and let the user decide.\n\n```bash\n[ -n \"$HF_TOKEN\" ] || echo \"HF_TOKEN unset — downloads will be throttled\"\n```\n\n## Step 1: dataset -> one combined RRD\n\nOnly needed if you want every episode in one file — to hand someone a single RRD, or to reprocess data you cannot re-read from the dataset directory.\nFor per-episode segments, skip to Step 2 and let `LeRobotReader` read the dataset directly.\n\n```python\nfrom huggingface_hub import snapshot_download\nimport rerun as rr\n\ndataset_dir = snapshot_download(repo_id=repo_id, repo_type=\"dataset\", local_dir=dest)\n\nwith rr.RecordingStream(\"rerun_example_lerobot\") as rec:\n rec.save(str(combined_rrd))\n rec.log_file_from_path(str(dataset_dir)) # the built-in importer\n```\n\nThe importer emits one recording per episode (recording ids like `episode_1`), plus a metadata-only root recording, all into the single RRD.\n\n`rr.RecordingStream` + `log_file_from_path` here is the **importer bootstrap** — the one place `RecordingStream` is correct in an ingestion pipeline (it drives the built-in importer, not per-message logging). Do not generalize it to `rr.log`-per-message loops; for everything after import, reprocess with a reader + lenses (see `rerun-chunk-processing`: Chunk API vs logging API).\n\n### Downloading only the first few episodes\n\n```bash\nuv run scripts/fetch_lerobot_episodes.py --episodes 5 # prints the dataset directory\n```\n\nIt reports the download size before starting, downloads `meta/*` plus the wanted `episode_%06d` files, trims `meta/` to match, verifies that exactly the requested episodes landed, and prints what it got.\nIts last line is the dataset directory; do not follow it with `find` or `du`.\nDo the equivalent by hand and three things bite:\n\n- The importer iterates the episode list in `meta/`, so an untrimmed `meta/` warns once per absent episode.\n- `snapshot_download` writes into an existing directory, so episodes left by an earlier run with a different N survive and inflate the result.\n- Both failures are invisible until you count the recordings in the viewer, which is why the script asserts instead of leaving you to look.\n\n### Just looking at it in the viewer?\n\nSkip the RRD entirely — the viewer imports the dataset directory itself.\nPass the plain path (`rerun /path/to/dataset`, or `open_url` over MCP).\nEach episode arrives as its own recording, with `recording_id` `episode_0`, `episode_1`, ….\n\n## Step 2: one RRD per episode\n\nCatalog segments are one-recording-per-file, and `recording_id` becomes the segment id on registration.\n\n`LeRobotReader` goes straight there, without building the combined RRD of Step 1 first:\n\n```python\nreader = rr.experimental.LeRobotReader(dataset_dir) # v2 or v3\nfor episode in reader.episodes():\n episode_id = f\"episode_{episode:05d}\" # zero-padded; see below\n reader.stream(episode).write_rrd(\n rrd_dir / f\"{episode_id}.rrd\",\n application_id=\"my_dataset\",\n recording_id=episode_id,\n )\n```\n\nStreaming is lazy end to end, so memory is bounded by chunk size rather than by episode or dataset size, and video is cut to the episode's time window.\nB-frame-free video (AV1, LeRobot's default codec) streams directly; a stream that must be re-encoded — H.264 with B-frames, or a window starting mid-GOP — needs `ffmpeg` on the `PATH`.\n`stream()` also takes `entity_path_prefix`, `timeline`, and `video_mode=\"skip\"` to drop video entirely.\n\nTo add lenses, drop topics, or fix data, put them between the stream and the write — see `rerun-chunk-processing`.\n\n### Splitting an RRD you already have\n\nIf the combined RRD of Step 1 already exists — or the data came from somewhere other than the dataset directory — split it with `RrdReader` instead:\n\n```python\nreader = rr.chunk.RrdReader(str(combined_rrd))\nfor entry in reader.recordings():\n store = reader.store(store=entry)\n if not store.schema().entity_paths(): # skip the metadata-only root recording\n continue\n episode_id = zero_pad(entry.recording_id) # episode_1 -> episode_00001\n with rr.RecordingStream(\"rerun_example_lerobot\", recording_id=episode_id, send_properties=False) as rec:\n rec.save(str(rrd_dir / f\"{episode_id}.rrd\"))\n rec.send_chunks(store)\n```\n\nTwo non-obvious moves:\n\n- **Zero-pad the episode id.** `episode_10` sorts before `episode_2`\n lexicographically; segment tables and viewers sort lexicographically. Pad to\n a fixed width when re-assigning `recording_id`.\n- **`send_properties=False`** on the new stream, so the copy doesn't inject\n fresh recording properties on top of the copied chunks.\n\n`send_chunks` does not preserve the source store's identity; the new stream's\n`recording_id` wins, which is exactly what makes the rename work.\n\nIf episodes need cleanup (drop topics, fix data, add derived components), run the store through lenses between read and write: `reader.stream(store=entry).drop(...).lenses(...)` then `collect().write_rrd(..., recording_id=episode_id)` (see `rerun-chunk-processing`).\n\nComputed layers and per-episode properties then follow the standard patterns in `rerun-data-model` (layer `recording_id` must equal the episode segment id).\n\n## Step 3: add the robot model (URDF)\n\nA LeRobot dataset ships joint values, never a robot.\nWithout a URDF you get camera views and joint plots but no 3D arm, which is usually not what the user pictured when they asked to look at the data.\nAdding one is a forward-kinematics layer — `rerun-urdf` owns that; this section is only the LeRobot-specific wiring.\n\n- **The URDF.** `meta/info.json` `robot_type` names it (`so100`, `so101`, `aloha`, …); `rerun-urdf` has the source table.\n- **The joint names.** `meta/info.json` `features.observation.state.names` lists them in column order.\n Map each source name explicitly to the corresponding URDF joint name; do not assume that removing a prefix produces a valid URDF name.\n- **The values.** `observation.state` is the measured pose and the one to run FK on; `action` is the commanded pose and belongs on the plots, not the robot.\n- **The units and calibration.** Confirm the units, sign, and offset against the dataset and robot documentation.\n Run `rerun-urdf`'s `check_joint_mapping.py` against the episode parquets before trusting a pose.\n\nThe FK source is the per-episode parquet, read directly rather than out of the imported RRD:\n\n```python\nParquetReader(str(episode_parquet)).stream(\n column_grouping=\"individual\",\n index_columns=[IndexColumn.sequence(\"frame_index\")],\n).drop(content=\"/__properties/**\")\n```\n\n`frame_index` is the index the importer uses too, so the FK transforms line up with the video and plots.\nMerge the URDF model stream plus the FK stream into each episode's RRD, keeping the episode's `recording_id` (see `rerun-urdf`'s \"Minimal shape\").\nFor a whole dataset, the model belongs in a shared asset rather than in every segment — also `rerun-urdf`.\n\n## Gotchas\n\n1. `log_file_from_path` must target the dataset **root directory**, not a file inside it. So does `LeRobotReader`.\n2. Unpadded episode ids sort incorrectly downstream; pad before registering.\n3. The combined RRD contains a metadata-only root recording; skip stores with no entity paths or you register an empty segment.\n4. A partially-downloaded dataset imports fine, but warns once per episode listed in `meta/` and missing on disk.\n5. Downloading a subset into a directory an earlier run already populated leaves the older episodes in place; start from an empty directory.\n6. `image`-dtype (inline) datasets can be ~100x larger per episode than `mp4` ones; check before downloading, not after.\n7. Check signs and units - e.g. an `observation.state` in degrees could produce wrong FK when fed into a robot model that expects radians.\n\n## References\n\n- `rerun-chunk-processing` (`LazyChunkStream`, lenses, `RrdReader`)\n- `rerun-urdf` (adding the robot model and the joint-value mapping)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/dataloader` `prepare_dataset.py` (download → import → split → register, complete and runnable) and `train.py` (training-side consumption via `rerun.experimental.dataloader`)\n\n\n---\nname: rerun-mcap\ndescription: Ingest MCAP files into Rerun chunk streams with rerun.chunk.McapReader. Read when converting an MCAP recording, selecting topics or decoders, decoding custom protobuf messages, or when an MCAP-derived stream comes out empty. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (what the topics should become).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun MCAP ingestion\n\n`McapReader` turns an MCAP file into a lazy chunk stream: one entity per topic\nat the topic's path, message payloads decoded by pluggable decoders. This\nskill covers the reader's options, what each topic becomes, and the failure\nmodes that yield an empty stream with no error. Stream mechanics (filter, drop,\nlenses, merge, write) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.chunk import McapReader\n\nreader = McapReader(mcap_path) # see help(McapReader) for the full option set\nstream = reader.stream()\n```\n\nA URDF embedded in the MCAP can be ingested as well (then see `rerun-urdf`).\n\n## What a topic becomes\n\nWith the default decoders (`decoders=None`), the message **schema name** decides what a topic becomes — **pass archetypes through, lens only the raw `:message` topics**:\n\n| MCAP schema name | decodes to | what to do |\n| --- | --- | --- |\n| `foxglove.FrameTransforms` | `Transform3D` | pass through; do **not** hand-build |\n| `foxglove.CameraCalibration` | `Pinhole` | pass through |\n| `foxglove.CompressedVideo` | `VideoStream` (real sample bytes) + `CoordinateFrame` | pass through |\n| other `foxglove.*` well-known types | the matching archetype | pass through |\n| ros2 well-known types (`ros2msg` / `ros2_reflection`) | archetype | pass through |\n| your own `schemas.proto.*` / custom protobuf | one `:message` struct | attach semantics with a `DeriveLens` + `Selector` |\n\nSo a camera topic already arrives as `Pinhole`, its video as `VideoStream`, and a `frame_transforms` topic as `Transform3D` — only custom messages (e.g. a custom joint states schema, a custom gripper status enum) come through reflection or raw only and need lenses.\n\nThe `foxglove` decoder does the schema→archetype mapping; because foxglove messages are protobuf-_encoded_ it rides on the `protobuf` decoder, so keep `decoders=None` (verified: `decoders=[\"protobuf\"]` alone leaves `foxglove.CameraCalibration` a raw `:message`; adding `foxglove` makes it a `Pinhole`). Confirm on your file: `McapReader(path).stream()`, then read `McapSchema:name` and a few `Chunk.format()` before deciding anything is missing or needs rebuilding.\n\n- Entity path = topic name (`/sensors/joint_states` stays `/sensors/joint_states`).\n Filter early: `McapReader(path).stream().filter(content=\"/sensors/**\")`.\n- A reflection-decoded message lands as one struct component named `:message`.\n Navigate it with `Selector` (`Selector(\".joint_positions\")`) inside lenses; this is how custom messages\n get Rerun semantics attached (see the DeriveLens patterns in `rerun-chunk-processing`).\n- Topic regexes use RE2 syntax and are **not anchored**: `cam` matches\n `/external/cam_low` and `/camera_info`.\n Anchor explicitly (`^/external/cam`) when it matters. Prefer reader-level topic filtering over `.filter(...)`\n when you can, so excluded topics are never decoded at all.\n\n## When to use the low-level `mcap` package instead\n\n`McapReader` keeps payloads in columnar chunk streams; that is almost always what you want.\nDrop to `mcap.reader.make_reader` only when you need raw record metadata without payloads, or when you need to rewrite the container itself (re-registering schemas, channels, and messages).\n\n## Gotchas\n\n1. Empty stream, no error: a topic regex that matched nothing, or a channel\n whose decoder produced no rows. Check `Chunk.format()` on a few chunks of\n `reader.stream().to_chunks()` against a tiny test file, or compare topic\n names with the `mcap` CLI / package first.\n2. Topic regexes are unanchored RE2; excludes run after includes.\n3. `timeline_type=\"timestamp\"` interprets MCAP log times as wall-clock ns\n since epoch. If the recording's clock is wrong, fix it at the reader with\n `timestamp_offset_ns` rather than mutating timestamps downstream.\n4. Decoder subsets silently skip topics no decoder claims; when a topic is\n missing, retry with `decoders=None` to rule out decoder selection.\n5. Example fix-lenses are dataset-specific. Before copying a `MutateLens` like\n the `Pinhole:resolution` swap from the `robot_data_preprocessing` example,\n read the raw component from `McapReader(path).stream()` and confirm the defect\n exists in _your_ data — applied blindly it corrupts correct calibration (a\n correct 648×480 flipped to 480×648).\n6. `foxglove` derives both the camera's `Pinhole:child_frame` and the video's\n `CoordinateFrame:frame` from each message's `.frame_id` (plus an image-plane\n suffix), so they **match** when the calibration and video topics share a\n `frame_id`. Only when those topics carry _different_ `frame_id`s does the\n video frame diverge and orphan the video from its image plane — re-home it\n then with a per-camera `MutateLens` on `CoordinateFrame:frame`.\n\n## References\n\n- End-to-end MCAP pipeline: `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing`\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-urdf` (FK from joint-state topics), `rerun-data-model` (modeling decisions)\n\n\n---\nname: rerun-mp4\ndescription: Ingest .mp4 video into Rerun chunk streams with rerun.experimental.Mp4Reader. Read when converting video into a VideoStream, choosing stream vs asset mode, transcoding (B-frames, output codec, GOP size) through FFmpeg, or aligning video PTS onto a recording's wall-clock timeline. Builds on rerun-chunk-processing (stream mechanics) and rerun-data-model (where the video belongs in the recording).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun mp4 ingestion\n\n`Mp4Reader` turns one `.mp4` file into a lazy chunk stream on **one entity**:\ncompressed video samples, no decode to pixels, nothing re-encoded unless it has\nto be. Everything is configured on the constructor; `stream()` takes no\narguments. Stream mechanics after `.stream()` (filter, map, merge, collect,\nwrite) are in `rerun-chunk-processing`.\n\n## The API\n\n```python\nfrom rerun.experimental import Mp4Reader, Mp4TranscodeOptions\n\nreader = Mp4Reader(video_path, entity_path=\"/camera/front\") # mode=\"stream\" by default\nstream = reader.stream() # lazy: nothing is decoded yet\n```\n\nEvery parameter after `path` is keyword-only. One reader handles one file — for\nseveral cameras, build one reader per file and `LazyChunkStream.merge(...)` them\n(see below).\n\n## Two modes\n\n| `mode` | emits | when |\n| -------------------- | ------------------------------------------------------- | ----------------------------------------------------------------- |\n| `\"stream\"` (default) | a codec chunk, per-GOP sample chunks, a keyframe marker | almost always — every frame is time-indexed and queryable |\n| `\"asset\"` | the whole file as one blob, plus a frame index | the codec cannot be a `VideoStream`, or the source starts mid-GOP |\n\nReach for stream mode; the next section covers exactly what it emits. Asset mode\ncopies the entire file into the recording as one blob, so nothing downstream can\nlook at a single frame without the whole asset. It is the fallback for codecs a\n`VideoStream` cannot carry (`mp4v`, image-sequence mp4), and it is what `rerun\nvideo.mp4` and the built-in file importer use.\n\n## What stream mode emits\n\nFor `tests/assets/video/Big_Buck_Bunny_1080_1s_h264_nobframes.mp4` (30 frames,\none GOP) with `entity_path=\"/camera/front\"`, the whole output is three chunks:\n\n```\n[0] entity=/camera/front static=True rows=1 timelines=[] cols=['VideoStream:codec']\n[1] entity=/camera/front static=False rows=30 timelines=['video'] cols=['VideoStream:sample']\n[2] entity=/camera/front static=False rows=1 timelines=['video'] cols=['VideoStream:is_keyframe']\n```\n\n- **One static chunk, one row**, holding `VideoStream:codec`.\n It carries **no timeline** — code that walks the stream must\n handle that (`if chunk.is_static: continue`).\n- **One temporal chunk per GOP**: a keyframe plus every sample that depends on\n it, up to (not including) the next keyframe. Samples only — the keyframe flags\n are not a column here.\n- **One trailing `is_keyframe` marker chunk**, holding a *sparse* `True` row at\n each keyframe's time and nothing else. Keeping it out of the sample chunks is\n what lets a keyframe-only query skip the sample payload, and what\n `collect(optimize=…)` accepts as canonical. It is list-per-row, so a row reads\n back as `[True]`, not `True`.\n- **Timeline `video`**, `duration[ns]`, values are the mp4 PTS from the start of\n the video.\n- **Every chunk on the same entity.** `entity_path=None` (the default) derives\n it from the file's *absolute* path, so `foo/video.mp4` run from `/data`\n becomes `/data/foo/video.mp4`. Pass `entity_path` explicitly in any real\n pipeline.\n\nA short clip is often a single GOP, so \"three chunks total\" is the normal shape —\nnot a sign that something was dropped. `chunk_by_gop=False` emits one Rerun\nchunk per sample instead (the marker chunk is unaffected). That is a debugging\nshape — one chunk per frame is a poor storage layout, and it also makes a\nfollowing optimize pass **6× more expensive** (measured below) — so leave the\ndefault alone unless you are inspecting individual samples.\n\nDistinguish the three by their columns, not by position or `is_static`: the codec\nchunk is the static one, sample chunks carry `VideoStream:sample`, and the marker\nchunk carries only `VideoStream:is_keyframe`.\n\nSupported stream-mode codecs are exactly the five `VideoCodec` values: H264,\nH265, AV1, VP8, VP9.\n\n## Transcoding through FFmpeg\n\n`VideoStream` cannot yet model DTS != PTS, so an H.264/H.265 source with\ncontainer-level B-frame reordering **cannot be emitted directly**. The reader\nhandles that itself: it re-encodes through FFmpeg with `-bf 0`, streams the\nresult back as a fragmented mp4, and turns each fragment into one GOP chunk, so\nonly one GOP is resident at a time. This is automatic and invisible in the API —\nthe output has the same shape as a clean source. It does need an `ffmpeg`\nexecutable. Asset mode is unaffected by B-frames.\n\n`Mp4TranscodeOptions` (stream mode only) additionally *requests* a transcode:\n\n| field | effect |\n| ----------------- | -------------------------------------------------------------------------- |\n| `output_codec` | re-encode to another `VideoCodec`; the emitted `VideoStream:codec` follows |\n| `gop_size` | force a keyframe every N frames — the knob for seek cost in the viewer |\n| `try_gpu` | best-effort hardware encode, NVENC / VideoToolbox only |\n| `ffmpeg_override` | use this `ffmpeg` instead of the one on `PATH` |\n\nRequesting the `output_codec` the source already uses is a **no-op**: it stays on\nthe direct, no-FFmpeg path. With `chunk_by_gop=True`, `gop_size=N` makes every\nGOP chunk but the last hold exactly N samples — a 30-frame clip at `gop_size=10`\ngives 10, 10, 10. `try_gpu` realistically covers H264/H265 plus AV1 on newer\nNVIDIA; VP8/VP9 always fall back to software, and it does nothing unless a\ntranscode is already happening.\n\n```python\nMp4Reader(\n video_path,\n entity_path=\"/camera/front\",\n # ~1s GOPs on a 60fps source, so seeking in the viewer stays snappy.\n transcode=Mp4TranscodeOptions(gop_size=64),\n).stream()\n```\n\n## Timelines: mp4 PTS is not your recording's clock\n\nThe emitted times are always PTS — elapsed nanoseconds from the start of *that\nvideo*. In a multi-sensor recording, that is almost never the timeline you want\nto align on.\n\n- `timeline_name=\"real_time\"` renames the timeline.\n- `timeline_type=\"timestamp\"` only **retypes** the same PTS values as\n nanoseconds since the Unix epoch. The reader does not shift them, so on its\n own it renders the video near 1970. It is meaningful only paired with a retag\n step.\n- **Retag with `stream.map(...)`.** Samples arrive in presentation order\n (B-frames are already stripped), so a running cursor maps sample `i` to\n `capture_times_ns[i]` — but the cursor must skip the keyframe marker chunk,\n which holds no samples. Map that one through the PTS the samples were already\n assigned:\n\n```python\nSAMPLE_COL = \"VideoStream:sample\"\n\n\ndef _reindex_to_capture_times(stream, capture_times_ns, timeline_name):\n cursor = 0\n pts_to_time = {}\n\n def _retag(chunk):\n nonlocal cursor\n if chunk.is_static: # the codec chunk carries no timeline\n return chunk\n batch = chunk.to_record_batch()\n col_index = batch.schema.get_field_index(timeline_name)\n old_field = batch.schema.field(col_index)\n old_pts = np.asarray(batch.column(col_index).cast(pa.int64()))\n\n if SAMPLE_COL in batch.schema.names:\n # Clamp in case the decoder yields a slightly different frame count.\n indices = np.clip(np.arange(cursor, cursor + chunk.num_rows), 0, len(capture_times_ns) - 1)\n cursor += chunk.num_rows\n new_times_ns = capture_times_ns[indices]\n pts_to_time.update(zip(old_pts.tolist(), new_times_ns.tolist()))\n else:\n # The sparse keyframe marker, emitted after every sample chunk.\n new_times_ns = np.array([pts_to_time[pts] for pts in old_pts.tolist()], dtype=np.int64)\n\n times = pa.array(new_times_ns.astype(\"datetime64[ns]\"))\n # `metadata=` is load-bearing — see gotcha 3.\n new_field = pa.field(old_field.name, times.type, nullable=old_field.nullable, metadata=old_field.metadata)\n return Chunk.from_record_batch(batch.set_column(col_index, new_field, times))[0]\n\n return stream.map(_retag)\n```\n\nThe cursor makes this order-dependent, so keep the retag on the single reader's\nstream, before any merge. This is the DROID loader's pattern — see the\nreferences.\n\nAny `map`/`flat_map` that assumes \"every non-static chunk is samples, in order\"\nhas this same bug: it advances over the marker chunk and stamps it with whatever\ntime comes next, silently moving the keyframe markers off their samples.\n\n## Several cameras into one recording\n\nOne reader per file, distinct entity paths, then merge:\n\n```python\nstreams = [\n Mp4Reader(path, entity_path=f\"/camera/{name}\", timeline_name=\"real_time\", timeline_type=\"timestamp\").stream()\n for name, path in cameras.items()\n]\n(\n LazyChunkStream\n .merge(*streams)\n .collect(optimize=OptimizationProfile.OBJECT_STORE)\n .write_rrd(out_path, application_id=app_id, recording_id=segment_id)\n)\n```\n\n`OBJECT_STORE` GOP-rebatches the video and preserves the reader's keyframe\nmarker as-is; no `fix_keyframe=True` is needed. If you *do* see\n`skipping GoP rebatching … is_keyframe data is incorrect`, something upstream\nrewrote the marker (see gotcha 4) — `fix_keyframe=True` re-derives it from the\nencoded samples as an escape hatch.\n\n## How the reader's chunks relate to optimize's\n\nThe reader and `collect(optimize=…)` agree on **where GOPs start** but not on\n**how many GOPs share a chunk**, and that is by design:\n\n- The reader emits the **finest GOP-aligned partition**: exactly one chunk per\n GOP. It cannot do better, because it does not know which profile the data is\n headed for.\n- Optimize then applies the size policy, **merging consecutive GOPs** up to the\n profile's `max_bytes` / `max_rows`. It never splits a GOP across chunks, so\n every boundary it keeps is one the reader already produced.\n\nSo optimize's partition is a pure *coarsening* of the reader's: same samples in\nthe same order, every optimized chunk a run of whole reader GOPs, keyframe marker\nuntouched. Where each GOP already sits near the profile's budget the two come out\nidentical (a 30-frame clip at `gop_size=10` is 3 chunks either way). Where GOPs\nare small they diverge sharply — a 12-GOP clip becomes **1** chunk under\n`OBJECT_STORE` and **4** under `LIVE`.\n\nThe practical consequence: **do not pre-merge or re-chunk the reader's output**\nto \"help\" optimize. Hand it the per-GOP stream and let the profile decide. And if\nyou skip optimize entirely (writing straight through `send_chunks`, as the DROID\nloader does), you are storing the finest partition — correct, but more chunks\nthan object storage wants.\n\n### The optimize pass is cheap on GOP-chunked input\n\nRunning optimize over the reader's output is not redundant work worth avoiding.\nOn an 89 MB / 1800-frame / 60-GOP H.264 file:\n\n| | `collect()` | `collect(optimize=OBJECT_STORE)` | delta |\n| ----------------------------- | ----------- | -------------------------------- | ------------------- |\n| `chunk_by_gop=True` (default) | 11.1 ms | 16.3 ms | **+5.2 ms** (1.5×) |\n| `chunk_by_gop=False` | 14.2 ms | 48.6 ms | **+34.4 ms** (3.4×) |\n\nNeither half of the pass is expensive on GOP-aligned input:\n\n- **Detecting GOP starts is header-only.** `build_sample_index` runs\n `detect_gop_start` on every sample, but that reads a few bytes of each sample's\n header — it never decodes. The whole optimize delta above (5.2 ms) is smaller\n than a Python loop calling the same detector over the same 1800 samples\n (6.0 ms), which is mostly FFI overhead.\n- **Rebuilding the chunks copies nothing.** `chunk_from_gop` calls\n `taken(0..n)`, and `re_arrow_util::take_array` returns the array **uncopied**\n when the indices are consecutive from zero over the whole array. One chunk per\n GOP means that is always the case, so the multi-MB sample buffers are never\n duplicated — hence 89 MB \"rebuilt\" in 5 ms.\n\nThat fast path is exactly what `chunk_by_gop=False` gives up: each GOP then spans\n30 source chunks, so `chunk_from_gop` has to `concat_and_sort` them and the sample\nbytes really are copied. `LIVE` also costs more than `OBJECT_STORE` (+13.2 ms vs\n+5.2 ms on the same file), because its much smaller chunk budget makes it attempt\ncompaction it cannot complete — a GOP is never split.\n\nRetag each camera *before* the merge — the retag above walks chunks with a\ncursor, so it must see one video's chunks in order.\n\n## Gotchas\n\n1. **Errors surface on the first pull, not at construction.** The constructor\n only checks that the file exists and validates its arguments; `stream()`\n builds a lazy pipeline and also succeeds. Codec support, keyframe layout, and\n FFmpeg availability are checked when the stream runs, so wrap the\n *consumption* (`to_chunks()`, `send_chunks`, iteration), not the\n `Mp4Reader(…)` call.\n2. **A codec outside the five `VideoCodec` values cannot be a `VideoStream`.**\n `mp4v` (MPEG-4 Part 2) is the one you will actually meet in robot datasets;\n it raises `RuntimeError: MP4 error: MP4 demux: Video track uses unsupported\ncodec \"mp4v\"`. Asset mode does accept the file, but only partly: the blob\n chunk is emitted and the `VideoFrameReference` index chunk is **skipped with\n a warning**, because the frame timestamps cannot be read either — so you get\n the bytes and no timeline. Skipping that camera and recording the fact as a\n recording property (what the DROID loader does) is usually better than a\n timeline-less blob, and either beats failing a whole episode over one camera.\n3. **A `map`/`flat_map` that rewrites the time column must carry the field\n metadata over.** `pa.field(...)` without `metadata=old_field.metadata` drops\n `rerun:kind: 'index'`, and the rebuilt chunk silently becomes **static** —\n no error, no timeline, and the samples land outside time entirely.\n4. **The trailing marker chunk is temporal but holds no samples**, so any\n per-chunk logic keyed on `not chunk.is_static` will process it as if it were\n samples. Key on the `VideoStream:sample` column instead. See the retag above.\n5. **Asset mode is capped at ~2 GiB** by Arrow's i32 offsets, and duplicates the\n file's bytes into the RRD.\n6. `mode=\"asset\"` rejects both `chunk_by_gop=False` and `transcode=` with\n `ValueError` — those are stream-mode-only knobs.\n7. `timeline_type=\"timestamp\"` on its own shifts nothing. Without a retag, the\n video sits at the epoch.\n8. **The reader emits compressed samples, never pixels.** Thumbnails, CLIP\n embeddings, or anything else needing RGB have to decode the file separately\n (OpenCV, PyAV); the `VideoStream` chunks cannot supply them.\n9. Every `stream()` call re-decodes the file from scratch, and every terminal\n call re-runs the pipeline. `collect()` once when you need more than one pass.\n10. Handle the static codec chunk explicitly in any `map`/`flat_map` — it has no\n timeline and no samples, and blind indexing into a time column will fail on\n it.\n11. **Samples before the first keyframe** are rejected in stream mode (a decoder\n cannot start mid-GOP); use asset mode for such a file.\n\n## References\n\n- Canonical worked examples: `rerun_py/tests/integration/test_mp4_reader.py`\n (both modes, `chunk_by_gop`, entity paths, `timeline_type`, transcode\n transforms, and the error cases).\n- Rust core: `crates/data_flow/re_mp4_reader/` (`stream.rs` for the GOP/transcode\n path, `asset.rs` for the blob+index path), with\n `crates/data_flow/re_mp4_reader/tests/stream.rs` covering codec pairs and GOP\n spacing.\n- `rerun-chunk-processing` (stream/lens mechanics), `rerun-data-model` (where\n video, calibration, and thumbnails belong in the recording).\n\n\n---\nname: rerun-parquet\ndescription: Ingest tabular Parquet files into Rerun chunk streams with rerun.experimental.ParquetReader. Read when converting trajectory or sensor tables (LeRobot-style parquet, exported logs) into entities and components — column grouping, timeline/index columns, static columns, and lenses (DeriveLens) that assemble the typed components (Transform3D, Scalars) from the reader's grouped struct/scalar output. Builds on rerun-chunk-processing and rerun-data-model.\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun parquet ingestion\n\n`ParquetReader` is a **pure reader**: it maps a flat table onto the Rerun\nmodel by turning raw columns into grouped, time-indexed chunks of struct and\nscalar components. Column-name prefixes become entities, grouped columns\nbecome a single struct component, designated columns become timelines. The\nreader does **not** assemble archetypes anymore — mapping struct fields into\ntyped Rerun components (Transform3D, Scalars, Points3D) is done with lenses on\nthe reader's `.stream()`. The whole reader job is configuration; fill in the\n`rerun-data-model` mapping table first, then express it through `stream()`.\nStream mechanics after `.stream()` are in `rerun-chunk-processing`.\n\n**The whole table is configuration, not code.** If you find yourself building\n`Chunk.from_columns` from a parquet, or munging it in pandas first, stop —\n`ParquetReader` plus a lens almost certainly expresses it. Anything the reader\ncannot express (per-row entity routing, derived values, unit conversion)\nbelongs in lenses downstream, not in pre-pandas munging; keep the pipeline\ncolumnar.\n\n## The API\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nreader = ParquetReader(table_path) # a lightweight handle over the file\nstream = reader.stream(\n entity_path_prefix=\"/world\", # prepended to every entity path\n column_grouping=\"prefix\", # \"prefix\" | \"individual\" | \"explicit_prefixes\"\n delimiter=\"_\", # split for column_grouping=\"prefix\"\n prefixes=None, # required for \"explicit_prefixes\"\n use_structs=True, # pack grouped columns into one struct component\n static_columns=[\"robot_type\"], # constant-per-file values, logged static\n index_columns=[IndexColumn.timestamp(\"timestamp\", input_unit=\"us\"), IndexColumn.sequence(\"frame_index\")],\n)\n```\n\nThe constructor takes only `path`; all loading options live on `stream()`, and every one is keyword-only.\nEach `stream()` call is independent, so one reader can drive several differently-configured streams over the same file.\nThe config is validated against the file's schema at `stream()` (missing index/static columns raise `ValueError` there, not mid-iteration).\nThere is no `column_rules` kwarg — typed-component assembly moved to lenses (below).\n\n## What the reader emits\n\nThe reader turns the table into chunks, one chunk per group, then leaves the\ndata as generic struct/scalar components for lenses to map. The naming is the\nkey thing the rest of the pipeline keys off:\n\n- **A grouped multi-column prefix `X`** → entity `/X`, with a single struct\n component named **`data`**. The struct's fields are the column names with the\n prefix (and delimiter, for `\"prefix\"`) stripped. So `A_pos_x`, `A_quat_w`\n under prefix `A` land as struct `data` with fields `pos_x`, `quat_w` on\n entity `/A`.\n- **A lone column with no group** → its own entity named after the column, and\n a raw component named after the column — *not* a `data` struct. So a `speed`\n column becomes entity `/speed`, component `speed`.\n- **A `/__properties` metadata chunk** built from the parquet file's schema\n metadata. You typically drop it right after `.stream()`:\n\n ```python\n stream = reader.stream().drop(content=\"/__properties/**\")\n ```\n\n## Column grouping: which columns share an entity\n\n- `\"prefix\"` (default): split each column name on `delimiter`, group by the\n first segment. `gripper_pos_x`, `gripper_pos_y` → entity `/gripper`, struct\n `data{pos_x, pos_y}`.\n- `\"explicit_prefixes\"`: group by the exact strings in `prefixes`, tried\n longest-first; the prefix is stripped from each struct field name (a raw\n string match, no delimiter — `foo` + `a` → field `a`). Columns matching no\n prefix become individual groups. Use this when names contain the delimiter\n ambiguously (`observation.state` vs `observation.images.top`: pass the full\n prefixes).\n- `\"individual\"`: every column is its own chunk/entity with a raw component\n named after the column — no struct packing at all, even for columns sharing a\n prefix. `use_structs` is ignored here. Rarely the model you want; reach for\n it only as a debugging baseline.\n\n`use_structs=True` (default) packs a group's columns into a single Arrow\nstruct component (the `data` field) for `\"prefix\"`/`\"explicit_prefixes\"`;\n`False` emits one component per column (the pre-struct flat layout, what\nqueries see as separate columns).\n\n## Timelines: `index_columns`\n\nEach entry is built with `IndexColumn`: `IndexColumn.timestamp(name, input_unit=...)` (since epoch), `IndexColumn.duration(name, input_unit=...)` (elapsed), or `IndexColumn.sequence(name)` (ordinal int).\n\n- `input_unit` describes what the raw integers in the column *are* (`\"ns\"` default,\n `\"us\"`, `\"ms\"`, `\"s\"`); Rerun rescales to ns internally. Sequences take no unit.\n\n**If omitted, a synthetic `row_index` sequence timeline is generated.** That\nis almost never the timeline you want to query or align against; always name\nthe real time columns. Stamp both a timestamp and a sequence timeline when the\ntable has both (multi-rate alignment, see `rerun-data-model`).\n\n## Static columns: `static_columns`\n\nListed columns are constant across all rows; they are emitted once as a single\nstatic (timeless) chunk, separate from the temporal data. A listed column that\nactually varies raises an error when the stream runs — that error is a\ndata-quality signal, not a reason to drop the static declaration.\n\n## Typed components via lenses\n\nThe reader's grouped output is generic struct (`data`) and scalar data. A\n`DeriveLens` reads that struct's fields, packs and casts them into real Rerun\ncomponents, and writes them to an output entity — this is what the old\n`column_rules` API used to do, now done downstream on the stream.\n\nConstruct a lens against the reader's struct component (`\"data\"` for grouped\nprefixes, or the column name for a lone/individual column), then add one or\nmore `.to_*` builder methods. Each builder returns a fresh lens, so they chain.\n\n| Builder | Produces | Argument order |\n|---|---|---|\n| `to_translation(x, y, z)` | `Transform3D:translation` | x, y, z |\n| `to_quaternion(x, y, z, w)` | `Transform3D:quaternion` | x, y, z, w (xyzw) |\n| `to_scale(x, y, z)` | `Transform3D:scale` | x, y, z |\n| `to_rotation_axis_angle(axis_x, axis_y, axis_z, angle)` | `Transform3D:rotation_axis_angle` | axis_x, axis_y, axis_z, angle (radians) |\n| `to_scalars(*fields)` | `Scalars:scalars` | one or more field names |\n| `to_packed_component(component, *fields)` | the given component | descriptor, then field names |\n| `to_component(component, selector)` | the given component | descriptor, then a `Selector` |\n| `to_timeline(name, type, selector)` | a timeline (not a component) | name, `\"sequence\"`/`\"duration_ns\"`/`\"timestamp_ns\"`, selector |\n\n`to_packed_component` packs the named struct fields (in order, at least one\nrequired) into the fixed-size list the component expects, and by default\n**auto-casts `f64`→`f32`** to match component types. The `to_translation`,\n`to_quaternion`, `to_scale` helpers are convenience wrappers over it, so they\nall auto-cast. `to_rotation_axis_angle` builds a `Struct{axis, angle}` and\nhard-casts axis and angle to `f32` internally. `to_scalars` with a single field\nemits a plain scalar per row (not a 1-element list); with multiple fields it\nemits one scalar series per field at the same entity.\n\nApply lenses with `.stream().lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")`:\n\n- `content` is a pre-filter on the *source* entity path — it scopes which\n chunks the lens may touch. Out-of-scope chunks pass through unchanged. Set it\n to the reader's grouped entity (e.g. `\"/A\"`).\n- `output_mode` decides the fate of in-scope-but-unmatched chunks:\n `\"drop_unmatched\"` (default, keep only lens output), `\"forward_unmatched\"`\n (output replaces matched, other originals survive), or `\"forward_all\"`\n (output plus all originals).\n- The lens's own `output_entity=` sets the *destination* entity — independent\n of `content`, which gates the input side.\n\nEnd-to-end Transform3D example. The reader groups `A_*` columns into a `data`\nstruct at `/A`; the lens reads the prefix-stripped field names (`pos_x`,\n`quat_w`), packs and casts them, and writes a full `Transform3D` to `/pose`:\n\n```python\nfrom rerun.chunk import DeriveLens, IndexColumn\nfrom rerun.experimental import ParquetReader\n\nlens = (\n DeriveLens(\"data\", output_entity=\"/pose\")\n .to_translation(\"pos_x\", \"pos_y\", \"pos_z\")\n .to_quaternion(\"quat_x\", \"quat_y\", \"quat_z\", \"quat_w\")\n)\n\nchunks = (\n ParquetReader(table_path)\n .stream(index_columns=[IndexColumn.sequence(\"frame_index\")])\n .lenses([lens], content=\"/A\", output_mode=\"drop_unmatched\")\n .to_chunks()\n)\n```\n\nChaining several `.to_*` on **one lens with a shared `output_entity`**\naccumulates multiple component columns into the same archetype at that entity —\nabove, both `Transform3D:translation` and `Transform3D:quaternion` land on\n`/pose`, forming a complete `Transform3D`. For a generic fixed-size-list\ncomponent, pass the descriptor to `to_packed_component`:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens\nfrom rerun.experimental import ParquetReader\n\nlens = DeriveLens(\"data\", output_entity=\"/points\").to_packed_component(\n rr.Points3D.descriptor_positions(), \"x\", \"y\", \"z\"\n)\n```\n\n## Selectors\n\nLens field paths use `Selector`, a jq-like grammar over Arrow columns\n(`.field` to access a struct field, `[]` to iterate a list, `[N]` to index, `?`\nto suppress errors on absent fields, `!` to assert non-null, `|` to pipe, and\n`pack(.x, .y, .z)` to zip paths into a fixed-size list). The `to_*` helpers\nbuild these selectors for you; reach for `to_component(component, Selector(\".x\"))`\nwhen you need a custom field path. Field paths reference the\n**prefix-stripped** struct field names — the lens sees `pos_x`, not `A_pos_x`.\n\n## Gotchas\n\n1. No `index_columns` → synthetic `row_index` timeline only. Queries that\n expect a timestamp timeline find nothing.\n2. The `unit` is the raw column's unit, not a desired output unit; a\n microsecond column declared `\"ns\"` lands 1000x in the past.\n3. `static_columns` raises if a listed column actually varies; that error is a\n data-quality signal, not a reason to drop the static declaration.\n Uniformity is data-dependent, so it is raised lazily when the stream runs;\n a listed column that doesn't *exist* raises eagerly at `stream()`.\n4. A grouped prefix's struct component is named **`data`** — that is the\n `input_component` string a `DeriveLens` matches against. A lone or\n `\"individual\"` column is instead a raw component named after the column.\n5. Selector field paths reference the **prefix-stripped** struct field names\n (`pos_x`, not `gripper_pos_x`).\n6. Drop the `/__properties` metadata chunk the reader emits from parquet schema\n metadata: `.stream().drop(content=\"/__properties/**\")`.\n7. Quaternion column order is x, y, z, w in `to_quaternion`; check the source's\n convention before wiring fields.\n8. `to_packed_component` (and the transform helpers built on it) auto-casts\n `f64`→`f32` to match component types; this is usually what you want for\n parquet's double columns.\n9. Anything the reader cannot express (per-row entity routing, derived values,\n unit conversion) belongs in lenses downstream, not in pre-pandas munging;\n keep the pipeline columnar (`rerun-chunk-processing`).\n\n## References\n\n- Lens builder source with full docstrings: `rerun/chunk/_lens.py` in\n the installed `rerun-sdk` package (`to_translation`, `to_quaternion`,\n `to_scale`, `to_rotation_axis_angle`, `to_scalars`, `to_packed_component`,\n `to_component`, `to_timeline`).\n- Reader source: `rerun/experimental/_parquet_reader.py`, or\n `python -c \"from rerun.experimental import ParquetReader; help(ParquetReader)\"`\n- Canonical worked examples: the integration tests\n `rerun_py/tests/integration/test_parquet_reader.py` (grouping, index/static\n columns, and the Transform3D / Points3D / Scalars lens flows) and\n `rerun_py/tests/integration/test_lazy_chunk_stream.py` (lens application,\n `content`/`output_mode`, selectors).\n- `rerun-lerobot` — LeRobot datasets store episodes as parquet; that skill\n covers the built-in importer route vs reading the parquet directly with\n this reader.\n- `rerun-data-model` (mapping decisions), `rerun-chunk-processing` (stream\n mechanics after `.stream()`)\n\n\n---\nname: rerun-urdf\ndescription: Drive the Rerun URDF API (rerun.urdf.UrdfTree) to ingest a URDF as a Transform3D layer on a robot recording. Read when logging a robot model, running forward kinematics from joint states, composing a fixed chain for sensor extrinsics, or when the transform tree will not connect from the data alone. Builds on rerun-chunk-processing (stream/lens mechanics) and rerun-data-model (entity paths, timeline, base-vs-layer).\nuser_invocable: true\nallowed-tools: Read, Grep, Bash, WebFetch\n---\n\n# Rerun URDF ingestion\n\nA URDF gives you a robot's geometry and its kinematic tree.\nIngesting it means two API calls on one `rerun.urdf.UrdfTree`: stream the static model, then drive it with forward kinematics from joint states. The transforms it produces are derived, so they are a **layer**, never base (the `URDF + joints (computed)` row of the `rerun-data-model` table).\n\nThis skill is the `UrdfTree` API and the two judgment calls it cannot make for\nyou: how your joint values map to URDF joints, and how the disconnected frames\nin the scene connect to one root. The stream/lens plumbing (`LazyChunkStream`,\n`DeriveLens`, `Selector`, writing and optimizing RRDs) is in\n`rerun-chunk-processing`; reach for it, do not re-derive it. Nothing below is\ntied to a data format: where your joint names, joint values, and calibration\ncome from is yours to wire in.\n\n## Getting the URDF\n\nDatasets rarely ship one, and the robot's own repo is not always the fastest route.\nIn order:\n\n- [`robot_descriptions.py`](https://github.com/robot-descriptions/robot_descriptions.py) — `pip install robot_descriptions`, then `from robot_descriptions import ur5e_description; ur5e_description.URDF_PATH`.\n It downloads and caches the upstream description and hands you a plain path with the meshes already next to it.\n Check its index first; it covers most of the table below.\n- The vendor repo, when the description is newer than the package's pin or is not in it.\n- [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie) — MJCF, not URDF, so it needs conversion. Worth it only when nothing else exists.\n\n| Robot | Source | Notes |\n| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |\n| SO-100 / SO-101 (LeRobot) | [TheRobotStudio/SO-ARM100](https://github.com/TheRobotStudio/SO-ARM100) | `Simulation/SO100/so100.urdf`, `Simulation/SO101/so101_new_calib.urdf`; plain URDF, relative mesh paths |\n| ALOHA (ViperX 300, WidowX) | [Interbotix/interbotix_ros_manipulators](https://github.com/Interbotix/interbotix_ros_manipulators) | xacro |\n| Franka Panda / FR3 | [frankaemika/franka_description](https://github.com/frankaemika/franka_description) | xacro |\n| Universal Robots UR3 to UR20 | [UniversalRobots/Universal_Robots_ROS2_Description](https://github.com/UniversalRobots/Universal_Robots_ROS2_Description) | xacro, arm parameters in YAML |\n| Kinova Gen3 | [Kinovarobotics/ros2_kortex](https://github.com/Kinovarobotics/ros2_kortex) | xacro |\n| UFactory xArm / Lite 6 | [xArm-Developer/xarm_ros2](https://github.com/xArm-Developer/xarm_ros2) | xacro |\n| Unitree Z1, G1, H1, Go2 | [unitreerobotics/unitree_ros](https://github.com/unitreerobotics/unitree_ros) | plain URDF per robot |\n\nTwo things bite before FK ever runs:\n\n- **xacro is not URDF.** First run `xacro arm.urdf.xacro > arm.urdf` (`pip install xacro`), passing whatever arguments the file requires (gripper variant, arm size).\n A xacro file handed to `UrdfTree.from_file_path` fails to parse.\n- **`package://` mesh URIs.** The importer resolves them by scanning `ROS_PACKAGE_PATH` (ROS 1) and `AMENT_PREFIX_PATH` (ROS 2) for the package, and otherwise treats the rest of the URI as relative to the URDF's own directory.\n Outside a sourced ROS workspace, either export one of those variables or lay the meshes out relative to the URDF.\n Meshes that fail to resolve give you a kinematically correct but invisible robot.\n\n## The API\n\n```python\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(\n urdf_path,\n entity_path_prefix=\"robot\", # links log under /robot/)`. The `recording_id` must equal the\nbase segment id or the layer never attaches; `application_id` is discarded on\nregistration.\n\n## Mapping joint state to URDF joints (you supply this; the data will not)\n\n`compute_joint_transform_batches` is only as right as the `names`/`values` you\nhand it, and the mapping is not in the URDF. Three things go wrong silently:\n\n- **Order.** Build an explicit `names` array aligned to the `values` you read.\n Never assume your message's field order matches the URDF's `` order.\n- **Count.** The URDF's non-`fixed` joint count rarely equals your reported value\n count. A gripper sent as one value is often two prismatic joints in the URDF;\n a mimic joint may be omitted from the message. Reconcile explicitly, and use\n the API to do it: iterate `urdf.joints()`, partition by `joint_type` and\n `mimic`. A joint with `mimic` set derives its value from the driver joint as\n `driver * multiplier + offset`; feed it that, not a message field. Confirm\n the count against `urdf.joints()`, not the message length.\n- **Units.** URDF joints are radians (revolute) and meters (prismatic). Convert\n if your source differs.\n\nGet any of these wrong and FK runs and writes a confident, wrong pose.\n\n### Verify joint mapping and limits\n\nJoint values in the source data aren't guaranteed to be within URDF joint limits.\nThis can happen with any format (MCAP, Parquet, etc).\nIt is therefore useful to check whether joint values in the data match the URDF model's limits.\nOut-of-range values can indicate issues like wrong indexing or model mismatch and should be clarified before calculating the forward kinematics.\n\nTo get limits of a URDF model:\n\n```python\nlimits = {joint.name: (joint.limit_lower, joint.limit_upper) for joint in urdf.joints() if joint.joint_type != \"fixed\"}\n# then compare each column's min/max against its joint's range\n```\n\n`scripts/check_joint_mapping.py` is that check worked out for one source shape — a LeRobot-style parquet column of per-frame joint values.\nRead it as the example to copy and adapt for your own format, and run it directly if your data is already a LeRobot-style parquet file:\n\n```bash\nuv run scripts/check_joint_mapping.py --column observation.state --degrees\n```\n\n**Important:** this is a validation meant to be done before full processing — don't run this for _every_ file in a larger dataset.\nSelect a single representative file or a small subset for this verification.\n\nFor each joint it tries the usual candidate corrections (identity, negate, `% 360`, ±90°/±180° offsets, and their combinations) and reports which ones land inside that joint's limits.\nIt reports the fit, not the answer: a joint with two surviving candidates is **ambiguous**, and no amount of range-checking will separate them.\nValidate ambiguous mappings against known poses or geometry for the dataset.\n\n**Read the failures by shape.**\nA whole column outside its range means a wrong sign, unit, or offset — a bug, and the pose will be confidently wrong.\nA few frames a few degrees past a limit is calibration slack — expected, and `clamp=True` is the fix, not a workaround.\n`clamp=True` warns once per offending row, so run it while working out the mapping: a flood of warnings is a mapping smell, and a thin trickle at the extremes of one joint is a healthy recording.\n\n## Make the joint states readable first\n\nWhere the joint values come from is not this skill's problem, but a dead joint\nsource produces an empty FK layer **with no error**: a source path that\nmatched nothing, a decoder that yielded zero rows, or a reader that dropped the\nmessage silently. Whatever the source, confirm the joint-state stream yields\nrows before debugging FK. The importer skill for your source format covers its\nown empty-stream failure modes.\n\n## How transforms compose (reason about this before logging anything)\n\nRerun resolves a pose by chaining transforms from a frame up to a root. There\nare two ways an edge in that chain gets defined, and a URDF ingest mixes both:\n\n- **By entity-path hierarchy.** A `Transform3D` on `/a/b` with no frame names is\n the transform of `/a/b` relative to its parent path `/a`. Composition follows\n the entity tree.\n- **By explicit frame graph.** A `Transform3D` that carries `parent_frame` and\n `child_frame` defines an edge between two **named frames**, independent of\n where in the entity tree it is logged. URDF FK uses this: every joint\n transform names its parent and child link frames.\n\nSo a URDF ingest is a graph of named frames. An edge exists only if some\n`Transform3D` names that exact `parent_frame -> child_frame` pair. A frame with\nno incoming edge is a root. The viewer renders every root at the world origin,\nwhich is why two unconnected robots silently overlap instead of erroring.\n\n## Resolving the transform forest (the part the data cannot always give you)\n\nA URDF is **one tree rooted at its base link**. FK and fixed joints supply every\nedge inside that tree. A real scene is a **forest** of roots the URDF never\nconnects: a world or scene frame, each robot's base, every camera or sensor\nframe. The edges that join those roots come from calibration (extrinsics in a\nsidecar, a TF static publisher, a hand-eye result), not from the URDF, and some\nare simply absent. A single connected tree is not always solvable. Resolve it\ndeliberately:\n\n1. **Enumerate every frame.** Iterate `urdf.joints()` and collect the\n `parent_link`/`child_link` pairs (the in-tree edges, frame-prefixed);\n `urdf.root_link()` is that URDF's root. Add every sensor/world frame the\n scene needs (from the `rerun-data-model` table). Decide the one intended\n root.\n2. **Classify each edge by source.** In-URDF edges (FK joints, `fixed` joints)\n come from the URDF plus joint values. Inter-root edges (root to each robot\n base, root to each fixed sensor, an arm link to a wrist-mounted camera) come\n from calibration and you must log them yourself.\n3. **Compose fixed chains from the URDF** when you need the transform between two\n links joined only by `fixed` joints (a camera bracket, a tool mount): walk\n parent links across `fixed` joints via `urdf.joints()`, turning each joint's\n `origin_xyz`/`origin_rpy` into a homogeneous matrix and multiplying along\n the chain. If the walk cannot reach\n the target link, **stop and say so** (\"no fixed chain from A to B; stuck at\n C\"). A broken chain is a wrong pose, not a missing one.\n4. **Build the edge set and find the roots.** Collect every `parent_frame ->\nchild_frame` pair you will log (URDF + calibration). Walk parents from each\n frame; any frame that does not reach the intended root is an unconnected root\n and names a **missing edge**. This is a pure graph check you can run before\n writing the RRD.\n5. **Resolve every missing edge, or fail loudly.** For each one:\n - If calibration supplies the transform, log it (next section).\n - If the data does not, the tree is unsolvable. Do **not** leave the frame\n disconnected (it collapses onto the origin and reads as one merged scene).\n Either abort and name the missing edge, or log identity and emit a loud\n warning naming the assumed edge. State which you did.\n6. **Match frame names across sources.** FK-derived `parent_frame`/`child_frame`\n must equal the names `urdf.stream()` emits for the static geometry, and your\n calibration edges must use those same names, or links float off the mesh. The\n `frame_prefix` is what keeps them identical; reuse it everywhere.\n\nA correct ingest has exactly one root, and a path from every frame to it.\n\n## Logging a connection correctly\n\nA connecting edge is a **static** `Transform3D` carrying the bridging frame\nnames. Static (no time index) so it holds for the whole recording; the frame\nnames, not the entity path, are what create the graph edge. This\n`Chunk.from_columns` is the rare **sidecar exception** — a calibration transform\nno reader or FK lens can produce; do not generalize it to transforms a reader\nemits (a `frame_transforms` topic → `Transform3D`) or that FK derives:\n\n```python\nimport rerun as rr\nfrom rerun.chunk import Chunk, LazyChunkStream\n\n# world -> this robot's base, from your calibration (translation + xyzw quaternion)\nedge = Chunk.from_columns(\n \"/world/robot_base\", # any sensible bridging entity path\n indexes=[], # no index == static\n columns=rr.Transform3D.columns(\n translation=[translation],\n quaternion=[quaternion_xyzw],\n parent_frame=[\"world\"], # must match the root frame name\n child_frame=[\"arm_base_link\"], # must match the URDF root frame (prefixed)\n ),\n)\nedges = LazyChunkStream.from_iter([edge]) # merge alongside model + FK streams\n```\n\nLog a fixed-chain result (step 3) the same way, with `parent_frame` and\n`child_frame` set to the two link frames the chain spans. Merge all edge chunks\ninto the same recording as the model and FK streams so they share the graph.\n\n## Minimal shape (one robot, generic joint source)\n\n```python\nimport rerun as rr\nfrom rerun.chunk import DeriveLens, LazyChunkStream, OptimizationProfile, Selector\nfrom rerun.urdf import UrdfTree\n\nurdf = UrdfTree.from_file_path(urdf_path, entity_path_prefix=\"robot\", static_transform_entity_path=\"robot/tf_static\")\nmodel = urdf.stream(include_joint_transforms=True).drop(content=\"/robot/**/collision_geometries/**\")\n\njoints = source_joint_state_stream() # your reader; one message column of names+values\nfk = (\n joints\n .lenses(\n DeriveLens(JOINT_MSG_COMPONENT, output_entity=\"/tmp/batches\").to_component(\n \"rerun.urdf.JointTransformBatch\",\n Selector(\".\").pipe(lambda msgs: urdf.compute_joint_transform_batches(read_names(msgs), read_values(msgs))),\n ),\n content=JOINT_SOURCE_PATH,\n output_mode=\"forward_all\",\n )\n .lenses(\n DeriveLens(\"rerun.urdf.JointTransformBatch\", output_entity=\"/robot/transforms\", scatter=True)\n .to_component(rr.Transform3D.descriptor_translation(), Selector(\"[].translation\"))\n .to_component(rr.Transform3D.descriptor_quaternion(), Selector(\"[].quaternion\"))\n .to_component(rr.Transform3D.descriptor_parent_frame(), Selector(\"[].parent_frame\"))\n .to_component(rr.Transform3D.descriptor_child_frame(), Selector(\"[].child_frame\")),\n content=\"/tmp/batches\",\n output_mode=\"drop_unmatched\",\n )\n .filter(content=\"/robot/transforms\")\n)\n\nLazyChunkStream.merge(model, fk).collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n out_path,\n application_id=\"urdf\",\n recording_id=segment_id,\n)\n```\n\n`read_names`, `read_values`, `JOINT_MSG_COMPONENT`, and `JOINT_SOURCE_PATH` are\nthe only data-specific pieces, and the mapping section above is what makes them\ncorrect.\n\n## One URDF for a whole dataset (assets)\n\n`urdf.stream()` emits only static chunks — meshes and fixed-joint transforms, with no time index.\nThe model does not belong in a per-episode layer, where every segment pays for the same meshes.\nRegister it once as a dataset **asset**:\n\n```python\nurdf.stream().collect(optimize=OptimizationProfile.OBJECT_STORE).write_rrd(\n model_rrd, application_id=\"urdf\", recording_id=\"so100_model\"\n)\nasset_id = dataset.register_asset(\"s3://bucket/so100_model.rrd\") # uri must be readable by the server\n```\n\nThe split is along static vs temporal: the **model** is an asset, and the **FK transforms** stay a per-segment layer keyed by `recording_id`.\nKeeping `include_joint_transforms=True` in the asset costs nothing and gives the rest pose for segments or time ranges the FK layer does not cover.\n\n- The viewer loads a dataset's assets alongside any segment it opens, and caches them across segments.\n- `dataset.segment_store(segment_id)` includes assets; pass `include_assets=False` to skip the extra manifest request per asset.\n- Dataframe queries never see assets — they stay on the dataset's own segments.\n\nWithout a catalog, the same split still pays: write the model RRD once and open it next to whichever episode you are looking at, rather than copying it into each one.\n\n## Gotchas that cause real failures\n\n1. Empty layer, no error: dead joint-state source (decoded to zero rows; see the importer skill for your format) or wrong `JOINT_SOURCE_PATH`/component name.\n2. Confident wrong pose: joint count, order, or units wrong.\n3. Layer writes but never attaches: `recording_id != segment_id`.\n4. Frames collide: two robots sharing a `frame_prefix`.\n5. Scene looks merged at the origin: unconnected roots logged as identity without a calibration edge.\n6. Catalog ingest rejects or misorders chunks: `OBJECT_STORE` optimization skipped.\n7. Layer opened in a running viewer never merges: opening an `.rrd` **by path** registers it as its own dataset and replaces its `application_id`, so it lands beside the segment instead of on it.\n To eyeball a layer locally, either merge model + FK + base into one RRD, or send both through a `RecordingStream(application_id, recording_id=segment_id)` pointed at the viewer with `rec.connect_grpc(url=\"rerun+http://127.0.0.1:9876/proxy\")`, which keeps the IDs and merges the stores.\n8. Robot invisible but transforms correct: meshes failed to resolve (see \"Getting the URDF\").\n9. No 3D view after adding a URDF to an existing recording: the auto-layout does not necessarily rebuild for the new entities. Send an explicit blueprint with `rrb.Spatial3DView(origin=\"/robot\")` (see `rerun-blueprint`).\n\n## References\n\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/robot_data_preprocessing` (FK two-lens pattern, two robots + scene URDFs, prefixes, recoloring, calibration offsets)\n- `https://github.com/rerun-io/rerun/tree/main/examples/python/animated_urdf` (classic logging API: `log_urdf_to_recording`, per-joint `compute_transform`)\n- [Catalog object model: assets](https://rerun.io/docs/concepts/query-and-transform/catalog-object-model#assets) (`register_asset`, the limits, and how assets are read back)\n- `rerun-data-model` (the mapping table this skill consumes)\n- the importer skill for your joint-state source format (making the source readable)\n- `rerun-chunk-processing` (lens/stream, write/optimize mechanics)\n\n\n# Claude Agent Guidelines & System Prompt\n\nPath: `CLAUDE.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/CLAUDE.md)\n\n\n# re_build_tools Documentation\n\nPath: `crates/build/re_build_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_build_tools/README.md)\n\n\n# re_dev_tools Documentation\n\nPath: `crates/build/re_dev_tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/README.md)\n\n\n# build_search_index Documentation\n\nPath: `crates/build/re_dev_tools/src/build_search_index/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_search_index/README.md)\n\n\n# build_web_viewer Documentation\n\nPath: `crates/build/re_dev_tools/src/build_web_viewer/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/rerun-io/rerun/HEAD/crates/build/re_dev_tools/src/build_web_viewer/README.md)\n "}