{"owner":"Kanaries","repo":"pygwalker","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md","AGENTS.md"],"skills":{"CLAUDE.md":"# CLAUDE.md\n\nThis project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a\nsingle source of truth for the architecture, dev-mode workflow, and log locations. Read it\nbefore making changes.\n\n@AGENTS.md\n\n## TL;DR for a coding agent\n\n- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e \".[dev]\"`,\n  then `cd app && yarn install && yarn build`.\n- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`\n  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).\n  Pass `--no-browser` for headless runs.\n- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),\n  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser\n  console.\n- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits\n  under `app/src/` hot-reload into the widget.\n- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`\n  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).\n- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,\n  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.\n","AGENTS.md":"# AGENTS.md — PyGWalker contributor & agent guide\n\nThis is the fast-start map of the PyGWalker repo for both human contributors and coding\nagents. It explains how the project is put together, how to run it in **dev mode with live\nfrontend reload**, and where all the logs go. Read this first — it is written to save you\nfrom re-deriving the architecture by grepping.\n\n> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),\n> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),\n> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).\n\n---\n\n## 1. What PyGWalker is (30-second model)\n\nPyGWalker turns a pandas / polars / pyarrow dataframe into an interactive\n[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,\nand plain web servers. It has **two halves that ship together**:\n\n- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,\n  `to_html`), data parsing, and the transports that talk to the UI.\n- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles\n  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python\n  side at render time.\n\nThe Python side never renders charts itself; it hands built JS + serialized data to a\nnotebook/browser and then answers data/spec requests over a message channel.\n\n---\n\n## 2. Repo map\n\n| Path | What lives here |\n|------|-----------------|\n| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |\n| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |\n| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |\n| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |\n| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |\n| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |\n| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |\n| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |\n| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |\n\n---\n\n## 3. How the two halves fit together (build & load model)\n\n```\napp/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it\n```\n\n**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):\n\n| Bundle | Built from | Loaded by |\n|--------|-----------|-----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app\nbundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.\n\n**The message protocol is generated, not hand-written.** Python Pydantic models in\n`pygwalker/communications/protocol.py` are the source of truth. Running\n`python scripts/generate_comm_protocol_ts.py` regenerates\n`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild\nthe frontend.** Never edit `comm.generated.ts` by hand.\n\n**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).\n`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget\nand slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.\n\n---\n\n## 4. First-time setup\n\nRequires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.\n\n```bash\n# Python (editable install with dev extras)\npython -m venv venv && source venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# Frontend deps + one full build so pygwalker/templates/dist/ is populated\ncd app && yarn install && yarn build && cd ..\n```\n\n---\n\n## 5. Dev mode: edit the frontend and see it live (anywidget HMR)\n\nThe default `pyg.walk(df)` uses the anywidget transport, which loads\n`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source\nchange and (b) let anywidget hot-reload it into open widgets. **One command starts\neverything and captures all logs:**\n\n```bash\nsource venv/bin/activate\npython scripts/dev.py\n```\n\nThis launches, and tees the output of, two long-running processes:\n\n- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into\n  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`\n- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its\n  environment (inherited by kernels). → `logs/jupyter.log`\n\nThen, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:\n\n```python\nimport pandas as pd, pygwalker as pyg\npyg.walk(pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]}))\n```\n\nEdit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`\n(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.\nFor a heavy change you can always re-run the cell.\n\n**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set\nto a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.\nanywidget then reads that file and watches it (via `watchfiles`), pushing new code to the\nfrontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the\nbundle is embedded as a string exactly as before — production behavior is unchanged.\n\nUseful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),\n`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),\n`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).\n\n> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is\n> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).\n> Prefer the anywidget HMR path above.\n\n---\n\n## 6. Logs: one place to look\n\n`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:\n\n| File | Contents |\n|------|----------|\n| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |\n| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |\n| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |\n\n**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):\n\n- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.\n- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).\n\n**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the\n**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as\nin-app toast notifications rather than `console.log`.\n\nAgent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a\nrebuild finished, and `logs/pygwalker.log` for kernel-side errors.\n\n---\n\n## 7. Everyday commands\n\n```bash\n# Frontend (run from app/)\nyarn build            # full build: typecheck + all 4 bundles (CI-equivalent)\nyarn build:app        # fast: only the two app bundles, no typecheck\nyarn dev:build        # rebuild-on-change (what scripts/dev.py runs)\nyarn typecheck        # tsc --noEmit\nyarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)\n\n# Python (run from repo root, venv active)\npython -m ruff check pygwalker tests scripts bin pygwalker_tools\npython -m ruff format --check pygwalker tests scripts bin pygwalker_tools\npython -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests\npython -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests\n\n# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py\npython scripts/generate_comm_protocol_ts.py\n\n# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)\npython scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow\n```\n\n---\n\n## 8. Rules & gotchas\n\n- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel\n  build (and CI) rebuilds it via the Hatch jupyter-builder hook.\n- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`\n  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves\n  Python and JS out of sync.\n- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or\n  `yarn typecheck`) before pushing frontend changes.\n- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /\n  iframe path for new work; it is removed in 0.7.0.\n- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.\n  Never rely on them being set at runtime for end users.\n- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from\n  disk; if it is missing you'll get a clear \"Missing PyGWalker frontend asset\" error — run the\n  frontend build (or wait for `scripts/dev.py`'s first build to finish).\n\n---\n\n## 9. Where to look for X\n\n| Question | Start here |\n|----------|-----------|\n| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |\n| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |\n| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |\n| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |\n| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |\n| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |\n"},"files":{"CLAUDE.md":"# CLAUDE.md\n\nThis project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a\nsingle source of truth for the architecture, dev-mode workflow, and log locations. Read it\nbefore making changes.\n\n@AGENTS.md\n\n## TL;DR for a coding agent\n\n- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e \".[dev]\"`,\n  then `cd app && yarn install && yarn build`.\n- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`\n  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).\n  Pass `--no-browser` for headless runs.\n- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),\n  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser\n  console.\n- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits\n  under `app/src/` hot-reload into the widget.\n- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`\n  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).\n- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,\n  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.\n","AGENTS.md":"# AGENTS.md — PyGWalker contributor & agent guide\n\nThis is the fast-start map of the PyGWalker repo for both human contributors and coding\nagents. It explains how the project is put together, how to run it in **dev mode with live\nfrontend reload**, and where all the logs go. Read this first — it is written to save you\nfrom re-deriving the architecture by grepping.\n\n> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),\n> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),\n> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).\n\n---\n\n## 1. What PyGWalker is (30-second model)\n\nPyGWalker turns a pandas / polars / pyarrow dataframe into an interactive\n[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,\nand plain web servers. It has **two halves that ship together**:\n\n- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,\n  `to_html`), data parsing, and the transports that talk to the UI.\n- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles\n  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python\n  side at render time.\n\nThe Python side never renders charts itself; it hands built JS + serialized data to a\nnotebook/browser and then answers data/spec requests over a message channel.\n\n---\n\n## 2. Repo map\n\n| Path | What lives here |\n|------|-----------------|\n| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |\n| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |\n| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |\n| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |\n| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |\n| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |\n| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |\n| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |\n| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |\n\n---\n\n## 3. How the two halves fit together (build & load model)\n\n```\napp/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it\n```\n\n**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):\n\n| Bundle | Built from | Loaded by |\n|--------|-----------|-----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app\nbundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.\n\n**The message protocol is generated, not hand-written.** Python Pydantic models in\n`pygwalker/communications/protocol.py` are the source of truth. Running\n`python scripts/generate_comm_protocol_ts.py` regenerates\n`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild\nthe frontend.** Never edit `comm.generated.ts` by hand.\n\n**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).\n`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget\nand slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.\n\n---\n\n## 4. First-time setup\n\nRequires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.\n\n```bash\n# Python (editable install with dev extras)\npython -m venv venv && source venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# Frontend deps + one full build so pygwalker/templates/dist/ is populated\ncd app && yarn install && yarn build && cd ..\n```\n\n---\n\n## 5. Dev mode: edit the frontend and see it live (anywidget HMR)\n\nThe default `pyg.walk(df)` uses the anywidget transport, which loads\n`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source\nchange and (b) let anywidget hot-reload it into open widgets. **One command starts\neverything and captures all logs:**\n\n```bash\nsource venv/bin/activate\npython scripts/dev.py\n```\n\nThis launches, and tees the output of, two long-running processes:\n\n- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into\n  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`\n- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its\n  environment (inherited by kernels). → `logs/jupyter.log`\n\nThen, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:\n\n```python\nimport pandas as pd, pygwalker as pyg\npyg.walk(pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]}))\n```\n\nEdit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`\n(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.\nFor a heavy change you can always re-run the cell.\n\n**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set\nto a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.\nanywidget then reads that file and watches it (via `watchfiles`), pushing new code to the\nfrontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the\nbundle is embedded as a string exactly as before — production behavior is unchanged.\n\nUseful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),\n`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),\n`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).\n\n> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is\n> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).\n> Prefer the anywidget HMR path above.\n\n---\n\n## 6. Logs: one place to look\n\n`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:\n\n| File | Contents |\n|------|----------|\n| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |\n| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |\n| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |\n\n**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):\n\n- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.\n- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).\n\n**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the\n**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as\nin-app toast notifications rather than `console.log`.\n\nAgent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a\nrebuild finished, and `logs/pygwalker.log` for kernel-side errors.\n\n---\n\n## 7. Everyday commands\n\n```bash\n# Frontend (run from app/)\nyarn build            # full build: typecheck + all 4 bundles (CI-equivalent)\nyarn build:app        # fast: only the two app bundles, no typecheck\nyarn dev:build        # rebuild-on-change (what scripts/dev.py runs)\nyarn typecheck        # tsc --noEmit\nyarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)\n\n# Python (run from repo root, venv active)\npython -m ruff check pygwalker tests scripts bin pygwalker_tools\npython -m ruff format --check pygwalker tests scripts bin pygwalker_tools\npython -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests\npython -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests\n\n# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py\npython scripts/generate_comm_protocol_ts.py\n\n# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)\npython scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow\n```\n\n---\n\n## 8. Rules & gotchas\n\n- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel\n  build (and CI) rebuilds it via the Hatch jupyter-builder hook.\n- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`\n  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves\n  Python and JS out of sync.\n- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or\n  `yarn typecheck`) before pushing frontend changes.\n- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /\n  iframe path for new work; it is removed in 0.7.0.\n- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.\n  Never rely on them being set at runtime for end users.\n- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from\n  disk; if it is missing you'll get a clear \"Missing PyGWalker frontend asset\" error — run the\n  frontend build (or wait for `scripts/dev.py`'s first build to finish).\n\n---\n\n## 9. Where to look for X\n\n| Question | Start here |\n|----------|-----------|\n| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |\n| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |\n| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |\n| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |\n| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |\n| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# CLAUDE.md\n\nThis project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a\nsingle source of truth for the architecture, dev-mode workflow, and log locations. Read it\nbefore making changes.\n\n@AGENTS.md\n\n## TL;DR for a coding agent\n\n- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e \".[dev]\"`,\n  then `cd app && yarn install && yarn build`.\n- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`\n  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).\n  Pass `--no-browser` for headless runs.\n- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),\n  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser\n  console.\n- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits\n  under `app/src/` hot-reload into the widget.\n- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`\n  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).\n- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,\n  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.\n","category":"root","tokens":325},{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# AGENTS.md — PyGWalker contributor & agent guide\n\nThis is the fast-start map of the PyGWalker repo for both human contributors and coding\nagents. It explains how the project is put together, how to run it in **dev mode with live\nfrontend reload**, and where all the logs go. Read this first — it is written to save you\nfrom re-deriving the architecture by grepping.\n\n> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),\n> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),\n> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).\n\n---\n\n## 1. What PyGWalker is (30-second model)\n\nPyGWalker turns a pandas / polars / pyarrow dataframe into an interactive\n[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,\nand plain web servers. It has **two halves that ship together**:\n\n- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,\n  `to_html`), data parsing, and the transports that talk to the UI.\n- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles\n  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python\n  side at render time.\n\nThe Python side never renders charts itself; it hands built JS + serialized data to a\nnotebook/browser and then answers data/spec requests over a message channel.\n\n---\n\n## 2. Repo map\n\n| Path | What lives here |\n|------|-----------------|\n| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |\n| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |\n| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |\n| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |\n| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |\n| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |\n| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |\n| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |\n| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |\n\n---\n\n## 3. How the two halves fit together (build & load model)\n\n```\napp/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it\n```\n\n**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):\n\n| Bundle | Built from | Loaded by |\n|--------|-----------|-----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app\nbundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.\n\n**The message protocol is generated, not hand-written.** Python Pydantic models in\n`pygwalker/communications/protocol.py` are the source of truth. Running\n`python scripts/generate_comm_protocol_ts.py` regenerates\n`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild\nthe frontend.** Never edit `comm.generated.ts` by hand.\n\n**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).\n`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget\nand slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.\n\n---\n\n## 4. First-time setup\n\nRequires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.\n\n```bash\n# Python (editable install with dev extras)\npython -m venv venv && source venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# Frontend deps + one full build so pygwalker/templates/dist/ is populated\ncd app && yarn install && yarn build && cd ..\n```\n\n---\n\n## 5. Dev mode: edit the frontend and see it live (anywidget HMR)\n\nThe default `pyg.walk(df)` uses the anywidget transport, which loads\n`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source\nchange and (b) let anywidget hot-reload it into open widgets. **One command starts\neverything and captures all logs:**\n\n```bash\nsource venv/bin/activate\npython scripts/dev.py\n```\n\nThis launches, and tees the output of, two long-running processes:\n\n- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into\n  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`\n- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its\n  environment (inherited by kernels). → `logs/jupyter.log`\n\nThen, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:\n\n```python\nimport pandas as pd, pygwalker as pyg\npyg.walk(pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]}))\n```\n\nEdit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`\n(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.\nFor a heavy change you can always re-run the cell.\n\n**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set\nto a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.\nanywidget then reads that file and watches it (via `watchfiles`), pushing new code to the\nfrontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the\nbundle is embedded as a string exactly as before — production behavior is unchanged.\n\nUseful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),\n`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),\n`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).\n\n> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is\n> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).\n> Prefer the anywidget HMR path above.\n\n---\n\n## 6. Logs: one place to look\n\n`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:\n\n| File | Contents |\n|------|----------|\n| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |\n| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |\n| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |\n\n**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):\n\n- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.\n- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).\n\n**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the\n**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as\nin-app toast notifications rather than `console.log`.\n\nAgent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a\nrebuild finished, and `logs/pygwalker.log` for kernel-side errors.\n\n---\n\n## 7. Everyday commands\n\n```bash\n# Frontend (run from app/)\nyarn build            # full build: typecheck + all 4 bundles (CI-equivalent)\nyarn build:app        # fast: only the two app bundles, no typecheck\nyarn dev:build        # rebuild-on-change (what scripts/dev.py runs)\nyarn typecheck        # tsc --noEmit\nyarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)\n\n# Python (run from repo root, venv active)\npython -m ruff check pygwalker tests scripts bin pygwalker_tools\npython -m ruff format --check pygwalker tests scripts bin pygwalker_tools\npython -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests\npython -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests\n\n# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py\npython scripts/generate_comm_protocol_ts.py\n\n# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)\npython scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow\n```\n\n---\n\n## 8. Rules & gotchas\n\n- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel\n  build (and CI) rebuilds it via the Hatch jupyter-builder hook.\n- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`\n  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves\n  Python and JS out of sync.\n- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or\n  `yarn typecheck`) before pushing frontend changes.\n- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /\n  iframe path for new work; it is removed in 0.7.0.\n- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.\n  Never rely on them being set at runtime for end users.\n- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from\n  disk; if it is missing you'll get a clear \"Missing PyGWalker frontend asset\" error — run the\n  frontend build (or wait for `scripts/dev.py`'s first build to finish).\n\n---\n\n## 9. Where to look for X\n\n| Question | Start here |\n|----------|-----------|\n| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |\n| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |\n| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |\n| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |\n| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |\n| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |\n","category":"root","tokens":2762}]}