{"owner":"Kanaries","repo":"pygwalker","hasSkills":true,"totalSkillsCount":3,"totalTokensCount":5553,"categories":["claude-rule","project-spec","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md","docs/ARCHITECTURE.md","AGENTS.md"],"skills":{"CLAUDE.md":"# CLAUDE.md\n\nThis project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a\nsingle source of truth for the architecture, dev-mode workflow, and log locations. Read it\nbefore making changes.\n\n@AGENTS.md\n\n## TL;DR for a coding agent\n\n- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e \".[dev]\"`,\n  then `cd app && yarn install && yarn build`.\n- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`\n  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).\n  Pass `--no-browser` for headless runs.\n- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),\n  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser\n  console.\n- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits\n  under `app/src/` hot-reload into the widget.\n- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`\n  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).\n- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,\n  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.\n","docs/ARCHITECTURE.md":"# PyGWalker Architecture\n\nHow the project is constructed across its Python and frontend halves, and how they load and\ntalk to each other at runtime. This is the orientation doc for contributors and agents; for\nthe hands-on dev workflow see [`DEVELOPMENT.md`](./DEVELOPMENT.md), and for the quick map see\n[`../AGENTS.md`](../AGENTS.md).\n\n## Two halves that ship together\n\nPyGWalker is a Python library plus a React frontend, packaged as one wheel:\n\n- **Python package** — `pygwalker/`. The public API, dataframe parsing, and the transports\n  that exchange data and chart specs with the UI.\n- **Frontend app** — `app/`. A React + Vite application built into JavaScript bundles that are\n  placed in `pygwalker/templates/dist/` and shipped inside the wheel.\n\nAt render time the Python side serializes the dataframe + config and hands the built\nJavaScript to a notebook/browser. The running UI then calls back into the Python kernel to\nfetch data, run queries, and save chart specs.\n\n```\n┌────────────────────────┐        build         ┌──────────────────────────────┐\n│  app/  (React + Vite)  │ ───────────────────▶ │ pygwalker/templates/dist/*.js │\n└────────────────────────┘   vite build          └──────────────┬───────────────┘\n                                                                 │ read at runtime\n                                                                 ▼\n┌───────────────────────────────────────────────────────────────────────────────┐\n│  pygwalker/ (Python)   walk()/render()/table()/Walker → transport → rendered UI │\n└───────────────────────────────────────────────────────────────────────────────┘\n                    ▲                                            │\n                    │      messages (data, spec, export)         │\n                    └────────────────────────────────────────────┘\n```\n\n## Python package layout\n\n| Module | Responsibility |\n|--------|----------------|\n| `api/adapter.py` | Top-level `walk`/`render`/`table`. Detects environment (`get_current_env`) and routes to the notebook (`api/jupyter.py`) or web-server (`api/webserver.py`) path. |\n| `api/jupyter.py` | Notebook dispatch. Maps `env` → a display method (`env_display_map`). Default `JupyterAnywidget`. |\n| `api/walker.py` | `Walker`, the reusable object (`.show()`, `.to_html()`, `.to_streamlit()`). |\n| `api/pygwalker.py` | `PygWalker`, the core object holding data source, spec, and props. |\n| `services/anywidget_widget.py` | **Default transport.** Wraps the built ESM in an `anywidget.AnyWidget`. |\n| `services/render.py` + `templates/*.html` | Iframe / static-HTML transport (Jinja templates). |\n| `services/jupyter_display.py` | The concrete display flows (anywidget, iframe, widgets, preview, convert). |\n| `services/global_var.py` | Process-wide runtime settings (`GlobalVarManager`): component URL, API keys, feature flags. |\n| `communications/` | Kernel⇄frontend transports + the shared message `protocol.py`. |\n| `data_parsers/` | Adapters that normalize pandas/polars/pyarrow/SQL/spark sources to a common interface. |\n| `utils/frontend_assets.py` | Locates and loads the built bundles from `templates/dist/`. |\n| `utils/log.py` | The shared `pygwalker` logger (console + optional file). |\n\n## Frontend app layout\n\n| Path | Responsibility |\n|------|----------------|\n| `src/index.tsx` | Entry point. Exports `GWalker` (iframe/http), `AnywidgetGWalkerApp` (anywidget), Streamlit and preview apps. Selects a transport by `env`. |\n| `src/utils/communication.tsx` | The four client transports: Jupyter widgets, HTTP (Streamlit/Gradio/web), and anywidget. |\n| `src/dataSource/` | Receives chunked data from the kernel and delegates kernel computation. |\n| `src/interfaces/comm.generated.ts` | **Generated** protocol types (request/response envelopes, action map). Do not hand-edit. |\n| `src/store/` | MobX stores for UI state, notifications, and the active communication instance. |\n| `src/tools/` | Toolbar actions: save, export chart, export dataframe, open-in-desktop. |\n| `src/lib/` | Standalone transforms (`dslToWorkflow.ts`, `vegaToDsl.ts`) built as separate UMD bundles. |\n\n## Build pipeline\n\n**Frontend → bundles.** `app/vite.config.ts` defines the build. `yarn build` runs a typecheck\nthen produces four artifacts into `pygwalker/templates/dist/`:\n\n| Bundle | Entry | Consumer |\n|--------|-------|----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | anywidget transport (default notebook path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build:app` is a fast subset (only the two app bundles, no typecheck).\n\n**Protocol types are generated.** The message contract is defined once, in Python Pydantic\nmodels at `pygwalker/communications/protocol.py`.\n`python scripts/generate_comm_protocol_ts.py` renders those models into\n`app/src/interfaces/comm.generated.ts`. Change the Python models, regenerate, then rebuild the\nfrontend — otherwise the two sides drift.\n\n**Wheel build.** `pyproject.toml` wires a Hatch `jupyter-builder` hook that runs the frontend\n`build` during `python -m build`, so a released wheel always contains freshly built bundles.\n`pygwalker/templates/dist/` is git-ignored and rebuilt on demand; never commit it.\n\n## Runtime flow (default anywidget path)\n\n1. `pyg.walk(df)` → `adapter.walk` detects a notebook → `jupyter.walk` with the default\n   `env='JupyterAnywidget'`.\n2. A `PygWalker` is created; `jupyter.py`'s `env_display_map` calls\n   `display_on_jupyter_use_anywidget`.\n3. `services/anywidget_widget.py` builds a `WalkerAnyWidget` whose `_esm` is the built\n   `pygwalker-app.es.js`, sets `props` (serialized config + a data sample), and registers an\n   `AnywidgetCommunication` channel.\n4. In the browser, `AnywidgetGWalkerApp` mounts Graphic Walker and, as the user explores,\n   sends typed requests (fetch data, run SQL, save spec, export) back over the anywidget\n   channel. `communications/` validates each message against `protocol.py` and dispatches it.\n\nThe **iframe transport** (`services/render.py` + `templates/pygwalker_main_page.html`) is an\nalternative that embeds the compressed `iife` bundle in an `iframe` `srcdoc`, or points the\niframe at a dev-server URL when `GlobalVarManager.component_url` is set. It backs\n`to_html()` and the deprecated low-level `PygWalker.display_on_jupyter()` compatibility\nmethod. The legacy `env='Jupyter'`/`'JupyterWidget'` aliases are coerced to anywidget and do\nnot select the iframe transport.\n\n## Message protocol & transports\n\nEvery transport speaks the same envelope (see `protocol.py` /\n`comm.generated.ts`): a request carries an `action`, a `data` payload, a request id `rid`, and\nthe widget/app id `gid`; a response carries `code`, `data`, and `message`. Actions include\ndata fetches (`get_datas`, `batch_get_datas_by_sql`), spec operations\n(`get_latest_vis_spec`, `save_chart`, `update_spec`), export, and AI features.\n\n| Transport | Module | Mechanism |\n|-----------|--------|-----------|\n| anywidget (default) | `communications/anywidget_comm.py` | anywidget model `send`/`on_msg` |\n| iframe widgets | `communications/hacker_comm.py` | ipywidgets text fields observed by the frontend |\n| Streamlit | `communications/streamlit_comm.py` | HTTP route under Streamlit's server |\n| Gradio | `communications/gradio_comm.py` | Starlette mount |\n| Reflex | `communications/reflex_comm.py` | Reflex endpoint |\n| web server | `api/webserver.py` | standalone HTTP handler |\n\n## Computation modes\n\n`computation` selects where data queries run: `browser` (all data sent to the client and\ncomputed there), `kernel` (a local DuckDB engine in the Python process answers queries on\ndemand — best for large data), or `cloud` (Kanaries cloud). In `kernel` mode the frontend\nparses the Graphic Walker DSL to SQL (using the `gw-dsl-parser` WASM / the UMD bundles) and\nasks the kernel to execute it, so only aggregated results cross the boundary.\n\n## How dev mode changes asset loading\n\nNormally `WalkerAnyWidget._esm` is the **contents** of `pygwalker-app.es.js` read once at\nimport — a plain string embedded in the widget. When `PYGWALKER_DEV=1` or `ANYWIDGET_HMR=1`\nis set, `_esm` instead points at the **file path** of that bundle. anywidget then reads it from\ndisk and, under HMR, watches it and live-reloads the widget whenever `vite build --watch`\nrewrites the file. This is entirely opt-in: with the flags unset, loading is byte-for-byte the\nsame as before. See [`DEVELOPMENT.md`](./DEVELOPMENT.md) for the workflow and\n[`../AGENTS.md`](../AGENTS.md) for the one-command dev stack.\n","AGENTS.md":"# AGENTS.md — PyGWalker contributor & agent guide\n\nThis is the fast-start map of the PyGWalker repo for both human contributors and coding\nagents. It explains how the project is put together, how to run it in **dev mode with live\nfrontend reload**, and where all the logs go. Read this first — it is written to save you\nfrom re-deriving the architecture by grepping.\n\n> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),\n> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),\n> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).\n\n---\n\n## 1. What PyGWalker is (30-second model)\n\nPyGWalker turns a pandas / polars / pyarrow dataframe into an interactive\n[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,\nand plain web servers. It has **two halves that ship together**:\n\n- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,\n  `to_html`), data parsing, and the transports that talk to the UI.\n- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles\n  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python\n  side at render time.\n\nThe Python side never renders charts itself; it hands built JS + serialized data to a\nnotebook/browser and then answers data/spec requests over a message channel.\n\n---\n\n## 2. Repo map\n\n| Path | What lives here |\n|------|-----------------|\n| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |\n| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |\n| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |\n| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |\n| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |\n| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |\n| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |\n| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |\n| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |\n\n---\n\n## 3. How the two halves fit together (build & load model)\n\n```\napp/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it\n```\n\n**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):\n\n| Bundle | Built from | Loaded by |\n|--------|-----------|-----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app\nbundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.\n\n**The message protocol is generated, not hand-written.** Python Pydantic models in\n`pygwalker/communications/protocol.py` are the source of truth. Running\n`python scripts/generate_comm_protocol_ts.py` regenerates\n`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild\nthe frontend.** Never edit `comm.generated.ts` by hand.\n\n**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).\n`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget\nand slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.\n\n---\n\n## 4. First-time setup\n\nRequires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.\n\n```bash\n# Python (editable install with dev extras)\npython -m venv venv && source venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# Frontend deps + one full build so pygwalker/templates/dist/ is populated\ncd app && yarn install && yarn build && cd ..\n```\n\n---\n\n## 5. Dev mode: edit the frontend and see it live (anywidget HMR)\n\nThe default `pyg.walk(df)` uses the anywidget transport, which loads\n`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source\nchange and (b) let anywidget hot-reload it into open widgets. **One command starts\neverything and captures all logs:**\n\n```bash\nsource venv/bin/activate\npython scripts/dev.py\n```\n\nThis launches, and tees the output of, two long-running processes:\n\n- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into\n  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`\n- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its\n  environment (inherited by kernels). → `logs/jupyter.log`\n\nThen, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:\n\n```python\nimport pandas as pd, pygwalker as pyg\npyg.walk(pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]}))\n```\n\nEdit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`\n(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.\nFor a heavy change you can always re-run the cell.\n\n**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set\nto a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.\nanywidget then reads that file and watches it (via `watchfiles`), pushing new code to the\nfrontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the\nbundle is embedded as a string exactly as before — production behavior is unchanged.\n\nUseful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),\n`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),\n`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).\n\n> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is\n> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).\n> Prefer the anywidget HMR path above.\n\n---\n\n## 6. Logs: one place to look\n\n`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:\n\n| File | Contents |\n|------|----------|\n| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |\n| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |\n| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |\n\n**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):\n\n- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.\n- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).\n\n**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the\n**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as\nin-app toast notifications rather than `console.log`.\n\nAgent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a\nrebuild finished, and `logs/pygwalker.log` for kernel-side errors.\n\n---\n\n## 7. Everyday commands\n\n```bash\n# Frontend (run from app/)\nyarn build            # full build: typecheck + all 4 bundles (CI-equivalent)\nyarn build:app        # fast: only the two app bundles, no typecheck\nyarn dev:build        # rebuild-on-change (what scripts/dev.py runs)\nyarn typecheck        # tsc --noEmit\nyarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)\n\n# Python (run from repo root, venv active)\npython -m ruff check pygwalker tests scripts bin pygwalker_tools\npython -m ruff format --check pygwalker tests scripts bin pygwalker_tools\npython -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests\npython -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests\n\n# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py\npython scripts/generate_comm_protocol_ts.py\n\n# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)\npython scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow\n```\n\n---\n\n## 8. Rules & gotchas\n\n- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel\n  build (and CI) rebuilds it via the Hatch jupyter-builder hook.\n- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`\n  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves\n  Python and JS out of sync.\n- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or\n  `yarn typecheck`) before pushing frontend changes.\n- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /\n  iframe path for new work; it is removed in 0.7.0.\n- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.\n  Never rely on them being set at runtime for end users.\n- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from\n  disk; if it is missing you'll get a clear \"Missing PyGWalker frontend asset\" error — run the\n  frontend build (or wait for `scripts/dev.py`'s first build to finish).\n\n---\n\n## 9. Where to look for X\n\n| Question | Start here |\n|----------|-----------|\n| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |\n| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |\n| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |\n| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |\n| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |\n| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |\n"},"files":{"CLAUDE.md":"# CLAUDE.md\n\nThis project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a\nsingle source of truth for the architecture, dev-mode workflow, and log locations. Read it\nbefore making changes.\n\n@AGENTS.md\n\n## TL;DR for a coding agent\n\n- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e \".[dev]\"`,\n  then `cd app && yarn install && yarn build`.\n- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`\n  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).\n  Pass `--no-browser` for headless runs.\n- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),\n  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser\n  console.\n- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits\n  under `app/src/` hot-reload into the widget.\n- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`\n  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).\n- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,\n  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.\n","docs/ARCHITECTURE.md":"# PyGWalker Architecture\n\nHow the project is constructed across its Python and frontend halves, and how they load and\ntalk to each other at runtime. This is the orientation doc for contributors and agents; for\nthe hands-on dev workflow see [`DEVELOPMENT.md`](./DEVELOPMENT.md), and for the quick map see\n[`../AGENTS.md`](../AGENTS.md).\n\n## Two halves that ship together\n\nPyGWalker is a Python library plus a React frontend, packaged as one wheel:\n\n- **Python package** — `pygwalker/`. The public API, dataframe parsing, and the transports\n  that exchange data and chart specs with the UI.\n- **Frontend app** — `app/`. A React + Vite application built into JavaScript bundles that are\n  placed in `pygwalker/templates/dist/` and shipped inside the wheel.\n\nAt render time the Python side serializes the dataframe + config and hands the built\nJavaScript to a notebook/browser. The running UI then calls back into the Python kernel to\nfetch data, run queries, and save chart specs.\n\n```\n┌────────────────────────┐        build         ┌──────────────────────────────┐\n│  app/  (React + Vite)  │ ───────────────────▶ │ pygwalker/templates/dist/*.js │\n└────────────────────────┘   vite build          └──────────────┬───────────────┘\n                                                                 │ read at runtime\n                                                                 ▼\n┌───────────────────────────────────────────────────────────────────────────────┐\n│  pygwalker/ (Python)   walk()/render()/table()/Walker → transport → rendered UI │\n└───────────────────────────────────────────────────────────────────────────────┘\n                    ▲                                            │\n                    │      messages (data, spec, export)         │\n                    └────────────────────────────────────────────┘\n```\n\n## Python package layout\n\n| Module | Responsibility |\n|--------|----------------|\n| `api/adapter.py` | Top-level `walk`/`render`/`table`. Detects environment (`get_current_env`) and routes to the notebook (`api/jupyter.py`) or web-server (`api/webserver.py`) path. |\n| `api/jupyter.py` | Notebook dispatch. Maps `env` → a display method (`env_display_map`). Default `JupyterAnywidget`. |\n| `api/walker.py` | `Walker`, the reusable object (`.show()`, `.to_html()`, `.to_streamlit()`). |\n| `api/pygwalker.py` | `PygWalker`, the core object holding data source, spec, and props. |\n| `services/anywidget_widget.py` | **Default transport.** Wraps the built ESM in an `anywidget.AnyWidget`. |\n| `services/render.py` + `templates/*.html` | Iframe / static-HTML transport (Jinja templates). |\n| `services/jupyter_display.py` | The concrete display flows (anywidget, iframe, widgets, preview, convert). |\n| `services/global_var.py` | Process-wide runtime settings (`GlobalVarManager`): component URL, API keys, feature flags. |\n| `communications/` | Kernel⇄frontend transports + the shared message `protocol.py`. |\n| `data_parsers/` | Adapters that normalize pandas/polars/pyarrow/SQL/spark sources to a common interface. |\n| `utils/frontend_assets.py` | Locates and loads the built bundles from `templates/dist/`. |\n| `utils/log.py` | The shared `pygwalker` logger (console + optional file). |\n\n## Frontend app layout\n\n| Path | Responsibility |\n|------|----------------|\n| `src/index.tsx` | Entry point. Exports `GWalker` (iframe/http), `AnywidgetGWalkerApp` (anywidget), Streamlit and preview apps. Selects a transport by `env`. |\n| `src/utils/communication.tsx` | The four client transports: Jupyter widgets, HTTP (Streamlit/Gradio/web), and anywidget. |\n| `src/dataSource/` | Receives chunked data from the kernel and delegates kernel computation. |\n| `src/interfaces/comm.generated.ts` | **Generated** protocol types (request/response envelopes, action map). Do not hand-edit. |\n| `src/store/` | MobX stores for UI state, notifications, and the active communication instance. |\n| `src/tools/` | Toolbar actions: save, export chart, export dataframe, open-in-desktop. |\n| `src/lib/` | Standalone transforms (`dslToWorkflow.ts`, `vegaToDsl.ts`) built as separate UMD bundles. |\n\n## Build pipeline\n\n**Frontend → bundles.** `app/vite.config.ts` defines the build. `yarn build` runs a typecheck\nthen produces four artifacts into `pygwalker/templates/dist/`:\n\n| Bundle | Entry | Consumer |\n|--------|-------|----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | anywidget transport (default notebook path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build:app` is a fast subset (only the two app bundles, no typecheck).\n\n**Protocol types are generated.** The message contract is defined once, in Python Pydantic\nmodels at `pygwalker/communications/protocol.py`.\n`python scripts/generate_comm_protocol_ts.py` renders those models into\n`app/src/interfaces/comm.generated.ts`. Change the Python models, regenerate, then rebuild the\nfrontend — otherwise the two sides drift.\n\n**Wheel build.** `pyproject.toml` wires a Hatch `jupyter-builder` hook that runs the frontend\n`build` during `python -m build`, so a released wheel always contains freshly built bundles.\n`pygwalker/templates/dist/` is git-ignored and rebuilt on demand; never commit it.\n\n## Runtime flow (default anywidget path)\n\n1. `pyg.walk(df)` → `adapter.walk` detects a notebook → `jupyter.walk` with the default\n   `env='JupyterAnywidget'`.\n2. A `PygWalker` is created; `jupyter.py`'s `env_display_map` calls\n   `display_on_jupyter_use_anywidget`.\n3. `services/anywidget_widget.py` builds a `WalkerAnyWidget` whose `_esm` is the built\n   `pygwalker-app.es.js`, sets `props` (serialized config + a data sample), and registers an\n   `AnywidgetCommunication` channel.\n4. In the browser, `AnywidgetGWalkerApp` mounts Graphic Walker and, as the user explores,\n   sends typed requests (fetch data, run SQL, save spec, export) back over the anywidget\n   channel. `communications/` validates each message against `protocol.py` and dispatches it.\n\nThe **iframe transport** (`services/render.py` + `templates/pygwalker_main_page.html`) is an\nalternative that embeds the compressed `iife` bundle in an `iframe` `srcdoc`, or points the\niframe at a dev-server URL when `GlobalVarManager.component_url` is set. It backs\n`to_html()` and the deprecated low-level `PygWalker.display_on_jupyter()` compatibility\nmethod. The legacy `env='Jupyter'`/`'JupyterWidget'` aliases are coerced to anywidget and do\nnot select the iframe transport.\n\n## Message protocol & transports\n\nEvery transport speaks the same envelope (see `protocol.py` /\n`comm.generated.ts`): a request carries an `action`, a `data` payload, a request id `rid`, and\nthe widget/app id `gid`; a response carries `code`, `data`, and `message`. Actions include\ndata fetches (`get_datas`, `batch_get_datas_by_sql`), spec operations\n(`get_latest_vis_spec`, `save_chart`, `update_spec`), export, and AI features.\n\n| Transport | Module | Mechanism |\n|-----------|--------|-----------|\n| anywidget (default) | `communications/anywidget_comm.py` | anywidget model `send`/`on_msg` |\n| iframe widgets | `communications/hacker_comm.py` | ipywidgets text fields observed by the frontend |\n| Streamlit | `communications/streamlit_comm.py` | HTTP route under Streamlit's server |\n| Gradio | `communications/gradio_comm.py` | Starlette mount |\n| Reflex | `communications/reflex_comm.py` | Reflex endpoint |\n| web server | `api/webserver.py` | standalone HTTP handler |\n\n## Computation modes\n\n`computation` selects where data queries run: `browser` (all data sent to the client and\ncomputed there), `kernel` (a local DuckDB engine in the Python process answers queries on\ndemand — best for large data), or `cloud` (Kanaries cloud). In `kernel` mode the frontend\nparses the Graphic Walker DSL to SQL (using the `gw-dsl-parser` WASM / the UMD bundles) and\nasks the kernel to execute it, so only aggregated results cross the boundary.\n\n## How dev mode changes asset loading\n\nNormally `WalkerAnyWidget._esm` is the **contents** of `pygwalker-app.es.js` read once at\nimport — a plain string embedded in the widget. When `PYGWALKER_DEV=1` or `ANYWIDGET_HMR=1`\nis set, `_esm` instead points at the **file path** of that bundle. anywidget then reads it from\ndisk and, under HMR, watches it and live-reloads the widget whenever `vite build --watch`\nrewrites the file. This is entirely opt-in: with the flags unset, loading is byte-for-byte the\nsame as before. See [`DEVELOPMENT.md`](./DEVELOPMENT.md) for the workflow and\n[`../AGENTS.md`](../AGENTS.md) for the one-command dev stack.\n","AGENTS.md":"# AGENTS.md — PyGWalker contributor & agent guide\n\nThis is the fast-start map of the PyGWalker repo for both human contributors and coding\nagents. It explains how the project is put together, how to run it in **dev mode with live\nfrontend reload**, and where all the logs go. Read this first — it is written to save you\nfrom re-deriving the architecture by grepping.\n\n> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),\n> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),\n> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).\n\n---\n\n## 1. What PyGWalker is (30-second model)\n\nPyGWalker turns a pandas / polars / pyarrow dataframe into an interactive\n[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,\nand plain web servers. It has **two halves that ship together**:\n\n- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,\n  `to_html`), data parsing, and the transports that talk to the UI.\n- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles\n  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python\n  side at render time.\n\nThe Python side never renders charts itself; it hands built JS + serialized data to a\nnotebook/browser and then answers data/spec requests over a message channel.\n\n---\n\n## 2. Repo map\n\n| Path | What lives here |\n|------|-----------------|\n| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |\n| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |\n| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |\n| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |\n| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |\n| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |\n| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |\n| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |\n| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |\n\n---\n\n## 3. How the two halves fit together (build & load model)\n\n```\napp/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it\n```\n\n**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):\n\n| Bundle | Built from | Loaded by |\n|--------|-----------|-----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app\nbundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.\n\n**The message protocol is generated, not hand-written.** Python Pydantic models in\n`pygwalker/communications/protocol.py` are the source of truth. Running\n`python scripts/generate_comm_protocol_ts.py` regenerates\n`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild\nthe frontend.** Never edit `comm.generated.ts` by hand.\n\n**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).\n`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget\nand slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.\n\n---\n\n## 4. First-time setup\n\nRequires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.\n\n```bash\n# Python (editable install with dev extras)\npython -m venv venv && source venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# Frontend deps + one full build so pygwalker/templates/dist/ is populated\ncd app && yarn install && yarn build && cd ..\n```\n\n---\n\n## 5. Dev mode: edit the frontend and see it live (anywidget HMR)\n\nThe default `pyg.walk(df)` uses the anywidget transport, which loads\n`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source\nchange and (b) let anywidget hot-reload it into open widgets. **One command starts\neverything and captures all logs:**\n\n```bash\nsource venv/bin/activate\npython scripts/dev.py\n```\n\nThis launches, and tees the output of, two long-running processes:\n\n- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into\n  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`\n- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its\n  environment (inherited by kernels). → `logs/jupyter.log`\n\nThen, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:\n\n```python\nimport pandas as pd, pygwalker as pyg\npyg.walk(pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]}))\n```\n\nEdit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`\n(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.\nFor a heavy change you can always re-run the cell.\n\n**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set\nto a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.\nanywidget then reads that file and watches it (via `watchfiles`), pushing new code to the\nfrontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the\nbundle is embedded as a string exactly as before — production behavior is unchanged.\n\nUseful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),\n`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),\n`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).\n\n> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is\n> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).\n> Prefer the anywidget HMR path above.\n\n---\n\n## 6. Logs: one place to look\n\n`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:\n\n| File | Contents |\n|------|----------|\n| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |\n| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |\n| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |\n\n**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):\n\n- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.\n- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).\n\n**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the\n**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as\nin-app toast notifications rather than `console.log`.\n\nAgent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a\nrebuild finished, and `logs/pygwalker.log` for kernel-side errors.\n\n---\n\n## 7. Everyday commands\n\n```bash\n# Frontend (run from app/)\nyarn build            # full build: typecheck + all 4 bundles (CI-equivalent)\nyarn build:app        # fast: only the two app bundles, no typecheck\nyarn dev:build        # rebuild-on-change (what scripts/dev.py runs)\nyarn typecheck        # tsc --noEmit\nyarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)\n\n# Python (run from repo root, venv active)\npython -m ruff check pygwalker tests scripts bin pygwalker_tools\npython -m ruff format --check pygwalker tests scripts bin pygwalker_tools\npython -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests\npython -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests\n\n# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py\npython scripts/generate_comm_protocol_ts.py\n\n# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)\npython scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow\n```\n\n---\n\n## 8. Rules & gotchas\n\n- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel\n  build (and CI) rebuilds it via the Hatch jupyter-builder hook.\n- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`\n  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves\n  Python and JS out of sync.\n- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or\n  `yarn typecheck`) before pushing frontend changes.\n- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /\n  iframe path for new work; it is removed in 0.7.0.\n- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.\n  Never rely on them being set at runtime for end users.\n- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from\n  disk; if it is missing you'll get a clear \"Missing PyGWalker frontend asset\" error — run the\n  frontend build (or wait for `scripts/dev.py`'s first build to finish).\n\n---\n\n## 9. Where to look for X\n\n| Question | Start here |\n|----------|-----------|\n| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |\n| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |\n| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |\n| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |\n| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |\n| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/Kanaries/pygwalker/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md — PyGWalker contributor & agent guide\n\nThis is the fast-start map of the PyGWalker repo for both human contributors and coding\nagents. It explains how the project is put together, how to run it in **dev mode with live\nfrontend reload**, and where all the logs go. Read this first — it is written to save you\nfrom re-deriving the architecture by grepping.\n\n> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),\n> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),\n> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).\n\n---\n\n## 1. What PyGWalker is (30-second model)\n\nPyGWalker turns a pandas / polars / pyarrow dataframe into an interactive\n[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,\nand plain web servers. It has **two halves that ship together**:\n\n- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,\n  `to_html`), data parsing, and the transports that talk to the UI.\n- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles\n  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python\n  side at render time.\n\nThe Python side never renders charts itself; it hands built JS + serialized data to a\nnotebook/browser and then answers data/spec requests over a message channel.\n\n---\n\n## 2. Repo map\n\n| Path | What lives here |\n|------|-----------------|\n| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |\n| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |\n| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |\n| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |\n| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |\n| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |\n| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |\n| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |\n| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |\n\n---\n\n## 3. How the two halves fit together (build & load model)\n\n```\napp/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it\n```\n\n**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):\n\n| Bundle | Built from | Loaded by |\n|--------|-----------|-----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app\nbundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.\n\n**The message protocol is generated, not hand-written.** Python Pydantic models in\n`pygwalker/communications/protocol.py` are the source of truth. Running\n`python scripts/generate_comm_protocol_ts.py` regenerates\n`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild\nthe frontend.** Never edit `comm.generated.ts` by hand.\n\n**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).\n`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget\nand slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.\n\n---\n\n## 4. First-time setup\n\nRequires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.\n\n```bash\n# Python (editable install with dev extras)\npython -m venv venv && source venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# Frontend deps + one full build so pygwalker/templates/dist/ is populated\ncd app && yarn install && yarn build && cd ..\n```\n\n---\n\n## 5. Dev mode: edit the frontend and see it live (anywidget HMR)\n\nThe default `pyg.walk(df)` uses the anywidget transport, which loads\n`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source\nchange and (b) let anywidget hot-reload it into open widgets. **One command starts\neverything and captures all logs:**\n\n```bash\nsource venv/bin/activate\npython scripts/dev.py\n```\n\nThis launches, and tees the output of, two long-running processes:\n\n- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into\n  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`\n- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its\n  environment (inherited by kernels). → `logs/jupyter.log`\n\nThen, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:\n\n```python\nimport pandas as pd, pygwalker as pyg\npyg.walk(pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]}))\n```\n\nEdit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`\n(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.\nFor a heavy change you can always re-run the cell.\n\n**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set\nto a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.\nanywidget then reads that file and watches it (via `watchfiles`), pushing new code to the\nfrontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the\nbundle is embedded as a string exactly as before — production behavior is unchanged.\n\nUseful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),\n`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),\n`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).\n\n> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is\n> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).\n> Prefer the anywidget HMR path above.\n\n---\n\n## 6. Logs: one place to look\n\n`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:\n\n| File | Contents |\n|------|----------|\n| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |\n| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |\n| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |\n\n**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):\n\n- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.\n- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).\n\n**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the\n**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as\nin-app toast notifications rather than `console.log`.\n\nAgent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a\nrebuild finished, and `logs/pygwalker.log` for kernel-side errors.\n\n---\n\n## 7. Everyday commands\n\n```bash\n# Frontend (run from app/)\nyarn build            # full build: typecheck + all 4 bundles (CI-equivalent)\nyarn build:app        # fast: only the two app bundles, no typecheck\nyarn dev:build        # rebuild-on-change (what scripts/dev.py runs)\nyarn typecheck        # tsc --noEmit\nyarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)\n\n# Python (run from repo root, venv active)\npython -m ruff check pygwalker tests scripts bin pygwalker_tools\npython -m ruff format --check pygwalker tests scripts bin pygwalker_tools\npython -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests\npython -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests\n\n# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py\npython scripts/generate_comm_protocol_ts.py\n\n# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)\npython scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow\n```\n\n---\n\n## 8. Rules & gotchas\n\n- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel\n  build (and CI) rebuilds it via the Hatch jupyter-builder hook.\n- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`\n  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves\n  Python and JS out of sync.\n- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or\n  `yarn typecheck`) before pushing frontend changes.\n- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /\n  iframe path for new work; it is removed in 0.7.0.\n- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.\n  Never rely on them being set at runtime for end users.\n- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from\n  disk; if it is missing you'll get a clear \"Missing PyGWalker frontend asset\" error — run the\n  frontend build (or wait for `scripts/dev.py`'s first build to finish).\n\n---\n\n## 9. Where to look for X\n\n| Question | Start here |\n|----------|-----------|\n| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |\n| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |\n| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |\n| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |\n| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |\n| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |\n","isInternal":false,"tokens":3005,"sizeBytes":11099},{"name":"ARCHITECTURE.md","path":"docs/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/Kanaries/pygwalker/HEAD/docs/ARCHITECTURE.md","title":"docs - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# PyGWalker Architecture\n\nHow the project is constructed across its Python and frontend halves, and how they load and\ntalk to each other at runtime. This is the orientation doc for contributors and agents; for\nthe hands-on dev workflow see [`DEVELOPMENT.md`](./DEVELOPMENT.md), and for the quick map see\n[`../AGENTS.md`](../AGENTS.md).\n\n## Two halves that ship together\n\nPyGWalker is a Python library plus a React frontend, packaged as one wheel:\n\n- **Python package** — `pygwalker/`. The public API, dataframe parsing, and the transports\n  that exchange data and chart specs with the UI.\n- **Frontend app** — `app/`. A React + Vite application built into JavaScript bundles that are\n  placed in `pygwalker/templates/dist/` and shipped inside the wheel.\n\nAt render time the Python side serializes the dataframe + config and hands the built\nJavaScript to a notebook/browser. The running UI then calls back into the Python kernel to\nfetch data, run queries, and save chart specs.\n\n```\n┌────────────────────────┐        build         ┌──────────────────────────────┐\n│  app/  (React + Vite)  │ ───────────────────▶ │ pygwalker/templates/dist/*.js │\n└────────────────────────┘   vite build          └──────────────┬───────────────┘\n                                                                 │ read at runtime\n                                                                 ▼\n┌───────────────────────────────────────────────────────────────────────────────┐\n│  pygwalker/ (Python)   walk()/render()/table()/Walker → transport → rendered UI │\n└───────────────────────────────────────────────────────────────────────────────┘\n                    ▲                                            │\n                    │      messages (data, spec, export)         │\n                    └────────────────────────────────────────────┘\n```\n\n## Python package layout\n\n| Module | Responsibility |\n|--------|----------------|\n| `api/adapter.py` | Top-level `walk`/`render`/`table`. Detects environment (`get_current_env`) and routes to the notebook (`api/jupyter.py`) or web-server (`api/webserver.py`) path. |\n| `api/jupyter.py` | Notebook dispatch. Maps `env` → a display method (`env_display_map`). Default `JupyterAnywidget`. |\n| `api/walker.py` | `Walker`, the reusable object (`.show()`, `.to_html()`, `.to_streamlit()`). |\n| `api/pygwalker.py` | `PygWalker`, the core object holding data source, spec, and props. |\n| `services/anywidget_widget.py` | **Default transport.** Wraps the built ESM in an `anywidget.AnyWidget`. |\n| `services/render.py` + `templates/*.html` | Iframe / static-HTML transport (Jinja templates). |\n| `services/jupyter_display.py` | The concrete display flows (anywidget, iframe, widgets, preview, convert). |\n| `services/global_var.py` | Process-wide runtime settings (`GlobalVarManager`): component URL, API keys, feature flags. |\n| `communications/` | Kernel⇄frontend transports + the shared message `protocol.py`. |\n| `data_parsers/` | Adapters that normalize pandas/polars/pyarrow/SQL/spark sources to a common interface. |\n| `utils/frontend_assets.py` | Locates and loads the built bundles from `templates/dist/`. |\n| `utils/log.py` | The shared `pygwalker` logger (console + optional file). |\n\n## Frontend app layout\n\n| Path | Responsibility |\n|------|----------------|\n| `src/index.tsx` | Entry point. Exports `GWalker` (iframe/http), `AnywidgetGWalkerApp` (anywidget), Streamlit and preview apps. Selects a transport by `env`. |\n| `src/utils/communication.tsx` | The four client transports: Jupyter widgets, HTTP (Streamlit/Gradio/web), and anywidget. |\n| `src/dataSource/` | Receives chunked data from the kernel and delegates kernel computation. |\n| `src/interfaces/comm.generated.ts` | **Generated** protocol types (request/response envelopes, action map). Do not hand-edit. |\n| `src/store/` | MobX stores for UI state, notifications, and the active communication instance. |\n| `src/tools/` | Toolbar actions: save, export chart, export dataframe, open-in-desktop. |\n| `src/lib/` | Standalone transforms (`dslToWorkflow.ts`, `vegaToDsl.ts`) built as separate UMD bundles. |\n\n## Build pipeline\n\n**Frontend → bundles.** `app/vite.config.ts` defines the build. `yarn build` runs a typecheck\nthen produces four artifacts into `pygwalker/templates/dist/`:\n\n| Bundle | Entry | Consumer |\n|--------|-------|----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | anywidget transport (default notebook path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build:app` is a fast subset (only the two app bundles, no typecheck).\n\n**Protocol types are generated.** The message contract is defined once, in Python Pydantic\nmodels at `pygwalker/communications/protocol.py`.\n`python scripts/generate_comm_protocol_ts.py` renders those models into\n`app/src/interfaces/comm.generated.ts`. Change the Python models, regenerate, then rebuild the\nfrontend — otherwise the two sides drift.\n\n**Wheel build.** `pyproject.toml` wires a Hatch `jupyter-builder` hook that runs the frontend\n`build` during `python -m build`, so a released wheel always contains freshly built bundles.\n`pygwalker/templates/dist/` is git-ignored and rebuilt on demand; never commit it.\n\n## Runtime flow (default anywidget path)\n\n1. `pyg.walk(df)` → `adapter.walk` detects a notebook → `jupyter.walk` with the default\n   `env='JupyterAnywidget'`.\n2. A `PygWalker` is created; `jupyter.py`'s `env_display_map` calls\n   `display_on_jupyter_use_anywidget`.\n3. `services/anywidget_widget.py` builds a `WalkerAnyWidget` whose `_esm` is the built\n   `pygwalker-app.es.js`, sets `props` (serialized config + a data sample), and registers an\n   `AnywidgetCommunication` channel.\n4. In the browser, `AnywidgetGWalkerApp` mounts Graphic Walker and, as the user explores,\n   sends typed requests (fetch data, run SQL, save spec, export) back over the anywidget\n   channel. `communications/` validates each message against `protocol.py` and dispatches it.\n\nThe **iframe transport** (`services/render.py` + `templates/pygwalker_main_page.html`) is an\nalternative that embeds the compressed `iife` bundle in an `iframe` `srcdoc`, or points the\niframe at a dev-server URL when `GlobalVarManager.component_url` is set. It backs\n`to_html()` and the deprecated low-level `PygWalker.display_on_jupyter()` compatibility\nmethod. The legacy `env='Jupyter'`/`'JupyterWidget'` aliases are coerced to anywidget and do\nnot select the iframe transport.\n\n## Message protocol & transports\n\nEvery transport speaks the same envelope (see `protocol.py` /\n`comm.generated.ts`): a request carries an `action`, a `data` payload, a request id `rid`, and\nthe widget/app id `gid`; a response carries `code`, `data`, and `message`. Actions include\ndata fetches (`get_datas`, `batch_get_datas_by_sql`), spec operations\n(`get_latest_vis_spec`, `save_chart`, `update_spec`), export, and AI features.\n\n| Transport | Module | Mechanism |\n|-----------|--------|-----------|\n| anywidget (default) | `communications/anywidget_comm.py` | anywidget model `send`/`on_msg` |\n| iframe widgets | `communications/hacker_comm.py` | ipywidgets text fields observed by the frontend |\n| Streamlit | `communications/streamlit_comm.py` | HTTP route under Streamlit's server |\n| Gradio | `communications/gradio_comm.py` | Starlette mount |\n| Reflex | `communications/reflex_comm.py` | Reflex endpoint |\n| web server | `api/webserver.py` | standalone HTTP handler |\n\n## Computation modes\n\n`computation` selects where data queries run: `browser` (all data sent to the client and\ncomputed there), `kernel` (a local DuckDB engine in the Python process answers queries on\ndemand — best for large data), or `cloud` (Kanaries cloud). In `kernel` mode the frontend\nparses the Graphic Walker DSL to SQL (using the `gw-dsl-parser` WASM / the UMD bundles) and\nasks the kernel to execute it, so only aggregated results cross the boundary.\n\n## How dev mode changes asset loading\n\nNormally `WalkerAnyWidget._esm` is the **contents** of `pygwalker-app.es.js` read once at\nimport — a plain string embedded in the widget. When `PYGWALKER_DEV=1` or `ANYWIDGET_HMR=1`\nis set, `_esm` instead points at the **file path** of that bundle. anywidget then reads it from\ndisk and, under HMR, watches it and live-reloads the widget whenever `vite build --watch`\nrewrites the file. This is entirely opt-in: with the flags unset, loading is byte-for-byte the\nsame as before. See [`DEVELOPMENT.md`](./DEVELOPMENT.md) for the workflow and\n[`../AGENTS.md`](../AGENTS.md) for the one-command dev stack.\n","isInternal":false,"tokens":2177,"sizeBytes":9452},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/Kanaries/pygwalker/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\nThis project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a\nsingle source of truth for the architecture, dev-mode workflow, and log locations. Read it\nbefore making changes.\n\n@AGENTS.md\n\n## TL;DR for a coding agent\n\n- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e \".[dev]\"`,\n  then `cd app && yarn install && yarn build`.\n- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`\n  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).\n  Pass `--no-browser` for headless runs.\n- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),\n  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser\n  console.\n- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits\n  under `app/src/` hot-reload into the widget.\n- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`\n  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).\n- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,\n  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.\n","isInternal":false,"tokens":371,"sizeBytes":1302}],"systemPromptSnippet":"<yakaai_skills repo=\"Kanaries/pygwalker\">\n<!-- File: AGENTS.md (Tokens: ~3005 | Category: root-instruction) -->\n# AGENTS.md — PyGWalker contributor & agent guide\n\nThis is the fast-start map of the PyGWalker repo for both human contributors and coding\nagents. It explains how the project is put together, how to run it in **dev mode with live\nfrontend reload**, and where all the logs go. Read this first — it is written to save you\nfrom re-deriving the architecture by grepping.\n\n> Deeper references: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) (how it is built),\n> [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) (dev workflow + troubleshooting),\n> [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) (validation & CI).\n\n---\n\n## 1. What PyGWalker is (30-second model)\n\nPyGWalker turns a pandas / polars / pyarrow dataframe into an interactive\n[Graphic Walker](https://github.com/Kanaries/graphic-walker) UI inside notebooks, Streamlit,\nand plain web servers. It has **two halves that ship together**:\n\n- **Python package** (`pygwalker/`) — public API (`walk`, `render`, `table`, `Walker`,\n  `to_html`), data parsing, and the transports that talk to the UI.\n- **Frontend app** (`app/`, React + Vite) — the UI. It is compiled into JavaScript bundles\n  that are checked into the wheel under `pygwalker/templates/dist/` and loaded by the Python\n  side at render time.\n\nThe Python side never renders charts itself; it hands built JS + serialized data to a\nnotebook/browser and then answers data/spec requests over a message channel.\n\n---\n\n## 2. Repo map\n\n| Path | What lives here |\n|------|-----------------|\n| `pygwalker/api/` | Public entry points. `adapter.py` picks jupyter vs webserver; `jupyter.py` = notebook dispatch; `walker.py` = the reusable `Walker`; `pygwalker.py` = the core `PygWalker`. |\n| `pygwalker/services/` | Rendering + display. `anywidget_widget.py` (default transport), `render.py` + `templates/*.html` (iframe transport), `global_var.py` (runtime globals), `jupyter_display.py`. |\n| `pygwalker/communications/` | Kernel⇄frontend transports: `anywidget_comm.py` (default), `hacker_comm.py` (iframe), `streamlit_comm.py`, `gradio_comm.py`, `reflex_comm.py`. `protocol.py` is the shared message schema. |\n| `pygwalker/data_parsers/` | Dataframe/connector adapters (pandas, polars, pyarrow, SQL, spark…). |\n| `pygwalker/templates/dist/` | **Build output** (git-ignored). The JS bundles the Python side loads. |\n| `pygwalker/utils/` | Helpers: `frontend_assets.py` (locate/load bundles), `log.py` (logging), encoders. |\n| `app/src/` | Frontend source. `index.tsx` = entry; `utils/communication.tsx` = transports; `dataSource/` = data ingest; `interfaces/comm.generated.ts` = **generated** protocol types; `store/` = MobX state. |\n| `scripts/` | `dev.py` (dev orchestrator), `compile.sh` (build frontend), `local_ci.py` (mirror CI), `generate_comm_protocol_ts.py` (regenerate protocol types). |\n| `tests/` | Python tests + `*.ipynb` notebooks run by `nbmake`. `app/tests/` holds Playwright smoke tests. |\n\n---\n\n## 3. How the two halves fit together (build & load model)\n\n```\napp/src/*  --(vite build)-->  pygwalker/templates/dist/*.js  --(read at runtime)-->  Python renders it\n```\n\n**Frontend build variants** (`app/vite.config.ts`, output to `pygwalker/templates/dist/`):\n\n| Bundle | Built from | Loaded by |\n|--------|-----------|-----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | **anywidget** transport (the default `pyg.walk` path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / static `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build` builds all four (+ typecheck). `yarn build:app` builds only the two app\nbundles (fast, no typecheck) — good for a quick manual rebuild, not for CI.\n\n**The message protocol is generated, not hand-written.** Python Pydantic models in\n`pygwalker/communications/protocol.py` are the source of truth. Running\n`python scripts/generate_comm_protocol_ts.py` regenerates\n`app/src/interfaces/comm.generated.ts`. **If you change `protocol.py`, regenerate and rebuild\nthe frontend.** Never edit `comm.generated.ts` by hand.\n\n**Transports.** The default notebook transport is **anywidget** (`env='JupyterAnywidget'`).\n`env='Jupyter'` / `env='JupyterWidget'` are deprecated aliases that are coerced to anywidget\nand slated for removal in 0.7.0. Streamlit/Gradio/Reflex/web-server have their own transports.\n\n---\n\n## 4. First-time setup\n\nRequires **Python 3.10+**, **Node.js 22.x**, **Yarn 1.x**.\n\n```bash\n# Python (editable install with dev extras)\npython -m venv venv && source venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install -e \".[dev]\"\n\n# Frontend deps + one full build so pygwalker/templates/dist/ is populated\ncd app && yarn install && yarn build && cd ..\n```\n\n---\n\n## 5. Dev mode: edit the frontend and see it live (anywidget HMR)\n\nThe default `pyg.walk(df)` uses the anywidget transport, which loads\n`pygwalker-app.es.js` from disk. In dev mode we (a) rebuild that bundle on every source\nchange and (b) let anywidget hot-reload it into open widgets. **One command starts\neverything and captures all logs:**\n\n```bash\nsource venv/bin/activate\npython scripts/dev.py\n```\n\nThis launches, and tees the output of, two long-running processes:\n\n- **frontend** — `cd app && yarn dev:build` (`vite build --watch`) rebuilds the bundles into\n  `pygwalker/templates/dist/` on every edit under `app/src/`. → `logs/frontend.log`\n- **jupyter** — `jupyter lab` started with `PYGWALKER_DEV=1` and `ANYWIDGET_HMR=1` in its\n  environment (inherited by kernels). → `logs/jupyter.log`\n\nThen, in a notebook cell, **no special setup is needed** — just use PyGWalker normally:\n\n```python\nimport pandas as pd, pygwalker as pyg\npyg.walk(pd.DataFrame({\"x\": [1, 2, 3], \"y\": [4, 5, 6]}))\n```\n\nEdit a file under `app/src/`, wait for the rebuild to finish in `logs/frontend.log`\n(`built in …`), and the widget hot-reloads in place — usually without re-running the cell.\nFor a heavy change you can always re-run the cell.\n\n**Why it works:** with `ANYWIDGET_HMR=1` (or `PYGWALKER_DEV=1`), `WalkerAnyWidget._esm` is set\nto a `pathlib.Path` pointing at the built `pygwalker-app.es.js` instead of an embedded string.\nanywidget then reads that file and watches it (via `watchfiles`), pushing new code to the\nfrontend when `vite build --watch` rewrites it. With the flags **off** (normal installs), the\nbundle is embedded as a string exactly as before — production behavior is unchanged.\n\nUseful flags: `--no-jupyter` (only rebuild the frontend), `--no-frontend` (only Jupyter),\n`--jupyter-port N`, `--no-browser` (auto-enabled when output is not a TTY, e.g. agent runs),\n`--log-dir DIR`. Stop everything with **Ctrl+C** (services are torn down cleanly).\n\n> Alternative (Vite dev server + browser refresh, using the legacy iframe transport) is\n> documented in [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#appendix-vite-dev-server-iframe-transport).\n> Prefer the anywidget HMR path above.\n\n---\n\n## 6. Logs: one place to look\n\n`scripts/dev.py` writes everything under `logs/` (git-ignored) at the repo root:\n\n| File | Contents |\n|------|----------|\n| `logs/frontend.log` | Vite build/watch output — watch for `built in …` (rebuild done) and TypeScript errors. |\n| `logs/jupyter.log` | JupyterLab server output — **the URL + token to open the notebook is here**. |\n| `logs/pygwalker.log` | Kernel-side PyGWalker Python logs (set via `PYGWALKER_LOG_FILE`). |\n\n**Python log controls** (honored by `pygwalker/utils/log.py`, independent of the orchestrator):\n\n- `PYGWALKER_LOG_FILE=/path/to/file.log` — also append Python logs to a file.\n- `PYGWALKER_LOG_LEVEL=DEBUG` — raise/lower verbosity (default `INFO`).\n\n**Frontend runtime logs** (what happens in the browser, e.g. comm errors) appear in the\n**browser devtools console**, not in `logs/`. The frontend surfaces user-facing errors as\nin-app toast notifications rather than `console.log`.\n\nAgent tip: tail `logs/jupyter.log` for the server URL, `logs/frontend.log` to know when a\nrebuild finished, and `logs/pygwalker.log` for kernel-side errors.\n\n---\n\n## 7. Everyday commands\n\n```bash\n# Frontend (run from app/)\nyarn build            # full build: typecheck + all 4 bundles (CI-equivalent)\nyarn build:app        # fast: only the two app bundles, no typecheck\nyarn dev:build        # rebuild-on-change (what scripts/dev.py runs)\nyarn typecheck        # tsc --noEmit\nyarn test:front_end   # Playwright smoke test (run `yarn playwright install chromium` once)\n\n# Python (run from repo root, venv active)\npython -m ruff check pygwalker tests scripts bin pygwalker_tools\npython -m ruff format --check pygwalker tests scripts bin pygwalker_tools\npython -X faulthandler -W error::DeprecationWarning:pygwalker -m pytest -o faulthandler_timeout=60 tests\npython -m pytest --nbmake --nbmake-kernel=python tests/*.ipynb   # notebook tests\n\n# Regenerate the JS protocol types after editing pygwalker/communications/protocol.py\npython scripts/generate_comm_protocol_ts.py\n\n# Run the whole CI flow locally (frontend build + smoke test + notebooks + python)\npython scripts/local_ci.py            # add --skip-frontend / --skip-notebooks to narrow\n```\n\n---\n\n## 8. Rules & gotchas\n\n- **Do not commit `pygwalker/templates/dist/`** — it is generated and git-ignored. The wheel\n  build (and CI) rebuilds it via the Hatch jupyter-builder hook.\n- **Regenerate + rebuild after protocol changes.** Editing `communications/protocol.py`\n  without running `scripts/generate_comm_protocol_ts.py` and rebuilding the frontend leaves\n  Python and JS out of sync.\n- **`yarn build:app` skips typecheck** and the DSL bundles. Run full `yarn build` (or\n  `yarn typecheck`) before pushing frontend changes.\n- **anywidget is the transport of record.** Don't reach for the deprecated `env='Jupyter'` /\n  iframe path for new work; it is removed in 0.7.0.\n- **Dev-mode flags are opt-in.** `PYGWALKER_DEV` / `ANYWIDGET_HMR` only affect a dev session.\n  Never rely on them being set at runtime for end users.\n- **First run needs a build.** In dev mode the widget loads `dist/pygwalker-app.es.js` from\n  disk; if it is missing you'll get a clear \"Missing PyGWalker frontend asset\" error — run the\n  frontend build (or wait for `scripts/dev.py`'s first build to finish).\n\n---\n\n## 9. Where to look for X\n\n| Question | Start here |\n|----------|-----------|\n| How does `pyg.walk()` decide what to display? | `pygwalker/api/adapter.py` → `api/jupyter.py` (`env_display_map`) |\n| How is the frontend bundle loaded / dev-swapped? | `pygwalker/services/anywidget_widget.py`, `utils/frontend_assets.py` |\n| What messages can the frontend send the kernel? | `pygwalker/communications/protocol.py` ⇄ `app/src/interfaces/comm.generated.ts` |\n| How is data sent to the browser? | `pygwalker/services/data_communication.py`, `app/src/dataSource/` |\n| How is a chart exported to PNG/SVG/code? | `pygwalker/services/chart_export.py`, `app/src/tools/` |\n| How does the iframe/static-HTML path render? | `pygwalker/services/render.py` + `pygwalker/templates/*.html` |\n\n<!-- File: docs/ARCHITECTURE.md (Tokens: ~2177 | Category: project-spec) -->\n# PyGWalker Architecture\n\nHow the project is constructed across its Python and frontend halves, and how they load and\ntalk to each other at runtime. This is the orientation doc for contributors and agents; for\nthe hands-on dev workflow see [`DEVELOPMENT.md`](./DEVELOPMENT.md), and for the quick map see\n[`../AGENTS.md`](../AGENTS.md).\n\n## Two halves that ship together\n\nPyGWalker is a Python library plus a React frontend, packaged as one wheel:\n\n- **Python package** — `pygwalker/`. The public API, dataframe parsing, and the transports\n  that exchange data and chart specs with the UI.\n- **Frontend app** — `app/`. A React + Vite application built into JavaScript bundles that are\n  placed in `pygwalker/templates/dist/` and shipped inside the wheel.\n\nAt render time the Python side serializes the dataframe + config and hands the built\nJavaScript to a notebook/browser. The running UI then calls back into the Python kernel to\nfetch data, run queries, and save chart specs.\n\n```\n┌────────────────────────┐        build         ┌──────────────────────────────┐\n│  app/  (React + Vite)  │ ───────────────────▶ │ pygwalker/templates/dist/*.js │\n└────────────────────────┘   vite build          └──────────────┬───────────────┘\n                                                                 │ read at runtime\n                                                                 ▼\n┌───────────────────────────────────────────────────────────────────────────────┐\n│  pygwalker/ (Python)   walk()/render()/table()/Walker → transport → rendered UI │\n└───────────────────────────────────────────────────────────────────────────────┘\n                    ▲                                            │\n                    │      messages (data, spec, export)         │\n                    └────────────────────────────────────────────┘\n```\n\n## Python package layout\n\n| Module | Responsibility |\n|--------|----------------|\n| `api/adapter.py` | Top-level `walk`/`render`/`table`. Detects environment (`get_current_env`) and routes to the notebook (`api/jupyter.py`) or web-server (`api/webserver.py`) path. |\n| `api/jupyter.py` | Notebook dispatch. Maps `env` → a display method (`env_display_map`). Default `JupyterAnywidget`. |\n| `api/walker.py` | `Walker`, the reusable object (`.show()`, `.to_html()`, `.to_streamlit()`). |\n| `api/pygwalker.py` | `PygWalker`, the core object holding data source, spec, and props. |\n| `services/anywidget_widget.py` | **Default transport.** Wraps the built ESM in an `anywidget.AnyWidget`. |\n| `services/render.py` + `templates/*.html` | Iframe / static-HTML transport (Jinja templates). |\n| `services/jupyter_display.py` | The concrete display flows (anywidget, iframe, widgets, preview, convert). |\n| `services/global_var.py` | Process-wide runtime settings (`GlobalVarManager`): component URL, API keys, feature flags. |\n| `communications/` | Kernel⇄frontend transports + the shared message `protocol.py`. |\n| `data_parsers/` | Adapters that normalize pandas/polars/pyarrow/SQL/spark sources to a common interface. |\n| `utils/frontend_assets.py` | Locates and loads the built bundles from `templates/dist/`. |\n| `utils/log.py` | The shared `pygwalker` logger (console + optional file). |\n\n## Frontend app layout\n\n| Path | Responsibility |\n|------|----------------|\n| `src/index.tsx` | Entry point. Exports `GWalker` (iframe/http), `AnywidgetGWalkerApp` (anywidget), Streamlit and preview apps. Selects a transport by `env`. |\n| `src/utils/communication.tsx` | The four client transports: Jupyter widgets, HTTP (Streamlit/Gradio/web), and anywidget. |\n| `src/dataSource/` | Receives chunked data from the kernel and delegates kernel computation. |\n| `src/interfaces/comm.generated.ts` | **Generated** protocol types (request/response envelopes, action map). Do not hand-edit. |\n| `src/store/` | MobX stores for UI state, notifications, and the active communication instance. |\n| `src/tools/` | Toolbar actions: save, export chart, export dataframe, open-in-desktop. |\n| `src/lib/` | Standalone transforms (`dslToWorkflow.ts`, `vegaToDsl.ts`) built as separate UMD bundles. |\n\n## Build pipeline\n\n**Frontend → bundles.** `app/vite.config.ts` defines the build. `yarn build` runs a typecheck\nthen produces four artifacts into `pygwalker/templates/dist/`:\n\n| Bundle | Entry | Consumer |\n|--------|-------|----------|\n| `pygwalker-app.es.js` | `src/index.tsx` | anywidget transport (default notebook path) |\n| `pygwalker-app.iife.js` | `src/index.tsx` | iframe transport / `to_html()` |\n| `dsl-to-workflow.umd.js` | `src/lib/dslToWorkflow.ts` | kernel-side DSL→workflow conversion |\n| `vega-to-dsl.umd.js` | `src/lib/vegaToDsl.ts` | kernel-side Vega→DSL conversion |\n\n`yarn build:app` is a fast subset (only the two app bundles, no typecheck).\n\n**Protocol types are generated.** The message contract is defined once, in Python Pydantic\nmodels at `pygwalker/communications/protocol.py`.\n`python scripts/generate_comm_protocol_ts.py` renders those models into\n`app/src/interfaces/comm.generated.ts`. Change the Python models, regenerate, then rebuild the\nfrontend — otherwise the two sides drift.\n\n**Wheel build.** `pyproject.toml` wires a Hatch `jupyter-builder` hook that runs the frontend\n`build` during `python -m build`, so a released wheel always contains freshly built bundles.\n`pygwalker/templates/dist/` is git-ignored and rebuilt on demand; never commit it.\n\n## Runtime flow (default anywidget path)\n\n1. `pyg.walk(df)` → `adapter.walk` detects a notebook → `jupyter.walk` with the default\n   `env='JupyterAnywidget'`.\n2. A `PygWalker` is created; `jupyter.py`'s `env_display_map` calls\n   `display_on_jupyter_use_anywidget`.\n3. `services/anywidget_widget.py` builds a `WalkerAnyWidget` whose `_esm` is the built\n   `pygwalker-app.es.js`, sets `props` (serialized config + a data sample), and registers an\n   `AnywidgetCommunication` channel.\n4. In the browser, `AnywidgetGWalkerApp` mounts Graphic Walker and, as the user explores,\n   sends typed requests (fetch data, run SQL, save spec, export) back over the anywidget\n   channel. `communications/` validates each message against `protocol.py` and dispatches it.\n\nThe **iframe transport** (`services/render.py` + `templates/pygwalker_main_page.html`) is an\nalternative that embeds the compressed `iife` bundle in an `iframe` `srcdoc`, or points the\niframe at a dev-server URL when `GlobalVarManager.component_url` is set. It backs\n`to_html()` and the deprecated low-level `PygWalker.display_on_jupyter()` compatibility\nmethod. The legacy `env='Jupyter'`/`'JupyterWidget'` aliases are coerced to anywidget and do\nnot select the iframe transport.\n\n## Message protocol & transports\n\nEvery transport speaks the same envelope (see `protocol.py` /\n`comm.generated.ts`): a request carries an `action`, a `data` payload, a request id `rid`, and\nthe widget/app id `gid`; a response carries `code`, `data`, and `message`. Actions include\ndata fetches (`get_datas`, `batch_get_datas_by_sql`), spec operations\n(`get_latest_vis_spec`, `save_chart`, `update_spec`), export, and AI features.\n\n| Transport | Module | Mechanism |\n|-----------|--------|-----------|\n| anywidget (default) | `communications/anywidget_comm.py` | anywidget model `send`/`on_msg` |\n| iframe widgets | `communications/hacker_comm.py` | ipywidgets text fields observed by the frontend |\n| Streamlit | `communications/streamlit_comm.py` | HTTP route under Streamlit's server |\n| Gradio | `communications/gradio_comm.py` | Starlette mount |\n| Reflex | `communications/reflex_comm.py` | Reflex endpoint |\n| web server | `api/webserver.py` | standalone HTTP handler |\n\n## Computation modes\n\n`computation` selects where data queries run: `browser` (all data sent to the client and\ncomputed there), `kernel` (a local DuckDB engine in the Python process answers queries on\ndemand — best for large data), or `cloud` (Kanaries cloud). In `kernel` mode the frontend\nparses the Graphic Walker DSL to SQL (using the `gw-dsl-parser` WASM / the UMD bundles) and\nasks the kernel to execute it, so only aggregated results cross the boundary.\n\n## How dev mode changes asset loading\n\nNormally `WalkerAnyWidget._esm` is the **contents** of `pygwalker-app.es.js` read once at\nimport — a plain string embedded in the widget. When `PYGWALKER_DEV=1` or `ANYWIDGET_HMR=1`\nis set, `_esm` instead points at the **file path** of that bundle. anywidget then reads it from\ndisk and, under HMR, watches it and live-reloads the widget whenever `vite build --watch`\nrewrites the file. This is entirely opt-in: with the flags unset, loading is byte-for-byte the\nsame as before. See [`DEVELOPMENT.md`](./DEVELOPMENT.md) for the workflow and\n[`../AGENTS.md`](../AGENTS.md) for the one-command dev stack.\n\n<!-- File: CLAUDE.md (Tokens: ~371 | Category: claude-rule) -->\n# CLAUDE.md\n\nThis project's contributor & agent guide is maintained in **[AGENTS.md](AGENTS.md)** — a\nsingle source of truth for the architecture, dev-mode workflow, and log locations. Read it\nbefore making changes.\n\n@AGENTS.md\n\n## TL;DR for a coding agent\n\n- **Setup:** `python -m venv venv && source venv/bin/activate && pip install -e \".[dev]\"`,\n  then `cd app && yarn install && yarn build`.\n- **Run everything in dev mode with live reload + centralized logs:** `python scripts/dev.py`\n  (starts the frontend watch build + JupyterLab with `PYGWALKER_DEV=1` / `ANYWIDGET_HMR=1`).\n  Pass `--no-browser` for headless runs.\n- **Logs:** `logs/frontend.log` (build/watch), `logs/jupyter.log` (server URL + token),\n  `logs/pygwalker.log` (kernel-side Python logs). Frontend runtime logs are in the browser\n  console.\n- **In a notebook:** just `import pygwalker as pyg; pyg.walk(df)` — no special setup; edits\n  under `app/src/` hot-reload into the widget.\n- **Before pushing:** `python scripts/local_ci.py` (or, narrower, `yarn typecheck` + `yarn build`\n  in `app/`, and `ruff check`/`ruff format --check`/`pytest` from the root).\n- **Don't:** commit `pygwalker/templates/dist/`, hand-edit `app/src/interfaces/comm.generated.ts`,\n  or rely on `PYGWALKER_DEV`/`ANYWIDGET_HMR` at runtime for end users.\n</yakaai_skills>"}