# Technical Documentation: saulpw/visidata

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/saulpw/visidata)

### Overview (`README.md`)

# VisiData v3.4

[](https://github.com/saulpw/visidata/actions/workflows/main.yml)
[](https://gitpod.io/#https://github.com/saulpw/visidata)

[](https://visidata.org/chat)
[](https://fosstodon.org/@saulpw)

A terminal interface for exploring and arranging tabular data.



VisiData supports tsv, csv, sqlite, json, xlsx (Excel), hdf5, and [many other formats](https://visidata.org/formats).

## Platform requirements

- Linux, OS/X, or Windows (with WSL)
- Python 3.8+
- additional Python modules are required for certain formats and sources

## Install

To install the latest release from PyPi:

    pip3 install visidata

To try VisiData without installing, use [pipx](https://pipx.pypa.io/) or [uv](https://docs.astral.sh/uv/):

    pipx run visidata          # or: uvx visidata

To install permanently (adds `vd` to your PATH):

    pipx install visidata      # or: uv tool install visidata

Additional Python packages are needed for some formats:

    pipx install visidata --preinstall openpyxl --preinstall lxml
    # or: uv tool install visidata --with openpyxl --with lxml

To add format packages to an existing pipx install:

    pipx inject visidata openpyxl lxml

To install the cutting edge `develop` branch (no warranty expressed or implied):

    pip3 install git+https://github.com/saulpw/visidata.git@develop

If VisiData reports `package X not installed` even after you installed it, `vd`
is almost certainly running from a different environment than the one you
installed the package into. Check `which vd`, then install the package into
*that* environment — e.g. `pipx inject visidata X` for a pipx install, or the
Homebrew Python for `brew install visidata`.

See [visidata.org/install](https://visidata.org/install) for detailed instructions for all available platforms and package managers.

### Usage

    $ vd <input>
    $ <command> | vd

Press `Ctrl+Q` to quit at any time.

Hundreds of other commands and options are also available; see the documentation.

### Documentation

* [VisiData documentation](https://visidata.org/docs)
* [Plugin Author's Guide and API Reference](https://visidata.org/docs/api)
* [Quick reference](https://visidata.org/man) (available within `vd` with `Ctrl+H`), which has a list of commands and options.
* [Intro to VisiData Tutorial](https://jsvine.github.io/intro-to-visidata/) by [Jeremy Singer-Vine](https://www.jsvine.com/)

### Help and Support

If you have a question, issue, or suggestion regarding VisiData, please [create an issue on Github](https://github.com/saulpw/visidata/issues) or chat with us at #visidata on [irc.libera.chat](https://libera.chat/).

If you use VisiData regularly, please [support me on Patreon](https://www.patreon.com/saulpw)!

## License

Code in the `stable` branch of this repository, including the main `vd` application, loaders, and plugins, is available for use and redistribution under GPLv3.

## Credits

VisiData is conceived and developed by Saul Pwanson `<vd@saul.pw>`.

Anja Kefala `<anja.kefala@gmail.com>` maintains the documentation and packages for all platforms.

Many thanks to numerous other [contributors](https://visidata.org/credits/), and to those wonderful users who provide feedback, for helping to make VisiData the awesome tool that it is.

---

### Overview (`dev/README.md`)

# dev/

Resources for VisiData contributors and maintainers.

## Contributor Guides

| File | Description |
|------|-------------|
| `STYLE.md` | Code style guide |
| `GIT.md` | Git workflow and branching conventions |
| `DOCS.md` | Documentation standards |
| `PERFORMANCE.md` | Performance guidelines |
| `TESTING.md` | Test framework guide (golden tests, pytest, writing new tests) |

## Checklists

| File | Description |
|------|-------------|
| `checklists/release.md` | Release process checklist |
| `checklists/feature.md` | New feature checklist (docs, tests, marketing, sample data) |
| `checklists/manual-tests.md` | Manual testing checklist (cmdlog, replay, options, split window, etc.) |
| `checklists/add-command.md` | Adding a new command |
| `checklists/add-aggregator.md` | Adding a new aggregator |

## Design Docs

See [design/README.md](design/README.md) for full index.

## Scripts

| File | Description |
|------|-------------|
| `test.sh` | Run test suite (batched, fast; `-d` for debug; see `TESTING.md`) |
| `diff-test.sh` | Diff-based test runner |
| `run-tests-individually.sh` | Run each test in its own process (isolated, slower; see `TESTING.md`) |
| `mkman.sh` | Generate manpage |
| `mkpandas-df.py` | Generate pandas DataFrame test fixtures |
| `zsh-completion.py` | Generate zsh completions |

## Data Files

| File | Description |
|------|-------------|
| `formats.jsonl` | Format metadata (builds visidata.org/formats) |
| `types.jsonl` | Type system structure |

## Test Fixtures

| File | Description |
|------|-------------|
| `stdin.vdj` | Used in CI (`.github/workflows/main.yml`) |
| `quit.vdx` | Quit test fixture |
| `formats.vd` | Format test fixture |
| `vduplot.vdx` | Unicode plot test |

## Packaging

| File | Description |
|------|-------------|
| `debian/` | Debian packaging files |
| `build-container` | Container build script |
| `visidata-brew.rb` | Homebrew formula (stale — pins v1.2) |
| `requirements-dev.txt` | Dev dependencies |

## Media

| File | Description |
|------|-------------|
| `kb-blank.svg` | Blank keyboard layout template |
| `kb-layout.svg` | VisiData keyboard layout |
| `vdlogo.svg` | VisiData logo |

## Other

| File | Description |
|------|-------------|
| `zsh-completion.in` | Zsh completion template |
| `workshop-outline.md` | Workshop outline |

---

### Overview (`docs/README.md`)

# VisiData Documentation Index

Guide to what each file covers, so it's clear which files to update when user-facing behavior changes.

## Common mappings

- New/changed commands or keybindings → man.md, plus the relevant topic file
- New/changed options → customize.md, man.md
- New/changed file format support → formats.md
- New/changed column or row operations → columns.md, rows.md, edit.md
- New features may need a new doc file (add it to this README.md too)

## User Guides

- index.md: Getting started, links to demos and tutorials
- loading.md: Specifying source files, piping data in
- usage.md: Loading files, exiting, specifying file types
- navigate.md: Movement commands, vim-style keys, scrolling, searching
- move.md: Moving around and searching (FAQ-style)
- columns.md: Column manipulation: pin, move, hide/unhide, resize
- rows.md: Selecting, filtering, and toggling rows
- edit.md: Cell editing, bulk updates, regex substitution, expressions
- crud.md: Creating blank sheets, rows, and columns
- group.md: Grouping data, aggregators, descriptive statistics
- freq.md: Frequency tables, binning (discrete/numeric), pivot sheets
- join.md: Combining datasets via the Sheets Sheet
- graph.md: Drawing graphs, visualizing numeric variables
- graphics.md: Low-res terminal graphics, braille Unicode, Canvas classes
- colors.md: Color themes, interface color customization
- customize.md: .visidatarc, command bindings, Options Sheet, XDG support
- macros.md: Recording and replaying command sequences
- menu.md: Toplevel menubar, arrow key/hjkl navigation
- mouse.md: Mouse interaction options
- pipes.md: Using VisiData in stdin/stdout pipelines
- dirsheet.md: Directory Sheet (DirSheet): browsing, editing, previewing files
- split.md: Split screen, viewing two sheets simultaneously
- save-restore.md: Saving and replaying sessions
- plugins.md: Installing plugins (builtin, pip, local plugins directory)
- shell.md: ZSH completion
- formats.md: Supported file formats, loaders/savers, dependencies
- internal_formats.md: .vd, .vdj, .vdx (command logs) and .vds (sheet state) formats
- gmail.md: Gmail access via OAuth 2.0 setup
- man.md: Quick reference / man page with full command and option listing

## Developer/Contributor Docs

- viewtsv.md: Annotated minimal 25-line VisiData app example
- test.md: Functional testing with .vd scripts
- contributing.md: Checklists for submitting loaders and plugins

## API Reference (api/)

Plugin/extension author documentation (reStructuredText, built with Sphinx).

- api/index.rst: Plugin Authors Guide overview
- api/sheets.rst: BaseSheet, openPath/openSource
- api/loaders.rst: Creating new loaders
- api/commands.rst: Command architecture, adding commands
- api/columns.rst: Column calculation engine (calcValue, getValue, etc.)
- api/plugins.rst: Plugin file structure and publishing
- api/options.rst: Creating configurable options
- api/async.rst: @asyncthread, threading, responsive UI
- api/data.rst: Selected rows API
- api/interface.rst: Drawing functions, window dimensions
- api/canvas.rst: Canvas graphics primitives
- api/extensible.rst: Extensible base class, monkey-patching

## Other Assets

- casts/: Asciinema terminal recordings embedded in various docs
- assets/: Screenshots and images (DirSheet, color options, Gmail OAuth steps, etc.)

---

### Overview (`dev/design/README.md`)

# Design Docs

Architecture and feature design documents for VisiData. See also [175-design-terms.md](175-design-terms.md) for the vocabulary used in these docs.

| File | Topic | Status | Description |
|------|-------|--------|-------------|
| `000-notes.md` | Founding vision | Implemented | Original plan (Oct 2016): sheet model, vi navigation, join design, loaders, outputs |
| `160-longnames.md` | Command naming | Implemented | Naming conventions for command longnames (syscopy, dive, slide, etc.) |
| `169-settings.md` | Settings architecture | Implemented | 7-layer option resolution order, command/keybinding API, options API |
| `173-benchmark.md` | Benchmark use cases | Future | 4 benchmark tasks for measuring VisiData performance |
| `174-keycols.md` | Key column rules | Implemented | 8 rules governing key column behavior |
| `175-design-terms.md` | Design vocabulary | Reference | Types of design statements: "is", "ideally", "someday", "currently", "bonus" |
| `176-miscrules.md` | Implementation gotchas | Reference | Rules about asynccache/calcValue behavior |
| `181-benchmark-data.md` | Benchmark dataset | Future | Spec for fictional benchmark dataset with unicode, errors, control chars, "100 mini-puzzles" |
| `200-splitpane.md` | Split pane | Implemented (v3) | Design and post-mortem: original use cases vs actual implementation |
| `232-input.md` | User input API | Implemented | VisiData.input(), confirm(), choose(), editCell(), completer/history |
| `260-push.md` | Push/reload behavior | Implemented | Gotcha: don't assign members after push (reload may overwrite) |
| `path.md` | visidata.Path | Implemented | Path wrapper API, comparison with pathlib.Path |

---

### Overview (`visidata/apps/galcon/README.md`)

**Note: This code has not been updated to run with the latest VisiData.**


# Galactic Conquest in VisiData

Take over the galaxy with your warship fleet!
Starting with only one planet and 10 ships, you compete with any number of enemies for the most planets and thus galactic glory.

At first, most planets are inhabited by savages with only a few spaceships for battle.
Some planets have better pilots, and ships from those planets have a higher kill rate in battle.

To capture a planet, you need to send a strong enough fleet.  Deploy ships from your planets and conquer the galaxy!

Galactic Conquest (or "galcon" as it is sometimes called) is turn-based.
When all players have submitted their moves, the turn is over.
Ships arrive, battles are resolved, and planets produce more ships.
Then the next turn begins.

## Instructions

### Starting a game

The host has to start a server:

    ./galcon-server.py [-p <port>]

This will output a URL for players to connect to.  All players then start their clients and login:

    vd -f galcon http://your.ip.address:8080

Of course, the given port on `your.ip.address` must be open to all players.

The first time a player name is used, the password is set.
Players join the game if it has not yet started (each server only hosts one game).

Each player presses `Ctrl+S` when they are ready for the game to start.

Press `Alt+M` to go to the Map for this game, or `Alt+P` for the list of planets.
Before the game has started, any player can change `Alt+G`ame options or press `Alt+N` to generate a New map.
`Ctrl+R` will reload the current sheet from data on the server.

When the map and all players are ready, the game begins.

### Playing the game

From any sheet:

- `Alt+M` opens the planet map
- `Alt+P` opens the planet sheet
- `Alt+U` opens the queued deployments sheet (not yet sent to server)
- `Alt+D` opens the full deployments sheet
- `Alt+E` opens the events sheet
- `Alt+Q` allows players to quit the game
- `Ctrl+S` submits the pending deployments to the server and signals the player is done

On the Planets sheet:

- `m` to mark the destination planet for a player's ships
- `f` to indicate the source planet
    - The player is then prompted to specify how many ships they wish to deploy from the source planet to the marked planet
- `gf` to send the provided number of ships from all selected planets to the marked planet

On the Map sheet:

- The colour of the planet indicates which player currently 'owns' it
- White planets are neutral and currently not in anybody's control
- `v` to cycle through `name`, `prod`, `killpct` and `numships` fields
- `m` and `f` to mark and deploy on the map sheet, same as on the Planets sheet

On the Queued Orders sheet:

- `e` to modify an order
- `d` to delete an order

### Using the Dockerfiles

Create an ssh deploy key that has permissions for vdplus. Here, it is called `bluebird_docker`

```
# build the image for the galcon server
sudo docker build --build-arg SSH_PRIVATE_KEY="$(cat bluebird_docker)" -t galcon-server -f Dockerfile.galcon-server .
# run the galcon server
sudo docker run -d -p 7777:8080 --rm galcon-server

# get ip address of where galcon server is running
sudo docker inspect GALCON_SERVER_CONTAINER_NUMBER | jq '.[0].NetworkSettings.IPAddress'

# build the image for the client
sudo docker build --build-arg SSH_PRIVATE_KEY="$(cat bluebird_docker)" -t galcon-client -f Dockerfile.galcon-client .

# for each person that is playing, run the docker client
sudo docker run -e IPADDR="GALCON_SERVER_IP_ADDRESS" -it --rm galcon-client
```

---

### Overview (`visidata/apps/vdsql/README.md`)

# vdsql: [VisiData](https://visidata.org) for Databases

A VisiData interface for databases.

Powered by [Ibis](https://ibis-project.org).

## Features

- query data in VisiData from any supported backend
- compose complex queries using VisiData commands instead of writing SQL
- output resulting query in SQL, Substrait, or Python

## Requirements

- Python 3.10+
- VisiData 3.0+
- Ibis 12.0+

## Installation

### Install from pip

This installs both:
  - the usual `vd` with the vdsql plugin available (use `-f vdsql` to use Ibis instead of builtin loaders),
  - the `vdsql` script that acts identically to `vd` but will use Ibis instead of VisiData's builtin loader.

    pip install vdsql

### Install manually (cutting edge development)

    git clone git@github.com:saulpw/visidata.git
    cd visidata/visidata/apps/vdsql
    pip3 install .

### Install Ibis backends

To minimize dependencies, only the sqlite backend is included by default.
[Install other backends for Ibis](https://ibis-project.org/backends/) directly, and they will be supported automatically:

    pip install 'ibis-framework[postgres]'

## Usage

### Connecting to databases

    vdsql <file_or_url>
    vd -f ibis <file_or_url>

where `file_or_url` is any connection string supported by `ibis.connect()` or any of the filetypes and options that VisiData itself supports.

#### Connection examples

    vdsql foo.sqlite          # or .sqlite3, .db
    vdsql foo.duckdb          # or .ddb
    vdsql mysql://...
    vdsql postgres://...
    vdsql clickhouse://play:clickhouse@play.clickhouse.com/?secure=1
    vdsql bigquery:///bigquery-public-data

### Commands

A decent amount of work has gone into making `vdsql` work just like VisiData.

You can learn about VisiData starting with the [Intro to VisiData Tutorial](https://jsvine.github.io/intro-to-visidata/) and the [VisiData documentation](https://visidata.org/docs).

There are a few differences, however:

- `"` (dup-sheet) runs a new base query, including added columns, filtering for the current selection, and applying the current sort order.
- `z"` creates a new sheet with a different row limit.
- `gz"` removes the row limit entirely (fetch all rows).
- `'` casts the current column to its given type, persisting into future queries (with `"`).
- `g'` freezes the current set of loaded rows into a plain VisiData sheet, where all VisiData commands are available.

Some VisiData commands aren't implemented using the database engine.
The base VisiData commands can only use the loaded rows (500 by default), and this might be misleading, so most not-implemented commands are disabled.
If you want to use them anyway, knowing the dataset is incomplete, use `g'` to freeze the sheet first.

### Sidebar

`vdsql` uses the VisiData sidebar to show the SQL query for the current view.

- `b` to toggle the sidebar on/off
- `zb` to choose a sidebar option (pending SQL, base SQL, etc.)
- `gb` to open the sidebar as its own sheet

In this way you can compose a SQL expression using VisiData commands, open the SQL sidebar, and save the resulting query to a file (or copy it into your system clipboard buffer).

### Options

- `ibis_limit` (default: `500`) - max number of rows to fetch per query
- `postgres_schema` (default: `''`, public only) - which PostgreSQL schemas to show (space-separated list; use `*` for all non-system schemas)
- `sql_always_count` (default: `False`) - include total row count in every query
- `disp_ibis_sidebar` (default: `pending_sql`) - which sidebar property to display

## Supported Backends

### Confirmed

- SQLite
- MySQL
- PostgreSQL
- DuckDB
- ClickHouse
- Google BigQuery
- Snowflake

### Backend-specific notes

#### PostgreSQL

By default, only tables from the `public` schema are shown.

To show tables from specific schemas:

    vdsql --postgres-schema='myschema otherschema' postgres://...

To show tables from all non-system schemas:

    vdsql --postgres-schema='*' postgres://...

Note: quote or escape `*` on the command line to prevent shell glob expansion.

#### MySQL

- Requires `libmysqlclient-dev` (on Debian/Ubuntu)
- If you get a timezone warning: `mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql mysql`

### Other backends supported by Ibis

These backends are supported by Ibis and should work, but haven't specifically been tested with vdsql.
If you have problems connecting, please [file an issue](https://github.com/saulpw/visidata/issues/new).

- Apache Impala
- Datafusion
- Dask
- PySpark
- HeavyAI

# License

`vdsql` is licensed under the Apache 2.0 license.

Share and enjoy!

---

### Overview (`visidata/apps/vgit/README.md`)

# vgit

## latest version: v0.2-dev (2018-12-xx)

vgit is a [VisiData package](https://visidata.org) that provides a sleek terminal user interface to git, supplanting the arcana of git command-line operations with intuitive keystroke commands.

Current features include:
- viewing, stashing, staging, unstaging, and committing of diff hunks.
- viewing the git log (history)
- branch and remotes management
- branch merges and commit cherry-picks
- popping, applying, and dropping of stashed diffs
- setting of local and global git config options

## Installing vgit

vgit is located as a module within the [VisiData repo](https://github.com/saulpw/visidata/tree/develop/visidata/apps/vgit). It has its own setup.py. Note that installing vgit will also install VisiData.

```
pip install git+https://github.com/saulpw/visidata.git@branch#subdirectory=visidata/apps/vgit
```

### Dependencies

- Python 3.8+
- [VisiData](https://visidata.org/)
- [sh](https://github.com/saulpw/sh)

## Running vgit

`vgit [<working_dir>]` automatically opens a git status [sheet](https://www.visidata.org/docs/api/sheets#sheets) if the directory has a `.git` subdirectory.

Further documentation is available [here](vgit-guide.md).

## License

vgit is released under a GPLv3 license.

---

### Overview (`visidata/experimental/daw/README.md`)

# VisiData Audio Workstation

* current goal: automate getting functional transcript (80%+)
  so that i can actually make progress editing with vdaw
  which will close the loop and take vdaw to become a functional transcript audio editor

## getting the input transcript

Given a separate .wav file per speaker.

For each input .wav file:

1. Cut out the low volume audio, so the non-speaker doesn't get transcribed.

    ffmpeg -i mike.wav -af "agate=" mike-only.wav

2.  Use whisperx to get word-level timestamps.

    uvx whisperx --model base.en --task transcribe --hf_token ${HF_API_TOKEN} --highlight_words True --compute_type int8 --device cpu mike-only.wav

3.  (Optional) get full transcript from other source as .md to serve as the best/canonical edit.

    xmd2json.py transcript.md > transcript.json

4.  Then merge the per-speaker transcripts.

    merge_transcripts.py transcript.json mike-only.json saul-only.json > editable-transcript.json

5.  Load into VisiData using the `transcript` file format.

    vd editable-transcript.json -f transcript

## Saving changes

Modifications can be saved via `Ctrl+S` to the original .json file, or use a `.transcript` suffix for a bit more convenience (doesn't require the `-f` flag above).

### Export markdown

[Ideally VisiData should make it possible for a sheet to override a saver, so vdaw could define a save_md.]

For now, save as `foo.xmd` to get a markdown transcript with strikethrough for deleted lines.

## Editing Commands

### Playback
- `p` - Play from cursor row (skipping cuts)
- `P` - Play from cursor row (including cuts)
- `zp` - Toggle pause
- `[` / `]` - Seek backward/forward 10 seconds
- `g[` / `g]` - Seek backward/forward 60 seconds
- `gg` - Jump cursor to playhead
- `F5` - Slow down playback speed by 50%
- `F8` - Speed up playback 2x
- `Ctrl+R` - Restart mpv process

### Cutting/Keeping Audio
- `x` - Cut audio for line at cursor row
- `y` - Keep/upvote audio for line at cursor row
- `gx` - Cut audio for selected rows
- `gy` - Keep audio for selected rows

### Splitting & Combining
- `a` - Split line at current playhead
- `za` - Split line at word (prompts for word)
- `g)` - Combine selected rows into one
- `(` - Expand row into subrows
- `g(` - Expand selected rows

### Navigation
- `<` / `>` - Move to previous/next section header
- `g<` / `g>` - Move to first/last section header

### Text Formatting
- `r` - Reformat row (wrap text to column width)
- `gr` - Reformat selected rows

### Audio Filters
- `f` - Open audio filter parameters sheet
- `0-9` - Set filter parameter to preset value (in filter sheet or main sheet)

### Utilities
- `c` - Clean/interpolate bad word timings

## Applying ffmpeg filters

- Use vdaw to experiment with ffmpeg filters via the `f` command

     - agate
     - compand
     - loudnorm

---

### Index (`docs/index.md`)

---
eleventyNavigation:
    key: Getting Started
    order: 1
---

## Getting started

* [VisiData video demo](https://youtu.be/N1CBDTgGtOU)
* [Installation](/install)

## Tutorials

* Beginner: [An Introduction to VisiData](https://jsvine.github.io/intro-to-visidata/) by [Jeremy Singer-Vine](https://www.jsvine.com/)
* Beginner: [Guida VisiData](https://ondata.github.io/guidaVisiData/) by [Andrea Borruso](https://twitter.com/@aborruso) ([onData association](https://twitter.com/ondatait))
* Intermediate/Advanced (English): [VisiData case study videos](https://www.youtube.com/watch?v=yhunJc8Nu4g&list=PLxu7QdBkC7drrAGfYzatPGVHIpv4Et46W&index=4) by creator [Saul Pwanson](http://saul.pw)

## References

* [quick reference guide](/man)
    * all available commands and options
    * also available as a manpage via `man vd` and from inside VisiData with `Ctrl+H`
* [VisiData Cheat Sheet](https://jsvine.github.io/visidata-cheat-sheet/en/)

## 'How to' recipes

* [Loading Data](/docs/loading)
    * [Specifying a source file](/docs/loading#specifying-a-source-file)
    * [Loading sources supported by pandas](/docs/loading#loading-sources-supported-by-pandas)
    * [Opening an R data frame with VisiData](/docs/loading#opening-an-r-data-frame-with-visidata)
    * [Loading multiple datasets simultaneously](/docs/loading#loading-multiple-datasets-simultaneously)
    * [Accessing other loaded or derived sheets](/docs/loading#accessing-other-loaded-or-derived-sheets)
    * [Converting a dataset from one supported filetype into another](/docs/loading#convert)
* [Navigation](/docs/navigate)
    * rapidly scroll through a sheet
    * search within a sheet
    * move between sheets
* [Rows](/docs/rows)
    * perform operations on a subset of rows
    * filter rows
    * filter a random subset of rows
    * select rows where the current column is not null
    * select rows where the current column is null
    * move, copy and remove rows
    * sort rows
* [Columns](/docs/columns)
    * manipulate columns
    * hide (remove) and unhide columns
    * specify column types
    * split a column
    * expand columns that contain nested data
    * create derivative columns
    * configure multiple columns
* [Editing contents](/docs/edit)
    * edit cells
    * rename columns
* [Grouping data and descriptive statistics](/docs/group)
    * set statistical aggregators
    * create a pivot table
    * create a frequency chart
    * calculate descriptive statistics
    * filter for grouped or described rows
* [Creating sheets, rows and columns](/docs/crud)
    * configure the cursor to move right after a successful edit
    * set up a sheet for data collection
    * add a new blank column
    * fill a column with a range of numbers
* [Combining datasets](/docs/join)
    * perform a join
    * append two datasets
* [Drawing graphs](/docs/graph)
    * graph a single column
    * graph multiple columns
    * interact with graphs
* [How to save and replay a VisiData session](/docs/save-restore)
* [Customizing VisiData](/docs/customize)
    * configure VisiData (user)
    * configure VisiData (dev)
    * have configurations persist
    * configure commands
* [Plugins - extending VisiData functionality](/docs/api/plugins)
* [STDOUT pipe/redirect](/docs/pipes)

---

### Colors (`docs/colors.md`)

---
eleventyNavigation:
    key: Customizing VisiData - Colors
    order: 12
update: 2023-10-12
version: VisiData v3.0
---

## Color Options

Interface elements and colors can be set on the command-line before a session, or in .visidatarc to take effect for every session, or within VisiData on the Options Sheet (`Shift+O`).  Color options in particular start with `color_`:

    vd --color-selected-row="magenta"

or in .visidatarc:

    options.color_selected_row = "magenta"

On the Options Sheet, if you edit a color (or any display element), you will see the changes immediately, both in the options value (which is itself colorized) and also in the interface.  `Shift+U` to undo if you don't like it.



## Color Specification

Colors in the terminal have 3 components: *foreground*, *background*, and *attributes*.
In most modern terminals, each of the foreground and background colors can be one of 256 colors in a standard palette.

Several attributes are also available: `bold`, `italic`, `reverse`, and `underline`.  (There are other attributes like `dim` and `blink`, but they are not widely supported).

In VisiData, colors are specified with a simple language:

    <attribute> <fg-color> on <bg-color>

where `<attribute>` is a plain text name as above, and each `<color>` is a number from 0-255, or a basic color name from the original 8-color palette: `black red green yellow blue magenta cyan white`.
Multiple attributes may be specified.

Most terminals support 256 colors, but because some do not, it's possible to specify a *fallback color*, which is a second color that's used in case the first color isn't available.
For instance, `215 yellow` means to use the color `215` (kind of orange-brown), but if it's not available, use `yellow` instead.

Multiple colors may be applied to a single interface element with different precedence, for instance with [colorizers](docs/api/interface).
The color used will be the highest precedence background and the highest precedence foreground, which may come from separate color strings.  All attributes will be applied regardless of precedence--there is no way to specify a negative attribute or "no" attributes.

Any of the color components may be omitted, letting that component be given by a lower-precedence colorizer instead, ultimately falling back to `color_default` (`white on black` in the default theme).
Many color strings only specify a single attribute.

The ordering above--attributes first, then foreground, and finally background--is suggested and preferred.
Attributes may be specified anywhere in the color string; this does not change their effect (e.g. `bold` applies to the font rendered, not to the foreground or background color, even if `bold` text makes the foreground color seem brighter).
The `reverse` attribute flips the foreground and background color.  Multiple `reverse` attributes do not undo each other.

## 256 Terminal Colors and the Colors Sheet

To see all the available colors and what they look like in your specific terminal, run the `open-colors` command.  (This command is not bound to any keystroke, so must be executed by its longname; press `Space`, then type `open-colors`, then press `Enter`.)



The first several rows on this sheet are the existing color strings that are currently in use.
After these are the numeric colors; note that colors 0-7 correspond to the named colors above, with 0 being black and 7 being standard white.  Colors 8-15 are unnamed brighter versions of these colors: 8 is gray ("bright black") and 15 is bright white.

Colors 16-231 form a regular color cube with a wide variety of gradations covering the color space.
Colors 232-255 are gradients of white, with 232 being the darkest possible gray (almost indistinguishable from the black of 0 and 16), and 255 being almost the brightest white (though 15 is brighter).

## [Changing Display Attributes within a string](#attrs) {#attrs}

In many interface elements, it's possible to change the display attributes inline, within a single string.
For example, the menu bar has a message displayed in the upper right (which by default shows "Ctrl+H for help menu"), and is configurable with `options.disp_menu_fmt`.
The base menu color is configurable with `options.color_menu`, which is also applied to this menubar message.
However, this can be changed by specifying a new color within the string itself, using syntax like `[:red]`:

    options.disp_menu_fmt = "[:red on black]{vd.motd}"

A color option name can also be used instead of a real color; for example, to use the `color_error` color, use `[:error]`:

    options.disp_menu_fmt = "[:error]{vd.motd}"

Attribute changes can happen multiple times in a single string.
Use `[/]` to clear the last inline color change, and `[:]` to clear all inline attributes:

    options.disp_menu_fmt = "[:underline]Note:[/] [:error]{vd.motd}"

What follows the `/` is not checked, so these are all valid:

    [:underline]underlined text[/underline]
    [:underline]also underlined[/]
    [:underline]underline ends at end of string

### onclick

In addition to changing the display attributes, an `onclick` attribute can be given with this inline syntax, which specifies either a VisiData command to run, or a url to open (in `$BROWSER`), when the interface element is clicked:

    options.disp_menu_fmt = "[:onclick https://jsvine.github.io/intro-to-visidata/]Click here to go to the tutorial[/]"

Or if you want a custom VisiData toolbar some of your favorite actions:

    vd --disp_menu_fmt="[:onclick freq-col] freq out [/] | [:onclick quit-sheet] back [/]"

Elements with `onclick` are displayed with `color_clickable`, which is by default `underline`, which is commonly understood as a clickable affordance.

## Setting the Theme

A set of configured display elements and attributes may be packaged into a *theme*, which can be set as an option before VisiData starts (on the CLI or in .visidatarc), and can be changed with the `theme-input` command (in the menu under `View > Set theme`).

By default VisiData uses a dark-mode color scheme and some suggestive Unicode characters.
A few non-default themes are also packaged with VisiData:

- [`light`](https://github.com/saulpw/visidata/blob/develop/visidata/themes/light.py): light-mode color scheme
- [`ascii8`](https://github.com/saulpw/visidata/blob/develop/visidata/themes/ascii8.py): ascii characters and basic 8-colors only
- [`asciimono`](https://github.com/saulpw/visidata/blob/develop/visidata/themes/asciimono.py): ascii characters and default colors only

To use VisiData on a classic [DEC VT102](https://terminals-wiki.org/wiki/index.php/DEC_VT102) terminal:

    vd --theme=asciimono

[I would [love to know](https://github.com/saulpw/visidata/discussions/1533) of anyone using VisiData in such a lo-fi situation!]

### Adding a new theme

To add another theme, add it to the `vd.themes` dictionary in .visidatarc.
If more than one person uses it, [submit a PR](https://github.com/saulpw/visidata/pulls) to include it in the [visidata/themes](https://github.com/saulpw/visidata/tree/develop/visidata/themes) directory.

    vd.themes['awesome'] = dict(
        color_default      = 'black on white',  # the default fg and bg colors
        color_key_col      = '20 blue',   # color of key columns
        color_edit_cell    = '234 black',     # cell color to use when editing cell
        # ... etc
    )

## Colorizers

`TableSheet` sheets (which is any sheet with a row/column grid, i.e. most sheets) have *colorizers*, functions that can provide display attributes on individual rows, columns, or cells.

See [docs/api/interface](/docs/api/interface#colors) for how to use colorizers.

## Conditional Formatting

For ad-hoc conditional formatting from inside a session, three commands attach a colorizer to the current sheet that paints cells, rows, or columns wherever a Python expression is truthy:

- `color-cell` -- color individual cells (a CellColorizer)
- `color-row` -- color whole rows (a RowColorizer)
- `color-col` -- color whole columns (a ColumnColorizer)

Each command pops up a two-field form prompting for an `expr:` (the Python expression) and a `color:`.  The `color:` field is pre-filled with the first color from `red green yellow cyan magenta blue` (and their `on <color>` background variants) that isn't already in use on this sheet.

The `color:` field accepts any color string (`red`, `bold green`, `on red`, `215 yellow on 17`) or any `color_*` option name -- with or without the `color_` prefix.  So `error` resolves to `color_error`, `current_row` to `color_current_row`, etc., picking up whatever your theme defines for that role.

Like `addcol-expr` and `select-expr`, bare column names in the expression resolve to that column's typed value for the current row, so `temperature > 100` Just Works on a sheet with a `temperature` column.  In addition, four bindings are always available, each under a short and long name:

- `s` / `sheet` -- the sheet
- `c` / `col` -- the Column object the colorizer is being asked about
- `r` / `row` -- the row object
- `v` / `value` -- the typed value of the current cell

Examples (assuming a sheet with columns `name`, `score`, `status`):

    color-cell  red          value < 0
    color-cell  bold green   score >= 90
    color-row   on red       status == 'failed'
    color-col   underline    c.name.startswith('total_')

### Notes

- **Row colorizers evaluate per cell, not per row.**  The colorizer dispatcher walks each cell as it draws and asks every colorizer whether to apply.  Inside a `color-row` expression, `v` and `value` therefore refer to *the current cell being drawn*, not "the row's value" -- so a row-level rule like `value > 100` will only light up cells that match, not entire rows.  For real per-row rules, reference the row by bare column name (`status == 'failed'`).
- **Empty `color:` is meaningful.**  If you blank out the color field, the colorizer falls back to using whatever the expression itself returned as the color string.  That lets an expression compute the color dynamically -- e.g. `'red' if value < 0 else 'green' if value > 0 else ''` -- though in practice you'll want a literal color most of the time.
- **Performance.**  These commands use the existing colorizer infrastructure that already drives selected-row highlighting, key-column coloring, etc.  The expression is compiled once at command time and re-evaluated only on cells that get drawn, so the cost is bounded by the visible viewport, not the size of the sheet.
- **Type errors are silent.**  Cells where the expression raises an exception (e.g. `value > 50` on a column that has not been typed as a number, so `value` is still a string) are simply not colored.  If a `color-cell` rule isn't lighting anything up, set the column type first (`#` for int, `%` for float) or wrap the comparison (`int(value) > 50`).  Run vd with `--debug` to see the raised exception in the status log.
- **Colors from a column.**  When the colors are already data, `color-row-from-col` asks for a column of color strings (defaulting to the current column) and colors each row with its own value.  `color-cell-from-col` colors each cell in the current column from the cell in the same row of the color column; on column `foo`, the color column defaults to `foo_color`.  The color column can be hidden.
- **No persistence yet.**  Conditional-format colorizers live on the sheet instance for the session and are lost when the sheet is closed.  To re-apply them in a future session, save the commands as a macro or add them to your `.visidatarc`.

---

### Columns (`docs/columns.md`)

---
eleventyNavigation:
    key: Columns
    order: 5
update: 2021-11-01
version: VisiData 2.6
---

## How to manipulate columns

Commands(s)     Operation
------------    -----------
`!`             pins the current column on the left as a key column
 `H`  `L`       slides the current column **one position** to the left/right
`gH` `gL`       slides the current column **all the way** to the left/right of its section

---

## Increment by key

{help.commands.addcol-incr-key}

Mark grouping columns as keys with `!`.  Numbering restarts at `options.incr_base` whenever any key column changes in the current row order.  A returning key starts a new count.  With no key columns, numbering continues across all rows, like `i`.

## How to hide (remove) and unhide (return) columns

###### How to hide columns

- Press `-` (hyphen) to hide the current column.

**or**

1. Press `Shift+C` on the source sheet to open its **Columns sheet**.
2. Move the cursor right to the **width** column.
3. Move the cursor down to the row which represents the column you wish to hide.
4. Press `e` followed by `0`.
5. Press `q` to return to the source sheet.

Hidden columns are drawn as 1-cell columns to the right of the visible ones.  Hidden columns can be viewed without unhiding by moving the cursor onto them.  At the last visible column, press `go-right` (Right arrow or `l`) twice to move past the visible columns into the hidden columns.

###### How to unhide columns

1. Press `gv` to unhide all columns on current sheet.

**or**

1. Press `Shift+C` on the source sheet to open its **Columns sheet**.
2. Move the cursor right to the **width** column.
3. Move the cursor down to the row which represents the column you wish to unhide.
4. Press `e` followed by a *positive number* to set the width.
5. Press `q` to return to the source sheet.

---

## How to specify column types

Command    Type
--------- --------
` ~`      string
` #`      int
` %`      float
` $`      currency
` @`      date
`z#`      vlen
`z~`      anytype

Columns usually begin as untyped (`anytype`). Errors when working with numerical or datetime data is often due to values being considered as strings, and the problem is solved by setting the correct type.

The `float` type uses Python's builtin `float()` constructor to parse the string, and it parses by using the decimal separator.

The `currency` type is a bit of a misnomer. It filters out any non-numeric characters, and then parses the remainder of the cell value as a float.
The reasons to prefer using `float` over `currency`, is performance (it is quite a bit slower than native parsing with `float`), or if any non-float characters should get reported as an error.

The `date` type parses dates into ISO8601 format. Those columns can then be used in mathematical calculations, and the calculations are interpreted for dates. E.g. 2020-01-01 + 1, is 2020-01-02.

The `vlen` type formats the cell value to the length of the content. For example, if the cell content is a list of length 3, then when `vlen` typed it will display a value of 3.

There is also the `floatlocale` type, which uses Python's `locale.atof` to parse the Column values. With `floatlocale`, you can set the `LC_NUMERIC` environment variable appropriately (before launching VisiData), such that `atof()` will parse the number based on your locale setting. There is a `type-floatlocale` command, which is unbound by default, because parsing this way is significantly slower than using the builtin float type.

If you need locale-specific float parsing regularly, you may want to [rebind](/docs/customize) `%` or `z%` (or maybe some other keystroke) to `type-floatlocale` instead.

The following example uses the file [sample.tsv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/sample.tsv).

<div class="asciicast">
    <div id="player-types"></div>
    <script type="text/javascript" src="/asciinema-player.js"></script>
    <script type="text/javascript">
        AsciinemaPlayer.create('../casts/types.cast', document.getElementById('player-types'), { poster: 'npt:0:20', rows: 27 });
    </script>
</div>

###### How to batch specify column types for more than one column

1. Press `Shift+C` to open the **Columns sheet**.
2. Press `s` or `t` to select the rows referencing the columns you wish to type.
3. Type `g` followed by the any of the above typing keystrokes to set the type for all selected columns on the source sheet.

---

## How to format columns

**Note**: Un-typed file formats, like tsvs and csvs, will save as they are displayed.

Some types have an option for their default display formatting.

Type      Option            Default
--------- ----------------  --------
int       `disp_int_fmt`      `{:.0f}`
float     `disp_float_fmt`    `{:.02f}`
currency  `disp_currency_fmt` `%0.2f`
date      `disp_date_fmt`     `%Y-%m-%d`

Ways to adjust the display formatting:

* The `fmtstr` column on the **Columns Sheet** allows you to specify the formatting for specific columns within that session, without affecting the default for the others.
* The `disp_TYPE_fmt` option can be changed on the **Options Sheet** to set the formatting for all columns of type `TYPE` in that session.
* The `--disp-TYPE-fmt` argument can be passed through the commandline to set the formatting for all columns of type `TYPE` in that session.
* The `options.disp_TYPE_fmt` can be set in the `~/.visidatarc` to change the default formatting for all columns of type `TYPE` for all sessions.

There are several formatting styles offered:

* Formatting that starts with `'%'` (e.g. `%0.2f`) will use [locale.format_string()](https://docs.python.org/3.6/library/locale.html#locale.format_string).
* Otherwise (e.g. `{:.02f}`), formatting will be passed to Python's [string.format()](https://docs.python.org/3/library/string.html#custom-string-formatting).
* Date fmtstr are passed to [strftime](https://strftime.org/).

The default for currency uses `locale.format_string()`. The default for int/float/date use `string.format()`.

###### How to format a specific numeric columns to contain a thousands separator within a session?

1. Set a column to a numeric type by pressing `#` (int), `%` (float), or `$` (currency).
2. Press `Shift+C` to open the **Columns Sheet**.
3. Move to the row referencing the column whose display you wish to format. Move the cursor to the fmtstr column.
4. Type `e` followed by `{:,.0f}` for an `int` type and `{:,.02f}` for a floating point type.

###### How to quickly adjust the precision of a float or date?

1. Ensure the column is typed as numeric column: float (`%`), floatsi (`z%`), dirty float (`$`), or date (`@`).
2. Press `Alt+Plus` or `Alt+Minus` to adjust the precision in the current column.
3. Or, press `Space` and use the longname `setcol-precision-more` or `setcol-precision-less`.

###### How to automatically set the type of a particular column

When loading sheets repeatedly with the same schema, it can be useful to pre-set the column types (and other metadata) in .visidatarc using the `knownCols` attribute on the Sheet class and its subclasses.

For example, to set the "timestamp" column in every sheet to the `date` type:

`Sheet.knownCols.timestamp.type = date`

Or to hide the `nCols` column on the SheetsSheet by default:

`SheetsSheet.knownCols.nCols.width = 0`

###### How to format all date columns as **month/day/year**.

The above method can be used to explicitly set the default `fmtstr` for specifically-named columns.
However, to use a default fmtstr for all date-typed columns, set the `disp_date_fmt` option.  In .visidatarc:

~~~
options.disp_date_fmt = '%m/%d/%Y'
~~~

or pass as a commandline argument:

~~~
vd --disp-date-fmt='%m/%d/%Y'
~~~

or set in the **Options Sheet**.

1. Press `Shift+O` to open the **Options Sheet**.
2. Move the cursor down to the relevant `disp_date_fmt` option.
3. Type `e` followed by `%m/%d/%Y`.

---

###### How to specify a comma decimal separator when typing floating point numbers?

1. Before launching VisiData, set the shell environment variable `LC_NUMERIC` to a locale which interprets commas as decimals. Any European locale should do; an example that works is `en_DK.UTF-8`.
2. Within VisiData, set a column to type `floatlocale` by pressing `Space` followed by `type-floatlocale`.

Note that `type-floatlocale` is significantly slower than `type-float`. However, if you wish to replace the current binding for `type-float` with `type-floatlocale`, add to your `~/.visidatarc`:

~~~
Sheet.unbindkey('%')
Sheet.bindkey('%', 'type-floatlocale')
~~~

or if you never use `type-floatsi`, you can do


~~~
Sheet.unbindkey('z%')
Sheet.bindkey('z%', 'type-floatlocale')
~~~

## How to split a column

Python regular expressions provide more finetuned column splitting. The following example
uses the commands for column splitting and transformation with [xd/puzzles.tsv](http://xd.saul.pw/xd-metadata.zip).

<div class="asciicast">
    <div id="player-split-regex"></div>
    <script type="text/javascript">
        AsciinemaPlayer.create('../casts/split-regex.cast', document.getElementById('player-split-regex'), { poster: 'npt:0:20', rows: 27 });
    </script>
</div>

###

- `:` adds new columns derived from splitting the current column at positions defined by a *regex pattern*. `options.default_sample_size` (default: 100) rows around the cursor will be used to determine the number of columns that will be created.
- `;` adds new columns derived from pulling the contents of the current column which match the *regex within capture groups*. The new columns are named using the capture group index, or if named capture groups are used, the capture group names. This command uses the `options.default_sample_size` (default:100) rows around the cursor as sample rows.
- `*` followed by *regex* and then `Tab` followed by the *replacement* replaces the text matching the *regex* with the contents of *replacement*. *replacement* may include backreferences to capture groups in the regex (`\1` etc).

## How to substitute text in a column

The `*` command can be used to do content transformations of cells. The `g*` variant transforms in-place, instead of creating a new column.

The following example uses [benchmarks.csv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/benchmarks.csv).

**Question** Transform the **SKU** values of *food* to *nutri*.

1. Move cursor to **SKU** column.
2. Press `gs` to select all rows.
3. Press `g*` to replace text in selected rows in current column.
4. Type `food` in the search field.
5. Press `Tab` to go to the replace field, then type `nutri`.
6. Press `Enter` to replace the text.

- tests/transform-cols.vd

---

## [How to expand columns that contain nested data](#expand) {#expand}

If a column includes container data such as JSON objects or arrays, the `(` family of commands can expand the child values into top-level columns:

Command       Operation
---------     --------
`  (`         expand _current_ column
` g(`         expand _all visible_ columns fully
` z(`         expand _current_ column to a specific depth (prompt for input)
`gz(`         expand _all visible_ columns to a specific depth (prompt for input)
`  )`         contract (unexpand) the current column
` zM`         expand _current_ column row-wise within that column

The following demo shows `(` commands applied to this data:

~~~
[
    [ "short", "array" ],
    [ "slightly", "longer", "array" ],
    { "nested": "data" },
    { "more": { "deeply": { "nested": "data" } } }
]
~~~

<div class="asciicast">
    <div id="player-expand-cols"></div>
    <script type="text/javascript">
        AsciinemaPlayer.create('../casts/expand-cols.cast', document.getElementById('player-expand-cols'), { poster: 'npt:0:20', rows: 13 });
    </script>
</div>

Note that by default the expansion logic will look for nested columns in **up to** `options.default_sample_size` (Default: 100) **rows surrounding the cursor**. This behavior can be controlled by adjusting `default_sample_size` in the **Options Sheet**, or setting `options.default_sample_size` in the `~/.visidatarc` file.

---

## [How to create derivative columns](#derived) {#derived}

The `=` command takes a Python expression as input and creates a new column, where each cell evaluates the expression in the context of its row.

To specify the column name, use `name=expr` (e.g. `total = Units * Unit_Cost`).  If no name is given, the expression itself is used as the column name.

These variables and functions are available in the scope of an expression:

- **Column names** evaluate to the typed value of the cell in the named column for the same row.
- **`vd`** attributes and methods; use `Ctrl+X vd` to view the vd object, or [see the API]().
- **`Sheet`** attributes and methods; use `g Ctrl+Y` to view the sheet object (or see the API).
- **Global** functions and variables (add your own in your .visidatarc).
- **modules** that have been `import`ed in Python
  - if you need a module that hasn't already been imported at runtime, use `g Ctrl+X import <modname>`.

- **`sheet`**: the current sheet (a TableSheet object)
- **`col`**: the current column (as a Column object; use for Column metadata)
- **`row`**: the current row (a Python object of the internal rowtype)
- **`curcol`**: evaluate to the typed value of this row in the column that the cursor was on at the time that the expression column was added.
- **`cursorCol`**: evaluate to the typed value of this row for the column the cursor is on. Changes as the cursor moves for `=`. Uses the column from the time the calculation was made for `g=`, `gz=`, and `z=`.
- **`currow`**: a convenience object for accessing column values of the current cursor row by name. e.g. `currow.Price` returns the typed value of the `Price` column for the cursor row.

Additional attributes can be added to sheets and columns.

`col` deliberately returns a Column object, but any other Column object is interpreted as the value within that column for the same row. For example, both `curcol` and `cursorCol` return values, not the object itself.


For example, this customizes addcol-expr to set the `curcol` attribute on the new ExprColumn to a snapshot of the current cursor column (at the time the expression column is added):

```
Sheet.addCommand('=', 'addcol-expr', 'addColumnAtCursor(ExprColumn(inputExpr("new column expr="), curcol=cursorCol))', 'create
 new column from Python expression, with column names as variables')
```

Then, an expression can use `curcol` as though it referred to the value in the saved column.

`Tab` autocompletion when inputting an expression will cycle through valid column names only.

The following examples use the file [sample.tsv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/sample.tsv).

**Question** On which days have we sold more than 10 **Item**s?

1. Scroll to the **Units** column. Set the type of the **Units** column by pressing `#` (int).
2. Type `=` followed by `Units > 10`. A new column will be created. The cells in this column will contain the value **True** in rows where the number of **Units** are greater than 10 and **False** otherwise.
3. Move to the new derived column.
4. Type `|` followed by `True` to select all rows where there were more than 10 **Units** sold.
5. Press `"` to open a duplicate sheet with only those selected rows.

**Question** I have a dataset with separate columns for **Year**, **Month** and **Day**. How can I concatenate them into a single date column?

1. Type `=` followed by `Year + '-' + Month + '-' + Day`.
2. Set the type of the new derived column by pressing `@` (date).
3. Type `^` followed by `Date` to rename the column to **Date**.

**Question** I have a dataset with **Date** column that is missing a prefix of '2020-'. How do I add it to the **Date** column?

When using `=`, and wanting to reference the current column, we recommend using `curcol`. When using `g=`, `gz=`, and `z=`, we recommend cursorCol. `=`, unlike the others, is dynamic and changes with adjustment of underlying values, which means it will change along with the movement of the cursor (tracked by `cursorCol`). `curcol` is a special attribute of a new **ExprColumn**, which remembers the cursorCol at the time of creation.

1. Move the cursor to **Date**.
2. Type `g=` followed by *f"2020-{cursorCol}"*.

**Question** I have a dataset with **file names**. How do I create a new column with the **file names** lower cased?

1. Move the cursor to **file names** column.
2. Type `=` followed by `curcol.casefold()`.
3. Move to the newly created column, and rename it with `^`, followed by the desired name.

---

## How to configure multiple columns

{help.commands.columns_all}

Hidden columns are included. Edit a hidden column's `width` to a positive number to unhide it on its source sheet.

Properties of columns on the source sheet can be changed by using [standard editing commands](/man#edit) on its **Columns sheet** (accessed with `Shift+C`). In particular, it facilitates the selection of multiple columns, followed by utilising one of the `g`-prefixed commands to modify all of them.

For a full list of available commands, see the [man page](/man#columns). Some example workflows follow.

The following examples use the file [sample.tsv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/sample.tsv).

###### How to set multiple statistical aggregators

**Question** What is the average daily revenue from sales of each **Item**?

1. Set the type of the **Units** column by pressing `#` (int).
2. Set the type of the **Total** column by pressing `%` (float).
3. Press `Shift+C` to open the **Columns sheet**.
4. Press `s` or `t` on the rows referencing the source sheet **Units** column and the **Total** column to select them.
5. Type `g+` followed by `avg` to add a **avg** statistical aggregator to the selected rows.
6. Press `q` to exit and return to the source sheet.
7. Scroll to the **Item** column. Press `Shift+F` to open the **Frequency table**.

**Question** What are the daily average and sum total number of **Units** sold for each **Item**?

1. Press `Shift+C` to open the **Columns sheet**.
2. Move the cursor to the row referencing the source sheet **Units** column.

    a. Press `s` or `t` to select it.
    b. Set the type for the source sheet **Units** columns by pressing `g#` (int).
    c. Move the cursor to the **aggregators** column.
    d. Type `e` to enter edit mode, followed by *sum avg*.

3. Press `q` to exit and return to the source sheet.
4. Move the cursor to the **Item** column. Press `Shift+F` to open the **Frequency table**.

---

---

### Contributing (`docs/contributing.md`)

---
eleventyNavigation:
  key: Checklists for Contributing to VisiData
  order: 99
---

## [Submitting a Core Loader](#loader) {#loader}
A deeper explanation of all of these steps can be found [in the loaders api documentation](https://www.visidata.org/docs/api/loaders.html).

- Create an `open_foo` function that returns the new `FooSheet`.
- Set an appropriate `rowtype` string.
- Provide a `# rowdef: ` comment. This describes the structure of a row (e.g. its base class) in a **Sheet**.
- If the loader's dependencies are not part of Python3 stdlib, note the additional dependencies in the `requirements.txt` (include a comment adjacent to the dep with the name of the loader).
- Check in a small sample dataset, in that format, to the `sample_data` folder
- Add a `load-foo.vd` to `tests/`. `load-foo.vd` should simply open the checked-in sample dataset and have a simple interaction if the source includes multiple tables.
- After replaying `vd load-foo.vd`, save the final sheet as `load-foo.tsv`. Save `load-foo.tsv` in `tests/golden/`.
- add a section on the loader to the [formats.jsonl](https://github.com/saulpw/visidata/blob/develop/dev/formats.jsonl).

## [Submitting an External Plugin](#plugins) {#plugins}
A deeper explanation of all of this framework can be found [in the plugins api documentation](https://www.visidata.org/docs/api/plugins.html).

- Host a single Python file containing all of the plugin's code.
- In the https://github.com/visidata/dlc/blob/stable/plugins.jsonl file in the `visidata:dlc` repo, add a row for each plugin with all of the necessary information.

---

### Crud (`docs/crud.md`)

---
eleventyNavigation:
    key: Creating sheets, rows and columns
    order: 8
Update: 2021-11-18
Version: VisiData 2.7.1
---



## How to set up a sheet for data collection

1. Type `Shift+A` to open a new blank sheet with one column.
2. Press

    a. `a` to add one blank row.

    **or**

    b. `ga` followed by a *number* to add that many blank rows.

---

## How to add a new blank column

1. Press

    a. `za` to add one blank column.

    **or**

    b. `gza` followed by a *number* to add that many blank columns.

2. Press `^` to edit the column name.

---

## How to fill a column with a range of numbers

1. Press `gs` to select all rows (or use other commands to select a subset of rows to fill).
2. Move the cursor to the column to be filled.
3. Press `gi`.

## How to edit a cell

1. Press `e` to edit an individual cell.
2. Type in the new value.
3. Press `Enter` to accept the value.

---

## How to move the cursor to the next cell after a successful edit

1. Press `Shift+Arrow` to accept the current value and move the cursor into edit mode in the next cell.

---

---

### Customize (`docs/customize.md`)

---
eleventyNavigation:
    key: Customizing VisiData
    order: 12
Updated: 2023-11-18
Version: VisiData v3.0
---

For a primer on configuring VisiData through setting options, see [jsvine's tutorial](https://jsvine.github.io/intro-to-visidata/advanced/configuring-visidata/).

## How to configure commands {#commands}

The **.visidatarc** in the user's home directory is plain Python code, and can contain additional commands or key bindings.

(Alternatively, since v2.9, VisiData has [XDG support](https://github.com/saulpw/visidata/pull/1420). If `$XDG_CONFIG_HOME` is set and the file `"$XDG_CONFIG_HOME"/visidata/config.py` exists, this will be loaded as the user's default configuration file.)

Longnames are names given to executable commands for ease of keystroke remapping. For example, the longname `select-row` is assigned to commands which select the current row in a sheet. On default, this longname is bound to the keystroke `s`.

From within VisiData, type `z Ctrl+H` to open the **Commands Sheet**. This is a reference for all of the commands available on the current sheet. For a deeper exploration of commands, check out [API reference manual](https://www.visidata.org/docs/api/commands.html).

### Setting/changing keybindings for existing commands

1. Learn the longname for a command. Longnames are usually 2-3 words, separated by hyphens. The first word is usually a verb, and the second usually a noun. When a command is executed, its longname appears in the lower right status, next to its keystroke. Alternatively, you can `z Ctrl+H` to open the **Commands Sheet** and discover the longname for the command in question.



2. a) To create a global keybinding, add `bindkey(keystroke, longname)` to your **.visidatarc**.
b) To set the binding for a particular sheet type, add `<Sheet>.bindkey(keystroke, longname)` to your **.visidatarc**, where `<Sheet>` is a **SheetType**.

~~~
Warning: bindings defined in a .visidatarc will overwrite default ones.
~~~

#### Example: Bind `i` to edit-cell globally

In VisiData, pressing `e` enters edit mode for the current cell. Seasoned vim users might prefer to press `i` instead.

1. Open `~/.visidatarc` in an editor.
2. Add the line `TableSheet.bindkey('i', 'edit-cell')` to globally bind the keystroke `i` to the longname `edit-cell`.
3. Launch VisiData, and press `i`.

#### Example: Unbind `i` from addcol-incr globally

If the above instructions are followed, a message will pop up that says "`i` was already bound to `addcol-incr`.

To unbind `i` before binding it:

1. Open `~/.visidatarc` in an editor.
2. Add the line `TableSheet.unbindkey('i')` before any piece of code where it is re-bound.
3. Launch VisiData.


### Customizing cell editing keybindings

`bindkey` changes keybindings in normal (non-editing) mode.  To change keybindings during cell editing, use `vd.editCellBindings` with `acceptThenFunc()`:

~~~
vd.editCellBindings['Enter'] = acceptThenFunc('go-down', 'edit-cell')
~~~

`acceptThenFunc()` takes one or more command longnames.  It saves the current edit, then executes each command in sequence.  For example, to make `Enter` save and move down to edit the next cell (spreadsheet-style):

~~~
vd.editCellBindings['Enter'] = acceptThenFunc('go-down', 'edit-cell')
~~~

See [Editing Contents](/docs/edit) for the full list of default cell editing keybindings.

### Creating new commands

At minimum, `<Sheet>.addCommand` requires a longname and execstr.

For example, to define a new command:

~~~
Sheet.addCommand('Ctrl+D', 'scroll-halfpage-down', 'cursorDown(nScreenRows//2); sheet.topRowIndex += nScreenRows//2')
~~~

To define a command interactively (without editing `.visidatarc`), use the `define-command` command.  The input is `longname execstr` (separated by the first space).  For example: `define-command show-item status(currow.Item)`.

Commands and keybindings are set on a particular Sheet Type in the class hierarchy. Use `BaseSheet` for commands which don't need a sheet at all--these will apply to all sheets.  Commands and bindings on more specific sheets will override more generic ones.  `Sheet` is a generic table, `ColumnsSheet` would be for the columns sheet, `FreqTableSheet` for frequency tables, and so on.

### Environment variables

`vd.setenv('LC_NUMERIC', 'en_US.UTF-8')` in `.visidatarc` sets the numeric locale and records it in saved command logs.
See [Environment and locale](save-restore.md#environment-and-locale).

### Adding custom aggregators {#aggregators}

Aggregators allow you to gather the rows within a single column, and interpret them using descriptive statistics. VisiData comes pre-loaded with a default set like mean, stdev, and sum.

To add your own custom aggregator `name`, add the following to your `.visidatarc`.

vd.aggregator('name', func, type=float)

Where `func` is a function of the form:

```
def func(list):
    return value
```

The `type` parameter is optional. It allows you to define the default type of the aggregated column.

Here is an example, that adds an aggregator for [numpy's internal rate of return](https://numpy.org/devdocs/reference/generated/numpy.irr.html) module.

```
import numpy as np
vd.aggregator('irr', np.irr, type=float)
```

**Bonus: How to choose which aggregators are columns within the DescribeSheet?**

Any numeric aggregator can be added!

Supply a space-separated list of aggregator names to `options.describe_aggrs` in your .visidatarc.

```
options.describe_aggrs = 'mean stdev irr'
```

### Saving options from the Options Sheet {#save-options}

Options edited on the Options Sheet (`Shift+O`) can be saved to the config file with `z Ctrl+S`.
Only options that differ from the default and aren't already in the config file will be appended.

### Turning off motd {#motd}

By default, the first time each day that VisiData is used, it downloads a single small file of startup messages.

This network request can be turned off by adding `options.motd_url=''` to your `~/.visidatarc`.

If you do decide to turn it off, we encourage you to [donate](https://www.patreon.com/saulpw/posts) to [support VisiData](https://github.com/sponsors/saulpw).

---

### Dirsheet (`docs/dirsheet.md`)

---
eleventyNavigation:
  key: Directory Sheet
  order: 99
---

# Directory Sheet (DirSheet)

Open any directory in VisiData to get a **DirSheet**: a browsable, editable view of the filesystem.

~~~
vd .
vd /path/to/directory
~~~

Within VisiData, use `open-dir-current` to open the current working directory.

## Columns

DirSheet shows these columns by default:

Column      Description
------      -----------
directory   parent directory path
filename    file name (key column)
ext         file extension (`/` for directories)
size        file size in bytes
modtime     last modification time

These columns are available but hidden by default (unhide with `-`):

Column      Description
------      -----------
abspath     absolute file path
owner       file owner (Unix)
group       file group (Unix)
mode        file permissions (octal)
filetype    output of `file --brief` (computed async)

Files are sorted by modification time (newest first) by default.

## Opening files

Keystroke(s)        Command                 Action
------------        -------                 ------
`Enter`             `open-row-file`         open file at cursor as a new sheet
`g Enter`           `open-rows`             open all selected files as new sheets
`z Enter`           `open-row-filetype`     open file at cursor, prompting for filetype
`` ` ``             `open-dir-parent`       open parent directory

VisiData detects the filetype from the file extension and opens the appropriate loader (csv, tsv, json, etc.).  Directories open as nested DirSheets.

## File preview

The `open-preview` command opens the file at the cursor in a split pane.  As you navigate the file list, the preview updates automatically to show the current file.

- Tables (csv, tsv, json, etc.) display as proper table sheets
- Text files display as TextSheet
- Directories display as nested DirSheet
- Preview sheets for visible rows are preloaded in the background

`open-preview` is unbound by default; add a keybinding in `.visidatarc`:

~~~
DirSheet.addCommand('p', 'open-preview', 'sheet.previewFile(cursorRow)')
~~~

Close the preview with `g Shift+Z` (close split).

## External editors

Keystroke(s)        Command                 Action
------------        -------                 ------
`Ctrl+O`            `sysopen-row`           open file in `$EDITOR`
`g Ctrl+O`          `sysopen-rows`          open selected files in `$EDITOR`

## Editing the filesystem

DirSheet uses **deferred mode**: edits are staged and applied all at once with `z Ctrl+S`.

### Renaming and moving files

Edit the **filename** column (`e`) to rename a file.  Edit the **directory** column to move it to a different directory (the destination is created if it doesn't exist).

### Deleting files

Mark rows for deletion with `d`.  Commit with `z Ctrl+S`.

- By default (`options.safety_first`), directory deletion uses `os.rmdir` (fails if not empty).
- With `safety_first` disabled, directory deletion uses `shutil.rmtree`.

### Editing metadata

Edit the **size**, **modtime**, **owner**, **group**, or **mode** columns directly.  Changes are applied on commit.

## Copying files

Keystroke(s)        Command                 Action
------------        -------                 ------
`y`                 `copy-row`              copy file at cursor to a destination directory
`gy`                `copy-selected`         copy selected files to a destination directory

## Shell integration

Keystroke(s)        Command                 Action
------------        -------                 ------
`z;`                `addcol-shell`          add column from shell command, with `$columnName` variables

Shell columns run a command for each row, interpolating `$columnName` references with column values.  Both stdout and stderr are captured as separate columns.

The prompt also asks for the minimum seconds between starting commands (0 for no delay), to e.g. stay under an API rate limit.  At most `options.max_threads` commands run at once.  `Ctrl+C` (`cancel-sheet`) kills the running commands.

## Options

Option          Default     Description
------          -------     -----------
`dir_depth`     `0`         folder recursion depth (0 = only immediate children)
`dir_hidden`    `False`     whether to show hidden files (dotfiles)

Options must be set before loading; reload the sheet after changing them.

## FileListSheet

Opening a `.fdir` file creates a **FileListSheet**: a DirSheet populated from a list of file paths (one per line) instead of a directory listing.

---

### Edit (`docs/edit.md`)

---
eleventyNavigation:
  key: Editing Contents
  order: 6
Update: 2018-08-19
Version: VisiData 1.3.1
---



## How to edit cells

For a summary of all editing commands, see the [man page](/man#edit).

Command                    Operation
--------                   ----------
 `e`                       edit contents of **current cell**
`ge` *text*                set contents of **current column for selected rows** to *text*
`g*` *regex*/*subst*       replace matching *regex* in **current column for selected rows** with *subst*
`g=` *expr*                evaluate Python *expr* over each selected row and set **current column** to the result

## Bulk corrections

`ge` unifies inconsistent values for the same logical item (e.g. "visidata", "Visidata", "VisiData", "vd") across selected rows. The initial input seeds from the current cell.

1. Select the rows with inconsistent values.
2. Move the cursor to a row with the correct value; it need not be selected.
3. Type `ge`, edit the value if needed, and press `Enter`.

All selected cells in the current column take the corrected value.

A Frequency Table (`Shift+F`) makes the distinct values easy to see. Editing a key cell there propagates the correction back to the source sheet (see [note](#note) below).

## note!

Modifications made to rows on derived sheets will be reflected on the source sheets.  This includes the Frequency Table: editing the key column there will change all instances on the source sheet, and if that sheet is derived from another source sheet, it will be reflected there, and so on.

This does not apply to adding or deleting rows, only changes to existing rows.

## Commands while editing

While in editing mode, or anytime VisiData expects input (with e.g. `=`, `;`), typical readline commands become available:

Command             Operation
--------            ----------
`Enter`             accepts input
`Ctrl+C`/`Esc`      aborts input
`Ctrl+O`            opens external $EDITOR to edit contents
`Ctrl+R`            reloads initial value
`Ctrl+A`/`Ctrl+E`   moves to beginning/end of line
`Backspace`         deletes previous character
`Up`/`Down`         sets contents to previous/next in history
`Tab`/`Shift-Tab`   autocompletes input (when available)

### Cell navigation while editing

When editing a cell (with `e`), these additional commands move to an adjacent cell and continue editing:

Command             Operation
--------            ----------
`Shift+Down`/`Shift+Up`       saves and moves to cell below/above
`Shift+Right`/`Shift+Left`    saves and moves to cell right/left
`Tab`/`Shift-Tab`             saves and moves to cell right/left

`Tab` wraps to the next row when at the last column; `Shift-Tab` wraps to the previous row when at the first column.

These bindings can be customized; see [Customize](/docs/customize#customizing-cell-editing-keybindings).

---

## How to rename columns

Command     Operation
--------    ----------
  `^`       edits name of **current** column
 `g^`       sets names of **all unnamed visible** columns to contents of **selected** rows (or **current** row)
 `z^`       sets name of **current** column to contents of **current** cell
`gz^`       sets name of **current** column to combined contents of **current** column for **selected** rows

In most cases, `^` is the preferred command. Examples which demo `^` can be seen in [Columns](/docs/columns#derived) and [Group](/docs/group#frequency).

###### How to set the header in an Excel sheet?

For most filetypes (e.g. csv, tsv, xls(x)) the loaders assume that the dataset's first `options.header` rows contain the column names.

If the Excel file has multiple sheets with varying number of header rows:

1. Pass `--header=0` while loading the file.

~~~
vd file.xlsx --header=0
~~~

2. For each sheet, press `s` or `t` to select the rows which represent the header rows.
3. Press `g^` to set the names of the headers to the contents of selected rows.

###### How to rename columns using the Columns sheet

1. Press `Shift+C` to open the **Columns sheet**.
2. Within the **name** column, move the cursor to the row which represents the source sheet.
3. Type `e` and then input *the new column name*. Press `Enter`.
4. Press `q` to return to the source sheet and see the renamed column.

---

---

### Formats (`docs/formats.md`)

---
eleventyNavigation:
    key: Supported Formats
    order: 99
---

| filetype              | format                                         | VisiData_loader | VisiData saver               | version_added | created | creator                                                       | PyPI dependencies    |
| --------------------- | ---------------------------------------------- | --------------- | ---------------------------- | ------------- | ------- | ------------------------------------------------------------- | -------------------- |
| [csv](#csv)           | Comma\-Separated Values                        | 0\.28           | displayed text               | 0\.28         | 1972    |                                                               |                      |
| [json](#json)         | Javascript Object Notation \(JSON\)            | 0\.28           | typed                        | 0\.28         | 2001    | Douglas Crockford                                             |                      |
| [tsv](#tsv)           | Tab\-Separated Values                          | 0\.28           | displayed text               | 0\.28         |         |                                                               |                      |
| xlsx                  | Excel spreadsheets                             | 0\.28           |                              | 0\.28         | 1987    | Microsoft                                                     | openpyxl             |
| zip                   | ZIP archive format                             | 0\.28           |                              | 0\.28         | 1989    | PKWARE                                                        |                      |
| hdf5                  | Hierarchical Data Format                       | 0\.28           |                              | 0\.28         | 199x    | NCSA                                                          | h5py                 |
| [sqlite](#sqlite)     | sqlite                                         | 0\.42           |                              | 0\.42         | 2000    | D\. Richard Hipp                                              |                      |
| [duckdb](#duckdb)     | DuckDB                                         | 3\.5            |                              | 3\.5          | 2019    | DuckDB Foundation                                             | duckdb               |
| xls                   | Excel spreadsheets                             | 0\.42           |                              | 0\.42         | 1987    | Microsoft                                                     | xlrd                 |
| [fixed](#fixed)       | fixed width text                               | 0\.97           |                              | 0\.97         |         |                                                               |                      |
| [postgres](#postgres) | PostgreSQL database                            | 0\.97           |                              | 0\.97         | 1996    |                                                               |                      |
| [imap](#imap)         | Internet Message Access Protocol               | 2\.12           |                              | 2\.12         | 1988    |                                                               |                      |
| [vd](#vd)             | VisiData command log (TSV)                     | 0\.97           | yes                          | 0\.97         | 2017    | VisiData                                                      |                      |
| [mbtiles](#mbtiles)   | MapBox Tileset                                 | 0\.98           |                              | 0\.98         | 2011    | MapBox                                                        | mapbox\-vector\-tile |
| pbf                   | Protocolbuffer Binary Format                   | 0\.98           |                              | 0\.98         | 2011    | OpenStreetMap                                                 |                      |
| [shp](#shp)           | Shapefile geographic data                      | 0\.98           |                              | 0\.98         | 1993    | ESRI                                                          | pyshp                |
| [html](#html)         | HTML tables                                    | 0\.99           | displayed text               | 0\.99         | 1996    | Dave Raggett                                                  | lxml                 |
| md                    | markdown table                                 |                 | displayed text               | 1\.1          | 2008    |                                                               |                      |
| [png](#png)           | Portable Network Graphics \(PNG\) image        | 1\.1            | from png                     | 1\.1          | 1996    | PNG Development Group                                         | pypng                |
| [ttf](#ttf)           | TrueType Font                                  | 1\.1            |                              | 1\.1          | 1991    | Apple                                                         | fonttools            |
| [dot](#pcap)          | Graphviz diagram                               |                 | from pcap                    | 1\.2          | 1991    |                                                               |                      |
| dta                   | Stata                                          | 1\.2            |                              | 1\.2          | 1985    | StataCorp                                                     | pandas               |
| [geojson](#shp)       | Geographic JSON                                | 2\.2            | yes \(from shp and geojson\) |               | 2008    | http://geojson\.org/                                          |                      |
| sas7bdat              | Statistical Analysis System \(SAS\)            | 1\.2            |                              | 1\.2          | 1976    | SAS Institute                                                 | sas7bdat             |
| sav                   | SPSS statistics                                | 1\.2            |                              | 1\.2          | 1968    | SPSS Inc                                                      |                      |
| spss                  | SPSS statistics                                | 1\.2            |                              | 1\.2          | 1968    | SPSS Inc                                                      | savReaderWriter      |
| xpt                   | Statistical Analysis System \(SAS\)            | 1\.2            |                              | 1\.2          | 1976    | SAS Institute                                                 | xport                |
| [jsonl](#json)        | JSON Lines                                     | 1\.3            | typed                        | 1\.3          | 2013    | Ian Ward                                                      |                      |
| [pandas](#pandas)     | all formats supported by pandas library        | 1\.3            |                              | 1\.3          | 2008    | Wes McKinney                                                  | pandas               |
| parquet               | Apache Parquet                                 | 1\.3            | yes                          | 3\.0          | 2013    | Apache Software Foundation                                    | pyarrow or pandas    |
| [pcap](#pcap)         | network packet capture                         | 1\.3            |                              | 1\.3          | 1988    | LBNL                                                          | dpkt dnslib          |
| pyprof                | Python Profile data                            | 1\.3            |                              | 1\.3          |         |                                                               |                      |
| [xml](#xml)           | eXtensible Markup Language \(XML\)             | 1\.3            | from xml                     | 1\.3          | 1998    | W3C                                                           | lxml                 |
| yaml                  | YAML Ain't Markup Language \(YAML\)            | 1\.3            |                              | 1\.3          | 2001    | Clark Evans                                                   | PyYAML               |
| frictionless          | Frictionless Data                              | 2\.0            |                              | 2\.0          |         | OpenKnowledge Institute                                       | datapackage          |
| jira                  | JIRA/Confluence table markup                   |                 | displayed text               | 2\.0          |         | Atlassian                                                     |                      |
| npy                   | NumPy array format                             | 2\.0            | typed                        | 2\.0          |         |                                                               | numpy                |
| tar                   | Unix Tape Archive                              | 2\.0            |                              | 2\.0          |         |                                                               |                      |
| usv                   | Unicode\-Separated Value                       | 2\.0            | displayed text               | 2\.0          | 1993    | Unicode                                                       |                      |
| xlsb                  | Excel binary format                            | 2\.0            |                              | 2\.0          |         | Microsoft                                                     | xlrd                 |
| [vdj](#vd)            | VisiData command log (JSON)                    | 2\.0            | yes                          | 2\.0          | 2020    | VisiData                                                      |                      |
| [mysql](#mysql)       | MySQL                                          | 2\.0            |                              |               | 1995    | MySQL AB                                                      | MySQLdb              |
| pdf                   | Portable Document Format                       | 2\.0            |                              |               | 1993    | Adobe                                                         | pdfminer\.six        |
| vcf                   | Virtual Contact File \(vCard\)                 | 2\.0            |                              |               | 1995    | Versit Consortium                                             |                      |
| rec                   | recutils database file                         | 2\.0            | displayed text               |               | 2010    | Jose E\. Marchesi                                             |                      |
| eml                   | Multipurpose Internet Mail Extensions \(MIME\) | 2\.0            |                              |               | 1996    | Nathaniel Borenstein and Ned Freed                            |                      |
| [vds](#vd)            | VisiData Sheet                                 | 2\.2            | yes                          | 2\.2          | 2021    | VisiData                                                      |                      |
| ods                   | OpenDocument Spreadsheet                       | 2\.7            |                              |               | 2006    | [OASIS](<https://en.wikipedia.org/wiki/OASIS_(organization)>) | odfpy                |
| lsv                   | awk-like key-value line-separated values       | 2\.7            |                              | v2\.7         |         |                                                               |                      |
| arrow                 | Arrow IPC file format                          | 2\.9            |                              |               | 2016    | Apache Software Foundation                                    | pyarrow              |
| arrows                | Arrow IPC streaming format                     | 2\.9            |                              |               | 2016    | Apache Software Foundation                                    | pyarrow              |
| [vdx](#vd)            | VisiData command log (text)                    | 2\.11           | yes                          | 2\.11         | 2022    | VisiData                                                      |                      |
| mailbox               | All formats supported by mailbox               | 3\.0            |                              |               | 1974    |                                                               | mailbox              |
| jrnl                  | CLI journal                                    | 3\.0            | yes                          | 3\.0          | 2012    | Micah Jerome Ellison                                          |                      |
| [reddit](#api)        | Reddit API                                     | 3\.0            |                              |               | 2005    |                                                               | praw                 |
| [matrix](#api)        | Matrix API                                     | 3\.0            |                              |               | 2014    | The Matrix.org Foundation                                     | matrix\_client        |
| [zulip](#api)         | Zulip API                                      | 3\.0            |                              |               | 2012    | Kandra Labs, Inc                                              | zulip                |
| [airtable](#api)      | Airtable API                                   | 3\.0            |                              |               | 2012    |                                                               | pyairtable           |
| orgmode               | Emacs Orgmode format                           | 3\.0            | yes                          | 3\.0          | 2003    | Carsten Dominik                                               |                      |
| s3                    | Amazon S3 paths and objects                    | 3\.0            |                              |               | 2006    | Amazon                                                        | s3fs                 |
| fec                   | Federal Election Commission                    | 3\.0            |                              |               |         | Federal Election Commission                                   | fecfile              |
| f5log                 | Parser for f5 logs                             | 3\.0            |                              |               |         | f5                                                            |                      |
| toml                  | Tom's Obvious Minimal Language                 | 3\.0            |                              |               |         | Tom Preston-Werner                                            | tomllib              |
| conll                 | CoNLL annotation scheme                        | 3\.0            |                              |               |         | Conference on Natural Language Learning                       | pyconll              |
| [grep](#grep)         | grep command-line utility                      | 3\.1            |                              |               | 1973    | AT&T Bell Laboratories                                        |                      |

# Extra notes about formats

## tsv (Tab Separated Values), as simple as it gets

- delimiter: field delimiter to use for tsv/usv filetype (default: .)
- row_delimiter: row delimiter to use for tsv/usv filetype (default: \n)
- tsv_safe_newline: replacement for newline character when saving to tsv (default: )
- tsv_safe_tab: replacement for tab character when saving to tsv (default: .)

Use `-f usv` for Unicode separators U+241F and U+241E.
Use `-f tsv` for awk-like records.
Use `--delimiter=` (an empty string) to make '\0' the value separator.
Use `--row-delimiter=` to make '\0' the row separator.

## csv (Comma Separated Values) for maximum compatibility {#csv}

.csv files are a scourge upon the earth, and still regrettably common.
All `csv_*` options are passed unchanged into csv.reader() and csv.writer().

- csv_dialect: dialect passed to csv.reader (default: excel)
- Accepted dialects are `excel-tab`, `unix`, and `excel`.
- csv_delimiter: delimiter passed to csv.reader (default: ,)
- csv_quotechar: quotechar passed to csv.reader (default: ")
- csv_skipinitialspace: skipinitialspace passed to csv.reader (default: True)
- csv_escapechar: escapechar passed to csv.reader (default: None)
- csv_lineterminator: lineterminator passed to csv.writer (default: \n)

## Saving TSV/CSV files

- save_filetype: specify default file type to save as (default: tsv)
- safety_first: sanitize input/output to handle edge cases, with a performance cost (default: False)

## Useful options for text formats in general

- regex_skip: regex of lines to skip in text sources (default: )
- save_encoding: encoding passed to codecs.open when saving a file (default: utf8)

## fixed {#fixed}
- loader-specific options
    - `fixed_rows` (default: 1000) number of rows to detect fixed width columns from
    - `fixed_maxcols` (default: 0) max number of fixed-width columns to create (0 is no max)

## json {#json}
- loader-specific options
    - `json_indent` (default: None) indent to use when saving json
    - `json_sort_keys` (default: False) sort object keys when saving to json
    - `default_colname` (default: '') column name to use for non-dict rows
- Cells containing lists (e.g. `[3]`) or dicts (e.g. `{3}`) can be expanded into new columns with `(` and unexpanded with `)`.
- All expanded subcolumns must be closed (with `)`) to retain the same structure.
- Support for jsonla was added in 3.0.

## xml {#xml}
- `v` show only columns in current row attributes
- `za` add column for xml attributes
- `dive-xml-table` open current element's children as a table, with attributes and child values as columns

## pcap {#pcap}
- loader-specific options
    - `pcap_internet` (default: 'n') (y/s/n) if save_dot includes all internet hosts separately (y), combined (s), or does not include the internet (n)

## postgres {#postgres}
- loader-specific options
    - `postgres_schema` (default: 'public') the desired schema for the Postgres database
- `vd postgres://`*username*`:`*password*`@`*hostname*`:`*port*`/`*database* opens a connection to the given postgres database.
    - any part may be omitted; libpq fills it from `PGUSER`, `PGPASSWORD`, `PGHOST`, `PGPORT`, or `~/.pgpass`, and `postgres:///`*database* connects over the local Unix socket.
- `vd postgres:///`*database*`/`*table* opens a single table directly.
- `exec-sql` on the tables sheet or a table opens a sheet with the results of a SELECT query; `edit-sql` edits that query.

## imap {#imap}
- `vd "imap://user@domain.com:passwordhere@imap-mailserver.com"` opens a connection to the IMAP server
    - e.g.  `vd "imap://someone@hotmail.com:pass123@imap-mail.outlook.com:993"`
    - e.g.  `vd "imap://someone@gmail.com@imap.gmail.com"`
        - note that you don't specify a password for gmail here -- instead, you will be prompted to follow some instructions

### using VisiData as a pager within psql

In psql:

~~~
\pset format csv
\pset pager always
\setenv PSQL_PAGER 'vd -f csv'
\pset pager_min_lines
~~~

## sqlite {#sqlite}
- supports saving for CREATE/INSERT (not wholesale updates)
- `z Ctrl+S` to commit any `add-row`/`edit-cell`/`delete-row`

## duckdb {#duckdb}
- loader-specific requirements
    - `duckdb` python module in path or virtual environment
- supports saving for CREATE/INSERT
- `z Ctrl+S` to commit deferred `add-row`/`edit-cell`/`delete-row` changes back to the source database

## mysql {#mysql}
- loader-specific requirements
    - working mysql / mariadb installation or at least the `libmysqlclient-dev` package (ubuntu; name might be different on other platforms)
    - `mysqlclient` python module in path or virtual environment (`pip install mysqlclient`)
- `vd mysql://`*username*`:`*password*`@`*hostname*`:`*port*`/`*database* opens a connection to the given mysql / mariadb database.

## html {#html}
- loader-specific options
    - `html_title` (default: `'<h2>{sheet.name}</h2>'`) table header when saving to html
- load all `<table>`s in a web page as VisiData sheets.


## shp {#shp}
- Can be edited in raw data form. Images can be plotted with `.` (dot).
- **.shp** files can be saved as **geoJSON**.

## mbtiles {#mbtiles}
- Can be edited in raw data form. Images can be plotted with `.` (dot).

## png {#png}
- Can be edited in raw data form. Images can be plotted with `.` (dot).

## ttf {#ttf}
- Can be edited in raw data form. Images can be plotted with `.` (dot).

## pandas {#pandas}

VisiData has an adapter for **pandas**. To load a file format which is supported by **pandas**, pass `-f pandas data.foo`. This will call `pandas.read_foo()`.

For example:

~~~
vd -f pandas data.parquet
~~~

loads a parquet file. When using the **pandas** loader, the `.fileformat` file extension is mandatory.

To load a hierarchy of parquet files located in folder `data/`, run

~~~
vd -f parquet data/
~~~

or rename the directory to `data.parquet` and run

~~~
vd data.parquet -f pandas
~~~

This should similarly work for any format that has a `pandas.read_format()` function.

## [VisiData Internal Formats](../internal_formats) (.vd, .vdj, .vdx, .vds) {#vd}

- .vd/vdj/.vdx are command log formats suitable for VisiData scripts and macros
- .vds is a multisheet save format that includes some sheet and column metadata

## API (reddit, matrix, zulip, airtable) {#api}
- loader-specific requirements
    - require setting authentication information in `~/.visidatarc` or on the CLI
    - launch the loader with `-f loadername` for steps to obtain and configure authentication credentials

## Grep {#grep}
A .grep file is a JSON lines file. It can be in two formats:
1) A simple container with three fields:
    file - a string with the path to the file where the match was found (absolute or relative path)
    line_no - an integer with the line number in the file where the match was found,
    text - a string with the text of the line that matched.
2) ripgrep `grep_printer` format, described here:
https://docs.rs/grep-printer/latest/grep_printer/struct.JSON.html

---

### Freq (`docs/freq.md`)

---
eleventyNavigation:
  key: Frequency Table Reference
  order: 99
---

# Frequency Table Reference

A frequency table groups source rows by one or more columns and counts the rows in each group.
A pivot table extends this by breaking each group into sub-groups based on a pivot column.
For tutorials and step-by-step workflows, see [Grouping data and descriptive statistics](/docs/group).

Everything below also applies to PivotSheet (`Shift+W`).

## Binning

### Discrete Binning

By default, each unique formatted value in the grouped column becomes its own bin.
Rows are sorted by count, descending.

### Numeric Binning

Set `options.numeric_binning` to True to bin numeric columns into ranges.

- `options.histogram_bins` controls the number of bins. When set to 0 (default), the bin count is the square root of the row count.
- Bin ranges display as "min - max".
- If all values are identical, a single bin is created.
- If there are more bins than distinct values, each value gets its own bin.

### Errors and Nulls

Null and error values get their own bins.
Error bins display as the stringified error value (e.g. `#ERR`).

## Commands

### On source sheet

- `Shift+F` (`freq-col`): open Frequency Table grouped on current column
- `g Shift+F` (`freq-keys`): open Frequency Table grouped by all key columns
- `z Shift+F` (`freq-summary`): open one-line summary for all rows and selected rows
- `Shift+W` (`pivot`): open Pivot Table: group rows by key columns and summarize current column

### On Frequency Table

- `Enter` (`open-row`): open copy of source sheet with rows in the current bin
- `g Enter` (`dive-selected`): open copy of source sheet with rows from all selected bins
- `z Enter` (`dive-except`): open copy of source sheet excluding rows in the current bin
- `gz Enter` (`dive-except-selected`): open copy of source sheet excluding rows in selected bins

## Selection Propagation

Selecting rows on the frequency table also selects the corresponding source rows.
Unselecting works the same way.
Undoing selection changes on the frequency table will also undo them on the source sheet.

## Columns

- **Key columns** from the grouped-by columns. Numeric binned columns use RangeColumn.
- **count**: number of source rows in the bin.
- **percent**: percentage of total source rows in the bin.
- **histogram**: visual bar chart (present when `disp_histogram` is set). Hidden automatically when aggregation columns are added.
- **Aggregation columns**: one column per aggregator set on source columns (via `+`).

## Options

- `options.disp_histogram` (default: `■`): histogram element character
- `options.histogram_bins` (default: `0`): number of bins for histogram of numeric columns (0 = auto)
- `options.numeric_binning` (default: `False`): bin numeric columns into ranges

## Tips

- `g '` (`freeze-sheet`) to create a plain data sheet snapshot of the frequency table.
- Add aggregators on source columns with `+` before opening the frequency table to see summary statistics per bin.
- Select bins with inconsistent values and use `ge` to unify them; the correction propagates back to the source sheet. See [bulk corrections](/docs/edit#bulk-corrections).

---

# Internals

## Row Type

Each row is a `PivotGroupRow` namedtuple with these components:

0. `discrete_keys`: tuple of formatted values for discrete key columns
1. `numeric_key`: range tuple `(min, max)` for numeric bins, or `(0, 0)` if not binned
2. `sourcerows`: list of source rows in this group
3. `pivotrows`: dict mapping pivot values to lists of source rows

## Architecture

- `FreqTableSheet` inherits from `PivotSheet`.
- `groupRows()` iterates source rows, assigns each to a bin by its formatted key values, and populates `sourcerows` and `pivotrows`.
- `addAggregateCols()` creates aggregation columns from any aggregators set on source columns.
- `nonKeyVisibleCols` determines which columns receive aggregation caching after load.

## Pivot-Specific

- `pivotrows` dict maps pivot values to row lists for each group.
- `Column.aggvalue` stores which pivot value a column represents.

## Key Source Files

- `visidata/freqtbl.py` — FreqTableSheet, HistogramColumn, commands
- `visidata/pivot.py` — PivotSheet, PivotGroupRow, RangeColumn, groupRows, addAggregateCols
- `visidata/aggregators.py` — aggregator definitions

---

### Gmail (`docs/gmail.md`)

# Using Gmail with OAuth 2.0

## Why

As of May 30 2022, Google [doesn't allow](https://support.google.com/accounts/answer/6010255?hl=en) you to log into your IMAP gmail account using only a username/password combination.
So to open your Gmail in Visidata you need to create a Google API App, attach some scopes to it, generate a client ID and secret, then let Visidata use that client ID and secret (in a json file).

## How

First, go to the [Google Console](https://console.cloud.google.com/apis/dashboard)

<br/>



<br/>

And click `CREATE PROJECT`

---




Give the project a name then click `CREATE`


---

Open a new tab and go to the [API Library](https://console.cloud.google.com/apis/library)

<br/>



Search for `gmail`

---



Click the search result `Gmail API`

---



Click `ENABLE`

---

Go back to your first tab

On the left, select `OAuth consent screen` 



Then select `External` and click `CREATE` to create an App 

---



Give the App a name and input your gmail address.

---



Click `ADD OR REMOVE SCOPES`

---



Search for `gmail`

---



Click the checkbox by the row with the scope value `https://mail.google.com/`

Then scroll to the bottom

---



And click `UPDATE`

---



You should see your selected scopes.

Click `SAVE AND CONTINUE`

---



Click `ADD USERS`

---



Type in your gmail email address then click `ADD`

---



On the left, click `Credentials`

---



Near the top click `CREATE CREDENTIALS`

Then click `OAuth client ID`

---



Select the application type `Desktop App` and give your OAuth 2.0 client a name then click `CREATE`

---



Click `DOWNLOAD JSON` and move the downloaded file into the visidata project directory at the path `vdplus/api/google/` and call the file `google-creds.json`

---

Now, on the command line run the equivalent for you:

`vd "imap://me@gmail.com@imap.gmail.com"`

Then you should get a web browser popup:



Select the account whose email address you have been using in these instructions.

---



Click `Select all`

---



See your gmail in Visidata.

<Chef's Kiss>

---

---

### Graph (`docs/graph.md`)

---
eleventyNavigation:
  key: Drawing graphs
  order: 10
Update: 2018-03-17
Version: VisiData 1.1
---



Graphs in VisiData can be used to visualise the relationship between numeric dependent variables vs a numeric independent variable. Optionally, a second independent variable, which is categorical,  adds an additional colored scatter plot for each category.

This example uses the file [StatusPR.csv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/StatusPR.csv).

<section id="hero">
    <div id="player"></div>
    <script type="text/javascript" src="/asciinema-player.js"></script>
    <script type="text/javascript">
        AsciinemaPlayer.create('../casts/pivot-graphs.cast', document.getElementById('player'), { poster: 'npt:0:30', rows: 27 });
    </script>
</section>

---

## How to graph a single column

1. Press `!` to set a column as the x-axis. This column must be numeric.
2. Set that column to a numeric type by pressing `#` (int), `%` (float), `$` (currency), or `@` (date).
3. Optional: Press `!` on a categorical key column to add it as an additional independent variable.
4. Set the type of the column you wish to set as the dependent variable with a numeric type.
5. Press `.` (dot=plot) on that column.

---

## How to graph multiple columns

1. Press `!` to set a column as the x-axis. This column must be numeric.
2. Set that column to a numeric type.
3. Optional: Press `!` on a categorical key column to add it as an additional independent variable.
4. Type all of the columns you wish to set as the dependent variables with a numeric type.
5. Press `-` (dash) to hide any numeric columns you don't wish graphed.
6. Press `g.`.

---

##How to interact with graphs

###### With the keyboard

Command(s)          Operation
----------          ----------
`1`-`9`             toggles display of each scatterplot layer
`h`  `j`  `k`  `l`  moves the cursor
`H`  `J`  `K`  `L`  expands and shrinks the cursor
`+`  `-`            increases/decreases the zoomlevel, centered on the cursor
`zz`                zooms into the cursor
`_` (underscore)    zooms to fit the full extent
 `s`   `t`   `u`    selects/toggles/unselects rows on the source sheet contained within the cursor
`gs`  `gt`  `gu`    selects/toggles/unselects rows visible as points on the screen
` d`                deletes rows on the source sheet contained within the cursor
`gd`                deletes all rows visible as points on the screen
 `Enter`            opens sheet of source rows contained within the cursor
`gEnter`            opens sheet of source rows which are visible on screen
`v`                 toggles the visibility of graph labels

###### With the mouse

Command                 Operation
-------                 -----------
Left-click and drag     sets the cursor
Right-click and drag    scrolls
Scroll-wheel            zooms in/out

---

---

### Graphics (`docs/graphics.md`)

---
eleventyNavigation:
  key: Terminal Graphics in VisiData
  order: 99
Updated: 2017-12-06
Version: VisiData 1.0
---



VisiData can display low-resolution terminal graphics with a reasonable amount of user interactivity.

The current implementation uses braille Unicode characters (inspired by [asciimoo/drawille](https://github.com/asciimoo/drawille)).  [Unicode blocks](https://en.wikipedia.org/wiki/Block_Elements) or the [sixel protocol](https://en.wikipedia.org/wiki/Sixel) may be supported in the future.

## Class hierarchy

- `Sheet`: the drawable context base class (part of core vdtui.py)
- `Plotter`: pixel-addressable display of entire terminal with (x,y) integer pixel coordinates
- `Canvas`: zoomable/scrollable virtual canvas with (x,y) coordinates in arbitrary units
- `InvertedCanvas`: a Canvas with inverted y-axis
- `Graph`: an InvertedCanvas with axis labels, a legend, and gridlines

### Summary

- The async `Graph.reload()` iterates over the given `sourceRows` (from its `source` Sheet) and calls `Canvas.polyline()` to indicate what to render.
- `Canvas.refresh()` triggers an async `Canvas.render()`, which iterates over the polylines and labels and calls the `Plotter.plot*` methods. 
- The `VisiData.run()` loop calls `Plotter.draw()`, which determines the characters and colors to represent the pixels.

### class `Plotter`

A `Plotter` is a [`Sheet`](/design/sheet) with a pixel-addressable drawing surface that covers the entire terminal (minus the status line).  Pixels and labels are plotted at exact locations in the terminal window, and must be recalculated after any zoomlevel change or terminal resizing.

`Plotter.draw(scr)` is called multiple times per second to update the screen, and chooses a curses attribute for each pixel.
By default, the most common attr is chosen for each pixel, but if `options.disp_graph_pixel_random` is set, an attr will be randomly chosen from the naturally weighted set of attrs (this may cause blocks of pixels to flicker between their possible attrs).
If an attr is in the `Canvas.hiddenAttrs` set, then it is not considered for display at all (and its rows will be ignored during selection).

All Plotter coordinates must be integer numbers of pixels.
[For performance reasons, they are presumed to already be integers, to save unnecessary calls to `round()`.]
Methods which plot multiple pixels on the canvas should be careful to gauge the display correctly; simply calling `round()` on each calculated float coordinate will work but can cause display artifacts.

#### `Plotter` methods

For Plotter methods, `x` and `y` must be integers, where `0 <= x < plotwidth`, and `0 <= y < plotheight`.  `(0,0)` is in the upper-left corner of the terminal window.

Pixels can be plotted directly onto a Plotter with these methods:

- `Plotter.plotpixel(x, y, attr, row=None)`
- `Plotter.plotline(x1, y1, x2, y2, attr, row=None)`
- `Plotter.plotlabel(x, y, text, attr)`

`attr` is a [curses attribute](/design/color), and `row` is the object associated with the pixel.

The above `plot*` methods append the `row` to `Plotter.pixels[y][x][attr]`.

These properties and methods are also available:

- `Plotter.plotwidth` is the width of the terminal, in pixels.
- `Plotter.plotheight` is the height of the terminal, in pixels.
- `Plotter.rowsWithin(bbox)` generates the rows plotted within the given region.
- `Plotter.hideAttr(attr, hide=True)` adds attr to `hiddenAttrs` if `hide`, and removes it otherwise.
- `Plotter.refresh()` is called whenever the screen size changes, and should also be invoked whenever new content is added.

`rowsWithin` takes a `Box` object (described below).  The `Box` class is otherwise unused by the Plotter.

### class `Canvas`

A **`Canvas`** is a `Plotter` with a virtual surface on which lines and labels can be rendered in arbitrary units.

The onscreen portion (the area within the visible bounds) is scaled and rendered onto the `Plotter`, with the minimum coordinates in the upper-left [same orientation as `Plotter`].

The [`Canvas` user interface](/docs/graph#commands) supports zoom, scroll, cursor definition, and selection of the underlying rows.  The `source` attribute should be the Sheet which owns the plotted `row` objects.

A call to `Canvas.refresh()` will trigger `Canvas.render()`, which is decorated with `@asyncthread` as it may take a perceptible amount of time for larger datasets.  Any active `render` threads are cancelled first.

#### `Box` and `Point` helper classes

While the Plotter API requires literal integer values for `x`/`y` and `width`/`height` parameters, `Canvas` methods generally take float values contained in either `Box` or `Point` classes.

##### `Point`

`Point` is simply a container for an `(x,y)` coordinate (passed to the constructor).  The individual components are stored as `.x` and `.y`, and the computed `.xy` property will return `(x,y)` as a simple tuple.  `Point` can also stringify itself reasonably.

##### `Box`

`Box` is effectively a rectangle stretching over some area of the canvas.  The constructor takes `(x,y,w,h)`, but a `Box` can also be constructed using the `BoundingBox(x1,y1,x2,y2)` helper.  [Note that in the BoundingBox case, the order of the individual points is not guaranteed; the individual coordinates may be swapped for convenience.]

`Box` has these members and properties:

- `xmin` and `ymin`: the minimum coordinates of the area.
- `xmax` and `ymax`: the maximum coordinates of the area.
- `xcenter` and `ycenter`: the central coordinates of the area.
- `w` and `h`: the width and height of the area.
- `xymin`: returns `Point(xmin,ymin)`.
- `center`: returns `Point(xcenter,ycenter)`.
- `contains(x, y)`: returns True if `(x,y)` is inside the bounding box.

#### `Canvas` methods

- `Canvas.polyline(vertices, attr, row=None)` adds a multi-segment line from the list of (x,y) `vertices`.  One vertex draws a point; two vertices draws a single line.  Note that the vertices are *not* Point objects (unlike parameters for other methods).
- `Canvas.label(xy, text, attr, row=None)` adds `text` at `xy` (Point in canvas units).
- `Canvas.fixPoint(xyplotter, xycanvas)` sets the position of the `visibleBox` so that `xycanvas` (Point in Canvas units) is plotted at `xyplotter` (Point in Plotter units).
- `Canvas.zoomTo(bbox)` sets the visible bounds so the given canvas coordinates will fill the entire Plotter area.  `aspectRatio` will still be obeyed.
- `Canvas.keyattr(key)` returns the `attr` for the given `key`, assigning a new color from `options.plot_colors` if `key` has not been seen before.  These keys are plotted as legends on the upper right corner of the canvas.  The last color is given out for all remaining keys and is labeled "other".
- `Canvas.resetBounds()` needs to be called after some or all points have been rendered, but before anything can be plotted.  It initializes the width and height of the canvas, visible area, and/or cursor.
- `Canvas.reset()` clears the canvas in preparation for `reload()`.

#### `Canvas` properties

- `Canvas.canvasBox` reflects the bounds of the entire canvas.
- `Canvas.visibleBox` defines the onscreen canvas area.
- `Canvas.cursorBox` defines the cursor region in canvas coordinates.
- `Canvas.zoomlevel` is a settable property, which sets the `visibleBox` size accordingly.  `zoomlevel` of 1.0 makes the entire canvas visible.  Does not change the position of the `visibleBox` (see `Canvas.fixPoint`).
- `Canvas.aspectRatio`, if set, maintains a proportional width and height of the `visibleBox` (considering also `plotwidth`/`plotheight`).  `aspectRatio` of 1.0 should be square.
- `Canvas.canvasCharWidth` and `Canvas.canvasCharHeight` is the width and height of one terminal character, in canvas units.

These properties reserve an area of the Plotter that is outside the visibleBox:
- `Canvas.leftMarginPixels`
- `Canvas.rightMarginPixels`
- `Canvas.topMarginPixels`
- `Canvas.bottomMarginPixels`

During a [mouse event](/design/commands#mouse), these properties indicate the mouse position for the current mouse event:

- `Canvas.canvasMouse`: a Point in canvas coordinates
- `Plotter.plotterMouse`: a Point in plotter (pixel) coordinates
- `Sheet.mouseX` and `Sheet.mouseY`: individual values in curses (character) coordinates

### class `InvertedCanvas`

An `InvertedCanvas` is a `Canvas` with a few internal methods overridden, such that the Y axis is inverted.  For an `InvertedCanvas`, the minimum coordinates are in the lower-left.

`InvertedCanvas` has not much else of interest.  It should be completely interchangeable with `Canvas`.

### class `Graph`

A `Graph` is an `InvertedCanvas` with axis labels and/or gridlines.

- `Graph.__init__(name, sheet, rows, xcols, ycols)` constructor
    - `sheet` is the `source` Sheet.
    - `rows` is a list of the rows to iterate over (from the given `source`).
    - `xcols` is a list of key columns forming the x-axis and color keys.
    - `ycols` is a list of numeric columns to be plotted on the y-axs.

---

---

### Group (`docs/group.md`)

---
eleventyNavigation:
  key: Grouping data and descriptive statistics
  order: 7
Update: 2018-08-19
Version: VisiData 1.3.1
---



## How to set statistical aggregators for a single column

Aggregators provide summary statistics for grouped rows.

Command             Operation
--------            ----------
 `+` *aggregator*   adds *aggregator* to current column
`z+` *aggregator*   displays result of *aggregator* over values in selected rows for current column

The following statistical aggregators are available:

Aggregator      Description
-----------     ------------
`min`           smallest value in the group
`max`           largest value in the group
`avg`/`mean`    average value of the group
`mode`          most frequently appearing value in group
`median`        median value in the group
`q3/q4/q5/q10`  add quantile aggregators to group (e.g. q4 adds p25, p50, p75)
`sum`           total summation of all numbers in the group
`distinct`      number of distinct values in the group
`count`         number of values in the group
`keymax`        key of the row with the largest value in the group
`list`          gathers values in column into a list
`stdev`         standard deviation of values

### Aggregation bar

When one or more columns have aggregators, the **aggregation bar** appears at the bottom of the sheet.
Each aggregator has its own row, showing the aggregator name and the computed value for each column with that aggregator.
Values are computed over all rows asynchronously, and recomputed after rows are added, deleted, or edited.

- {help.options.color_aggregator}

The follow howtos will have examples of workflows involving grouping of data and statistical aggregation.

---
The following examples use the file [sample.tsv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/sample.tsv).

## How to group data (frequency table, pivot table, describe table,)

## How to make a pivot table frequency table, pivot table, describe table,)

<div class="asciicast">
    <div id="player"></div>
    <script type="text/javascript" src="/asciinema-player.js"></script>
    <script type="text/javascript">
        AsciinemaPlayer.create('../casts/pivot.cast', document.getElementById('player'), { poster: 'npt:0:10', rows: 27 });
    </script>
</div>

1. Move to the column A with the independent variable, and press `!` to mark it as a key column to group by.
2. Move to a numeric column B, and press `+` to add an *aggregator* to that column, so that the aggregator will be applied to each group of values.
3. Make sure column B has a numeric type: int (`#`),  float (`%`),  currency (`$`), or date (`@`).
4. Move to the column C with the dependent categorical variable to be pivoted, and press `Shift+W` (`pivot`)  to pivot on that column.  The pivoted sheet then will have a column for each distinct value in the source column.

---

## [How to create a frequency chart](#frequency) {#frequency}


### How to make a histogram

**Question** How many of each **Item** were sold?

1. Move the cursor to the **Item** column.
2. Press `Shift+F` to open the **Frequency table**.

### How to use the Frequency table to view the results of statistical aggregation

**Question** What was the monthly revenue?

1. On the **OrderDate** column, type `;` followed by `(\d+-\d+)` to create a column with only the year and the month using regex capture groups.
2. Press `-` to hide the **OrderDate** column.
3. On the **OrderDate_re0** column, type `^` followed by `OrderDate` to rename it.
4. On the **Total** column, press `$` to set its type to currency.
5. Type `+` followed by `sum` to add a statistical aggregator to **Total**.
6. On the **OrderDate** column type `Shift+F` to open the **Frequency table**.
7. On the **OrderDate** column, press `[` to sort the table in chrononological order.
8. On the **sum_Total** column, type `^` followed by `Revenue` to rename the column.

---

### How to calculate some descriptive statistics

1. Press `Shift+I` to open the **Describe sheet**.

---

### How to filter for grouped rows

1. Press `Shift+F` to open the **Frequency table**.
2. Press `s` or `t` on the groups you are interested in to select those entries in the source sheet.
3. Press `q` or `Ctrl+^` to return to the source sheet.
4. Press `"` to open a duplicate sheet with selected rows.

**or**

1. Press `Shift+F` to open the **Frequency table**.
2. Press `Enter` on the grouping you are interested in to open a sheet of the source rows that are part of that group.

### How to filter for described rows

1. Press `Shift+I` to open the **Describe sheet**.
2. Use `zs` to select rows on source sheet which are being described in the cells of interest.
3. Press `q` or `Ctrl+^` to return to the source sheet.
4. Press `"` to open a duplicate sheet with selected rows.

**or**

1. Press `Shift+I` to open the **Describe sheet**.
2. Press `zEnter` open copy of source sheet with rows being described in the current cell of interest.

### How to filter for the rows aggregated in a pivot table

`Enter`/`zEnter` can both be used in the **Pivot table** to open a sheet of the source rows which are aggregated in the current pivot row/cell.

---

### How to combine cells in one for column for matching rows in another

This tutorial shows you how to go from this dataset:

field           value
------          -----
A               1
A               2
B               3
C               4
C               5

to

field           value
------          -----
A               1;2
B               3
C               4;5

in VisiData.

1. On the **value** column, type `+` followed by `list` to add to it a list aggregator.
2. Press `Shift+F` on the **field** column to open the **Frequency table**.
3. Type `=` followed by `';'.join(value_list)` to add an expression column with the joined values.

---

### Internal Formats (`docs/internal_formats.md`)

---
eleventyNavigation:
    key: VisiData Internal Formats
    order: 99
---

VisiData created these formats that it uses:

- .vd, .vdj, and .vdx to save a Command Log (for macros and replays);
- .vds to save/restore the current set of data sheets and columns.

All internal formats have comments being lines starting with `#`, so they can all be scripts on unix if they are executable and the first line starts with hash-bang (`#!`).

## .vd

.vd is the original replay format: a TSV save of the Command Log (`g Shift+D`).  Deprecated in favor of .vdj.

## .vdj

.vdj is the next generation format, a JSONL of the Command Log.
This enables support for multi-input commands (where input is a dictionary) among other things.

Macros are stored in .vdj format (since 3.0).  Macros from previous versions in .vd format are still accepted.

## .vdx

.vdx is an experimental format for the Command Log which does not require a data editor to create.
Since much of the time the sheet/col/row cursor position columns are blank, only the command longname and input are necessary to execute a command.
So the .vdx format is lines of commands with the command longnamefirst, and then a space and input (if used).  Comments start with `#`.
The cursor can be moved specifically with the `col` and `row` commands.

The .vdx loader also handles .vd and .vdj lines, so `--play -` (stdin) always uses the `vdx` filetype, which accepts any of the three command log formats.

This format is interesting for some use cases but not compelling enough to replace .vdj.
There is no current intent to replace the .vdj format with .vdx.

## .vds

.vds is a new, simple, multisheet format based on JSONL that includes some sheet metadata (mostly column definitions).
If you care only about the data and columns on the sheets, and not any provenance or exactitude or any of that stuff you get from going directly to the source, then you can save to .vds.
Opening a .vds pushes its sheets onto the stack in the order they were saved, so a file written with `save-all` (`g Ctrl+S`) reopens with the same sheet on top.
Each sheet retains the data and layout it was saved with, but without any history.
Use `open-source` (`` ` ``) on any of them to see the index of all sheets in the file.
Each row is a JSON object keyed by column name; a column whose name repeats an earlier one is keyed `name_2`, `name_3`, etc., with that key saved as `key` in its column metadata.
A key missing from a row stays missing, so sparse JSON rows keep their shape; an explicit null is saved as null.

---

### Join (`docs/join.md`)

---
eleventyNavigation:
  key: Combining datasets
  order: 9
Update: 2020-10-27
Version: VisiData 2.0.1
---



## How to join two datasets

1.  Open the datasets in VisiData.

    a. `vd d1.tsv d2.tsv`

    **or**

    b. Press `o` and enter a filepath for each file.
2. Press `S` to open up the **Sheets Sheet**. Through here, you can navigate to every sheet by pressing `Enter` on the row it is referenced in.
3. Navigate to the sheets you want the join, and set their shared columns as key columns with `!`.
4. Press `S` to return to the **Sheets sheet**. Select the sheets you want to merge with `s`.
5. Optional: If performing a left outer join, use `Shift+J` or `Shift+K` to reorder the sheets. The first sheet will be the one for whom all rows will be retained.
6. Type `&` to open the join-chooser, and select your desired jointype with `Enter`.

jointype            description
---------           -------------
`inner`             keeps only rows which match keys on all sheets
`outer`             keeps all rows from first selected sheet
`full`              keeps all rows from all sheets (union)
`diff`              keeps only rows NOT in all sheets
`extend`            keeps all rows and retain **SheetType** from first selected sheet
`merge`             Merges differences from other sheets into first sheet
`append`            keeps all rows from all sheets; columns from all sheets
`concat`            keeps all rows from all sheets; matches columns by position, with names and types from first sheet

`concat` requires the same sheet type and number of visible columns in each sheet. `append` matches columns by name.

## How to append two datasets

1. Open the datasets with VisiData.
2. Press `Shift+S` to open the **Sheets sheet**.
3. Use `s` or `t` to select the sheets to merge.
4. Type `&` and press `Enter` on `append` to concatenate the selected datasets.

## Identifying source rows

The `append` and `concat` join types add a hidden `origin_sheet` column that shows which source sheet each row came from. Unhide all hidden columns with `gv`.

---

### Loading (`docs/loading.md`)

---
eleventyNavigation:
  key: Loading data
  order: 2
  parent: Getting Started
update: 2018-12-18
version: VisiData 1.5.1
---

## Specifying a source file

The first way to open a dataset in VisiData, specify it directly after invoking `vd`.

~~~
vd filename.ext
~~~

Alternatively, you can pipe or redirect data in through stdin.

~~~
ps aux | vd
vd - < sample.tsv
~~~

Finally, you can launch the [DirSheet](), navigate to the file you wish to load, and press `Enter`.

~~~
vd .
~~~

In VisiData, a [loader](/docs/api/loaders) is a module which directs how VisiData structures and engages with a particular data source. [These sources](/formats) are currently supported.

On default, the file extension determines which loader is used. Unknown filetypes are loaded as **Text sheets**. To force a particular loader, pass `-f` with the filetype or format name.

~~~
vd -f sqlite bar.db
ls -l | vd -f fixed
~~~

To read an Arrow IPC stream from stdin (requires `pyarrow`):

~~~
vd -f arrows - < data.arrows
~~~

### How CLI options apply to files

`-f`/`-if`/`--input-filetype` applies to all subsequent input paths on the command line, not just the one immediately following.  The output file given with `-o` has its own flag, `-of`/`--output-filetype`; without it the output is saved by its extension.

`-f` attaches only to input paths given on the command line (and to piped stdin).  Files opened later from within the session use their own extensions; the `open-file` and `save-sheet` commands have their own filetype field (the *as filetype:* prompt) for overriding it.

You can reset back to extension-based detection for later files with `-f ""`:

See the [manpage](/man#commandline-options) for more details.

~~~
vd -f csv data.txt -f "" other.jsonl
~~~

This loads `data.txt` as csv, and `other.jsonl` using its file extension as usual.

---

## Loading sources supported by pandas

VisiData has an adapter for pandas. To load a file format which is supported by pandas, execute `vd -f pandas data.foo`. This will call `pandas.read_foo()`.

For example:

~~~
vd -f pandas data.parquet
~~~

loads a parquet file. When using the pandas loader, the `.fileformat` file extension is mandatory.

Note that if you are using Python v3.7, then you will need to manually install pandas >=0.23.2 (our requirements.txt file installs v0.19.2 as the last version compatible with 3.4).

---

## Opening an R data frame with VisiData

[@paulklemm](https://github.com/paulklemm) has wonderfully developed a small `R` package which bridges the gap between `R` and VisiData.

To install `rvisidata` using [devtools](https://cran.r-project.org/web/packages/devtools/index.html) run:

~~~
devtools::install_github('paulklemm/rvisidata')
~~~

from within the `R` interpreter.

Any data frame can then be opened by VisiData:

~~~
vd(iris)
~~~

Please note, that this tool opens the data frame in readonly mode. Any changes made will be discarded.

For more more details, questions, and feedback, check out the [rvisidata repository](https://github.com/paulklemm/rvisidata).

---

## Loading multiple datasets simultaneously

Multiple files can be passed as inputs through the commandline.

~~~
vd birdsdiet.tsv surveys.csv sunshinelist.html
~~~

Upon launching, the final dataset to load (in this case, sunshinelist.html) will be displayed on top.

To load files from within a VisiData session, press `o` and enter a filepath.

---

## Accessing other loaded or derived sheets

1. Press `Shift+S` to open the **Sheets sheet**.
2. Move the cursor to the row containing the desired sheet.
3. Press `Enter` to jump to the sheet referenced in that current cursor row.

---

## [How to convert a dataset from one supported filetype into another](#convert) {#convert}

~~~
vd -b countries.fixed -o countries.tsv
~~~

The output format follows the `-o` filename's extension.  To save to a filename whose extension does not match the desired format, give `-of`/`--output-filetype`:

~~~
vd -b countries.fixed -of csv -o countries.txt
~~~

**Note**: Not all filetypes which are supported as loaders are also supported as savers. See the [formats page](/formats#output) for the supported output formats.


---

---

### Macros (`docs/macros.md`)

---
eleventyNavigation:
    key: Macros
    order: 5
update: 2023-11-18
version: VisiData 3.0
---

# Macros

Macros allow you to bind a series of commands to a key and then replay those commands within a session by using that keystroke.

The basic usage is:
1. Press `m` (macro-record) to begin recording the macro.
2. Go through the commands you wish to record.
3. Then type `m` again to complete the recording, and prompt for the keystroke or longname to bind it to.

The macro will then be executed every time the provided keystroke is used. Note: the Alt+keys and the function keys are left unbound; overriding other keys may conflict with existing bindings, now or in the future.

Executing a macro will the series of commands starting on the current row and column on the current sheet.

## Keystroke Binding

When binding a macro to a longname (like `happy-time`), you will also be prompted for an optional keystroke shortcut.
Type a keystroke (like `Alt+s`) and press Enter to bind it, or just press Enter to skip.

The keystroke is shown on the Macros Sheet and can be edited there.

## Input Parameters

Macro command inputs can use `{param}` or `{param=default}` placeholders in the `input` or `col` fields.
When a parameterized macro is run, VisiData prompts for parameter values before executing.

To add parameters to an existing macro:

1. Open the Macros Sheet with `gm`.
2. Press `Enter` on the macro to open its command log.
3. Edit the `input` or `col` cells to replace literal values with `{param}` or `{param=default}` placeholders.
4. Save the macro with `z Ctrl+S`.

For example, recording a macro that adds a column with `=name + " suffix"`, then editing the input to `{col} + " {text=suffix}"`, will prompt for `col` and `text` values each time the macro runs.

Parameter names must be valid Python identifiers, so `{3}` in a regex is left alone.
To keep literal braces in a parameterized macro, double them: `f"{{name}}-{suffix}"` runs as `f"{name}-..."` and prompts only for `suffix`.

If a recorded command's input or column already contains `{name}` (for example an f-string expression like `=f"{first} {last}"`), the macro is recorded with `braces` set to `literal`, and its braces run as typed with no parameter prompts.
The `braces` column on the Macros Sheet shows this; set it to `param` to use parameters in such a macro, after doubling any braces that should stay literal.

# The Macros Sheet

Use `gm` (`macro-sheet`) to open an index of existing macros.

To delete a macro, move the cursor to it and press `d` to mark it for deletion. Then press `z Ctrl+S` to commit the change. This removes the macro's key binding and deletes its source file.

The `helpstr` column can be edited to add a description for each macro. This description will show up in the command palette and the Commands Sheet. Save changes with `z Ctrl+S`.

The `keystroke` column can be edited to bind or change a keystroke shortcut for a macro.

`Enter` will open the macro in the current row, and you can view the series of commands composing it.

---

### Menu (`docs/menu.md`)

---
eleventyNavigation:
    key: Command Menus
    order: 12
Updated: 2021-09-02
Version: VisiData v2.6
---

At the top of the screen, there's a typical application toplevel menubar:

    Sheet  Edit  View  Column  ...

You can press `Ctrl+H` to activate the Help menu (shown in the upper right).
Then you can use the arrow keys (or `hjkl` of course) to navigate the various commands and submenus.

`Enter` or `Space` will execute the command or dive into the submenu.

`q` or `Esc` or `Ctrl+Q` will leave the menu without taking an action.

## Other ways of opening the menu

`Alt+S` or `Alt+E` etc will open that specific submenu (see the underlined letter).
On Macos, `Option+S` etc should work the same (with the exception of `Option+E`, which needs to be `Option+E+Space`).

You can also click on the menu with the mouse to open it, as well as clicking on the various menu items to navigate to them.
Click on the already active menu item will execute the command or navigate into the submenu.

Click off the menu to deactivate it.

## Menu annotations

The `…` annotation, like in traditional GUI menus, indicates that the command requires further input.
The `⎘` annotation on a command is specific to VisiData, and means that the command will push a sheet onto the sheet stack, so your data context will change.  You can always return to the previous sheet with `q` (or if you like taking the long way around, the "Sheet»Quit»Top sheet" command in the menu).

## Configuration Options

To make the top menu line disappear (as it looked pre-2.6)

    options.disp_menu = False

The menu can still be activated with `Ctrl+H` or the Alt+ keys as above.  The menu system cannot be entirely disabled.

### Theme options

As with most VisiData display elements, the menu can be altered to taste.  These options are available to change the colors and displayed chars:

- `color_menu`
- `color_menu_active`
- `color_menu_spec`
- `color_menu_help`
- `disp_menu_boxchars`
- `disp_menu_more`
- `disp_menu_input`
- `disp_menu_push`

## Plugin API for adding new menu items

The menu can be extended with additional commands.  See the [plugin API](/docs/api/interface).

---

### Mouse (`docs/mouse.md`)

---
eleventyNavigation:
    key: Mouse
    order: 15
Update: 2022-06-19
Version: VisiData v2.8
---

## How to disable the mouse in VisiData?

To disable for all sessions, add to your `~/.visidatarc`:

    options.mouse_interval = 0 # disables the mouse-click
    options.scroll_incr = 0    # disables the scroll wheel

To disable in the current session, press `SPACE`, and type the longname *mouse-disable*.

---

### Move (`docs/move.md`)

---
eleventyNavigation:
  key: How can I move around and search?
  order: 99
---

- basic movement just like vim
  - 'G' 'gg' '^G' work as in vim, as do some of the 'z' scroll commands
- 'g' prefix goes all the way
.  'gh' (or g Left) already goes to the first column, but maybe '0' should also.
- (call out no number prefix?)
- '/' and '?' search for regex, 'z/' and 'z?' search by Python expr. 'n' and 'N' continue the last search of either type.  These should work just like in vim but only within a single column.
  - 'g' searches across all columns
- 'zr'/'zc' go to a specific row or column by number
- 'r' and 'c' go by row key or column name

i
  - 'g' and 'z' are also command prefixes in vd, but in vd they generally means something like 'g'lobal/embi'g'gen and 'z'croll/'z'mallify.

---

### Navigate (`docs/navigate.md`)

---
eleventyNavigation:
  key: Navigation
  order: 3
updated: 2020-07-18
version: VisiData 2.0
---


## How to rapidly scroll through a sheet

Command(s)                    Operation
--------------                ---------------
` ← ↑ → ↓ PgUp PgDn Home End` move as expected
` h`  ` j`  ` k`  ` l`        move cursor **one cell** left/down/up/right (like in vim)
`gh`  `gj`  `gk`  `gl`        move **all the way** to the left/bottom/top/right of sheet
` <`  ` >`                    move up/down the current column to the next **value which differs from current cell**
` {`  ` }`                    move up/down the current column to the next **[selected](/docs/rows#subset) row**

---

## Go to a numbered row or column

`z r` (`go-row-number`) and `z c` (`go-col-number`) go to a 0-based row or column number.
Negative numbers count from the end: `-1` is the last row or column.
Column numbers include visible columns first, followed by hidden columns.
Out-of-bounds numbers move to the nearest end and warn with the resulting position.

---

## How to search within a sheet

Command(s)              Operation
--------------          ---------------
` /`  ` ?` *regex*      search for *regex* matches up/down the **current** column
`g/`  `g?` *regex*      search for *regex* matches up/down over **all visible** columns
` n`  `Shift+N`              move to next/previous match from last search
`z/`  `z?` *expr*       search by Python *expr* up/down (with column names as variables)

By default, search matches are highlighted in the sheet (like vim's `hlsearch`).

Command(s)              Operation
--------------          ---------------
`highlight-col`         highlight a regex in the current column (without moving)
`highlight-sheet`       highlight a regex in all columns (without moving)
`highlight-clear`       clear all highlight patterns

Undo of a search command does not undo the highlighting; use `highlight-clear` to remove all highlights.

**Options**

Option                         Default                 Description
-----                          -----                   -----
`highlight_search`             `True`                  whether to highlight search matches
`color_highlight_search`       `21 blue on 15 white`   color for highlighted search matches

The following example uses [sample.tsv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/sample.tsv).

**Question** Has there been a day where we sold more than 95 **Item**s?

1. Set the type of the **Units** column to integer by moving to the **Units** column and pressing `#`.
2. Type `z/` followed by `Units > 95`.

**Question** What is the **longname** for `gk`?

1. Press `z Ctrl+H` to open the **Commands sheet**.
2. Move to the `keystrokes` column and press `/`, followed by `gk`.
3. Press `c` followed by `longname` to move the cursor to the **longname** column.

---

## Input history

When entering a search regex or other input that has been used before, a history palette appears showing previous inputs.

- `Up`/`Down` cycle through previous inputs (like shell history).
- `Down` past the newest entry restores the original typed text.
- Type to fuzzy search through history.
- `PgUp`/`PgDn` to scroll through the history palette.

The history palette uses the same visual style as the command palette, and is enabled when `cmdpalette` is included in `options.disp_help_flags`.

---

## How to move between sheets

The Sheets Sheet is a list of all sheets that have ever been opened (in order of opening).

The Sheets Stack (`z Shift+S`is the list of active sheets (most recently used at top).

###### Jumping to sheets

1. Press `Shift+S` to open the **Sheets sheet**.
2. Move the cursor to the row containing the desired sheet.
3. Press `Enter` to jump to the sheet referenced in that current cursor row.

###### Jumping away from sheets

Command(s)              Operation
--------------          ---------------
`Ctrl+^`                jump to the previous sheet, without closing the current one
`q`                     quit the current sheet (closes it)

**Note**: A quit/closed sheet is grayed out on the Sheets Sheet.
Closed sheets are removed from the active sheets stack, and the next sheet is then shown.  When the active sheets stack is empty, VisiData exits.

---

---

### Pipes (`docs/pipes.md`)

---
eleventyNavigation:
   key: Pipes
   order: 14
Update: 2019-09-11
Version: VisiData 2.0
---


## stdin/stdout pipe/redirect

VisiData works with other tools in a pipeline. VisiData will read any piped input from stdin as a sheet, and save the
results of any remaining sheets to stdout Visdata allows interactive edits in the middle of a pipeline.

   - Use it to manually update (sort, filter, edit) tabular data in a pipeline  `mysql < query.sql | vd | awk 'awkity {awk}'
   - Use it to interactively pick processes to kill `ps -ef | vd | tail -n +2 | xargs --no-run-if-empty kill`

When redirecting VisiData output:

   - `Ctrl+Q` will output current sheet (as it quits with the top sheet still on the stack)
   - `q` (or `gq`) will output nothing (as it quits by dropping all sheets from the stack)

Use `vd -o-` to send directly to the terminal when not redirecting stdout (it's not necessary if already redirected).

To output a single column without the column header, make sure only that column is visible and save as .txt.  For example, `vd . --save-filetype txt | lpr`.

---

### Plugins (`docs/plugins.md`)

---
eleventyNavigation:
  key: Plugins
  order: 13
Updated: 2026-10-16
Version: VisiData 3.5
---

A *plugin* is an optional Python module to modify or extend VisiData functionality. Once installed, all plugins are automatically imported when `vd` is started.

# How to install a plugin

Plugins can be installed in 3 places:

1. They can come packaged with the VisiData install.
2. They can be installed via pip.
3. They can be copied into the `plugins/` subdirectory of the [VisiData directory](#the-visidata-directory).

## builtin plugins

The VisiData release includes additional functionality that isn't enabled by default, but which can be made available by importing the module.
For example, there are some features available in `visidata.experimental`, like `digit_autoedit`, which starts editing the current cell when a numeric digit is pressed (like Excel).
VisiData intentionally does not work this way, but people who prefer their terminal applications to act more like Excel can add this to their .visidatarc:

    import visidata.experimental.digit_autoedit

## via pip

Any package with a `visidata.plugins` entry point will be loaded automatically by default.
These plugins must be managed via pip (to upgrade or remove).
Plugins can be published to PyPI or downloaded as wheels or installed directly from source.

For example, `DarkDraw` is a VisiData plugin that allows drawing text-art.
Once the Python module is installed, you can use `vd` to open and save a `.ddw` file, which is a text image or animation specified in JSON format.
It also adds other commands and options.

## copied manually into local plugins directory

Python modules in the [VisiData directory](#the-visidata-directory) (`<vd_dir>` below) are available for import from `.visidatarc`.

To install a plugin manually:

1. Make a `plugins` directory: `mkdir -p <vd_dir>/plugins`
2. Copy the plugin Python file there: `cp myplugin.py <vd_dir>/plugins`
3. Add a line to `~/.visidatarc` to import the plugin: `import plugins.myplugin`
4. Install the dependencies for the plugin (if any).

# How to Disable Plugins

The nuclear option is `--nothing` or `-N` flag, which forces the base configuration, not loading any .visidatarc or plugins.
This is especially useful for debugging; when filing an issue, please make sure the bug reproduces with `-N` (if not, it is likely something in your .visidatarc)

To load `.visidatarc` but not autoload plugins installed with pip, set `options.plugins_autoload` on the command-line:

    `vd --plugins-autoload=False`

Plugins explicitly imported in .visidatarc will still be installed.

# The VisiData directory

The VisiData directory holds local plugins (`plugins/`) and their dependencies (`plugins-deps/`), saved macros, Google API tokens, the HTTP cache for the scrape and reddit loaders, and REPL history.
VisiData adds it to the Python import path at startup.

The default VisiData directory is:

- Linux: `~/.config/visidata`
- macOS: `~/Library/Preferences/visidata`
- Windows: `%LOCALAPPDATA%\visidata\visidata`

On Linux and macOS, `XDG_CONFIG_HOME` changes the default to `$XDG_CONFIG_HOME/visidata`.

To use another directory, set the `VD_DIR` environment variable before starting `vd`, or pass `--visidata-dir=<vd_dir>` on the command line.
The command-line option takes precedence over `VD_DIR`.
Setting `options.visidata_dir` in `.visidatarc` does not work, because the VisiData directory is set up before `.visidatarc` is loaded.
Before VisiData 3.2, the default was `~/.visidata`.
To keep using it, add this to the shell startup file:

    export VD_DIR="$HOME/.visidata"

## Other storage locations

VisiData stores some files outside the VisiData directory.
`VD_DIR` and `--visidata-dir` do not move these.

- The data directory holds input history (`input_history.jsonl`) and the list of saved macros (`macros.jsonl`).
  Default: `~/.local/share/visidata` (Linux), `~/Library/Application Support/visidata` (macOS), `%LOCALAPPDATA%\visidata\visidata` (Windows), or `$XDG_DATA_HOME/visidata`.
- The cache directory holds cached downloads, like the message of the day and currency rates.
  Default: `~/.cache/visidata` (Linux), `~/Library/Caches/visidata` (macOS), `%LOCALAPPDATA%\visidata\visidata\Cache` (Windows), or `$XDG_CACHE_HOME/visidata`.
- The config file is `config.py` in the default VisiData directory if it exists, otherwise `~/.visidatarc`.
  Set `VD_CONFIG` or pass `--config=<file>` to use another file.

# VisiData API reference

For help making plugins, see the [VisiData API reference](https://visidata.org/docs/api).

---

### Rows (`docs/rows.md`)

---
eleventyNavigation:
  key: Rows
  order: 4
Update: 2023-10-12
Version: VisiData 3.0
---


## [How to perform operations on a subset of rows](#subset) {#subset}

Many commands can be finetuned to operate on rows which are 'selected'.

###### How to specify the subset of rows which are selected

Command(s)         Operation
-----------------  -------------
 `s`  `t`  `u`     select/toggle/unselect **current** row
`gs` `gt` `gu`     select/toggle/unselect **all** rows
 `|`  `\` *regex*  select/unselect rows matching *regex* in **current** column
`g|` `g\` *regex*  select/unselect rows matching *regex* in **any visible** column
`z|` `z\` *expr*   select/unselect rows matching Python *expr* in **any visible** column
 `,`               select rows matching current cell in **current** column
`g,`               select rows matching **entire current** row in **all visible** columns

An example usage follows.

---

## How to filter rows

1. Press `s` or `t` on the rows to be filtered.

2. Press

    a. `gd` to delete the selected rows.

    **or**

    b. `"` to open a duplicate sheet which has references to the selected rows.  Edits performed within the duplicate sheet will also propagate to the source sheet.

    **or**

    c. `g"` to open a duplicate sheet which has references to all rows, and keeping the selected rows selected.  The selection can be modified on the parent sheet and the new sheet independently, but any changes will not be reflected on both sheets.

    **or**

    d. `z"` to open a copy of the sheet which has copies of the selected rows.  Any changes will not affect the source sheet.

    **or**

    d. `gz"` to open a copy of the sheet which has copies of all rows.  Any changes will be reflected on the source sheet


The following example uses the file [sample.tsv](https://raw.githubusercontent.com/saulpw/visidata/stable/sample_data/sample.tsv).


**Question** On which days have we sold less than 10 Binders?

1. Scroll to the `Units` column. Set the type of the `Units` column by pressing `#` (int).
2. Type `z|` followed by `Item == 'Binder' and Units < 10` to select all of the rows where the `Item` is Binder and the number of `Units` is less than 10.
3. Press `"` to open a duplicate sheet with only those selected rows.

---

## How to filter a random subset of rows

1. Type `Space` `random-rows` followed by the *number* of rows you wish included in your random population sample.

---

## How to select rows where the current column is not null or empty?

'Null' cells, by default, are cells which contain `None`. This can be changed with `options.null_value`. Null cells can be set with `zd` (set current cell to `None`) or `gzd` (set selected rows in current column to `None`). Null cells are distinguished with a yellow ∅' symbol on the right hand corner. They are distinct from empty cells (which are `''` in columns of string type.)

1. Type `|` followed by *.* to select all rows without empty or null cells in the current column.

---

## How to select rows where the current column is null?

There are several different options:

- `z,` matches by **typed value** while `,` matches by **display value**. Move to an empty or `None` cell in the column of interest and press `,` to select all empty and `None` cells in that column. Type `z,` on a `None` cell to select all rows that contain `None` in that column. Type `z.` on an empty cell to select all rows that have an empty cell in that column.
- Open a **DescribeSheet** for the current sheet with `Shift+I`. Move to the **nulls** column, and then move to the row which references the source column of interest. Type `zs` to select all null rows for that column.
- `z|` is a command which allows selection by provided Python expression criteria (using column names as variables). For non-numerical columns `z|` followed by **not ColumnName**, will select all empty cells and `None` cells for that column. For numerical columns it will also select cells with `0`.
- [A typed value of Python None is always a `TypedWrapper`](https://www.visidata.org/docs/api/columns.html#user-defined-types), which allows sorting, etc., on columns with None values. For this reason `z|` *ColumnName is None* will not work, thought `z|` *ColumnName.val is None* will work. This is not a recommended approach, though at the moment, it is available.

---

## How to move, copy and remove rows

Command(s)         Operation
-----------------  -------------
 `J`  `K`          move cursor row down/up
`gJ` `gK`          move cursor row all the way to the bottom/top of sheet


###### How to copy and paste a single row

1. Press `y` to copy the row to the clipboard.
2. Move the cursor to the desired location.
3. Press `p`/`Shift+P` to paste the row after/before current row.

###### How to copy and paste multiple rows

1. Press `s`/`t` on those rows to select them.
2. Press `gy` to copy all selected rows to the clipboard.
3. Move the cursor to the desired location.
3. Press `p`/`Shift+P` to paste those rows after/before current row.

###### Note

 VisiData has a universal paste mechanism: it creates new rows on the target sheet and then fills them with values from the copied rows from the previous sheet.

 This value-filling happens positionally, so if columns are missing or in a different order, the values will be in different columns,

---

## How to sort rows

Command(s)         Operation
-----------------  -------------
 `[`  `]`          sorts ascending/descending by **current** column; replace any existing sort criteria
`g[` `g]`          sorts ascending/descending by **all key** columns; replace any existing sort criteria
`z[` `z]`          sorts ascending/descending by **current** column; add to existing sort criteria
`gz[` `gz]`        sorts ascending/descending by **all key** column; add to existing sort criteria

###### How to sort a numerical column from highest number to lowest:

1. Set the type of the column being sorted by pressing `#` (int) or `%` (float).
2. Press `[` to sort the column from highest to lowest.

###### How to sort a date column in chronological order:

1. Set the type of the column being sorted by pressing `@` (date).
2. Press `]` to sort the column chronologically.

###### How to sort based on multiple columns

1. Press `!` on those columns to designate them as key columns.
2. Press `g[` or `g]` to sort.

**or**

1. Sort the first column with `[` or `]`.
2. Sort the next column with `z[` or `z]` to add sorting to the existing criteria.

###### How to increase row height

Press `v` on any **TableSheet** to toggle multi-line rows. This dynamically lengthens rows so that the full content of the column is visible.

Multi-line rows have some limitations; they can't be paged, for instance.  The full contents of a cell can be viewed or edited in an external program like `emacs` or `less`:

* Press `e` on a cell to enter Editing mode. Then scroll left and right to explore its contents.
* Press `Ctrl+O` while in editing mode, to open the contents of the current cell in an external *$EDITOR*.

**TextSheet-specific options**
**TextSheet**s are used for loading `.txt` files in VisiData. They are also the default loaders used for un-identified sources. They are notable for having a single column which has the name "text".

**TextSheet**s have a bonus option `wrap` which will wrap the text into multiple rows, so that it fits the window width.

* On **TextSheet**, press `Shift+O` to open its **OptionsSheet**. Press `Enter` on the `wrap` option, to set it to *True*. Press `q` to return to your **TextSheet**. Reload it with `Ctrl+R`. Note that **reload** undoes an previous modifications you may have made to the **TextSheet**.
* On the commandline, the option `--wrap` will set the `wrap` option to *True* for all **TextSheet**s in that session.
* In your `~.visidatarc`, `options.wrap = True` will set the `wrap` option to *True* for all **TextSheet**s in every session.

---

---

### Save Restore (`docs/save-restore.md`)

---
eleventyNavigation:
    key: Save and Restore
    order: 11
Updated: 2018-01-17
Version: VisiData 0.99
---


## How to save and replay a VisiData session

This example creates and then uses the file [pivot.vdj](https://raw.githubusercontent.com/saulpw/visidata/stable/tests/pivot.vdj).

<section id="hero">
    <div id="player"></div>
    <script type="text/javascript" src="/asciinema-player.js"></script>
    <script type="text/javascript">
        AsciinemaPlayer.create('../casts/save-restore.cast', document.getElementById('player'), { poster: 'npt:0:41', rows: 27 });
    </script>
</section>

To save and restore a session in VisiData:

1. Save the cmdlog using *one* of the following options:

    a. Press `Ctrl+D` to save the cmdlog to a `fn.vdj` file.

    *or*

    b. Press `Shift+D` to view the `CommandLog Sheet`, then press `Ctrl+S` and save it with a `.vdj` suffix.

2. Press `gq` to quit VisiData.
3. Replay the cmdlog, on the commandline: `vd -p fn.vdj`.

---

## Environment and locale

`set-envvar` sets an environment variable and records it in the command logs, like a global option.
Setting `LANG` or any `LC_*` variable also reapplies the locale, which affects `type-floatlocale`, number formatting, and date formatting.

To record a locale in every saved cmdlog, add this to `.visidatarc`:

```python
vd.setenv('LC_NUMERIC', 'en_US.UTF-8')
```

The locale must be installed on the replaying machine (`locale -a` lists them), and a nonempty `LC_ALL` overrides the other `LC_*` variables.

## How to replay a cmdlog on a different file

Cmdlogs record the exact filename and sheet names from the original session.
To reuse a cmdlog with a different input file, you need to make the cmdlog filename-independent.

### Method: Edit the `.vd`/`.vdj` to remove hardcoded references

1. Save your cmdlog as usual (`Shift+D` to open the cmdlog, `Ctrl+S` to save).
2. Edit the `.vdj` file:
   - **Remove** the `open-file` line (the line with `"longname": "open-file"`).
   - **Erase all `"sheet"` values** (for instance with `gs gzd`). A blank sheet name means "use the current sheet", so commands operate on whatever is on top of the stack.
3. Replay on a new file:

```
vd -b -p cmdlog.vdj newfile.csv -o output.tsv
```

The new file is passed as a positional argument, and `-p` replays the cmdlog against it. Use `-b` for batch (non-interactive) mode and `-o` to write the result.

**Limitations:** Blank sheet names only work for linear workflows where you never switch between sheets. If your analysis involves joining sheets or switching back to a previous sheet, you'll need to either keep the sheet names, or use stack-relative commands like `jump-prev` (`Ctrl+^`, jump to previous sheet) or `jump-first` (`gCtrl+^`, jump to first sheet) to navigate between sheets without relying on their names.

### Cmdlog formats

VisiData supports several cmdlog formats:

- **`.vd`** (TSV) — original format with tab-separated columns
- **`.vdj`** (JSONL) — JSON objects, one per line; fields include `sheet`, `col`, `row`, `longname`, `input`, `keystrokes`, `comment`
- **`.vdx`** (simplified) — human-readable, one command per line; only emits `sheet`/`col`/`row` lines when they change

When replaying from stdin (`vd -p -`), VisiData uses the `.vdx` loader, which accepts all three formats.

---

---

### Shell (`docs/shell.md`)

---
eleventyNavigation:
    key: Shell
    order: 10
update: 2022-08-16
version: VisiData v2.9
---

## How to add ZSH Completion

1. Clone the VisiData repo: `git clone https://github.com/saulpw/visidata`
2. `cd visidata`
3. Run the script that will create the list of autocompletes for visidata

~~~
dev/zsh_completion.py
~~~
4. `echo $fpath | grep site-functions` to learn where your zsh site-functions folder is located. (Possible locations `/usr/share/zsh/site-functions`, `/usr/local/share/zsh/site-functions`)
5. Move the visidata compdefs to where zsh can find them: `sudo mv _visidata /path/to/site-functions`
6. `compinit`
7. Check that the autocompletes are configured with `grep -n visidata ~/.zcompdump`

---

### Split (`docs/split.md`)

---
eleventyNavigation:
  key: Split view
  order: 99
---

# Viewing two sheets simultaneously

VisiData supports the viewing of two sheets simultaneously with a feature called split screen.

## Entering split screen mode

Keystroke(s)        Action
------------        ------
`Shift+Z`           split screen in half, placing second sheet on stack visibly in the second pane
`z Shift+Z` *n*     split screen, setting the height of the second screen to *n*


When there is a single pane, all currently-open sheets are situated within the same sheets stack. The more recent that sheet was viewed, the higher it is in the [stack](https://jsvine.github.io/intro-to-visidata/basics/understanding-sheets/#how-to-use-the-sheets-sheet).

Split-pane opens a second window, and creates a second stack for it. This keeps the sheet stably within a specific window.

For instance, pressing `Shift+C` on the [sample.tsv]() will give us a stack with two columns, the sample **Sheet** and its **Columns Sheet**.

![]

Pressing `Shift+Z` will now give us a view of them both, with the sample **Sheet** moved to the second window.

![]

## Split screen mode specific commands

While in split screen mode, you can move between windows, adjust the sheets that appear in the current window, and full-screen the current pane.

Keystroke(s)        Action
------------        ------
`z Shift+Z` *n*     adjust the height of the second window to *n*
`g Shift+Z`         full screen the current window
`Tab`               make the other window the active window
`Ctrl+^`            swap top two sheets in stack in current window, second sheet in stack is now visible
`g Ctrl+^`          cycle through sheets in stack in current window

For instance, we can use `Tab` to make the sample **Sheet** the active window.

![]

Moving around it, we can see the **Columns Sheet** in the top window continuing to be updated.

We can press `Shift+F` on the **Item** column to view its [Frequency Table]().

Now, the visible sheet of the second window has changed, while the visible sheet on the top window has stayed the same.

![]

We can use `^^`, so that sample **Sheet** is once again visible, and full-screen it with `g Shift+Z`, effectively leaving split screen mode.

![]

---

### Test (`docs/test.md`)

---
eleventyNavigation:
    key: Contributing Tests
    order: 99
Update: 2024-01-10
Version: VisiData 3.0
---


The `tests` folder contains functional tests in the form of `.vd` scripts, each of which records a session of VisiData commands.  These ensure that data processing works consistently and reliably.

`dev/test.sh` (run from the git root) will execute all tests.  The final sheet of each test is saved as .tsv and compared to the respective expected output checked into the `tests/golden` directory.

As of January 2024, to pass all the tests, you will need to install the `test` extras:

```
git clone https://github.com/saulpw/visidata.git
cd visidata
pip3 install ".[test]"
```

To show each step of a test with a delay of 1 second between commands:

    $ bin/vd -p tests/foo.vd -w 1

To build a `.vd` file:

1. Go through all of the steps of the workflow, ending on the sheet with the final result.
2. Press `Shift+D` to view the `CommandLog Sheet`.
3. Edit the commandlog to minimize the number of commands.  Cells may be parameterized like `{foo}`, to be specified on the commandline:

    $ vd cmdlog.vd --foo=value

4. Save the cmdlog:

    Press `Shift+D` to open the `commanDlog Sheet`, and press `Ctrl+S` to save it with a `.vd` suffix.


There are also unit tests in visidata/tests. To run the unit tests:

```
pytest -sv visidata/tests
```

---

---

### Usage (`docs/usage.md`)

---
eleventyNavigation:
        key: Usage
        order: 99
---

# How can I load files and exit the program?

If you use the command line, 'vd' should pretty much work as you expect:

        $ vd foo.csv

You can pipe in tabular data:

        $ find | vd

If the file is not plain ASCII .txt or [tsv](/tsv), you will have to specify the filetype with -f and/or other options:

        $ ls -al | vd -f fixed --skip 1 --headers 0

Many [data formats](/formats) are supported, with more being added every day.

# How do I quit?

You can just bang on 'q' until all the sheets are gone (there are usually only one or two anyway).

   Note: If a prompt or editbox is waiting for input, you may have to press '^C' first.

It requires a bit more dexterity, but Control-Q (aka '^Q') will abort the program quite rudely.[1]

If you wish to be a bit gentler, 'gq' will quit all sheets ('global quit').


[1] Control-Q is the only builtin command (necessary sometimes during development).  All other [commands can be overridden](/howto/commands).

# How do I confirm before quitting?

- `options.quitguard = True` in `.visidatarc`: confirm before quitting a sheet with unsaved changes.
- `vd --quitguard foo.csv`: guard the sheets opened from the command line.  Quitting a guarded sheet always asks, even with no changes.
- `guard-sheet` (File > Guard > on): guard the current sheet.  `guard-sheet-off` removes the guard.

When exiting VisiData, one confirmation covers all sheets.

---

### Viewtsv (`docs/viewtsv.md`)

---
eleventyNavigation:
    key: viewtsv Annotated
    order: 99
Date: 2017-12-27
Version: 1.0
---


# viewtsv

[viewtsv.py](https://github.com/saulpw/visidata/blob/stable/bin/viewtsv.py)
is a great example of a minimal VisiData application.  This is an extremely functional utility in 25 lines of code, leveraging the essence of the VisiData architecture.  Here it is in its entirety, with line by line annotations:

    #!/usr/bin/env python3

VisiData 2.x requires Python 3.6+.

    import sys
    from visidata import Sheet, ColumnItem, options, asyncthread, run

Import the relevant components used below.

    class TsvSheet(Sheet):
        rowtype = 'rows'  # rowdef: tuple of values

All tabular data sheets inherit from `Sheet`.  The rowtype is displayed on the right status bar.  The `rowdef` comment declares the structure of every row object, and should be present for every Sheet.

        columns = [ColumnItem('tsv', 0)]

An initial column.  Generally the class-level `columns` is set to the actual columns of the sheet, but in this case, the columns aren't known until the source is loaded.
(See the `reload()` function below where they are set from the contents of the first row.)  This line is not strictly necessary but makes loading feel a bit more responsive.


        @asyncthread

@[asyncthread](/docs/async) marks the function to spawn a new thread whenever it is called.

        def reload(self):

The [`reload()`](/docs/loaders) function collects data from the source and puts it into `rows`.  It's called once automatically when first pushed, and manually with `^R`.

            self.rows = []

`rows` is a list of Python objects.  The row definition ('rowdef') for the TsvSheet is a tuple of values, with each position corresponding to one column.

            with open(self.source, encoding=options.encoding) as fp:

`source` is the filename, which has been passed to the constructor (see the last line with `run()`).

                for line in fp:
                    line = line[:-1]
                    if line:
                        self.addRow(line.split('\t'))

For each line, strip the included newline character, and filter out any blank lines.  Add each split tuple to `rows`.

            for i, colname in enumerate(self.rows[0]):
                self.addColumn(ColumnItem(colname, i))

The actual columns are set from the first (header) row.
`ColumnItem` is a builtin, which creates a column to use getitem/setitem with the given key/index.

            self.rows = self.rows[1:]

The header row is removed from the list of rows.  (Column names are displayed on the first row anyway).

    run(*(TsvSheet(fn, source=fn) for fn in sys.argv[1:]))

`run(*sheets)` is the toplevel entry point for a VisiData application.



---

---

### Guides MovementGuide (`guides/MovementGuide.md`)

# Movement and Search

The `g` prefix extends most movement commands to go all the way to the edge
of the sheet, and the `z` prefix gives a more precise or smaller variant.

## Basic movement

- {help.commands.go-left}
- {help.commands.go-down}
- {help.commands.go-up}
- {help.commands.go-right}
- {help.commands.go-leftmost}
- {help.commands.go-bottom}
- {help.commands.go-top}
- {help.commands.go-rightmost}
- {help.commands.scroll-middle}

## Skip by value

- {help.commands.go-next-value}
- {help.commands.go-prev-value}
- {help.commands.go-next-null}
- {help.commands.go-prev-null}
- {help.commands.go-next-selected}
- {help.commands.go-prev-selected}

## Search

Search operates on the current column's displayed values by default. Use the `g` prefix to search across all visible columns.

- {help.commands.go-col-regex}
- {help.commands.search-keys}
- {help.commands.search-col}
- {help.commands.searchr-col}
- {help.commands.search-cols}
- {help.commands.searchr-cols}
- {help.commands.search-expr}
- {help.commands.searchr-expr}
- {help.commands.search-next}
- {help.commands.searchr-next}

## Sheets

- {help.commands.sheets-stack}
- {help.commands.jump-prev}

---

### Api Async (`docs/api/async.rst`)

.. _threads:

=========================
Threads
=========================

VisiData is not necessarily the fastest at processing large amounts of data, but great care has been taken to make sure that it remains *responsive*.
It is better to wait 10 seconds with active indication of progress, than 2 seconds with the program frozen and not accepting input (or worse; queueing up commands without ability to cancel).

To this end, VisiData has a main interface thread for display and input, and creates threads for any task that may be time-consuming.
In fact, anything that iterates through all rows (there may be millions) or all columns (there may be thousands), should probably be in its own thread.

``@asyncthread``
~~~~~~~~~~~~~~~~

Use the ``@asyncthread`` decorator on a function to spawn a new thread for each call to that function.
The return value is the spawned thread (which can often be ignored by the caller), and the return value of the original function is effectively lost.

Cells which are being computed in a separate thread will have the Thread object as their value until the result is available.
[This will show the ``options.disp_pending`` cell note and allow the user to interact with the specific thread (via e.g. :kbd:`z Ctrl+Y` or :kbd:`z Ctrl+C`).]

Each thread is added to ``Sheet.currentThreads`` on the current sheet, and to ``vd.threads``.
Note that a thread spawned by calling a function on a different sheet will add the thread to the currentThreads for the topmost/current sheet instead.

.. autoclass:: visidata.asyncthread

.. data:: visidata.vd.threads

A list of all threads.

.. data:: visidata.BaseSheet.currentThreads

A list of all threads started from commands on this sheet.

.. autofunction:: visidata.vd.execAsync
.. autofunction:: visidata.vd.sync
.. autofunction:: visidata.vd.cancelThread

.. note::

    ``vd.cancelThread(*threads)`` sends each thread an ``EscapeException``, which percolates up the stack to be caught by the thread entry point.
    ``EscapeException`` inherits from Python's ``BaseException`` instead of ``Exception``, so that functions can still have catchall try-blocks with ``except Exception:``, without catching explicit user cancellations.

    An unqualified ``except:`` clause is bad practice (as always).
    When used in an async function, it will make the thread uncancelable.

.. _progress:

Progress counters
~~~~~~~~~~~~~~~~~~

To provide the progress meter in the right status bar, every function that could iterate over a large collection of objects (usually rows or columns) or loop a large number of times, should use an ``Progress`` object somewhere to track its progress.

.. note::

    Many internal functions will contribute their own ``Progress`` counters (including ``visidata.Path.read``), so an async function may not need its own, but it can still be helpful to provide a more precise *gerund* to show to the user.

.. autoclass:: visidata.vd.Progress

.. function:: visidata.vd.Progress.addProgress(n)

    Increase the progress count by *n*.


Progress as iterable
^^^^^^^^^^^^^^^^^^^^^

When iterating over a potentially large sequence:

::

    for item in Progress(iterable, gerund='working'):

This is equivalent to ``for item in iterable``.  The `gerund` specifies the action being performed.

- These ``Progress`` only display if used in another thread, and has no effect otherwise.  So any iteration over rows and columns should have ``Progress``, even if it is not ``@asyncthread``.
- Use ``Progress`` around the innermost iterations for maximum precision in the progress meter.
- But this incurs a small amount of overhead, so if a tight loop needs every last optimization, use it with an outer iterator instead.
- Multiple ``Progress`` objects used in parallel will stack properly.
- Multiple ``Progress`` objects used serially will make the progress indicator reset (which is better than having no indicator at all).

If *iterable* does not know its own length, then pass the length or an approximation as the *total* keyword argument:

::

    for item in Progress(iterable, gerund='working', total=approx_size):

Each iteration of the ``Progress`` object contributes 1 towards the total.
To contribute a different amount, use ``Progress.addProgress(n)`` (n-1 if being used as an iterable, as 1 will be added automatically).

Progress as context manager
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To manage ``Progress`` without wrapping an iterable, use it as a context manager with only a ``total`` keyword argument, and call ``addProgress`` as progress is made:

::

    with Progress(gerund='working', total=amt) as prog:
        while amt > 0:
            some_amount = some_work()
            prog.addProgress(some_amount)
            amt -= some_amount

- Using ``Progress`` other than as an iterable or a context manager will have no effect.




Threads Sheet
~~~~~~~~~~~~~~

The Threads Sheet is useful for analyzing the performance of long-running async commands.

All threads (active, aborted, and completed) are added to ``vd.threads``, which can be viewed as the ThreadsSheet via :kbd:`Ctrl+T`.
Threads which take very little time (currently less than 10ms) are removed, to reduce clutter.

- Press :kbd:`Ctrl+_` (*toggle-profile*) to toggle profiling of the main thread.
- Press :kbd:`Enter` (*open-row*) on a thread to view its performance profile (if ``options.profile`` was set when the thread started).

Profiling
^^^^^^^^^

The performance profile sheet in VisiData is the output from ``pstats.Stats.print_stats()``.

- :kbd:`z Ctrl+S` on the performance profile will call ``dump_stats()`` and save the profile data to the given filename, for analysis with e.g. `pyprof2calltree <https://libraries.io/pypi/pyprof2calltree>`__ and `kcachegrind <https://kcachegrind.github.io/html/Home.html>`__.

---

### Api Canvas (`docs/api/canvas.rst`)

Canvas
=========

.. autofunction visidata.BoundingBox

.. autofunction visidata.Canvas.point
.. autofunction visidata.Canvas.line
.. autofunction visidata.Canvas.polyline
.. autofunction visidata.Canvas.polygon
.. autofunction visidata.Canvas.qcurve
.. autofunction visidata.Canvas.label

.. autofunction visidata.Canvas.plotterCursorBox
.. autofunction visidata.Canvas.cursorBox
.. autofunction visidata.Canvas.plotterMouse
.. autofunction visidata.Canvas.canvasMouse
.. autofunction visidata.Canvas.zoomTo
.. autofunction visidata.Canvas.resetBounds
.. autofunction visidata.Canvas.rowsWithin
.. autofunction visidata.Canvas.fixPoint

.. autofunction visidata.Canvas.plotpixel
.. autofunction visidata.Canvas.plotlabel

Canvas
=========
.. autofunction visidata.GraphSheet.formatX
.. autofunction visidata.GraphSheet.formatY
.. autofunction visidata.GraphSheet.parseX
.. autofunction visidata.GraphSheet.parseY

---

### Api Columns (`docs/api/columns.rst`)

.. _columns:

Columns
====================================

Columns are the heart of the VisiData computation engine.

Each column can **calculate** a value from a row object; and it might also be able to **put** a different value into the row object (for a later calculate to re-derive).

A Column subclass can override ``calcValue`` and ``putValue`` to define its fundamental interaction with the row object.
This is often the only thing a Column subclass has to do.

``calcValue`` and ``putValue`` should generally not be called by application code.
Instead, apps and plugins should call ``getValue`` and ``setValue``, which provide appropriate layers of caching.

.. autoclass:: visidata.Column
.. autoattribute:: visidata.Column.name
.. autoattribute:: visidata.Column.type
.. autoattribute:: visidata.Column.width
.. autoattribute:: visidata.Column.visibleWidth
.. versionadded:: 2.1

.. autoattribute:: visidata.Column.fmtstr

.. autoattribute:: visidata.Column.hidden

.. autofunction:: visidata.Column.calcValue
.. autofunction:: visidata.Column.putValue

.. autofunction:: visidata.Column.getValue
.. autofunction:: visidata.Column.getTypedValue
.. autofunction:: visidata.Column.getDisplayValue
.. autofunction:: visidata.Column.formatValue

.. autofunction:: visidata.Column.getValueRows
.. autofunction:: visidata.Column.getValues


.. autofunction:: visidata.Column.setValue
.. autofunction:: visidata.Column.setValues
.. autofunction:: visidata.Column.setValuesTyped
.. autofunction:: visidata.Column.setValuesFromExpr

.. autofunction:: visidata.BaseSheet.evalExpr
.. autofunction:: visidata.Column.recalc
.. autofunction:: visidata.TableSheet.recalc
.. autofunction:: visidata.Column.isError

- ``calcValue`` may be arbitrarily expensive or even asynchronous, so once the value is calculated, it is cached until ``Column.recalc()`` is called.
- ``putValue`` may modify the source data directly (for instance, if the row object represents a row in a database table).  VisiData will *never* modify source data without an explicit ``save`` command.  So applications (and all other code) must call ``setValue`` to change the value of any cell.

- ``delete-cell`` actually just calls setValue with None.

Columns are added to Sheets with :ref:`addColumn <enumerate-columns>`.

Example
~~~~~~~~~

::

    class SpecialColumn(Column):
        'Column for special fields on special rows.'
        def calcValue(self, row):
            return row.special()[self.expr]

        def putValue(self, row, val):
            row.special()[self.expr] = val

    c = SpecialColumn('field1', expr='field1_key')
    sheet.addColumn(c)


Column Subclasses
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: visidata.ItemColumn
.. autoclass:: visidata.AttrColumn
.. autoclass:: visidata.ExprColumn
.. autoclass:: visidata.SettableColumn
.. autoclass:: visidata.SubColumnFunc

Key Columns
============

.. autofunction:: visidata.TableSheet.setKeys
.. autofunction:: visidata.TableSheet.unsetKeys
.. autofunction:: visidata.TableSheet.rowkey
.. autofunction:: visidata.TableSheet.keystr
.. autoattribute:: visidata.TableSheet.keyCols
.. autoattribute:: visidata.TableSheet.nonKeyVisibleCols

Types
======

The value returned by getValue could be many different things:

   - a string
   - numerically typed
   - a list or dict
   - None
   - a null value (according to ``options.null_value``)
   - an Exception (error getting)
   - a Thread (async pending)
   - any python object

This value may need to be parsed and/or converted to a consistent type.
So, every column has a ``type`` attribute, which affects how it is parsed, displayed, grouped, sorted, and more.

The default column type is ``anytype``, which lets the underlying value pass through unaltered.
This is the only ``type`` for which ``Column.getTypedValue`` can return arbitrary types.

The classic VisiData column types are:

============  ==================  =========  =================  ============
type          description         numeric    command            keystrokes
============  ==================  =========  =================  ============
``anytype``   pass-through                   ``type-anytype``   :kbd:`z~`
``str``       string                         ``type-str``       :kbd:`~`
``date``      date/time           Y          ``type-date``      :kbd:`@`
``int``       integer             Y          ``type-int``       :kbd:`#`
``float``     decimal             Y          ``type-float``     :kbd:`%`
``currency``  decimal with units  Y          ``type-currency``  :kbd:`$`
``vlen``      sequence length     Y          ``type-vlen``      :kbd:`z#`
============  ==================  =========  =================  ============

The default keybindings for setting types are all on the shifted top left keys on a US keyboard.

User-defined Types
~~~~~~~~~~~~~~~~~~~

Fundamentally, a type is a function which takes the underlying value and returns an object of a specific type.
This function should accept a string and do a reasonable conversion, like Python ``int`` and ``float`` do.
And like those builtin types, this function should produce a reasonable baseline zero (arithmetic identity) when passed no parameters or None.

Computations should generally call ``getTypedValue``, so that the values being used are consistently typed.

If the underlying value is None, the result will be a ``TypedWrapper``, which provides the baseline zero value for purposes of comparison, but a stringified version of the underlying value for display.
For a ``calcValue`` which raises or returns an Exception, ``getTypedValue`` will return a ``TypedExceptionWrapper`` with similar behavior.


In the following, ``TYPE`` is the type (like ``int``, ``date``, etc), and ``typedval`` is an instance of that type.

A VisiData type function or constructor must have certain properties:

    - ``TYPE()`` must return a reasonable default value for *TYPE*.
    - ``TYPE(typedval)`` must return an exact copy of *typedval*.
    - ``TYPE(str)`` must convert from a reasonable string representation into *TYPE*.
    - ``TYPE.__name__`` must be set to the official name of the type function or constructor.

Objects returned by ``TYPE(...)`` must be:

    - comparable (for sorting)
    - hashable
    - formattable
    - roundable (if numeric, for binning)
    - idempotent (``TYPE(TYPE(v)) == TYPE(v)``)

Types API
~~~~~~~~~~~

.. autoclass:: visidata.vd.addType
.. autofunction:: visidata.vd.isNumeric
.. versionadded:: 2.1
.. autofunction:: visidata.isNumeric
.. deprecated:: 2.1
    Use ``vd.isNumeric`` instead.

Examples
~~~~~~~~~

::
    # Add an ip_address type.
    vd.addType(ipaddress.ip_address, icon=':', formatter=lambda fmt,ip: str(ip))

    TableSheet.addCommand(None, 'type-ipaddr', 'cursorCol.type=ipaddress.ip_address', 'set type of current column to IP address'

Nulls
~~~~~~

VisiData has a crude concept of null values.  These interact with:
   a. aggregators: the denominator counts only non-null values
   b. the Describe Sheet: only null values count in the nulls column
   c. the ``fill-nulls`` command
   d. the ``melt`` command only keeps non-null values
   e. the ``prev-null`` and ``next-null`` commands (:kbd:`z<` and :kbd:`z>`)

- The null values are Python ``None`` and options.null_value if set.

.. note::

    ``options.null_value`` can be used to specify a null value in addition to Python ``None``.
    The CLI can only set this option to a string like ``''`` (empty string) or ``'NA'``.
    The option can be set to a different type in ``.visidatarc`` or other code, though:

    ::

        options.null_value = 0.0
        options.null_value = date("1980-01-01")

.. autofunction:: visidata.BaseSheet.isNullFunc

There is no direct isNull function, because the possible null values can change at runtime via the above option, and getting an option value is very expensive to do in a bulk operation.

Aggregators
===========

Aggregators allow you to gather the rows within a single column, and interpret them using descriptive statistics.
VisiData comes pre-loaded with a default set like mean, stdev, and sum.

.. autofunction:: visidata.TableSheet.addAggregators

.. autofunction:: visidata.vd.aggregator

The `type` parameter is optional. It allows you to define the default type of the aggregated column.

Example
~~~~~~~

Add aggregator for `numpy's internal rate of return <https://numpy.org/doc/stable/reference/generated/numpy.irr.html>`__ module:

::

    import numpy as np
    vd.aggregator('irr', np.irr, type=float)

---

### Api Commands (`docs/api/commands.rst`)

.. _commands:

Commands
--------

VisiData is **command-driven**, which means that it only does something when you tell it to.
Otherwise, it just sits there, waiting for your next command.

Every command is a discrete unit of behavior that does a defined task and runs to completion.
Functions that could take longer than a couple hundred milliseconds should run in their own thread (see :ref:`Threads <threads>`).

Every command should be **reproducible**: given the same sheet, cursor position, and :ref:`input <input>` string (if relevant), a command should yield identical output (with a few obvious exceptions, like :kbd:random-rows).

Any command which makes changes to a :ref:`saveable sheet<sheets>` is appended to that sheet's **Command Log**.
Since all state changes must be initiated by a reproducible command, this command log can be `replayed </docs/save-restore>`__.

Adding new commands is a natural way to extend VisiData's functionality.

Command Overview
~~~~~~~~~~~~~~~~

Commands and Keybindings are managed in a similar way to options.
The same precedence hierarchy is used, so that commands can be created or overridden for a specific type of sheet, or even a specific sheet itself.

Since all Sheets ultimately inherit from ``BaseSheet``, a command on ``BaseSheet`` is effectively a global command.

.. _command-context:

Command Context
^^^^^^^^^^^^^^^

The *execstr* is a string of Python code passed to ``exec()`` when the command is run.

``exec()`` looks up symbols in this order:

-  the current ``sheet``
-  the ``vd`` object
-  the ``visidata`` module (see ``addGlobals()`` and ``getGlobals()`` :ref:`below<other-commands>`)

The ``vd`` and ``sheet`` symbols are available to specify explicitly.

``currow`` is available on TableSheets; it provides access to column values for the current cursor row by name (e.g. ``currow.Price`` returns the typed value of the ``Price`` column for the cursor row).

.. note::

    Unqualified ``options`` in a command execstr will use the sheet-specific options context for the current sheet.

.. warning::

   In an *execstr*, while you can **get** an attribute on ``vd`` or ``sheet`` without specifying the object, to **set** an attribute does require an explicit object.  e.g. instead of ``cursorRowIndex=2``, it must be ``sheet.cursorRowIndex=2``.

Commands API
~~~~~~~~~~~~

.. autofunction:: visidata.BaseSheet.addCommand

Keybindings
~~~~~~~~~~~~

   - Use "``Ctrl+X``" for :kbd:`Ctrl+X`.
   - Primarily, plugin authors and users should use ``0-9``, function keys (``F2``), ``Alt+`` for custom keybindings; these are purposefully left available for user keybindings.
   - Consider not providing a default at all, for infrequently used commands.
   - Instead give it an easy and memorable longname, and/or a unique *helpstr* which can be searched for in the **Command Help** (:kbd:`g Ctrl+H`) with :kbd:`g/`.
   - Many other keycodes can be returned from the curses library as strings.
   - To discover what to use for some unknown key, press that key in VisiData and use the keystroke shown in the status bar.

.. autoattribute:: visidata.vd.allPrefixes

``vd.allPrefixes`` is a list of *prefixes* (keystrokes that don't trigger the end of a command sequence).
New prefixes can be added to this list, and then they can also be used as prefixes in keybindings.

.. note::

    Combinations of prefixes are allowed, but only in the specified order: ``g`` must come before ``z``, which must come before ``Alt+``.

.. _other-commands:

Other helpful functions
~~~~~~~~~~~~~~~~~~~~~~~~

.. autofunction:: visidata.vd.bindkey
.. autofunction:: visidata.BaseSheet.bindkey
.. autofunction:: visidata.BaseSheet.execCommand
.. autofunction:: visidata.addGlobals
.. autofunction:: visidata.getGlobals

Rules for command longnames
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1) 3 words max, 2 words if possible.  A longname should be short and fit on a keymap.

2) Command classes should be unique in their first 3 chars and ideally mostly in their first 2.

3) Command longnames should be intuitively understandable, or at least not jargony.

4) Longnames should evoke interface, when possible.

5) Use existing verb-object-input structure if possible:

Verbs:

-  ``open``: push new sheet
-  ``jump``: push existing sheet
-  ``dup``: push copy of sheet
-  ``show``: display on status
-  ``go``: move the cursor
-  ``scroll``: change the visible screen area without changing the cursor
-  ``addcol``: add new column to this sheet
-  ``setcol``: set selected cells in this column
-  ``search``: search one or more columns
-  ``searchr``: search reverse
-  ``select/unselect/stoggle``: add/remove rows from selected rows
-  ``syscopy/syspaste``: copy/paste to system clipboard
-  ``sysopen``: open with $EDITOR or other external program

Objects:

- ``-all``: all sheets or all visible columns
- ``-col``: cursorCol
- ``-cols``: all visible columns
- ``-cells``: this column, selected rows
- ``-selected``: selected rows

Inputs:

- ``-expr``: python expression
- ``-regex``: python regex

Many others are used, see the full command list for inspiration.

Examples
~~~~~~~~

::

    @Sheet.api
    def show_hello(sheet):
        vd.status(sheet.options.disp_hello)

    # `sheet` members and `vd` members are available in the execstr scope
    BaseSheet.addCommand(None, 'show-hello', 'show_hello()', 'show a warm greeting')

    # bind Shift+H, Ctrl+H, and Alt+H to this command
    BaseSheet.bindkey('H', 'show-hello')
    BaseSheet.bindkey('Ctrl+H', 'show-hello')
    BaseSheet.bindkey('Alt+h', 'show-hello')

---

### Api Data (`docs/api/data.rst`)

Selected Rows
-------------

Each TableSheet has a set of *selected rows*, which is a strict subset of the rows on the sheet.

.. autoattribute:: visidata.TableSheet.selectedRows
.. autoattribute:: visidata.TableSheet.someSelectedRows
.. versionchanged:: 2.1
.. autoattribute:: visidata.TableSheet.onlySelectedRows
.. versionchanged:: 2.1
.. autoattribute:: visidata.TableSheet.nSelectedRows

.. autofunction:: visidata.TableSheet.selectRow
.. autofunction:: visidata.TableSheet.unselectRow
.. autofunction:: visidata.TableSheet.isSelected

.. autofunction:: visidata.TableSheet.clearSelected

.. autofunction:: visidata.TableSheet.selectByIdx
.. autofunction:: visidata.TableSheet.unselectByIdx

.. autofunction:: visidata.TableSheet.select
.. autofunction:: visidata.TableSheet.unselect
.. autofunction:: visidata.TableSheet.deleteSelected


.. note::

    To clear the set of selected rows before any bulk selection, set ``options.bulk_select_clear`` to True.
    The status message will include "instead" as a reminder that the option is enabled.

.. warning::

    ``selectedRows`` feels like a list, but it's actually a property that iterates over all rows to generate the selected rows in sheet order.
    With large datasets, collecting the list of selected rows itself can take a large time, regardless of the number of rows that are actually selected.
    So instead of using selectedRows in the execstr, call an ``@asyncthread`` ``@TableSheet.api`` function which uses sheet.selectedRows.
    Use it as a parameter immediately or save it to a local variable on the first usage, to avoid unnecessary work.

Sorting
-------

.. autofunction:: visidata.TableSheet.orderBy
.. autofunction:: visidata.TableSheet.sort

---

### Api Extensible (`docs/api/extensible.rst`)

API Extensions
===============

One of VisiData's core design goals is **extensibility**.
Many of its features can exist in isolation, and can be enabled or disabled independently, without affecting other features.

So, VisiData encourages plugin authors to provide new features in a similarly modular form.
These features can be enabled by importing the module, or left disabled by not importing it.
Modules should degrade or fail gracefully if they depend on another module which is not available.

The ``Extensible`` base class
-----------------------------

Python classes usually implement all methods and members in the class definition, which exists in a single source file. A modular VisiData feature, by contrast, is self-contained within a separate source file, while providing methods on previously-defined classes.

Core functionality can be changed through 'monkey-patching', which is the ability for modules loaded after startup to add or change methods on existing classes.

To make this a bit easier, the core classes in VisiData inherit from the ``Extensible`` class, which provides some helper functions and decorators to make monkey-patching easier and more consistent.
All of their subclasses are then also naturally Extensible.

Extensible API
--------------

.. autoclass:: visidata.Extensible

.. autofunction:: visidata.Extensible.api

This decorator defines a member function on a specific class.

Because this is a member function, the first parameter is the instance itself.
If this function were defined in the class, the first parameter would be named ``self`` by Python convention.
When members are defined in other files, it is better to use a specific local object name instead of ``self``.
Use ``sheet`` for any Sheet type, ``col`` for any Column type, and ``vd`` for VisiData (which will shadow the global ``vd`` object, but as it is a singleton, they will be identical).

::

        @VisiData.api
        def vd_func(vd, ...):
            pass

        @Sheet.api
        def sheet_func(sheet, ...):
            pass

        @Column.api
        def col_func(col, ...):
            pass

``Extensible.api`` can be used either to add new member functions, or to
override existing members. To call the original function, use
``func.__wrapped__``:

::

        @Sheet.api
        def addRow(sheet, row):
            # do something first
            addRow.__wrapped__(sheet, row)

        ....

        sheet.addRow(row)

.. autofunction:: visidata.Extensible.class_api

.. autofunction:: visidata.Extensible.property

This acts just like the ``@property`` decorator, if it were defined inline to the class.

.. autofunction:: visidata.Extensible.lazy_property

This works like ``@property``, except it only computes the value on first access, and then caches it for every subsequent usage.

.. note::

    Because of how Python instantiates classes, extensions monkey-patched into a class are not also added to already-instantiated objects.
    So global sheets defined by a plugin should be added to the VisiData object as a ``@VisiData.lazy_property``.
    This way, they are not created until their first use, which allows them to take advantage of Sheet extensions that were loaded after the plugin.


.. autofunction:: visidata.Extensible.before

Runs *beforefunc* before the original method.  The return value of *beforefunc* is ignored.

::

    @Sheet.before
    def reload(sheet):
        vd.status('reloading...')

.. autofunction:: visidata.Extensible.after

Runs *afterfunc* after the original method.  The return value of *afterfunc* is ignored; the original's return value is preserved.

::

    @DrawablePane.after
    def execCommand2(sheet, cmd, *args, **kwargs):
        vd.usedInputs[cmd.longname] += 1

.. autofunction:: visidata.Extensible.around

Wraps the original method, passing it as the first argument (*oldfunc*) to *aroundfunc*.
*aroundfunc* can call *oldfunc* with modified arguments, return a different value, or skip calling it entirely.

::

    @VisiData.around
    def input(vd, oldfunc, prompt, **kwargs):
        if should_intercept(kwargs):
            return custom_input(vd, prompt, **kwargs)
        return oldfunc(vd, prompt, **kwargs)

.. autofunction:: visidata.Extensible.init

If a module wants to store some data on an Extensible class, it can add
a member with a call to that class' ``init()``:

::

    TableSheet.init('foo', dict)

This monkey-patches ``TableSheet.__init__`` to add the instance member
``foo`` to every TableSheet on construction, and to initialize it with an
empty dict. To provide an initial non-object value:

::

    TableSheet.init('bar', lambda: 42)

.. note::

    This will not work with the ``vd`` because it is instantiated very early.
    Instead, assign member variables directly on ``vd`` in the toplevel scope:

    ::

        vd.bar = []

    This member can then be used like any other member of the class.

By default, when an instance of the class is copied, a member specified with this ``init()`` is reset to a newly constructed value (by calling the constructor again).
If *copy* is ``True``, then a copy is made of the member for the new instance.

.. autofunction:: visidata.vd.global_api

All features (and plugins) should expose functions and classes to plugins in one of these ways:

1. with `@VisiData.api` (or @Sheet.api): for most methods

These can be used in an execstr as though they were global (attributes on `vd` and `sheet` are both implicitly in scope in an execstr).

Outside of an execstr, use `vd.funcname(...)` (or `sheet.funcname(...)`).

Note that classes can be annotated with `@VisiData.api` also.

2. with `addGlobals({"funcname": funcname})`: for classes and methods internal to VisiData

These can be used via direct import:

`from visidata import SomeInternalClass`
`from plugins.myplugin import HelperClass`

This is acceptable for commonly-used classes.

See :ref:`getGlobals() and addGlobals() <other-commands>`.

What to extend: ``Sheet``, ``Column``, ``VisiData``, or globals?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Look at what the function uses.
If it uses a specific column, use ``@Column.api`` with ``col`` as the first "self" argument, and if you need access to the sheet, use ``col.sheet``.

If the function naturally takes a sheet, use ``@Sheet.api`` with first argument of ``sheet``.

Otherwise, extend the VisiData object if neither Column nor Sheet are relevant.
``vd`` is always available as a global regardless.

Classes and functions which don't use ``vd`` or ``sheet`` at all are candidates for the list of bare globals in ``__all__``.


The ``vd`` singleton
-----------------------------

The VisiData class is a singleton object, containing all of VisiData's global functions and state for the current session.
This object should always be available as ``vd``.

Calling conventions
~~~~~~~~~~~~~~~~~~~

When calling functions on ``vd`` or ``sheet`` outside of a Command *execstr*, they should be properly qualified:

::

    @Sheet.api
    def show_hello(sheet):
        vd.status(sheet.options.disp_hello)

    BaseSheet.addCommand(None, 'show-hello', 'show_hello()')

The current **Sheet** and the **VisiData** object are both in scope for :ref:`execstrs <command-context>`, so within an *execstr*, the ``sheet.`` or ``vd.`` may be omitted, as in the hello world example:

::

    BaseSheet.addCommand(None, 'show-hello', 'status(options.disp_hello)')

---

### Api Guides (`docs/api/guides.rst`)

Guides
=======

A Guide is an in-app writeup of a particular feature that gives a basic synopsis of how to use it and explains how it works together with other features.

Open the Guide Index with ``Space open-guide-index`` within VisiData or ``vd -P open-guide-index`` from the CLI.


.. note::

    Guides that have not been written yet are grayed out.
    We love to get help with documentation, please get in touch if you want to write one or have other suggestions!

Here's an outline for adding a guide, with our writing style preferences:

1. Launch **GuideIndex** and find the ``name`` of the guide.
2. Create a markdown file ``visidata/guides/{name}.md``.
3. Write the guide.


Hello Guide
------------

This same general structure and process should work for most guides.

Step 1. Launch **GuideIndex** and find the ``name`` of the guide
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. image:: ../assets/guide.png
    :width: 1000

Within VisiData, ``Space open-guide-index`` to see the Table of Contents.

``gv`` to show the **name** column. Choose a guide to work on, and note down its **name**.
For example: **MacrosSheet**.

::

Step 2. Create a markdown file ``visidata/guides/{name}.md``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this example, we would create a ``visidata/guides/MacrosSheet.md``.

At the top of the file, optionally place template front matter, using YAML syntax:

::

    ---
    sheettype: Sheet
    ---

This front matter will override the default configuration for guides.

At the moment, the main configurable option is ``sheettype``. ``sheettype`` is a string for the type of sheet that the guide involves. VisiData uses this to auto-complete command + options patterns (see "stylings of note").

Its default value is ``Sheet`` (aka TableSheet), the base class for every table sheet.

.. note::

    The vast majority of guides will not need to set the front matter or change the default sheet type. Only set it if command + option patterns are failing to auto-complete.

    Feel free to ask us if unsure which sheet type to use.



Step 3. Write the guide
~~~~~~~~~~~~~~~~~~~~~~~

Next, fill out the text for the guide:

::

    ---
    sheettype: Sheet
    ---
    # Macros
    Macros allow you to bind a command sequence to a keystroke or longname, to replay when that
    keystroke is pressed or the command is executed by longname.

    The basic usage is:
        1. {help.commands.macro-record}
        2. Execute a series of commands.
        3. `m` again to complete the recording, and prompt for the keystroke or longname to bind it to.

    # The Macros Sheet

    - {help.commands.macro-sheet}

    - `d` (`delete-row`) to mark macros for deletion.
    - {help.commands.commit_sheet}
    - `Enter` (`open-row`) to open the macro in the current row, and view the series of commands composing it.
    '''

Read the `style guide </docs/api/style>`_ for an overview of the stylings and markup.

---

### Api Index (`docs/api/index.rst`)

.. VisiData documentation master file, created by
   sphinx-quickstart on Wed Sep 23 19:53:19 2020.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

=====================================
VisiData Plugin Authors Guide (v2.0)
=====================================

VisiData is designed to be **extensible**.
Anything that can be done with Python (which is basically everything) can be exposed in a VisiData interface with a minimal amount of code.

VisiData is also designed to be **modular**.
Many of its features can exist in isolation, and can be enabled or disabled independently, without affecting other features.
Modules should degrade or fail gracefully if they depend on another module which is not loaded.

.. _hello-world:

Hello World
===========

As a simple example, this code can be placed in ``~/.visidatarc``:

::

    BaseSheet.addCommand('1', 'hello-world', 'status("Hello world!")')

This extends VisiData so that when :kbd:`1` is pressed on any sheet, the command called ``hello-world`` with the Python code ``status("Hello world!")`` is executed, and this prints the string "Hello world!" on the status bar.

----

.. toctree::
   :maxdepth: 1
   :caption: Table of contents
   :name: mastertoc

   options
   commands
   loaders
   extensible
   sheets
   columns
   data
   modify
   interface
   async
   guides
   style
   plugins

----

Toplevel functions
===================

.. autofunction:: visidata.vd.view
.. autofunction:: visidata.vd.view_pandas
.. autofunction:: visidata.vd.run

---

### Api Interface (`docs/api/interface.rst`)

Drawing
========

These functions should rarely be necessary to use.
``BaseSheet.draw`` must be overridden on non-tabular sheets.

.. autofunction:: visidata.BaseSheet.refresh
.. autofunction:: visidata.vd.redraw
.. autofunction:: visidata.BaseSheet.draw

.. autoclass:: visidata.SuspendCurses

Window
========

.. autoattribute:: visidata.BaseSheet.windowWidth
.. autoattribute:: visidata.BaseSheet.windowHeight

.. data:: visidata.TableSheet.topRowIndex

Top row in the window, as an index into ``TableSheet.rows``.  Settable.

.. data:: visidata.TableSheet.leftVisibleColIndex

Leftmost column in the window (after key columns), as an index into ``TableSheet.visibleCols``.  Settable.

Cursor
======

Every Sheet has a cursor, which makes it easy to interact individual elements and slices of data.

A ``TableSheet`` has a row cursor and a column cursor, which overlap on a single cell.

.. data:: visidata.TableSheet.cursorVisibleColIndex

Column cursor, as an index into ``TableSheet.visibleCols``.  Settable.

.. data:: visidata.TableSheet.cursorRowIndex

Row cursor as an index into ``TableSheet.rows``.  Settable.

.. autofunction:: visidata.BaseSheet.checkCursor

.. autoattribute:: visidata.TableSheet.cursorCol
.. autoattribute:: visidata.TableSheet.cursorValue
.. autoattribute:: visidata.TableSheet.cursorTypedValue
.. autoattribute:: visidata.TableSheet.cursorDisplay

.. autofunction:: visidata.TableSheet.cursorDown
.. autofunction:: visidata.TableSheet.cursorRight
.. autofunction:: visidata.TableSheet.moveToNextRow
.. autofunction:: visidata.TableSheet.moveToCol
.. autofunction:: visidata.TableSheet.moveToRow

.. _input:

Input
======

.. autofunction:: visidata.vd.input
.. autofunction:: visidata.vd.choose
.. autofunction:: visidata.vd.chooseOne
.. autofunction:: visidata.vd.chooseMany
.. autofunction:: visidata.vd.confirm
.. autofunction:: visidata.vd.launchEditor

.. _status:

Status
======

.. autofunction:: visidata.vd.status
.. autofunction:: visidata.vd.warning
.. autofunction:: visidata.vd.fail
.. autofunction:: visidata.vd.exceptionCaught
.. autoclass:: visidata.ExpectedException

Examples
^^^^^^^^

::

    with SuspendCurses():
        try:
            return subprocess.call('less', '/etc/passwd')
        except Exception as e:
            vd.exceptionCaught(e)

::

    passwd = vd.input("password: ", display=False)

    # initial value is the formatted value under the cursor
    vd.status(vd.input("text to show: ", value=cursorDisplay))

.. _colors:

Colors
======

The TableSheet allows rows, columns, or individual cells to be colorized according to a Python function.

See `Color Docs <../colors>`__ for information on how to configure specific colors and attributes.

.. data:: visidata.TableSheet.colorizers

List of Colorizers for this class.  Class member.

Do not manually update this list to add colorizers on a specific sheet.
Instead use addColorizer and removeColorizer.

.. autofunction:: visidata.TableSheet.addColorizer
.. autofunction:: visidata.TableSheet.removeColorizer

.. autoclass:: visidata.DisplayWrapper

.. autoclass:: visidata.RowColorizer
.. autoclass:: visidata.ColumnColorizer
.. autoclass:: visidata.CellColorizer

- *precedence*: use color from colorizer with the greatest *precedence* value.
- *coloropt*: name of color option, or ``None``.
- *func*: ``func(sheet, col, row, value)`` is called for each cell as it is being rendered. Return True when *coloropt* should be applied to the cell.  If *coloropt* is None, ``func(...)`` can return the relevant coloropt (or None) instead.

- Attributes (underline, bold, reverse) combine, regardless of *precedence*.  Only colors will override other.

- The *value* passed to the Colorizer *func* is a ``DisplayWrapper``, which encapsulates the entirety of a displayed cell.  Use ``val.value`` to get the typed value (or None), or ``val.display`` for the displayed string.

.. note::

    Colorizers are called for headers and separators, in addition to cells.
    For a header, *row* will be None; for a separator, *col* will be None.
    So a RowColorizer needs to include a check that *row* is valid, a ColumnColorizer needs to check that *col* is valid, and a cell colorizer needs to check both.
    A simple check for non-False (see Examples) is sufficient.



Examples
^^^^^^^^

::

    class DescribeSheet(ColumnsSheet):
        colorizers = [
            # colors key 'rows' on the like key columns
            RowColorizer(7, 'color_key_col', lambda s,c,r,v: r and r in r.sheet.keyCols),
        ]

    class OptionsSheet(TableSheet):
        colorizers = [
            # colors cells
            CellColorizer(3, None, lambda s,c,r,v: v.value if r and c in s.columns[1:3] and r.name.startswith('color_') else None)
        ]

Menu System
===========

Features and plugins can add commands to the hierarchical menu system (new in v2.6).

.. autofunction:: visidata.vd.runMenu
.. autofunction:: visidata.vd.addMenu
.. autofunction:: visidata.vd.addMenuItem

.. autoclass:: visidata.Menu
.. data:: visidata.vd.menus

`vd.menus` contains the hierarchical list all Menu() objects, for exploration.

Examples
^^^^^^^^

::

    # add a single command to an existing menusubmenu with multiple commands
    vd.addMenuItem('Column', 'Add column', 'enhance', 'enhance-col')

    # add a submenu with multiple commands
    vd.addMenu(Menu('Data', Menu('Enhance',
        Menu('this column', 'enhance-col'),
        Menu('all columns', 'enhance-cols'),
    )))

    # activate menu and navigate to Data»Enhance
    vd.runMenu('Data', 'Enhance')

---

### Api Loaders (`docs/api/loaders.rst`)

Loaders
=======

.. note::

    You are welcome to submit new loaders to core VisiData, or as plugins. Please, see our `checklists for contribution <https://visidata.org/docs/contributing>`__.

Creating a new loader for a data source is simple and straightforward.

1. ``open_filetype`` boilerplate
2. ``FooSheet`` subclass with rowtype and rowdef
3. ``FooSheet`` reload or iterload
4. ``FooSheet.columns``

Hello Loader
------------

Here's a step-by-line breakdown of a basic loader, which reads in a text file as a series of lines.
This same general structure and process should work for all loaders.

Step 1. ``open_<filetype>`` boilerplate
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    @VisiData.api
    def open_readme(vd, p):
        return ReadmeSheet(p.base_stem, source=p)

This is used for filetype ``readme``, which is used for files with extension ``.readme``, or when specified manually with the ``filetype`` option like ``--filetype=readme`` or ``-f readme`` on the command line.

The ``open_<filetype>`` function usually looks exactly like this, with only the type of :ref:`Sheet <sheets>` changed.

The *p* argument is a :ref:`visidata.Path<vd-path>`.

The actual loading happens in the Sheet.  An existing :ref:`sheet type<sheets>` can be used, or a new sheet type can be created.

::

Step 2. Create a Sheet subclass
-------------------------------

::

    class ReadmeSheet(TableSheet):
        rowtype = 'lines'   # rowdef: [str]

-  TableSheet (and its alias ``Sheet``) is the basic tabular sheet of rows
   and columns. Most loader sheets will inherit from TableSheet, but
   some might inherit from more specialized sheets if they share
   functionality, or from ``BaseSheet`` if they are not tabular (like
   the ``Canvas``).

-  The ``rowtype`` member is only displayed on the :ref:`right-hand
   status <status>`. It should be **plural**. If not given, it is
   "``rows``". It's helpful to give the user an subconscious check of
   the kind of sheet being shown.

-  The ``rowdef`` should be given for all loaders, even though it is
   only a comment. It specifies the expected Pythonic structure of the
   rows on this sheet. This is important because nearly every other
   component of the sheet depends on this structure.

Step 3. Load data into rows, and yield them one-by-one
------------------------------------------------------

``reload()`` is called when the Sheet is first pushed, and thereafter by the user with :kbd:`Ctrl+R`.
The default ``TableSheet.reload()`` iterates through the rows returned by ``TableSheet.iterload()``, and takes care of a few common tasks (like running async and resetting the ``rows`` member to a new list).

Each loader for a tabular sheet should overload ``iterload()``, which uses the Sheet ``source`` to populate and then yield each row one-by-one.

::

    class ReadmeSheet(TableSheet):
        rowtype = 'lines'   # rowdef: [str]

        def iterload(self):
            for line in self.source:
                yield [line]

.. warning::
   ``str`` by itself is not a valid rowdef.

   Each row must have a unique *rowid*, which by default is the Python ``id()`` of the row.
   Because Python interns common strings, strings with the same value will have the same *id*.
   This would break a lot of features, like row selection for instance.

   Also, as an immutable type, it would be annoying to not be able to modify it.

   So it needs to be wrapped in a Python ``list``, which is guaranteed to be unique, and also mutable.


-  ``sheet.source`` is the :ref:`visidata.Path<vd-path>` given as the *source* kwarg to ``ReadmeSheet()`` in ``open_readme``. 

  .. note::

     Any *kwarg* passed to a Sheet constructor will be stored on the sheet in an attribute of the same name.

  .. note::

    `visidata.Path <vd-path>` objects are Path-like but have some additional features, like being iterable (yielding their contents one line at a time).

   While there is a ``visidata.Path.read_text()`` function, do **not** use ``for line in p.read_text().splitlines()`` in a loader, as that will read the entire file before returning the first line.
   A loader must be able to handle arbitrary amounts of data (including data too large to fit in memory), so this will not work.

   ``Path.__iter__`` is optimized to read the file a small amount at a time, so ``for line in path`` is workable for a textual line-based file format.

- If the loader requires a third-party library, import it inside ``iterload()`` or ``reload()`` (or ``open_<filetype>`` if necessary).
  Do not import at the toplevel, or ``vd`` will fail to start when the library is not installed.
    - preferably, import it using ``modname = importExternal(modname, pythonPackageName``. If the user does not have the package installed, it will output instructions to ``pip3 install pythonPackageName``.

.. autofunction:: visidata.vd.importExternal

By default, a Sheet has one Column which just displays a string representation of the row.
So the above example is a good starting point for any loader; just get the rows however they come most easily from the source, and launch ``vd`` with a sample dataset in that format.
Then use :kbd:`Ctrl+Y` to explore the resulting Python object, to find what attributes to show on the sheet.

reload()
~~~~~~~~

For more control over the whole loading process, ``BaseSheet.reload()`` can be overridden instead of ``iterload()``:

::

        @asyncthread
        def reload(self):
            self.rows = []
            for line in self.source:
                self.addRow([line])

-  ``@asyncthread`` launches the decorated function in its own thread.  See :ref:`Threads<threads>`
-  ``sheet.rows`` must always be reset to a new list.  **Never** call ``sheet.rows.clear()``.
-  Always add rows using :ref:`addRow()<sheets>`.

Supporting asynchronous loaders
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Loading a large dataset in the main thread will cause the interface to freeze.
However, the basic TableSheet ``reload`` and ``iterload`` structure results in an :ref:`asynchronous <threads>` loader by default.
Since rows are yielded **one at a time**, they become available as they are loaded, and ``reload`` itself is decorated with an ``@asyncthread``, which causes it to be launched in a new thread.

- All row iterators should be wrapped with :ref:`Progress<progress>`.
  This updates the progress percentage as it passes each element through.

- Do not depend on the order of ``rows`` after they are added; e.g. do not reference ``rows[-1]``.  The order of rows may change during an asynchronous loader.

- Catch any ``Exception`` that might be raised while handling a specific row, and add them as the row instead.  Uncaught exceptions will cause the loader thread to abort.

- Do not use a bare ``except:`` clause or the loader thread will not be cancelable with :kbd:`Ctrl+C`.

Progress and Exception example
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

        class FooSheet(Sheet):
            ...
            def iterload(self):
                for bar in Progress(foolib.iterfoo(self.source.open_text())):
                    try:
                        r = foolib.parse(bar)
                    except Exception as e:
                        r = e
                    yield r

Testing for Loader Performance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Test the loader with a very large dataset to make sure that:

-  the first rows appear immediately;
-  the progress percentage is being updated;
-  the loader can be cancelled (with :kbd:`Ctrl+C`).

.. _enumerate-columns:

Step 4. Enumerate the Columns
------------------------------

Each sheet has a ``columns`` attribute with a unique list of ``Column`` objects. Each ``Column`` provides a
different view into the row.

::

        class FooSheet(Sheet):
            rowtype = 'foobits'  # rowdef: foolib.Bar object

            columns = [
                ColumnAttr('name'),  # foolib.Bar.name
                Column('bar', getter=lambda col,row: row.inside[2],
                              setter=lambda col,row,val: row.set_bar(val)),
                Column('baz', type=int, getter=lambda col,row: row.inside[1]*100)
            ]

In general, set ``columns`` as a class member containing a list of
static columns. If the columns aren't known until data is loaded,
reload/iterload can add new columns using :ref:`addColumn() <tablesheet>`.

If the rowdef is a ``list``, and the columns are dynamic, :ref:`SequenceSheet.reload() <other-sheets>` could handle the Column creation.

::

    class FooSheet(SequenceSheet):
        rowtype = 'foobits'  # rowdef: a list, which is a sequence of values

        def iterload(self):
            with foolib.iterfoo(self.source.open_text() as f:
                r = foolib.parse(bar)
                yield r


Column attributes
~~~~~~~~~~~~~~~~~

Columns have several attributes; all except *name* are **optional** arguments to the constructor:

-  *name*: should be a valid Python identifier and unique among the column names on the sheet.  (Otherwise the column cannot be used in an expression.)
-  *type*: can be ``str``, ``int``, ``float``, ``date``, ``currency``, or a custom type.  By default it is ``anytype``, which passes the original value through unmodified.
-  *width*: the initial width for the column. ``0`` means hidden; ``None`` (default) means calculate on first draw.

Column getters can be any function, but many loaders are satisfied with a static list of ``ItemColumn`` (for values in dict and list rowdefs) and/or ``AttrColumn`` (for a members or attributes directly on the row object).
This is dependent on the loader function; some loaders may prefer to do less parsing to load faster, and then the Columns will need to be correspondingly more complicated.

See the :ref:`Columns section <columns>` for a complete API.

Passthrough options
~~~~~~~~~~~~~~~~~~~

Loaders which use a Python library (internal or external) are encouraged to pass its kwargs using ``**options.getall("foo_")`` interface.
For modules like ``csv`` which expose them as kwargs to some function or constructor, this is very easy:

::

        rdr = csv.reader(fp, **csvoptions())

Full Example
~~~~~~~~~~~~

This is a completely functional loader for the ``sas7bdat`` (SAS dataset file) format, thanks to Jared Hobbs' `sas7bdat package <https://bitbucket.org/jaredhobbs/sas7bdat>`__.

::

    from visidata import Sheet, ItemColumn, Progress

    @VisiData.api
    def open_sas7bdat(vd, p):
        return SasSheet(p.base_stem, source=p)

    class SasSheet(Sheet):
        def iterload(self):
            import sas7bdat
            SASTypes = { 'string': str, 'number': float, }

            self.dat = sas7bdat.SAS7BDAT(str(self.source),
                                         skip_header=True,
                                         log_level=logging.CRITICAL)

            self.columns = []
            for col in self.dat.columns:
                self.addColumn(ItemColumn(col.name.decode('utf-8'),
                                         col.col_id,
                                         type=SASTypes.get(col.type, anytype)))

            with self.dat as fp:
                yield from Progress(fp, total=self.dat.properties.row_count)


Guessing Filetypes
==================

When loading a file, VisiData tries to infer its filetype by peeking at the initial lines of the file and guessing from its structure.

``vd.guess_<filetype>(path)`` contains this logic for checking whether a file might be ``<filetype``.

If those structures are not present, the function should return nothing. If they are, the function should return a dictionary with:

- ``filetype`` being the filetype they detect (corresponding to the ``vd.open_<filetype>``)
- ``_likelihood`` (optional) being a number from 0-10, 10 being most likely and 0 meaning a last ditch effort if nothing else will take it
- any other key/values will be set as options on the *Sheet* the ``open_<filetype>`` function returns

`Examples of guess_filetype functions <https://github.com/saulpw/visidata/commit/4743f92bb855cf931d896e65845c549ce6027e2f>`_


::

        @VisiData.api
        def guess_foo(vd, p):
            import foobar
            if p.open_text().read(8).startswith("#Foo"):
                enc = foobar.encoding(p)
                return dict(filetype='foo', foo_encoding=enc)


Savers
=======

A full-duplex loader requires a **saver**.
The saver iterates over all ``rows`` and ``visibleCols``, calling ``getValue``, ``getDisplayValue`` or ``getTypedValue`` as the saving format allows, and saves the results in its format to the given *path*.
Savers should be decorated with ``@VisiData.api`` in order to make them available through the ``vd`` object's scope.

.. autofunction:: visidata.vd.save_txt

-  *p* is a :ref:`visidata.Path <vd-path>` object referencing the file being written to.
-  *sheets* is a list of 1 or more sheets to be saved.

The saver should preserve the column names and translate their types into ``foolib`` semantics, but other attributes on the Columns are generally not saved.

Savers which can handle typed values should use ``Column.getTypedValue``, and displayable savers (like html, markdown, csv) should use ``Column.getDisplayValue`` (which takes into account the column's *fmtstr*).

Example
^^^^^^^

With this example, saving as filetype ``table`` will call the `tabulate library <https://github.com/astanin/python-tabulate>`__ to save the data in any number of text formats, specified by the ``tbl_tablefmt`` option.
(Several built-in savers use ``tabulate`` also, but those savers work a little differently, as each tablefmt is available as a direct save filetype.)

::

    vd.option('tbl_tablefmt', 'simple', 'file format to save with "table" filetype')

    def get_rows(sheet, cols):
        for row in Progress(sheet.rows):
            yield [ col.getDisplayValue(row) for col in cols ]

    @VisiData.api
    def save_table(path, *sheets):
        import tabulate

        with path.open_text(mode='w') as fp:
            for vs in sheets:
                fp.write(tabulate.tabulate(
                    get_rows(vs, vs.visibleCols),
                    headers=[ col.name for col in vs.visibleCols ],
                    **options.getall('tbl_')))

.. _vd-path:

visidata.Path
=============

``visidata.Path`` is a wrapper around Python's builtin ``pathlib.Path`` that can also handle non-filesystem files (URLs, stdin, files within archives).

The ``given`` attribute is new to ``visidata.Path``.
Other functions listed here are wrappers around the equivalent ``pathlib.Path`` functions, with specialized functionality as needed for non-filesystem files.
All other accesses are forwarded to the inner ``pathlib.Path`` object, but will probably not work for non-filesystem files.

.. autoattribute:: visidata.Path.given

.. autofunction:: visidata.Path.exists
.. autofunction:: visidata.Path.open
.. autofunction:: visidata.Path.open_text
.. autofunction:: visidata.Path.read_text
.. autofunction:: visidata.Path.open_bytes
.. autofunction:: visidata.Path.read_bytes
.. autofunction:: visidata.Path.stat
.. autofunction:: visidata.Path.with_name

URL Scheme Loaders
---------------------------------------

When VisiData tries to open a URL with schemetype of ``foo`` (i.e.
starting with ``foo://``), it calls ``openurl_foo(urlpath, filetype)``.
``urlpath`` is a ``UrlPath`` object, with attributes for each of the
elements of the parsed URL.

``openurl_foo`` should return a Sheet or call ``error()``. If the URL
indicates a particular type of Sheet (like ``magnet://``), then it
should construct that Sheet itself. If the URL is just a means to get to
another filetype, then it can call ``openSource`` with a Path-like
object that knows how to fetch the URL:

::

        def openurl_foo(p, filetype=None):
            return openSource(FooPath(p.url), filetype=filetype)

---

### Api Modify (`docs/api/modify.rst`)

===============
Modifying Data
===============


When adding rows, deleting rows, or modifying cell values, **always use the functions documented in this API**.
Use ``addRow()`` to add a row, ``deleteBy()`` or ``deleteSelected()`` to delete a row, and the ``setValue()`` family of functions to modify values.
Even though ``setValue`` will call ``putValue`` and that may be defined by the plugin itself, do not modify rows manually.
This also means that rows should not be added or removed from ``sheet.rows`` directly.

Always use these functions, as they do some important bookkeeping for features like deferred modifications.

.. autofunction:: visidata.TableSheet.newRow
.. autofunction:: visidata.TableSheet.addNewRows
.. autofunction:: visidata.TableSheet.editCell
.. autofunction:: visidata.TableSheet.deleteBy

Deferred Modification
======================

On sheets where ``putValue`` changes the source directly, like **SQLite** and **DirSheet**, it would be undesirable for modifications to happen immediately.
For these cases, the modifications are cached, shown visually on the sheet in a different color, and deferred until they are committed with an explicit ``commit-sheet`` (bound to :kbd:`z Ctrl+S`).

Sheet types can add this deferring behavior by setting their ``defer`` attribute to ``True`` (either on the class type or on the individual sheet instance), but they may have to implement their own ``Sheet.putChanges``.

.. autofunction:: visidata.Column.getSourceValue

.. autofunction:: visidata.TableSheet.commit
.. autofunction:: visidata.TableSheet.getDeferredChanges
.. autofunction:: visidata.TableSheet.putChanges

Undo
-------

When a function adds, removes, or modifies data--including metadata--as part of a user action, it needs to ensure that operation is **undoable**.
For many operations, this is already handled internally by existing functions.
For instance, if the user executes a command which modifies cell values via ``Column.setValues``, the ``undo`` command will undo those modifications automatically.

If a new operation changes the data or metadata and undo is not already handled by existing functionality, then the new operation needs to call ``addUndo()`` to add an *undo function* to the currently executing command.

For performance reasons, undo functions should generally be as large as possible.
For instance, ``Column.setValues`` adds a single undo function to undo all changes in one go.
By contrast, ``Column.setValue`` does not add an undo function at all.
Therefore, commands should always use ``Column.setValues``, unless for some special reason the modification should not be undoable.
Undo functions, however, should use ``Column.setValue``, so it doesn't add an undo function for the undo itself.
(Redo is handled by replaying the previous command, not by undoing the undo).

.. autofunction:: visidata.vd.addUndo
.. autofunction:: visidata.vd.addUndoSetValues


Example
--------

.. note::

    ``Sheet.editCell`` requires row and column *indexes* for its parameters.
    For example, ``cursorVisibleColIndex`` and ``cursorRowIndex`` are sheet attributes that can be passed to ``Sheet.editCell``.

::

    Sheet.addCommand('e', 'edit-cell',
                     'cursorCol.setValues([cursorRow], editCell(cursorVisibleColIndex))',
                     'edit contents of current cell')


::

    # equivalent to Sheet.deleteSelected but slower
    sheet.deleteBy(sheet.isSelected)

::

    sheet.addRow(row)

---

### Api Modules (`docs/api/modules.rst`)

# Module abilities

Internal modules and external plugins can:

- define new commands
- add new options
- set existing options
- maintain state on vd singleton or sheets or columns
- add arbitrary Python code before/after most functions
- replace functions
- add new sheet types
- new file formats and urls
- have lazy dependencies (with vd.importExternal)

# Module imports

By default, VisiData loads all installed plugins and all of its own internal modules.
Some modules (esp loaders) may have dependencies in PyPI that aren't installed.

Modules must lazily import any external dependencies only when the feature or loader is attempted to be used.  i.e. modules must not import external dependencies at file scope.  For loaders this should happen in reload/iterload() instead of vd.open_foo, if possible.

The import should be with vd.importExternal so that VisiData can give a command to install the library via pip.

Some modules are not compatible with each other; for instance, the traditional visidata sqlite loader which loads all data into memory, and the vdsql sqlite loader which uses Ibis to compose SQL queries that get executed against the sqlite database itself.

The last module to be imported will have its functions registered for the filetype or command.  This should work fine for most cases (latter modules are generally more advanced), but to use the former module, this module must not be loaded in the first place.

---

### Api Options (`docs/api/options.rst`)

Options
========

Adding to the :ref:`hello world<hello-world>` example from the intro, the displayed text could be made configurable with an option:

::

    vd.option('disp_hello', 'Hello world!', 'string to display for hello-world command')

    BaseSheet.addCommand('1', 'hello-world', 'status(options.disp_hello)')


Now the user can set the option to modify which text is displayed during their session when they press :kbd:`1`.
For example, on the command line (note that the underscores can be converted to hyphens):

::

    vd --disp-hello="¡Hola mundo!"

The user can override it persistently for every session by adding a line to their ``.visidatarc``:

::

    options.disp_hello = 'Bonjour monde!'


Options Context
~~~~~~~~~~~~~~~

Options can have different values depending on the context in which they're used.
For instance, one TSV sheet might need its ``delimiter`` set to "``|``", while another TSV sheet in the same session might need to use the default ("``\t``", or TAB) instead.

Options can be overridden globally, or for all sheets of a specific type, or only for one specific sheet.

The options context should be referenced directly when setting:

    - ``sheet.options`` to *set* an option on a specific sheet instance (**sheet override**).
    - ``<SheetType>.options`` to *set* a option default for a particular type of Sheet (**class override**).
    - ``vd.options`` (or plain ``options``) to *set* an option globally for all sheets, except for sheets that have a sheet override (**global override**).

.. note::

    Class overrides prior to v2.10
    ------------------------------

    In VisiData v2.10 and earlier, class overrides had to be specified using ``<SheetType>.class_options``.
    In subsequent versions, ``.options`` can be used for getting and setting options at all levels.

Use ``sheet.options`` to *get* an option within the context of a specific sheet.
This is strongly preferred, so the user can override the option setting on a sheet-specific basis.
However, some options and situations are truly sheet-agnostic, and so ``vd.options`` (or plain ``options``) will *get* an option using the context of the **top sheet**.

When getting an option value, VisiData will look for a sheet override first, then class overrides next (from most specific subclass all the way up to BaseSheet), then a global override, before returning the default value from the option definition itself.

In general, plugins should use ``FooSheet.options`` to override values for the plugin-specific sheet type.

.. note::

    Performance
    ------------------

    Because it allows so much flexibility, getting option values is a comparatively slow operation.  For tight loops, save the option into a local variable in an outer block.  An option value can only change between commands anyway.

Options API
~~~~~~~~~~~~~~~

.. autofunction:: visidata.vd.options.__getattr__

``x = sheet.options.hello_world`` is the preferred style for getting a single option value.

.. autofunction:: visidata.vd.options.__setattr__

``sheet.options.hello_world = "Привет мир"`` is the preferred style for setting a single option value.

.. autofunction:: visidata.vd.options.get
.. autofunction:: visidata.vd.options.set
.. autofunction:: visidata.vd.options.unset
.. versionadded:: 2.1
.. autofunction:: visidata.vd.options.getall

The dict returned by ``options.getall('foo_')`` is designed to be used as kwargs to other loaders, so that their options can be passed through VisiData transparently.
For example, to pass all csv options through to the builtin Python ``csv`` module:

::

    csv.reader(fp, **sheet.options.getall('csv_'))

.. autofunction:: visidata.vd.option

.. note::

    If the option affects loading, transforming, or saving, then set *replay* to True.

Rules for naming options
^^^^^^^^^^^^^^^^^^^^^^^^

- Options defined within a plugin should all start with the same short module abbreviation, like "``mod_``".
- Except for theme option names, which should start with "``disp_``" for a displayed string and "``color_``" for a color option (see :ref:`Colors<colors>`).
- Use common abbreviations instead of full words.
- Use "``_``" (underscore) to separate words.
- Keep the option name length under 20 characters.  Maximum of 3 words (2 separators).

.. note::

    * Consider whether some subset of options can be passed straight through to the underlying Python library via kwargs (maximum power with minimal effort).

Option type
^^^^^^^^^^^^

When setting an option, strings and other types will be coerced to the type of the *default*.  An ``Exception`` is raised if conversion fails.  A *default* of ``None`` allows any type.

Examples
~~~~~~~~~

::

    vd.option('disp_hello', 'Hello world!', 'string to display for hello-world')

    # option without regard to a current sheet (global override only)
    vd.options.disp_hello = 'こんにちは世界'

    # show the option value given the context of a specific sheet
    vd.status(sheet.options.disp_hello)

    # option set on specific sheet only
    sheet.options.color_current_row = 'bold blue'

    # option set for all DirSheets
    DirSheet.options.color_current_row = 'reverse green'

---

### Api Plugins (`docs/api/plugins.rst`)

Plugins
========


Plugin file structure
----------------------

Most features can be self-contained in their own .py file, so that the feature is enabled or disabled by ``import``-ing that .py file or not.
A plugin is usually a single .py file, installed on the same computer as visidata.
`Instructions on how to install a plugin </docs/plugins/>`__

Plugins often start as a small snippet in ``.visidatarc``, and then the code is migrated to a separate file when it gets too large to share with other people via a short code snippet.
The actual code in either case should be identical.

.. note::

    To quickly install a personal plugin, place the plugin in the ``$HOME/.visidata/plugins/`` directory, and add an ``import`` statement to ``$HOME/.visidata/plugins/__init__.py``.
    At startup, VisiData will automatically import this ``plugins`` package and all the included plugins.

To publish a plugin, create a public repo with a .py file. To package it within VisiData, `open a PR on the VisiData repo <https://github.com/saulpw/visidata/pulls>`_, placing the code in the `the experimental folder <https://github.com/saulpw/visidata/tree/develop/visidata/experimental>`_.


Complete "Hello world" plugin example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

hello.py
^^^^^^^^^^^^^^^^

::

    '''This plugin adds the ``hello-world`` command to all sheets.
    Press ``0`` to show display options.disp_hello on the status line.'''

    __author__ = 'Jo Baz <jobaz@example.com>'

    vd.option('disp_hello', 'Hello world!', 'string to display for hello-world command')

    BaseSheet.addCommand('0', 'hello-world', 'status(options.disp_hello)')

.. note::

Notes:

- There should be a searchable docstring description of the core features.
- Optionally include author metadata.
- Options at the top, commands at the bottom.
- Avoid toplevel imports of non-stdlib Python extensions.

---

### Api Requirements (`docs/api/requirements.txt`)

recommonmark
sphinx
sphinx-markdown-tables
sphinx-argparse
git+https://github.com/saulpw/visidata.git@develop

---

### Api Sheets (`docs/api/sheets.rst`)

.. _sheets:

Sheets
====================================

A Sheet is a representation of some source, usually a :ref:`Path <vd-path>`.
A path can be opened, returning a sheet, and/or a sheet can be saved to path.

.. autofunction:: visidata.vd.openPath
.. autofunction:: visidata.vd.openSource

.. autofunction:: visidata.vd.saveSheets

Sheets API
-----------

.. autoclass:: visidata.BaseSheet

.. autoattribute:: visidata.BaseSheet.name

.. autofunction:: visidata.BaseSheet.__len__
.. function:: visidata.BaseSheet.__copy__(self)

    Return shallow copy of sheet.

.. autofunction:: visidata.BaseSheet.reload

.. _tablesheet:

TableSheet
^^^^^^^^^^^

Sheets are generally specialized for their rowtype, and TableSheet, used for sheets with rows and columns, is the most common base class.
(So common, that it was originally just called ``Sheet``.  For clarity, ``TableSheet`` is preferred, but ``Sheet`` is a valid alias that will never be deprecated.)

.. autoclass:: visidata.TableSheet

.. data:: visidata.TableSheet.rows

List of row objects on this sheet.

.. data:: visidata.TableSheet.columns

List of all Column objects on this sheet (including hidden columns).


.. autoattribute:: visidata.TableSheet.nRows
.. autoattribute:: visidata.TableSheet.nCols

.. autoattribute:: visidata.TableSheet.visibleCols
.. autoattribute:: visidata.TableSheet.nVisibleCols

.. autofunction:: visidata.TableSheet.addRow
.. autofunction:: visidata.TableSheet.addColumn

.. autofunction:: visidata.TableSheet.iterload

.. autofunction:: visidata.TableSheet.rowid
.. autofunction:: visidata.TableSheet.column

.. autofunction:: visidata.TableSheet.openRow
.. autofunction:: visidata.TableSheet.openCell
.. autofunction:: visidata.TableSheet.gatherBy

.. _other-sheets:

Other Common Sheet Types
^^^^^^^^^^^^^^^^^^^^^^^^^^

These can be used as base classes, or instantiated as they are, or :ref:`used as templates <enumerate-columns>` for rows with similar structure,

.. autoclass:: visidata.IndexSheet
.. autoclass:: visidata.TextSheet
.. autoclass:: visidata.SequenceSheet
.. autoclass:: visidata.PyobjSheet

The Sheet Stack
-----------------

.. data:: visidata.vd.sheets

The "sheet stack", a list of active sheets (available as the **Sheets Sheet** via :kbd:`Shift+S`).
The top sheet, or the displayed sheet, is the first item in the list (``vd.sheets[0]``).

.. data:: visidata.vd.sheet

The top sheet on the sheet stack.

.. autofunction:: visidata.vd.push
.. autofunction:: visidata.vd.replace
.. autofunction:: visidata.vd.remove
.. autofunction:: visidata.vd.quit

.. autofunction:: visidata.vd.getSheet

---

### Api Style (`docs/api/style.rst`)

Guide Style Guide
=================

General Notes
-------------

- Review first sentences `with this in mind <https://jamesg.blog/2023/12/03/first-sentences/>`_.
- In general, do not use second person (you/your) outside of tutorials.

Referenced from `Simplified Technical English <https://en.wikipedia.org/wiki/Simplified_Technical_English>`_:

- Make instructions as clear and specific as possible.
- Use the approved forms of the verb:
    - The infinitive
    - The imperative
    - The simple present tense
    - The simple past tense
    - The past participle (only as an adjective)
    - The future tense
- Do not use helping verbs to make `complex verb structures <https://www.utsa.edu/twc/documents/Complex%20Verb%20Phrases.pdf>`_.
- Use the "-ing" form of a verb only as a technical name or as a modifier in a technical name.
- Do not use passive voice in procedures.
- Use the active voice as much as possible in descriptive texts.
- Prefer short sentences and simpler word choices.
- Do not omit parts of the sentence (e.g. verb, subject, article) to make your text shorter.
- Use vertical lists for complex text.
- Write one instruction per sentence.
- Write only one topic per paragraph.

Color/Markup Standards
-----------------------

- VisiData documentation uses `basic markdown <https://www.markdownguide.org/basic-syntax/>`_ like # Headings, \*\*bold\*\*, \*italics\*, \`code snippets\`, and \_underscore\_.

- Internally, VisiData has its `own display attribute syntax </docs/colors#attrs>`_. Some example usage:
    - ``[:onclick <url>]<text>[/]`` formats ``<text>`` into a clickable url that will open in ``$BROWSER``.
    - ``[:red on black]<sentence>[/]`` changes the color of ``<sentence>`` to be red text on black background.
      Any VisiData color option name can be used after ``:``, like ``[:warning]``, ``[:error]``, ``[:menu]``.
      Preference is for using ``[:option_name]`` instead of hard-coded colors when possible.
    - e.g. Use ``[:keys]`` for keystrokes and longnames, and ``[:code]`` for Python and other actual code.

- List relevant commands with the general pattern: ``- [:keys]<keystroke>[/] ([:code]<longname>[/]) to <command helpstring>``.
    - Use ``{help.commands.<longname>}`` to get the **GuideSheet** to replace it with the properly formatted string above. You can write it out manually if that would result in more detail or clarity, but it's much more preferred to adjust the docstring so that this pattern would work. 

- List relevant options with ``[:onclick options-sheet <option name>][:code]option name>[/] to <option helpstring> (default: <option default value>)``.
  - Similarly, use ``{help.options.<option-name>}`` to expand into the above, and prefer to modify the helpstring instead of writing it out manually.

---

--- METRICS ---
- Files Extracted: 57
- Estimated Token Budget: ~60799 tokens
- Canonical Reference: https://codewiki.google/github.com/saulpw/visidata