visidata Documentation

Technical Documentation: saulpw/visidata

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

VisiData v3.4

A terminal interface for exploring and arranging tabular data.

VisiData supports tsv, csv, sqlite, json, xlsx (Excel), hdf5, and many other formats.

Platform requirements Copy Chapter

• Linux, OS/X, or Windows (with WSL)
• Python 3.8+
• additional Python modules are required for certain formats and sources

Install Copy Chapter

To install the latest release from PyPi:

pip3 install visidata

To try VisiData without installing, use pipx or uv:

pipx run visidata          # or: uvx visidata

To install permanently (adds vd to your PATH):

pipx install visidata      # or: uv tool install visidata

Additional Python packages are needed for some formats:

pipx install visidata --preinstall openpyxl --preinstall lxml
# or: uv tool install visidata --with openpyxl --with lxml

To add format packages to an existing pipx install:

pipx inject visidata openpyxl lxml

To install the cutting edge develop branch (no warranty expressed or implied):

pip3 install git+https://github.com/saulpw/visidata.git@develop

If VisiData reports package X not installed even after you installed it, vd
is almost certainly running from a different environment than the one you
installed the package into. Check which vd, then install the package into
that environment — e.g. pipx inject visidata X for a pipx install, or the
Homebrew Python for brew install visidata.

See visidata.org/install for detailed instructions for all available platforms and package managers.

Usage Copy Section

$ vd 
$ <command> | vd

Press Ctrl+Q to quit at any time.

Hundreds of other commands and options are also available; see the documentation.

Documentation Copy Section

• VisiData documentation
• Plugin Author's Guide and API Reference
• Quick reference (available within vd with Ctrl+H), which has a list of commands and options.
• Intro to VisiData Tutorial by Jeremy Singer-Vine

Help and Support Copy Section

If you have a question, issue, or suggestion regarding VisiData, please create an issue on Github or chat with us at #visidata on irc.libera.chat.

If you use VisiData regularly, please support me on Patreon!

License Copy Chapter

Code in the stable branch of this repository, including the main vd application, loaders, and plugins, is available for use and redistribution under GPLv3.

Credits Copy Chapter

VisiData is conceived and developed by Saul Pwanson <[email protected]>.

Anja Kefala <[email protected]> maintains the documentation and packages for all platforms.

Many thanks to numerous other contributors, and to those wonderful users who provide feedback, for helping to make VisiData the awesome tool that it is.

Overview (`dev/README.md`) Copy Section

dev/

Resources for VisiData contributors and maintainers.

Contributor Guides Copy Chapter

Checklists Copy Chapter

Design Docs Copy Chapter

See design/README.md for full index.

Scripts Copy Chapter

Data Files Copy Chapter

Test Fixtures Copy Chapter

Packaging Copy Chapter

Media Copy Chapter

Other Copy Chapter

Overview (`docs/README.md`) Copy Section

VisiData Documentation Index

Guide to what each file covers, so it's clear which files to update when user-facing behavior changes.

Common mappings Copy Chapter

• New/changed commands or keybindings → man.md, plus the relevant topic file
• New/changed options → customize.md, man.md
• New/changed file format support → formats.md
• New/changed column or row operations → columns.md, rows.md, edit.md
• New features may need a new doc file (add it to this README.md too)

User Guides Copy Chapter

• index.md: Getting started, links to demos and tutorials
• loading.md: Specifying source files, piping data in
• usage.md: Loading files, exiting, specifying file types
• navigate.md: Movement commands, vim-style keys, scrolling, searching
• move.md: Moving around and searching (FAQ-style)
• columns.md: Column manipulation: pin, move, hide/unhide, resize
• rows.md: Selecting, filtering, and toggling rows
• edit.md: Cell editing, bulk updates, regex substitution, expressions
• crud.md: Creating blank sheets, rows, and columns
• group.md: Grouping data, aggregators, descriptive statistics
• freq.md: Frequency tables, binning (discrete/numeric), pivot sheets
• join.md: Combining datasets via the Sheets Sheet
• graph.md: Drawing graphs, visualizing numeric variables
• graphics.md: Low-res terminal graphics, braille Unicode, Canvas classes
• colors.md: Color themes, interface color customization
• customize.md: .visidatarc, command bindings, Options Sheet, XDG support
• macros.md: Recording and replaying command sequences
• menu.md: Toplevel menubar, arrow key/hjkl navigation
• mouse.md: Mouse interaction options
• pipes.md: Using VisiData in stdin/stdout pipelines
• dirsheet.md: Directory Sheet (DirSheet): browsing, editing, previewing files
• split.md: Split screen, viewing two sheets simultaneously
• save-restore.md: Saving and replaying sessions
• plugins.md: Installing plugins (builtin, pip, local plugins directory)
• shell.md: ZSH completion
• formats.md: Supported file formats, loaders/savers, dependencies
• internal_formats.md: .vd, .vdj, .vdx (command logs) and .vds (sheet state) formats
• gmail.md: Gmail access via OAuth 2.0 setup
• man.md: Quick reference / man page with full command and option listing

Developer/Contributor Docs Copy Chapter

• viewtsv.md: Annotated minimal 25-line VisiData app example
• test.md: Functional testing with .vd scripts
• contributing.md: Checklists for submitting loaders and plugins

API Reference (api/) Copy Chapter

Plugin/extension author documentation (reStructuredText, built with Sphinx).

• api/index.rst: Plugin Authors Guide overview
• api/sheets.rst: BaseSheet, openPath/openSource
• api/loaders.rst: Creating new loaders
• api/commands.rst: Command architecture, adding commands
• api/columns.rst: Column calculation engine (calcValue, getValue, etc.)
• api/plugins.rst: Plugin file structure and publishing
• api/options.rst: Creating configurable options
• api/async.rst: @asyncthread, threading, responsive UI
• api/data.rst: Selected rows API
• api/interface.rst: Drawing functions, window dimensions
• api/canvas.rst: Canvas graphics primitives
• api/extensible.rst: Extensible base class, monkey-patching

Other Assets Copy Chapter

• casts/: Asciinema terminal recordings embedded in various docs
• assets/: Screenshots and images (DirSheet, color options, Gmail OAuth steps, etc.)

Overview (`dev/design/README.md`) Copy Section

Design Docs

Architecture and feature design documents for VisiData. See also 175-design-terms.md for the vocabulary used in these docs.

Overview (`visidata/apps/galcon/README.md`) Copy Section

Note: This code has not been updated to run with the latest VisiData.

Galactic Conquest in VisiData

Take over the galaxy with your warship fleet!
Starting with only one planet and 10 ships, you compete with any number of enemies for the most planets and thus galactic glory.

At first, most planets are inhabited by savages with only a few spaceships for battle.
Some planets have better pilots, and ships from those planets have a higher kill rate in battle.

To capture a planet, you need to send a strong enough fleet. Deploy ships from your planets and conquer the galaxy!

Galactic Conquest (or "galcon" as it is sometimes called) is turn-based.
When all players have submitted their moves, the turn is over.
Ships arrive, battles are resolved, and planets produce more ships.
Then the next turn begins.

Instructions Copy Chapter

Starting a game Copy Section

The host has to start a server:

./galcon-server.py [-p ]

This will output a URL for players to connect to. All players then start their clients and login:

vd -f galcon http://your.ip.address:8080

Of course, the given port on your.ip.address must be open to all players.

The first time a player name is used, the password is set.
Players join the game if it has not yet started (each server only hosts one game).

Each player presses Ctrl+S when they are ready for the game to start.

Press Alt+M to go to the Map for this game, or Alt+P for the list of planets.
Before the game has started, any player can change Alt+Game options or press Alt+N to generate a New map.
Ctrl+R will reload the current sheet from data on the server.

When the map and all players are ready, the game begins.

Playing the game Copy Section

From any sheet:

• Alt+M opens the planet map
• Alt+P opens the planet sheet
• Alt+U opens the queued deployments sheet (not yet sent to server)
• Alt+D opens the full deployments sheet
• Alt+E opens the events sheet
• Alt+Q allows players to quit the game
• Ctrl+S submits the pending deployments to the server and signals the player is done

On the Planets sheet:

• m to mark the destination planet for a player's ships
• f to indicate the source planet
  • The player is then prompted to specify how many ships they wish to deploy from the source planet to the marked planet
• gf to send the provided number of ships from all selected planets to the marked planet

On the Map sheet:

• The colour of the planet indicates which player currently 'owns' it
• White planets are neutral and currently not in anybody's control
• v to cycle through name, prod, killpct and numships fields
• m and f to mark and deploy on the map sheet, same as on the Planets sheet

On the Queued Orders sheet:

• e to modify an order
• d to delete an order

Using the Dockerfiles Copy Section

Create an ssh deploy key that has permissions for vdplus. Here, it is called bluebird_docker

# build the image for the galcon server
sudo docker build --build-arg SSH_PRIVATE_KEY="$(cat bluebird_docker)" -t galcon-server -f Dockerfile.galcon-server .
# run the galcon server
sudo docker run -d -p 7777:8080 --rm galcon-server

# get ip address of where galcon server is running
sudo docker inspect GALCON_SERVER_CONTAINER_NUMBER | jq '.[0].NetworkSettings.IPAddress'

# build the image for the client
sudo docker build --build-arg SSH_PRIVATE_KEY="$(cat bluebird_docker)" -t galcon-client -f Dockerfile.galcon-client .

# for each person that is playing, run the docker client
sudo docker run -e IPADDR="GALCON_SERVER_IP_ADDRESS" -it --rm galcon-client

Overview (`visidata/apps/vdsql/README.md`) Copy Section

vdsql: [VisiData](https://visidata.org) for Databases

A VisiData interface for databases.

Powered by Ibis.

Features Copy Chapter

• query data in VisiData from any supported backend
• compose complex queries using VisiData commands instead of writing SQL
• output resulting query in SQL, Substrait, or Python

Requirements Copy Chapter

• Python 3.10+
• VisiData 3.0+
• Ibis 12.0+

Installation Copy Chapter

Install from pip Copy Section

This installs both:

• the usual vd with the vdsql plugin available (use -f vdsql to use Ibis instead of builtin loaders),

• the vdsql script that acts identically to vd but will use Ibis instead of VisiData's builtin loader.

  pip install vdsql

Install manually (cutting edge development) Copy Section

git clone [email protected]:saulpw/visidata.git
cd visidata/visidata/apps/vdsql
pip3 install .

Install Ibis backends Copy Section

To minimize dependencies, only the sqlite backend is included by default.
Install other backends for Ibis directly, and they will be supported automatically:

pip install 'ibis-framework[postgres]'

Usage Copy Chapter

Connecting to databases Copy Section

vdsql <file_or_url>
vd -f ibis <file_or_url>

where file_or_url is any connection string supported by ibis.connect() or any of the filetypes and options that VisiData itself supports.

Connection examples

vdsql foo.sqlite          # or .sqlite3, .db
vdsql foo.duckdb          # or .ddb
vdsql mysql://...
vdsql postgres://...
vdsql clickhouse://play:[email protected]/?secure=1
vdsql bigquery:///bigquery-public-data

Commands Copy Section

A decent amount of work has gone into making vdsql work just like VisiData.

You can learn about VisiData starting with the Intro to VisiData Tutorial and the VisiData documentation.

There are a few differences, however:

• " (dup-sheet) runs a new base query, including added columns, filtering for the current selection, and applying the current sort order.
• z" creates a new sheet with a different row limit.
• gz" removes the row limit entirely (fetch all rows).
• ' casts the current column to its given type, persisting into future queries (with ").
• g' freezes the current set of loaded rows into a plain VisiData sheet, where all VisiData commands are available.

Some VisiData commands aren't implemented using the database engine.
The base VisiData commands can only use the loaded rows (500 by default), and this might be misleading, so most not-implemented commands are disabled.
If you want to use them anyway, knowing the dataset is incomplete, use g' to freeze the sheet first.

Sidebar Copy Section

vdsql uses the VisiData sidebar to show the SQL query for the current view.

• b to toggle the sidebar on/off
• zb to choose a sidebar option (pending SQL, base SQL, etc.)
• gb to open the sidebar as its own sheet

In this way you can compose a SQL expression using VisiData commands, open the SQL sidebar, and save the resulting query to a file (or copy it into your system clipboard buffer).

Options Copy Section

• ibis_limit (default: 500) - max number of rows to fetch per query
• postgres_schema (default: '', public only) - which PostgreSQL schemas to show (space-separated list; use * for all non-system schemas)
• sql_always_count (default: False) - include total row count in every query
• disp_ibis_sidebar (default: pending_sql) - which sidebar property to display

Supported Backends Copy Chapter

Confirmed Copy Section

• SQLite
• MySQL
• PostgreSQL
• DuckDB
• ClickHouse
• Google BigQuery
• Snowflake

Backend-specific notes Copy Section

PostgreSQL

By default, only tables from the public schema are shown.

To show tables from specific schemas:

vdsql --postgres-schema='myschema otherschema' postgres://...

To show tables from all non-system schemas:

vdsql --postgres-schema='*' postgres://...

Note: quote or escape * on the command line to prevent shell glob expansion.

MySQL

• Requires libmysqlclient-dev (on Debian/Ubuntu)
• If you get a timezone warning: mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql mysql

Other backends supported by Ibis Copy Section

These backends are supported by Ibis and should work, but haven't specifically been tested with vdsql.
If you have problems connecting, please file an issue.

• Apache Impala
• Datafusion
• Dask
• PySpark
• HeavyAI

License

vdsql is licensed under the Apache 2.0 license.

Share and enjoy!

Overview (`visidata/apps/vgit/README.md`) Copy Section

vgit

latest version: v0.2-dev (2018-12-xx) Copy Chapter

vgit is a VisiData package that provides a sleek terminal user interface to git, supplanting the arcana of git command-line operations with intuitive keystroke commands.

Current features include:

• viewing, stashing, staging, unstaging, and committing of diff hunks.
• viewing the git log (history)
• branch and remotes management
• branch merges and commit cherry-picks
• popping, applying, and dropping of stashed diffs
• setting of local and global git config options

Installing vgit Copy Chapter

vgit is located as a module within the VisiData repo. It has its own setup.py. Note that installing vgit will also install VisiData.

pip install git+https://github.com/saulpw/visidata.git@branch#subdirectory=visidata/apps/vgit

Dependencies Copy Section

• Python 3.8+
• VisiData
• sh

Running vgit Copy Chapter

vgit [<working_dir>] automatically opens a git status sheet if the directory has a .git subdirectory.

Further documentation is available here.

License Copy Chapter

vgit is released under a GPLv3 license.

Overview (`visidata/experimental/daw/README.md`) Copy Section

VisiData Audio Workstation

• current goal: automate getting functional transcript (80%+)
  so that i can actually make progress editing with vdaw
  which will close the loop and take vdaw to become a functional transcript audio editor

getting the input transcript Copy Chapter

Given a separate .wav file per speaker.

For each input .wav file:

1. Cut out the low volume audio, so the non-speaker doesn't get transcribed.

   ffmpeg -i mike.wav -af "agate=" mike-only.wav

2. Use whisperx to get word-level timestamps.

   uvx whisperx --model base.en --task transcribe --hf_token ${HF_API_TOKEN} --highlight_words True --compute_type int8 --device cpu mike-only.wav

3. (Optional) get full transcript from other source as .md to serve as the best/canonical edit.

   xmd2json.py transcript.md > transcript.json

4. Then merge the per-speaker transcripts.

   merge_transcripts.py transcript.json mike-only.json saul-only.json > editable-transcript.json

5. Load into VisiData using the transcript file format.

   vd editable-transcript.json -f transcript

Saving changes Copy Chapter

Modifications can be saved via Ctrl+S to the original .json file, or use a .transcript suffix for a bit more convenience (doesn't require the -f flag above).

Export markdown Copy Section

[Ideally VisiData should make it possible for a sheet to override a saver, so vdaw could define a save_md.]

For now, save as foo.xmd to get a markdown transcript with strikethrough for deleted lines.

Editing Commands Copy Chapter

Playback Copy Section

• p - Play from cursor row (skipping cuts)
• P - Play from cursor row (including cuts)
• zp - Toggle pause
• [ / ] - Seek backward/forward 10 seconds
• g[ / g] - Seek backward/forward 60 seconds
• gg - Jump cursor to playhead
• F5 - Slow down playback speed by 50%
• F8 - Speed up playback 2x
• Ctrl+R - Restart mpv process

Cutting/Keeping Audio Copy Section

• x - Cut audio for line at cursor row
• y - Keep/upvote audio for line at cursor row
• gx - Cut audio for selected rows
• gy - Keep audio for selected rows

Splitting & Combining Copy Section

• a - Split line at current playhead
• za - Split line at word (prompts for word)
• g) - Combine selected rows into one
• ( - Expand row into subrows
• g( - Expand selected rows

Navigation Copy Section

• < / > - Move to previous/next section header
• g< / g> - Move to first/last section header

Text Formatting Copy Section

• r - Reformat row (wrap text to column width)
• gr - Reformat selected rows

Audio Filters Copy Section

• f - Open audio filter parameters sheet
• 0-9 - Set filter parameter to preset value (in filter sheet or main sheet)

Utilities Copy Section

• c - Clean/interpolate bad word timings

Applying ffmpeg filters Copy Chapter

• Use vdaw to experiment with ffmpeg filters via the f command

  • agate
  • compand
  • loudnorm

Index (`docs/index.md`) Copy Section

eleventyNavigation:
key: Getting Started
order: 1

Getting started Copy Chapter

• VisiData video demo
• Installation

Tutorials Copy Chapter

• Beginner: An Introduction to VisiData by Jeremy Singer-Vine
• Beginner: Guida VisiData by Andrea Borruso (onData association)
• Intermediate/Advanced (English): VisiData case study videos by creator Saul Pwanson

References Copy Chapter

• quick reference guide
  • all available commands and options
  • also available as a manpage via man vd and from inside VisiData with Ctrl+H
• VisiData Cheat Sheet

'How to' recipes Copy Chapter

• Loading Data
  • Specifying a source file
  • Loading sources supported by pandas
  • Opening an R data frame with VisiData
  • Loading multiple datasets simultaneously
  • Accessing other loaded or derived sheets
  • Converting a dataset from one supported filetype into another
• Navigation
  • rapidly scroll through a sheet
  • search within a sheet
  • move between sheets
• Rows
  • perform operations on a subset of rows
  • filter rows
  • filter a random subset of rows
  • select rows where the current column is not null
  • select rows where the current column is null
  • move, copy and remove rows
  • sort rows
• Columns
  • manipulate columns
  • hide (remove) and unhide columns
  • specify column types
  • split a column
  • expand columns that contain nested data
  • create derivative columns
  • configure multiple columns
• Editing contents
  • edit cells
  • rename columns
• Grouping data and descriptive statistics
  • set statistical aggregators
  • create a pivot table
  • create a frequency chart
  • calculate descriptive statistics
  • filter for grouped or described rows
• Creating sheets, rows and columns
  • configure the cursor to move right after a successful edit
  • set up a sheet for data collection
  • add a new blank column
  • fill a column with a range of numbers
• Combining datasets
  • perform a join
  • append two datasets
• Drawing graphs
  • graph a single column
  • graph multiple columns
  • interact with graphs
• How to save and replay a VisiData session
• Customizing VisiData
  • configure VisiData (user)
  • configure VisiData (dev)
  • have configurations persist
  • configure commands
• Plugins - extending VisiData functionality
• STDOUT pipe/redirect

Colors (`docs/colors.md`) Copy Section

eleventyNavigation:
key: Customizing VisiData - Colors
order: 12
update: 2023-10-12

version: VisiData v3.0 Copy Chapter

Color Options Copy Chapter

Interface elements and colors can be set on the command-line before a session, or in .visidatarc to take effect for every session, or within VisiData on the Options Sheet (Shift+O). Color options in particular start with color_:

vd --color-selected-row="magenta"

or in .visidatarc:

options.color_selected_row = "magenta"

On the Options Sheet, if you edit a color (or any display element), you will see the changes immediately, both in the options value (which is itself colorized) and also in the interface. Shift+U to undo if you don't like it.

Color Specification Copy Chapter

Colors in the terminal have 3 components: foreground, background, and attributes.
In most modern terminals, each of the foreground and background colors can be one of 256 colors in a standard palette.

Several attributes are also available: bold, italic, reverse, and underline. (There are other attributes like dim and blink, but they are not widely supported).

In VisiData, colors are specified with a simple language:

<attribute> <fg-color> on

where <attribute> is a plain text name as above, and each <color> is a number from 0-255, or a basic color name from the original 8-color palette: black red green yellow blue magenta cyan white.
Multiple attributes may be specified.

Most terminals support 256 colors, but because some do not, it's possible to specify a fallback color, which is a second color that's used in case the first color isn't available.
For instance, 215 yellow means to use the color 215 (kind of orange-brown), but if it's not available, use yellow instead.

Multiple colors may be applied to a single interface element with different precedence, for instance with colorizers.
The color used will be the highest precedence background and the highest precedence foreground, which may come from separate color strings. All attributes will be applied regardless of precedence--there is no way to specify a negative attribute or "no" attributes.

Any of the color components may be omitted, letting that component be given by a lower-precedence colorizer instead, ultimately falling back to color_default (white on black in the default theme).
Many color strings only specify a single attribute.

The ordering above--attributes first, then foreground, and finally background--is suggested and preferred.
Attributes may be specified anywhere in the color string; this does not change their effect (e.g. bold applies to the font rendered, not to the foreground or background color, even if bold text makes the foreground color seem brighter).
The reverse attribute flips the foreground and background color. Multiple reverse attributes do not undo each other.

256 Terminal Colors and the Colors Sheet Copy Chapter

To see all the available colors and what they look like in your specific terminal, run the open-colors command. (This command is not bound to any keystroke, so must be executed by its longname; press Space, then type open-colors, then press Enter.)

The first several rows on this sheet are the existing color strings that are currently in use.
After these are the numeric colors; note that colors 0-7 correspond to the named colors above, with 0 being black and 7 being standard white. Colors 8-15 are unnamed brighter versions of these colors: 8 is gray ("bright black") and 15 is bright white.

Colors 16-231 form a regular color cube with a wide variety of gradations covering the color space.
Colors 232-255 are gradients of white, with 232 being the darkest possible gray (almost indistinguishable from the black of 0 and 16), and 255 being almost the brightest white (though 15 is brighter).

[Changing Display Attributes within a string](#attrs) {#attrs} Copy Chapter

In many interface elements, it's possible to change the display attributes inline, within a single string.
For example, the menu bar has a message displayed in the upper right (which by default shows "Ctrl+H for help menu"), and is configurable with options.disp_menu_fmt.
The base menu color is configurable with options.color_menu, which is also applied to this menubar message.
However, this can be changed by specifying a new color within the string itself, using syntax like [:red]:

options.disp_menu_fmt = "[:red on black]{vd.motd}"

A color option name can also be used instead of a real color; for example, to use the color_error color, use [:error]:

options.disp_menu_fmt = "[:error]{vd.motd}"

Attribute changes can happen multiple times in a single string.
Use [/] to clear the last inline color change, and [:] to clear all inline attributes:

options.disp_menu_fmt = "[:underline]Note:[/] [:error]{vd.motd}"

What follows the / is not checked, so these are all valid:

[:underline]underlined text[/underline]
[:underline]also underlined[/]
[:underline]underline ends at end of string

onclick Copy Section

In addition to changing the display attributes, an onclick attribute can be given with this inline syntax, which specifies either a VisiData command to run, or a url to open (in $BROWSER), when the interface element is clicked:

options.disp_menu_fmt = "[:onclick https://jsvine.github.io/intro-to-visidata/]Click here to go to the tutorial[/]"

Or if you want a custom VisiData toolbar some of your favorite actions:

vd --disp_menu_fmt="[:onclick freq-col] freq out [/] | [:onclick quit-sheet] back [/]"

Elements with onclick are displayed with color_clickable, which is by default underline, which is commonly understood as a clickable affordance.

Setting the Theme Copy Chapter

A set of configured display elements and attributes may be packaged into a theme, which can be set as an option before VisiData starts (on the CLI or in .visidatarc), and can be changed with the theme-input command (in the menu under View > Set theme).

By default VisiData uses a dark-mode color scheme and some suggestive Unicode characters.
A few non-default themes are also packaged with VisiData:

• light: light-mode color scheme
• ascii8: ascii characters and basic 8-colors only
• asciimono: ascii characters and default colors only

To use VisiData on a classic DEC VT102 terminal:

vd --theme=asciimono

[I would love to know of anyone using VisiData in such a lo-fi situation!]

Adding a new theme Copy Section

To add another theme, add it to the vd.themes dictionary in .visidatarc.
If more than one person uses it, submit a PR to include it in the visidata/themes directory.

vd.themes['awesome'] = dict(
    color_default      = 'black on white',  # the default fg and bg colors
    color_key_col      = '20 blue',   # color of key columns
    color_edit_cell    = '234 black',     # cell color to use when editing cell
    # ... etc
)

Colorizers Copy Chapter

TableSheet sheets (which is any sheet with a row/column grid, i.e. most sheets) have colorizers, functions that can provide display attributes on individual rows, columns, or cells.

See docs/api/interface for how to use colorizers.

Conditional Formatting Copy Chapter

For ad-hoc conditional formatting from inside a session, three commands attach a colorizer to the current sheet that paints cells, rows, or columns wherever a Python expression is truthy:

• color-cell -- color individual cells (a CellColorizer)
• color-row -- color whole rows (a RowColorizer)
• color-col -- color whole columns (a ColumnColorizer)

Each command pops up a two-field form prompting for an expr: (the Python expression) and a color:. The color: field is pre-filled with the first color from red green yellow cyan magenta blue (and their on <color> background variants) that isn't already in use on this sheet.

The color: field accepts any color string (red, bold green, on red, 215 yellow on 17) or any color_* option name -- with or without the color_ prefix. So error resolves to color_error, current_row to color_current_row, etc., picking up whatever your theme defines for that role.

Like addcol-expr and select-expr, bare column names in the expression resolve to that column's typed value for the current row, so temperature > 100 Just Works on a sheet with a temperature column. In addition, four bindings are always available, each under a short and long name:

• s / sheet -- the sheet
• c / col -- the Column object the colorizer is being asked about
• r / row -- the row object
• v / value -- the typed value of the current cell

Examples (assuming a sheet with columns name, score, status):

color-cell  red          value < 0
color-cell  bold green   score >= 90
color-row   on red       status == 'failed'
color-col   underline    c.name.startswith('total_')

Notes Copy Section

• Row colorizers evaluate per cell, not per row. The colorizer dispatcher walks each cell as it draws and asks every colorizer whether to apply. Inside a color-row expression, v and value therefore refer to the current cell being drawn, not "the row's value" -- so a row-level rule like value > 100 will only light up cells that match, not entire rows. For real per-row rules, reference the row by bare column name (status == 'failed').
• Empty color: is meaningful. If you blank out the color field, the colorizer falls back to using whatever the expression itself returned as the color string. That lets an expression compute the color dynamically -- e.g. 'red' if value < 0 else 'green' if value > 0 else '' -- though in practice you'll want a literal color most of the time.
• Performance. These commands use the existing colorizer infrastructure that already drives selected-row highlighting, key-column coloring, etc. The expression is compiled once at command time and re-evaluated only on cells that get drawn, so the cost is bounded by the visible viewport, not the size of the sheet.
• Type errors are silent. Cells where the expression raises an exception (e.g. value > 50 on a column that has not been typed as a number, so value is still a string) are simply not colored. If a color-cell rule isn't lighting anything up, set the column type first (# for int, % for float) or wrap the comparison (int(value) > 50). Run vd with --debug to see the raised exception in the status log.
• Colors from a column. When the colors are already data, color-row-from-col asks for a column of color strings (defaulting to the current column) and colors each row with its own value. color-cell-from-col colors each cell in the current column from the cell in the same row of the color column; on column foo, the color column defaults to foo_color. The color column can be hidden.
• No persistence yet. Conditional-format colorizers live on the sheet instance for the session and are lost when the sheet is closed. To re-apply them in a future session, save the commands as a macro or add them to your .visidatarc.

Columns (`docs/columns.md`) Copy Section

eleventyNavigation:
key: Columns
order: 5
update: 2021-11-01

version: VisiData 2.6 Copy Chapter

How to manipulate columns Copy Chapter

Commands(s) Operation

! pins the current column on the left as a key column
H L slides the current column one position to the left/right
gH gL slides the current column all the way to the left/right of its section

Increment by key Copy Chapter

{help.commands.addcol-incr-key}

Mark grouping columns as keys with !. Numbering restarts at options.incr_base whenever any key column changes in the current row order. A returning key starts a new count. With no key columns, numbering continues across all rows, like i.

How to hide (remove) and unhide (return) columns Copy Chapter

How to hide columns

• Press - (hyphen) to hide the current column.

or

1. Press Shift+C on the source sheet to open its Columns sheet.
2. Move the cursor right to the width column.
3. Move the cursor down to the row which represents the column you wish to hide.
4. Press e followed by 0.
5. Press q to return to the source sheet.

Hidden columns are drawn as 1-cell columns to the right of the visible ones. Hidden columns can be viewed without unhiding by moving the cursor onto them. At the last visible column, press go-right (Right arrow or l) twice to move past the visible columns into the hidden columns.

How to unhide columns

1. Press gv to unhide all columns on current sheet.

or

1. Press Shift+C on the source sheet to open its Columns sheet.
2. Move the cursor right to the width column.
3. Move the cursor down to the row which represents the column you wish to unhide.
4. Press e followed by a positive number to set the width.
5. Press q to return to the source sheet.

How to specify column types Copy Chapter

Command Type

~ string
# int
% float
$ currency
@ date
z# vlen
z~ anytype

Columns usually begin as untyped (anytype). Errors when working with numerical or datetime data is often due to values being considered as strings, and the problem is solved by setting the correct type.

The float type uses Python's builtin float() constructor to parse the string, and it parses by using the decimal separator.

The currency type is a bit of a misnomer. It filters out any non-numeric characters, and then parses the remainder of the cell value as a float.
The reasons to prefer using float over currency, is performance (it is quite a bit slower than native parsing with float), or if any non-float characters should get reported as an error.

The date type parses dates into ISO8601 format. Those columns can then be used in mathematical calculations, and the calculations are interpreted for dates. E.g. 2020-01-01 + 1, is 2020-01-02.

The vlen type formats the cell value to the length of the content. For example, if the cell content is a list of length 3, then when vlen typed it will display a value of 3.

There is also the floatlocale type, which uses Python's locale.atof to parse the Column values. With floatlocale, you can set the LC_NUMERIC environment variable appropriately (before launching VisiData), such that atof() will parse the number based on your locale setting. There is a type-floatlocale command, which is unbound by default, because parsing this way is significantly slower than using the builtin float type.

If you need locale-specific float parsing regularly, you may want to rebind % or z% (or maybe some other keystroke) to type-floatlocale instead.

The following example uses the file sample.tsv.

<script type="text/javascript" src="/asciinema-player.js"></script>
<script type="text/javascript">
    AsciinemaPlayer.create('../casts/types.cast', document.getElementById('player-types'), { poster: 'npt:0:20', rows: 27 });
</script>

How to batch specify column types for more than one column

1. Press Shift+C to open the Columns sheet.
2. Press s or t to select the rows referencing the columns you wish to type.
3. Type g followed by the any of the above typing keystrokes to set the type for all selected columns on the source sheet.

How to format columns Copy Chapter

Note: Un-typed file formats, like tsvs and csvs, will save as they are displayed.

Some types have an option for their default display formatting.

Type Option Default

int disp_int_fmt {:.0f}
float disp_float_fmt {:.02f}
currency disp_currency_fmt %0.2f
date disp_date_fmt %Y-%m-%d

Ways to adjust the display formatting:

• The fmtstr column on the Columns Sheet allows you to specify the formatting for specific columns within that session, without affecting the default for the others.
• The disp_TYPE_fmt option can be changed on the Options Sheet to set the formatting for all columns of type TYPE in that session.
• The --disp-TYPE-fmt argument can be passed through the commandline to set the formatting for all columns of type TYPE in that session.
• The options.disp_TYPE_fmt can be set in the ~/.visidatarc to change the default formatting for all columns of type TYPE for all sessions.

There are several formatting styles offered:

• Formatting that starts with '%' (e.g. %0.2f) will use locale.format_string().
• Otherwise (e.g. {:.02f}), formatting will be passed to Python's string.format().
• Date fmtstr are passed to strftime.

The default for currency uses locale.format_string(). The default for int/float/date use string.format().

How to format a specific numeric columns to contain a thousands separator within a session?

1. Set a column to a numeric type by pressing # (int), % (float), or $ (currency).
2. Press Shift+C to open the Columns Sheet.
3. Move to the row referencing the column whose display you wish to format. Move the cursor to the fmtstr column.
4. Type e followed by {:,.0f} for an int type and {:,.02f} for a floating point type.

How to quickly adjust the precision of a float or date?

1. Ensure the column is typed as numeric column: float (%), floatsi (z%), dirty float ($), or date (@).
2. Press Alt+Plus or Alt+Minus to adjust the precision in the current column.
3. Or, press Space and use the longname setcol-precision-more or setcol-precision-less.

How to automatically set the type of a particular column

When loading sheets repeatedly with the same schema, it can be useful to pre-set the column types (and other metadata) in .visidatarc using the knownCols attribute on the Sheet class and its subclasses.

For example, to set the "timestamp" column in every sheet to the date type:

Sheet.knownCols.timestamp.type = date

Or to hide the nCols column on the SheetsSheet by default:

SheetsSheet.knownCols.nCols.width = 0

How to format all date columns as **month/day/year**.

The above method can be used to explicitly set the default fmtstr for specifically-named columns.
However, to use a default fmtstr for all date-typed columns, set the disp_date_fmt option. In .visidatarc:

options.disp_date_fmt = '%m/%d/%Y'

or pass as a commandline argument:

vd --disp-date-fmt='%m/%d/%Y'

or set in the Options Sheet.

1. Press Shift+O to open the Options Sheet.
2. Move the cursor down to the relevant disp_date_fmt option.
3. Type e followed by %m/%d/%Y.

How to specify a comma decimal separator when typing floating point numbers?

1. Before launching VisiData, set the shell environment variable LC_NUMERIC to a locale which interprets commas as decimals. Any European locale should do; an example that works is en_DK.UTF-8.
2. Within VisiData, set a column to type floatlocale by pressing Space followed by type-floatlocale.

Note that type-floatlocale is significantly slower than type-float. However, if you wish to replace the current binding for type-float with type-floatlocale, add to your ~/.visidatarc:

Sheet.unbindkey('%')
Sheet.bindkey('%', 'type-floatlocale')

or if you never use type-floatsi, you can do

Sheet.unbindkey('z%')
Sheet.bindkey('z%', 'type-floatlocale')

How to split a column Copy Chapter

Python regular expressions provide more finetuned column splitting. The following example
uses the commands for column splitting and transformation with xd/puzzles.tsv.

<script type="text/javascript">
    AsciinemaPlayer.create('../casts/split-regex.cast', document.getElementById('player-split-regex'), { poster: 'npt:0:20', rows: 27 });
</script>

Copy Section

• : adds new columns derived from splitting the current column at positions defined by a regex pattern. options.default_sample_size (default: 100) rows around the cursor will be used to determine the number of columns that will be created.
• ; adds new columns derived from pulling the contents of the current column which match the regex within capture groups. The new columns are named using the capture group index, or if named capture groups are used, the capture group names. This command uses the options.default_sample_size (default:100) rows around the cursor as sample rows.
• * followed by regex and then Tab followed by the replacement replaces the text matching the regex with the contents of replacement. replacement may include backreferences to capture groups in the regex (\1 etc).

How to substitute text in a column Copy Chapter

The * command can be used to do content transformations of cells. The g* variant transforms in-place, instead of creating a new column.

The following example uses benchmarks.csv.

Question Transform the SKU values of food to nutri.

1. Move cursor to SKU column.
2. Press gs to select all rows.
3. Press g* to replace text in selected rows in current column.
4. Type food in the search field.
5. Press Tab to go to the replace field, then type nutri.
6. Press Enter to replace the text.

• tests/transform-cols.vd

[How to expand columns that contain nested data](#expand) {#expand} Copy Chapter

If a column includes container data such as JSON objects or arrays, the ( family of commands can expand the child values into top-level columns:

Command Operation

( expand current column
g( expand all visible columns fully
z( expand current column to a specific depth (prompt for input)
gz( expand all visible columns to a specific depth (prompt for input)
) contract (unexpand) the current column
zM expand current column row-wise within that column

The following demo shows ( commands applied to this data:

[
    [ "short", "array" ],
    [ "slightly", "longer", "array" ],
    { "nested": "data" },
    { "more": { "deeply": { "nested": "data" } } }
]

<script type="text/javascript">
    AsciinemaPlayer.create('../casts/expand-cols.cast', document.getElementById('player-expand-cols'), { poster: 'npt:0:20', rows: 13 });
</script>

Note that by default the expansion logic will look for nested columns in up to options.default_sample_size (Default: 100) rows surrounding the cursor. This behavior can be controlled by adjusting default_sample_size in the Options Sheet, or setting options.default_sample_size in the ~/.visidatarc file.

[How to create derivative columns](#derived) {#derived} Copy Chapter

The = command takes a Python expression as input and creates a new column, where each cell evaluates the expression in the context of its row.

To specify the column name, use name=expr (e.g. total = Units * Unit_Cost). If no name is given, the expression itself is used as the column name.

These variables and functions are available in the scope of an expression:

• Column names evaluate to the typed value of the cell in the named column for the same row.

• vd attributes and methods; use Ctrl+X vd to view the vd object, or see the API.

• Sheet attributes and methods; use g Ctrl+Y to view the sheet object (or see the API).

• Global functions and variables (add your own in your .visidatarc).

• modules that have been imported in Python

  • if you need a module that hasn't already been imported at runtime, use g Ctrl+X import <modname>.

• sheet: the current sheet (a TableSheet object)

• col: the current column (as a Column object; use for Column metadata)

• row: the current row (a Python object of the internal rowtype)

• curcol: evaluate to the typed value of this row in the column that the cursor was on at the time that the expression column was added.

• cursorCol: evaluate to the typed value of this row for the column the cursor is on. Changes as the cursor moves for =. Uses the column from the time the calculation was made for g=, gz=, and z=.

• currow: a convenience object for accessing column values of the current cursor row by name. e.g. currow.Price returns the typed value of the Price column for the cursor row.

Additional attributes can be added to sheets and columns.

col deliberately returns a Column object, but any other Column object is interpreted as the value within that column for the same row. For example, both curcol and cursorCol return values, not the object itself.

For example, this customizes addcol-expr to set the curcol attribute on the new ExprColumn to a snapshot of the current cursor column (at the time the expression column is added):

Sheet.addCommand('=', 'addcol-expr', 'addColumnAtCursor(ExprColumn(inputExpr("new column expr="), curcol=cursorCol))', 'create
 new column from Python expression, with column names as variables')

Then, an expression can use curcol as though it referred to the value in the saved column.

Tab autocompletion when inputting an expression will cycle through valid column names only.

The following examples use the file sample.tsv.

Question On which days have we sold more than 10 Items?

1. Scroll to the Units column. Set the type of the Units column by pressing # (int).
2. Type = followed by Units > 10. A new column will be created. The cells in this column will contain the value True in rows where the number of Units are greater than 10 and False otherwise.
3. Move to the new derived column.
4. Type | followed by True to select all rows where there were more than 10 Units sold.
5. Press " to open a duplicate sheet with only those selected rows.

Question I have a dataset with separate columns for Year, Month and Day. How can I concatenate them into a single date column?

1. Type = followed by Year + '-' + Month + '-' + Day.
2. Set the type of the new derived column by pressing @ (date).
3. Type ^ followed by Date to rename the column to Date.

Question I have a dataset with Date column that is missing a prefix of '2020-'. How do I add it to the Date column?

When using =, and wanting to reference the current column, we recommend using curcol. When using g=, gz=, and z=, we recommend cursorCol. =, unlike the others, is dynamic and changes with adjustment of underlying values, which means it will change along with the movement of the cursor (tracked by cursorCol). curcol is a special attribute of a new ExprColumn, which remembers the cursorCol at the time of creation.

1. Move the cursor to Date.
2. Type g= followed by f"2020-{cursorCol}".

Question I have a dataset with file names. How do I create a new column with the file names lower cased?

1. Move the cursor to file names column.
2. Type = followed by curcol.casefold().
3. Move to the newly created column, and rename it with ^, followed by the desired name.

How to configure multiple columns Copy Chapter

{help.commands.columns_all}

Hidden columns are included. Edit a hidden column's width to a positive number to unhide it on its source sheet.

Properties of columns on the source sheet can be changed by using standard editing commands on its Columns sheet (accessed with Shift+C). In particular, it facilitates the selection of multiple columns, followed by utilising one of the g-prefixed commands to modify all of them.

For a full list of available commands, see the man page. Some example workflows follow.

The following examples use the file sample.tsv.

How to set multiple statistical aggregators

Question What is the average daily revenue from sales of each Item?

1. Set the type of the Units column by pressing # (int).
2. Set the type of the Total column by pressing % (float).
3. Press Shift+C to open the Columns sheet.
4. Press s or t on the rows referencing the source sheet Units column and the Total column to select them.
5. Type g+ followed by avg to add a avg statistical aggregator to the selected rows.
6. Press q to exit and return to the source sheet.
7. Scroll to the Item column. Press Shift+F to open the Frequency table.

Question What are the daily average and sum total number of Units sold for each Item?

1. Press Shift+C to open the Columns sheet.

2. Move the cursor to the row referencing the source sheet Units column.

   a. Press s or t to select it.
   b. Set the type for the source sheet Units columns by pressing g# (int).
   c. Move the cursor to the aggregators column.
   d. Type e to enter edit mode, followed by sum avg.

3. Press q to exit and return to the source sheet.

4. Move the cursor to the Item column. Press Shift+F to open the Frequency table.

Contributing (`docs/contributing.md`) Copy Section

eleventyNavigation: key: Checklists for Contributing to VisiData order: 99 Copy Chapter

[Submitting a Core Loader](#loader) {#loader} Copy Chapter

A deeper explanation of all of these steps can be found in the loaders api documentation.

• Create an open_foo function that returns the new FooSheet.
• Set an appropriate rowtype string.
• Provide a # rowdef: comment. This describes the structure of a row (e.g. its base class) in a Sheet.
• If the loader's dependencies are not part of Python3 stdlib, note the additional dependencies in the requirements.txt (include a comment adjacent to the dep with the name of the loader).
• Check in a small sample dataset, in that format, to the sample_data folder
• Add a load-foo.vd to tests/. load-foo.vd should simply open the checked-in sample dataset and have a simple interaction if the source includes multiple tables.
• After replaying vd load-foo.vd, save the final sheet as load-foo.tsv. Save load-foo.tsv in tests/golden/.
• add a section on the loader to the formats.jsonl.

[Submitting an External Plugin](#plugins) {#plugins} Copy Chapter

A deeper explanation of all of this framework can be found in the plugins api documentation.

• Host a single Python file containing all of the plugin's code.
• In the https://github.com/visidata/dlc/blob/stable/plugins.jsonl file in the visidata:dlc repo, add a row for each plugin with all of the necessary information.

Crud (`docs/crud.md`) Copy Section

eleventyNavigation:
key: Creating sheets, rows and columns
order: 8
Update: 2021-11-18

Version: VisiData 2.7.1 Copy Chapter

How to set up a sheet for data collection Copy Chapter

1. Type Shift+A to open a new blank sheet with one column.

2. Press

   a. a to add one blank row.

   or

   b. ga followed by a number to add that many blank rows.

How to add a new blank column Copy Chapter

1. Press

   a. za to add one blank column.

   or

   b. gza followed by a number to add that many blank columns.

2. Press ^ to edit the column name.

How to fill a column with a range of numbers Copy Chapter

1. Press gs to select all rows (or use other commands to select a subset of rows to fill).
2. Move the cursor to the column to be filled.
3. Press gi.

How to edit a cell Copy Chapter

1. Press e to edit an individual cell.
2. Type in the new value.
3. Press Enter to accept the value.

How to move the cursor to the next cell after a successful edit Copy Chapter

1. Press Shift+Arrow to accept the current value and move the cursor into edit mode in the next cell.

Customize (`docs/customize.md`) Copy Section

eleventyNavigation:
key: Customizing VisiData
order: 12
Updated: 2023-11-18

Version: VisiData v3.0 Copy Chapter

For a primer on configuring VisiData through setting options, see jsvine's tutorial.

How to configure commands {#commands} Copy Chapter

The .visidatarc in the user's home directory is plain Python code, and can contain additional commands or key bindings.

(Alternatively, since v2.9, VisiData has XDG support. If $XDG_CONFIG_HOME is set and the file "$XDG_CONFIG_HOME"/visidata/config.py exists, this will be loaded as the user's default configuration file.)

Longnames are names given to executable commands for ease of keystroke remapping. For example, the longname select-row is assigned to commands which select the current row in a sheet. On default, this longname is bound to the keystroke s.

From within VisiData, type z Ctrl+H to open the Commands Sheet. This is a reference for all of the commands available on the current sheet. For a deeper exploration of commands, check out API reference manual.

Setting/changing keybindings for existing commands Copy Section

1. Learn the longname for a command. Longnames are usually 2-3 words, separated by hyphens. The first word is usually a verb, and the second usually a noun. When a command is executed, its longname appears in the lower right status, next to its keystroke. Alternatively, you can z Ctrl+H to open the Commands Sheet and discover the longname for the command in question.

2. a) To create a global keybinding, add bindkey(keystroke, longname) to your .visidatarc.
   b) To set the binding for a particular sheet type, add <Sheet>.bindkey(keystroke, longname) to your .visidatarc, where <Sheet> is a SheetType.

Warning: bindings defined in a .visidatarc will overwrite default ones.

Example: Bind `i` to edit-cell globally

In VisiData, pressing e enters edit mode for the current cell. Seasoned vim users might prefer to press i instead.

1. Open ~/.visidatarc in an editor.
2. Add the line TableSheet.bindkey('i', 'edit-cell') to globally bind the keystroke i to the longname edit-cell.
3. Launch VisiData, and press i.

Example: Unbind `i` from addcol-incr globally

If the above instructions are followed, a message will pop up that says "i was already bound to addcol-incr.

To unbind i before binding it:

1. Open ~/.visidatarc in an editor.
2. Add the line TableSheet.unbindkey('i') before any piece of code where it is re-bound.
3. Launch VisiData.

Customizing cell editing keybindings Copy Section

bindkey changes keybindings in normal (non-editing) mode. To change keybindings during cell editing, use vd.editCellBindings with acceptThenFunc():

vd.editCellBindings['Enter'] = acceptThenFunc('go-down', 'edit-cell')

acceptThenFunc() takes one or more command longnames. It saves the current edit, then executes each command in sequence. For example, to make Enter save and move down to edit the next cell (spreadsheet-style):

vd.editCellBindings['Enter'] = acceptThenFunc('go-down', 'edit-cell')

See Editing Contents for the full list of default cell editing keybindings.

Creating new commands Copy Section

At minimum, <Sheet>.addCommand requires a longname and execstr.

For example, to define a new command:

Sheet.addCommand('Ctrl+D', 'scroll-halfpage-down', 'cursorDown(nScreenRows//2); sheet.topRowIndex += nScreenRows//2')

To define a command interactively (without editing .visidatarc), use the define-command command. The input is longname execstr (separated by the first space). For example: define-command show-item status(currow.Item).

Commands and keybindings are set on a particular Sheet Type in the class hierarchy. Use BaseSheet for commands which don't need a sheet at all--these will apply to all sheets. Commands and bindings on more specific sheets will override more generic ones. Sheet is a generic table, ColumnsSheet would be for the columns sheet, FreqTableSheet for frequency tables, and so on.

Environment variables Copy Section

vd.setenv('LC_NUMERIC', 'en_US.UTF-8') in .visidatarc sets the numeric locale and records it in saved command logs.
See Environment and locale.

Adding custom aggregators {#aggregators} Copy Section

Aggregators allow you to gather the rows within a single column, and interpret them using descriptive statistics. VisiData comes pre-loaded with a default set like mean, stdev, and sum.

To add your own custom aggregator name, add the following to your .visidatarc.

vd.aggregator('name', func, type=float)

Where func is a function of the form:

def func(list):
    return value

The type parameter is optional. It allows you to define the default type of the aggregated column.

Here is an example, that adds an aggregator for numpy's internal rate of return module.

import numpy as np
vd.aggregator('irr', np.irr, type=float)

Bonus: How to choose which aggregators are columns within the DescribeSheet?

Any numeric aggregator can be added!

Supply a space-separated list of aggregator names to options.describe_aggrs in your .visidatarc.

options.describe_aggrs = 'mean stdev irr'

Saving options from the Options Sheet {#save-options} Copy Section

Options edited on the Options Sheet (Shift+O) can be saved to the config file with z Ctrl+S.
Only options that differ from the default and aren't already in the config file will be appended.

Turning off motd {#motd} Copy Section

By default, the first time each day that VisiData is used, it downloads a single small file of startup messages.

This network request can be turned off by adding options.motd_url='' to your ~/.visidatarc.

If you do decide to turn it off, we encourage you to donate to support VisiData.

Dirsheet (`docs/dirsheet.md`) Copy Section

eleventyNavigation: key: Directory Sheet order: 99 Copy Chapter

Directory Sheet (DirSheet)

Open any directory in VisiData to get a DirSheet: a browsable, editable view of the filesystem.

vd .
vd /path/to/directory

Within VisiData, use open-dir-current to open the current working directory.

Columns Copy Chapter

DirSheet shows these columns by default:

Column Description

directory parent directory path
filename file name (key column)
ext file extension (/ for directories)
size file size in bytes
modtime last modification time

These columns are available but hidden by default (unhide with -):

Column Description

abspath absolute file path
owner file owner (Unix)
group file group (Unix)
mode file permissions (octal)
filetype output of file --brief (computed async)

Files are sorted by modification time (newest first) by default.

Opening files Copy Chapter

Keystroke(s) Command Action

Enter open-row-file open file at cursor as a new sheet
g Enter open-rows open all selected files as new sheets
z Enter open-row-filetype open file at cursor, prompting for filetype
` open-dir-parent open parent directory

VisiData detects the filetype from the file extension and opens the appropriate loader (csv, tsv, json, etc.). Directories open as nested DirSheets.

File preview Copy Chapter

The open-preview command opens the file at the cursor in a split pane. As you navigate the file list, the preview updates automatically to show the current file.

• Tables (csv, tsv, json, etc.) display as proper table sheets
• Text files display as TextSheet
• Directories display as nested DirSheet
• Preview sheets for visible rows are preloaded in the background

open-preview is unbound by default; add a keybinding in .visidatarc:

DirSheet.addCommand('p', 'open-preview', 'sheet.previewFile(cursorRow)')

Close the preview with g Shift+Z (close split).

External editors Copy Chapter

Keystroke(s) Command Action

Ctrl+O sysopen-row open file in $EDITOR
g Ctrl+O sysopen-rows open selected files in $EDITOR

Editing the filesystem Copy Chapter

DirSheet uses deferred mode: edits are staged and applied all at once with z Ctrl+S.

Renaming and moving files Copy Section

Edit the filename column (e) to rename a file. Edit the directory column to move it to a different directory (the destination is created if it doesn't exist).

Deleting files Copy Section

Mark rows for deletion with d. Commit with z Ctrl+S.

• By default (options.safety_first), directory deletion uses os.rmdir (fails if not empty).
• With safety_first disabled, directory deletion uses shutil.rmtree.

Editing metadata Copy Section

Edit the size, modtime, owner, group, or mode columns directly. Changes are applied on commit.

Copying files Copy Chapter

Keystroke(s) Command Action

y copy-row copy file at cursor to a destination directory
gy copy-selected copy selected files to a destination directory

Shell integration Copy Chapter

Keystroke(s) Command Action

z; addcol-shell add column from shell command, with $columnName variables

Shell columns run a command for each row, interpolating $columnName references with column values. Both stdout and stderr are captured as separate columns.

The prompt also asks for the minimum seconds between starting commands (0 for no delay), to e.g. stay under an API rate limit. At most options.max_threads commands run at once. Ctrl+C (cancel-sheet) kills the running commands.

Options Copy Chapter

Option Default Description

dir_depth 0 folder recursion depth (0 = only immediate children)
dir_hidden False whether to show hidden files (dotfiles)

Options must be set before loading; reload the sheet after changing them.

FileListSheet Copy Chapter

Opening a .fdir file creates a FileListSheet: a DirSheet populated from a list of file paths (one per line) instead of a directory listing.

Edit (`docs/edit.md`) Copy Section

eleventyNavigation: key: Editing Contents order: 6 Update: 2018-08-19 Version: VisiData 1.3.1 Copy Chapter

How to edit cells Copy Chapter

For a summary of all editing commands, see the man page.

Command Operation

e edit contents of current cell
ge text set contents of current column for selected rows to text
g* regex/subst replace matching regex in current column for selected rows with subst
g= expr evaluate Python expr over each selected row and set current column to the result

Bulk corrections Copy Chapter

ge unifies inconsistent values for the same logical item (e.g. "visidata", "Visidata", "VisiData", "vd") across selected rows. The initial input seeds from the current cell.

1. Select the rows with inconsistent values.
2. Move the cursor to a row with the correct value; it need not be selected.
3. Type ge, edit the value if needed, and press Enter.

All selected cells in the current column take the corrected value.

A Frequency Table (Shift+F) makes the distinct values easy to see. Editing a key cell there propagates the correction back to the source sheet (see note below).

note! Copy Chapter

Modifications made to rows on derived sheets will be reflected on the source sheets. This includes the Frequency Table: editing the key column there will change all instances on the source sheet, and if that sheet is derived from another source sheet, it will be reflected there, and so on.

This does not apply to adding or deleting rows, only changes to existing rows.

Commands while editing Copy Chapter

While in editing mode, or anytime VisiData expects input (with e.g. =, ;), typical readline commands become available:

Command Operation

Enter accepts input
Ctrl+C/Esc aborts input
Ctrl+O opens external $EDITOR to edit contents
Ctrl+R reloads initial value
Ctrl+A/Ctrl+E moves to beginning/end of line
Backspace deletes previous character
Up/Down sets contents to previous/next in history
Tab/Shift-Tab autocompletes input (when available)

Cell navigation while editing Copy Section

When editing a cell (with e), these additional commands move to an adjacent cell and continue editing:

Command Operation

Shift+Down/Shift+Up saves and moves to cell below/above
Shift+Right/Shift+Left saves and moves to cell right/left
Tab/Shift-Tab saves and moves to cell right/left

Tab wraps to the next row when at the last column; Shift-Tab wraps to the previous row when at the first column.

These bindings can be customized; see Customize.

How to rename columns Copy Chapter

Command Operation

^ edits name of current column
g^ sets names of all unnamed visible columns to contents of selected rows (or current row)
z^ sets name of current column to contents of current cell
gz^ sets name of current column to combined contents of current column for selected rows

In most cases, ^ is the preferred command. Examples which demo ^ can be seen in Columns and Group.

How to set the header in an Excel sheet?

For most filetypes (e.g. csv, tsv, xls(x)) the loaders assume that the dataset's first options.header rows contain the column names.

If the Excel file has multiple sheets with varying number of header rows:

1. Pass --header=0 while loading the file.

vd file.xlsx --header=0

2. For each sheet, press s or t to select the rows which represent the header rows.
3. Press g^ to set the names of the headers to the contents of selected rows.

How to rename columns using the Columns sheet

1. Press Shift+C to open the Columns sheet.
2. Within the name column, move the cursor to the row which represents the source sheet.
3. Type e and then input the new column name. Press Enter.
4. Press q to return to the source sheet and see the renamed column.

Formats (`docs/formats.md`) Copy Section

eleventyNavigation:
key: Supported Formats
order: 99

Extra notes about formats

tsv (Tab Separated Values), as simple as it gets Copy Chapter

• delimiter: field delimiter to use for tsv/usv filetype (default: .)
• row_delimiter: row delimiter to use for tsv/usv filetype (default: \n)
• tsv_safe_newline: replacement for newline character when saving to tsv (default: )
• tsv_safe_tab: replacement for tab character when saving to tsv (default: .)

Use -f usv for Unicode separators U+241F and U+241E.
Use -f tsv for awk-like records.
Use --delimiter= (an empty string) to make '\0' the value separator.
Use --row-delimiter= to make '\0' the row separator.

csv (Comma Separated Values) for maximum compatibility {#csv} Copy Chapter

.csv files are a scourge upon the earth, and still regrettably common.
All csv_* options are passed unchanged into csv.reader() and csv.writer().

• csv_dialect: dialect passed to csv.reader (default: excel)
• Accepted dialects are excel-tab, unix, and excel.
• csv_delimiter: delimiter passed to csv.reader (default: ,)
• csv_quotechar: quotechar passed to csv.reader (default: ")
• csv_skipinitialspace: skipinitialspace passed to csv.reader (default: True)
• csv_escapechar: escapechar passed to csv.reader (default: None)
• csv_lineterminator: lineterminator passed to csv.writer (default: \n)

Saving TSV/CSV files Copy Chapter

• save_filetype: specify default file type to save as (default: tsv)
• safety_first: sanitize input/output to handle edge cases, with a performance cost (default: False)

Useful options for text formats in general Copy Chapter

• regex_skip: regex of lines to skip in text sources (default: )
• save_encoding: encoding passed to codecs.open when saving a file (default: utf8)

fixed {#fixed} Copy Chapter

• loader-specific options
  • fixed_rows (default: 1000) number of rows to detect fixed width columns from
  • fixed_maxcols (default: 0) max number of fixed-width columns to create (0 is no max)

json {#json} Copy Chapter

• loader-specific options
  • json_indent (default: None) indent to use when saving json
  • json_sort_keys (default: False) sort object keys when saving to json
  • default_colname (default: '') column name to use for non-dict rows
• Cells containing lists (e.g. [3]) or dicts (e.g. {3}) can be expanded into new columns with ( and unexpanded with ).
• All expanded subcolumns must be closed (with )) to retain the same structure.
• Support for jsonla was added in 3.0.

xml {#xml} Copy Chapter

• v show only columns in current row attributes
• za add column for xml attributes
• dive-xml-table open current element's children as a table, with attributes and child values as columns

pcap {#pcap} Copy Chapter

• loader-specific options
  • pcap_internet (default: 'n') (y/s/n) if save_dot includes all internet hosts separately (y), combined (s), or does not include the internet (n)

postgres {#postgres} Copy Chapter

• loader-specific options
  • postgres_schema (default: 'public') the desired schema for the Postgres database
• vd postgres://username:password@hostname:port/database opens a connection to the given postgres database.
  • any part may be omitted; libpq fills it from PGUSER, PGPASSWORD, PGHOST, PGPORT, or ~/.pgpass, and postgres:///database connects over the local Unix socket.
• vd postgres:///database/table opens a single table directly.
• exec-sql on the tables sheet or a table opens a sheet with the results of a SELECT query; edit-sql edits that query.

imap {#imap} Copy Chapter

• vd "imap://[email protected]:[email protected]" opens a connection to the IMAP server
  • e.g. vd "imap://[email protected]:[email protected]:993"
  • e.g. vd "imap://[email protected]@imap.gmail.com"
    • note that you don't specify a password for gmail here -- instead, you will be prompted to follow some instructions

using VisiData as a pager within psql Copy Section

In psql:

\pset format csv
\pset pager always
\setenv PSQL_PAGER 'vd -f csv'
\pset pager_min_lines

sqlite {#sqlite} Copy Chapter

• supports saving for CREATE/INSERT (not wholesale updates)
• z Ctrl+S to commit any add-row/edit-cell/delete-row

duckdb {#duckdb} Copy Chapter

• loader-specific requirements
  • duckdb python module in path or virtual environment
• supports saving for CREATE/INSERT
• z Ctrl+S to commit deferred add-row/edit-cell/delete-row changes back to the source database

mysql {#mysql} Copy Chapter

• loader-specific requirements
  • working mysql / mariadb installation or at least the libmysqlclient-dev package (ubuntu; name might be different on other platforms)
  • mysqlclient python module in path or virtual environment (pip install mysqlclient)
• vd mysql://username:password@hostname:port/database opens a connection to the given mysql / mariadb database.

html {#html} Copy Chapter

• loader-specific options
  • html_title (default: '<h2>{sheet.name}</h2>') table header when saving to html
• load all ``s in a web page as VisiData sheets.

shp {#shp} Copy Chapter

• Can be edited in raw data form. Images can be plotted with . (dot).
• .shp files can be saved as geoJSON.

mbtiles {#mbtiles} Copy Chapter

• Can be edited in raw data form. Images can be plotted with . (dot).

png {#png} Copy Chapter

• Can be edited in raw data form. Images can be plotted with . (dot).

ttf {#ttf} Copy Chapter

• Can be edited in raw data form. Images can be plotted with . (dot).

pandas {#pandas} Copy Chapter

VisiData has an adapter for pandas. To load a file format which is supported by pandas, pass -f pandas data.foo. This will call pandas.read_foo().

For example:

vd -f pandas data.parquet

loads a parquet file. When using the pandas loader, the .fileformat file extension is mandatory.

To load a hierarchy of parquet files located in folder data/, run

vd -f parquet data/

or rename the directory to data.parquet and run

vd data.parquet -f pandas

This should similarly work for any format that has a pandas.read_format() function.

[VisiData Internal Formats](../internal_formats) (.vd, .vdj, .vdx, .vds) {#vd} Copy Chapter

• .vd/vdj/.vdx are command log formats suitable for VisiData scripts and macros
• .vds is a multisheet save format that includes some sheet and column metadata

API (reddit, matrix, zulip, airtable) {#api} Copy Chapter

• loader-specific requirements
  • require setting authentication information in ~/.visidatarc or on the CLI
  • launch the loader with -f loadername for steps to obtain and configure authentication credentials

Grep {#grep} Copy Chapter

A .grep file is a JSON lines file. It can be in two formats:

1. A simple container with three fields:
   file - a string with the path to the file where the match was found (absolute or relative path)
   line_no - an integer with the line number in the file where the match was found,
   text - a string with the text of the line that matched.
2. ripgrep grep_printer format, described here:
   https://docs.rs/grep-printer/latest/grep_printer/struct.JSON.html

Freq (`docs/freq.md`) Copy Section

eleventyNavigation: key: Frequency Table Reference order: 99 Copy Chapter

Frequency Table Reference

A frequency table groups source rows by one or more columns and counts the rows in each group.
A pivot table extends this by breaking each group into sub-groups based on a pivot column.
For tutorials and step-by-step workflows, see Grouping data and descriptive statistics.

Everything below also applies to PivotSheet (Shift+W).

Binning Copy Chapter

Discrete Binning Copy Section

By default, each unique formatted value in the grouped column becomes its own bin.
Rows are sorted by count, descending.

Numeric Binning Copy Section

Set options.numeric_binning to True to bin numeric columns into ranges.

• options.histogram_bins controls the number of bins. When set to 0 (default), the bin count is the square root of the row count.
• Bin ranges display as "min - max".
• If all values are identical, a single bin is created.
• If there are more bins than distinct values, each value gets its own bin.

Errors and Nulls Copy Section

Null and error values get their own bins.
Error bins display as the stringified error value (e.g. #ERR).

Commands Copy Chapter

On source sheet Copy Section

• Shift+F (freq-col): open Frequency Table grouped on current column
• g Shift+F (freq-keys): open Frequency Table grouped by all key columns
• z Shift+F (freq-summary): open one-line summary for all rows and selected rows
• Shift+W (pivot): open Pivot Table: group rows by key columns and summarize current column

On Frequency Table Copy Section

• Enter (open-row): open copy of source sheet with rows in the current bin
• g Enter (dive-selected): open copy of source sheet with rows from all selected bins
• z Enter (dive-except): open copy of source sheet excluding rows in the current bin
• gz Enter (dive-except-selected): open copy of source sheet excluding rows in selected bins

Selection Propagation Copy Chapter

Selecting rows on the frequency table also selects the corresponding source rows.
Unselecting works the same way.
Undoing selection changes on the frequency table will also undo them on the source sheet.

Columns Copy Chapter

• Key columns from the grouped-by columns. Numeric binned columns use RangeColumn.
• count: number of source rows in the bin.
• percent: percentage of total source rows in the bin.
• histogram: visual bar chart (present when disp_histogram is set). Hidden automatically when aggregation columns are added.
• Aggregation columns: one column per aggregator set on source columns (via +).

Options Copy Chapter

• options.disp_histogram (default: ■): histogram element character
• options.histogram_bins (default: 0): number of bins for histogram of numeric columns (0 = auto)
• options.numeric_binning (default: False): bin numeric columns into ranges

Tips Copy Chapter

• g ' (freeze-sheet) to create a plain data sheet snapshot of the frequency table.
• Add aggregators on source columns with + before opening the frequency table to see summary statistics per bin.
• Select bins with inconsistent values and use ge to unify them; the correction propagates back to the source sheet. See bulk corrections.

Internals

Row Type Copy Chapter

Each row is a PivotGroupRow namedtuple with these components:

0. discrete_keys: tuple of formatted values for discrete key columns
1. numeric_key: range tuple (min, max) for numeric bins, or (0, 0) if not binned
2. sourcerows: list of source rows in this group
3. pivotrows: dict mapping pivot values to lists of source rows

Architecture Copy Chapter

• FreqTableSheet inherits from PivotSheet.
• groupRows() iterates source rows, assigns each to a bin by its formatted key values, and populates sourcerows and pivotrows.
• addAggregateCols() creates aggregation columns from any aggregators set on source columns.
• nonKeyVisibleCols determines which columns receive aggregation caching after load.

Pivot-Specific Copy Chapter

• pivotrows dict maps pivot values to row lists for each group.
• Column.aggvalue stores which pivot value a column represents.

Key Source Files Copy Chapter

• visidata/freqtbl.py — FreqTableSheet, HistogramColumn, commands
• visidata/pivot.py — PivotSheet, PivotGroupRow, RangeColumn, groupRows, addAggregateCols
• visidata/aggregators.py — aggregator definitions

Gmail (`docs/gmail.md`) Copy Section

Using Gmail with OAuth 2.0

Why Copy Chapter

As of May 30 2022, Google doesn't allow you to log into your IMAP gmail account using only a username/password combination.
So to open your Gmail in Visidata you need to create a Google API App, attach some scopes to it, generate a client ID and secret, then let Visidata use that client ID and secret (in a json file).

How Copy Chapter

First, go to the Google Console

And click CREATE PROJECT

Give the project a name then click CREATE

Open a new tab and go to the API Library

Search for gmail

Click the search result Gmail API

Click ENABLE

Go back to your first tab

On the left, select OAuth consent screen

Then select External and click CREATE to create an App

Give the App a name and input your gmail address.

Click ADD OR REMOVE SCOPES

Search for gmail

Click the checkbox by the row with the scope value https://mail.google.com/

Then scroll to the bottom

And click UPDATE

You should see your selected scopes.

Click SAVE AND CONTINUE

Click ADD USERS

Type in your gmail email address then click ADD

On the left, click Credentials

Near the top click CREATE CREDENTIALS

Then click OAuth client ID

Select the application type Desktop App and give your OAuth 2.0 client a name then click CREATE

Click DOWNLOAD JSON and move the downloaded file into the visidata project directory at the path vdplus/api/google/ and call the file google-creds.json

Now, on the command line run the equivalent for you:

vd "imap://[email protected]@imap.gmail.com"

Then you should get a web browser popup:

Select the account whose email address you have been using in these instructions.

Click Select all

See your gmail in Visidata.

<Chef's Kiss>

Graph (`docs/graph.md`) Copy Section

eleventyNavigation: key: Drawing graphs order: 10 Update: 2018-03-17 Version: VisiData 1.1 Copy Chapter

Graphs in VisiData can be used to visualise the relationship between numeric dependent variables vs a numeric independent variable. Optionally, a second independent variable, which is categorical, adds an additional colored scatter plot for each category.

This example uses the file StatusPR.csv.

<script type="text/javascript" src="/asciinema-player.js"></script>
<script type="text/javascript">
    AsciinemaPlayer.create('../casts/pivot-graphs.cast', document.getElementById('player'), { poster: 'npt:0:30', rows: 27 });
</script>

How to graph a single column Copy Chapter

1. Press ! to set a column as the x-axis. This column must be numeric.
2. Set that column to a numeric type by pressing # (int), % (float), $ (currency), or @ (date).
3. Optional: Press ! on a categorical key column to add it as an additional independent variable.
4. Set the type of the column you wish to set as the dependent variable with a numeric type.
5. Press . (dot=plot) on that column.

How to graph multiple columns Copy Chapter

1. Press ! to set a column as the x-axis. This column must be numeric.
2. Set that column to a numeric type.
3. Optional: Press ! on a categorical key column to add it as an additional independent variable.
4. Type all of the columns you wish to set as the dependent variables with a numeric type.
5. Press - (dash) to hide any numeric columns you don't wish graphed.
6. Press g..

##How to interact with graphs

With the keyboard

Command(s) Operation

1-9 toggles display of each scatterplot layer
h j k l moves the cursor
H J K L expands and shrinks the cursor
+ - increases/decreases the zoomlevel, centered on the cursor
zz zooms into the cursor
_ (underscore) zooms to fit the full extent
s t u selects/toggles/unselects rows on the source sheet contained within the cursor
gs gt gu selects/toggles/unselects rows visible as points on the screen
d deletes rows on the source sheet contained within the cursor
gd deletes all rows visible as points on the screen
Enter opens sheet of source rows contained within the cursor
gEnter opens sheet of source rows which are visible on screen
v toggles the visibility of graph labels

With the mouse

Command Operation

Left-click and drag sets the cursor
Right-click and drag scrolls
Scroll-wheel zooms in/out

Graphics (`docs/graphics.md`) Copy Section

eleventyNavigation: key: Terminal Graphics in VisiData order: 99 Updated: 2017-12-06 Version: VisiData 1.0 Copy Chapter

VisiData can display low-resolution terminal graphics with a reasonable amount of user interactivity.

The current implementation uses braille Unicode characters (inspired by asciimoo/drawille). Unicode blocks or the sixel protocol may be supported in the future.

Class hierarchy Copy Chapter

• Sheet: the drawable context base class (part of core vdtui.py)
• Plotter: pixel-addressable display of entire terminal with (x,y) integer pixel coordinates
• Canvas: zoomable/scrollable virtual canvas with (x,y) coordinates in arbitrary units
• InvertedCanvas: a Canvas with inverted y-axis
• Graph: an InvertedCanvas with axis labels, a legend, and gridlines

Summary Copy Section

• The async Graph.reload() iterates over the given sourceRows (from its source Sheet) and calls Canvas.polyline() to indicate what to render.
• Canvas.refresh() triggers an async Canvas.render(), which iterates over the polylines and labels and calls the Plotter.plot* methods.
• The VisiData.run() loop calls Plotter.draw(), which determines the characters and colors to represent the pixels.

class `Plotter` Copy Section

A Plotter is a Sheet with a pixel-addressable drawing surface that covers the entire terminal (minus the status line). Pixels and labels are plotted at exact locations in the terminal window, and must be recalculated after any zoomlevel change or terminal resizing.

Plotter.draw(scr) is called multiple times per second to update the screen, and chooses a curses attribute for each pixel.
By default, the most common attr is chosen for each pixel, but if options.disp_graph_pixel_random is set, an attr will be randomly chosen from the naturally weighted set of attrs (this may cause blocks of pixels to flicker between their possible attrs).
If an attr is in the Canvas.hiddenAttrs set, then it is not considered for display at all (and its rows will be ignored during selection).

All Plotter coordinates must be integer numbers of pixels.
[For performance reasons, they are presumed to already be integers, to save unnecessary calls to round().]
Methods which plot multiple pixels on the canvas should be careful to gauge the display correctly; simply calling round() on each calculated float coordinate will work but can cause display artifacts.

`Plotter` methods

For Plotter methods, x and y must be integers, where 0 <= x < plotwidth, and 0 <= y < plotheight. (0,0) is in the upper-left corner of the terminal window.

Pixels can be plotted directly onto a Plotter with these methods:

• Plotter.plotpixel(x, y, attr, row=None)
• Plotter.plotline(x1, y1, x2, y2, attr, row=None)
• Plotter.plotlabel(x, y, text, attr)

attr is a curses attribute, and row is the object associated with the pixel.

The above plot* methods append the row to Plotter.pixels[y][x][attr].

These properties and methods are also available:

• Plotter.plotwidth is the width of the terminal, in pixels.
• Plotter.plotheight is the height of the terminal, in pixels.
• Plotter.rowsWithin(bbox) generates the rows plotted within the given region.
• Plotter.hideAttr(attr, hide=True) adds attr to hiddenAttrs if hide, and removes it otherwise.
• Plotter.refresh() is called whenever the screen size changes, and should also be invoked whenever new content is added.

rowsWithin takes a Box object (described below). The Box class is otherwise unused by the Plotter.

class `Canvas` Copy Section

A Canvas is a Plotter with a virtual surface on which lines and labels can be rendered in arbitrary units.

The onscreen portion (the area within the visible bounds) is scaled and rendered onto the Plotter, with the minimum coordinates in the upper-left [same orientation as Plotter].

The Canvas user interface supports zoom, scroll, cursor definition, and selection of the underlying rows. The source attribute should be the Sheet which owns the plotted row objects.

A call to Canvas.refresh() will trigger Canvas.render(), which is decorated with @asyncthread as it may take a perceptible amount of time for larger datasets. Any active render threads are cancelled first.

`Box` and `Point` helper classes

While the Plotter API requires literal integer values for x/y and width/height parameters, Canvas methods generally take float values contained in either Box or Point classes.

`Point`

Point is simply a container for an (x,y) coordinate (passed to the constructor). The individual components are stored as .x and .y, and the computed .xy property will return (x,y) as a simple tuple. Point can also stringify itself reasonably.

`Box`

Box is effectively a rectangle stretching over some area of the canvas. The constructor takes (x,y,w,h), but a Box can also be constructed using the BoundingBox(x1,y1,x2,y2) helper. [Note that in the BoundingBox case, the order of the individual points is not guaranteed; the individual coordinates may be swapped for convenience.]

Box has these members and properties:

• xmin and ymin: the minimum coordinates of the area.
• xmax and ymax: the maximum coordinates of the area.
• xcenter and ycenter: the central coordinates of the area.
• w and h: the width and height of the area.
• xymin: returns Point(xmin,ymin).
• center: returns Point(xcenter,ycenter).
• contains(x, y): returns True if (x,y) is inside the bounding box.

`Canvas` methods

• Canvas.polyline(vertices, attr, row=None) adds a multi-segment line from the list of (x,y) vertices. One vertex draws a point; two vertices draws a single line. Note that the vertices are not Point objects (unlike parameters for other methods).
• Canvas.label(xy, text, attr, row=None) adds text at xy (Point in canvas units).
• Canvas.fixPoint(xyplotter, xycanvas) sets the position of the visibleBox so that xycanvas (Point in Canvas units) is plotted at xyplotter (Point in Plotter units).
• Canvas.zoomTo(bbox) sets the visible bounds so the given canvas coordinates will fill the entire Plotter area. aspectRatio will still be obeyed.
• Canvas.keyattr(key) returns the attr for the given key, assigning a new color from options.plot_colors if key has not been seen before. These keys are plotted as legends on the upper right corner of the canvas. The last color is given out for all remaining keys and is labeled "other".
• Canvas.resetBounds() needs to be called after some or all points have been rendered, but before anything can be plotted. It initializes the width and height of the canvas, visible area, and/or cursor.
• Canvas.reset() clears the canvas in preparation for reload().

`Canvas` properties

• Canvas.canvasBox reflects the bounds of the entire canvas.
• Canvas.visibleBox defines the onscreen canvas area.
• Canvas.cursorBox defines the cursor region in canvas coordinates.
• Canvas.zoomlevel is a settable property, which sets the visibleBox size accordingly. zoomlevel of 1.0 makes the entire canvas visible. Does not change the position of the visibleBox (see Canvas.fixPoint).
• Canvas.aspectRatio, if set, maintains a proportional width and height of the visibleBox (considering also plotwidth/plotheight). aspectRatio of 1.0 should be square.
• Canvas.canvasCharWidth and Canvas.canvasCharHeight is the width and height of one terminal character, in canvas units.

These properties reserve an area of the Plotter that is outside the visibleBox:

• Canvas.leftMarginPixels
• Canvas.rightMarginPixels
• Canvas.topMarginPixels
• Canvas.bottomMarginPixels

During a mouse event, these properties indicate the mouse position for the current mouse event:

• Canvas.canvasMouse: a Point in canvas coordinates
• Plotter.plotterMouse: a Point in plotter (pixel) coordinates
• Sheet.mouseX and Sheet.mouseY: individual values in curses (character) coordinates

class `InvertedCanvas` Copy Section

An InvertedCanvas is a Canvas with a few internal methods overridden, such that the Y axis is inverted. For an InvertedCanvas, the minimum coordinates are in the lower-left.

InvertedCanvas has not much else of interest. It should be completely interchangeable with Canvas.

class `Graph` Copy Section

A Graph is an InvertedCanvas with axis labels and/or gridlines.

• Graph.__init__(name, sheet, rows, xcols, ycols) constructor
  • sheet is the source Sheet.
  • rows is a list of the rows to iterate over (from the given source).
  • xcols is a list of key columns forming the x-axis and color keys.
  • ycols is a list of numeric columns to be plotted on the y-axs.

Group (`docs/group.md`) Copy Section

eleventyNavigation: key: Grouping data and descriptive statistics order: 7 Update: 2018-08-19 Version: VisiData 1.3.1 Copy Chapter

How to set statistical aggregators for a single column Copy Chapter

Aggregators provide summary statistics for grouped rows.

Command Operation

+ aggregator adds aggregator to current column
z+ aggregator displays result of aggregator over values in selected rows for current column

The following statistical aggregators are available:

Aggregator Description

min smallest value in the group
max largest value in the group
avg/mean average value of the group
mode most frequently appearing value in group
median median value in the group
q3/q4/q5/q10 add quantile aggregators to group (e.g. q4 adds p25, p50, p75)
sum total summation of all numbers in the group
distinct number of distinct values in the group
count number of values in the group
keymax key of the row with the largest value in the group
list gathers values in column into a list
stdev standard deviation of values

Aggregation bar Copy Section

When one or more columns have aggregators, the aggregation bar appears at the bottom of the sheet.
Each aggregator has its own row, showing the aggregator name and the computed value for each column with that aggregator.
Values are computed over all rows asynchronously, and recomputed after rows are added, deleted, or edited.

• {help.options.color_aggregator}

The follow howtos will have examples of workflows involving grouping of data and statistical aggregation.

The following examples use the file sample.tsv.

How to group data (frequency table, pivot table, describe table,) Copy Chapter

How to make a pivot table frequency table, pivot table, describe table,) Copy Chapter

<script type="text/javascript" src="/asciinema-player.js"></script>
<script type="text/javascript">
    AsciinemaPlayer.create('../casts/pivot.cast', document.getElementById('player'), { poster: 'npt:0:10', rows: 27 });
</script>

1. Move to the column A with the independent variable, and press ! to mark it as a key column to group by.
2. Move to a numeric column B, and press + to add an aggregator to that column, so that the aggregator will be applied to each group of values.
3. Make sure column B has a numeric type: int (#), float (%), currency ($), or date (@).
4. Move to the column C with the dependent categorical variable to be pivoted, and press Shift+W (pivot) to pivot on that column. The pivoted sheet then will have a column for each distinct value in the source column.

[How to create a frequency chart](#frequency) {#frequency} Copy Chapter

How to make a histogram Copy Section

Question How many of each Item were sold?

1. Move the cursor to the Item column.
2. Press Shift+F to open the Frequency table.

How to use the Frequency table to view the results of statistical aggregation Copy Section

Question What was the monthly revenue?

1. On the OrderDate column, type ; followed by (\d+-\d+) to create a column with only the year and the month using regex capture groups.
2. Press - to hide the OrderDate column.
3. On the OrderDate_re0 column, type ^ followed by OrderDate to rename it.
4. On the Total column, press $ to set its type to currency.
5. Type + followed by sum to add a statistical aggregator to Total.
6. On the OrderDate column type Shift+F to open the Frequency table.
7. On the OrderDate column, press [ to sort the table in chrononological order.
8. On the sum_Total column, type ^ followed by Revenue to rename the column.

How to calculate some descriptive statistics Copy Section

1. Press Shift+I to open the Describe sheet.

How to filter for grouped rows Copy Section

1. Press Shift+F to open the Frequency table.
2. Press s or t on the groups you are interested in to select those entries in the source sheet.
3. Press q or Ctrl+^ to return to the source sheet.
4. Press " to open a duplicate sheet with selected rows.

or

1. Press Shift+F to open the Frequency table.
2. Press Enter on the grouping you are interested in to open a sheet of the source rows that are part of that group.

How to filter for described rows Copy Section

1. Press Shift+I to open the Describe sheet.
2. Use zs to select rows on source sheet which are being described in the cells of interest.
3. Press q or Ctrl+^ to return to the source sheet.
4. Press " to open a duplicate sheet with selected rows.

or

1. Press Shift+I to open the Describe sheet.
2. Press zEnter open copy of source sheet with rows being described in the current cell of interest.

How to filter for the rows aggregated in a pivot table Copy Section

Enter/zEnter can both be used in the Pivot table to open a sheet of the source rows which are aggregated in the current pivot row/cell.

How to combine cells in one for column for matching rows in another Copy Section

This tutorial shows you how to go from this dataset:

field value

A 1
A 2
B 3
C 4
C 5

to

field value

A 1;2
B 3
C 4;5

in VisiData.

1. On the value column, type + followed by list to add to it a list aggregator.
2. Press Shift+F on the field column to open the Frequency table.
3. Type = followed by ';'.join(value_list) to add an expression column with the joined values.

Internal Formats (`docs/internal_formats.md`) Copy Section

eleventyNavigation:
key: VisiData Internal Formats
order: 99

VisiData created these formats that it uses:

• .vd, .vdj, and .vdx to save a Command Log (for macros and replays);
• .vds to save/restore the current set of data sheets and columns.

All internal formats have comments being lines starting with #, so they can all be scripts on unix if they are executable and the first line starts with hash-bang (#!).

.vd Copy Chapter

.vd is the original replay format: a TSV save of the Command Log (g Shift+D). Deprecated in favor of .vdj.

.vdj Copy Chapter

.vdj is the next generation format, a JSONL of the Command Log.
This enables support for multi-input commands (where input is a dictionary) among other things.

Macros are stored in .vdj format (since 3.0). Macros from previous versions in .vd format are still accepted.

.vdx Copy Chapter

.vdx is an experimental format for the Command Log which does not require a data editor to create.
Since much of the time the sheet/col/row cursor position columns are blank, only the command longname and input are necessary to execute a command.
So the .vdx format is lines of commands with the command longnamefirst, and then a space and input (if used). Comments start with #.
The cursor can be moved specifically with the col and row commands.

The .vdx loader also handles .vd and .vdj lines, so --play - (stdin) always uses the vdx filetype, which accepts any of the three command log formats.

This format is interesting for some use cases but not compelling enough to replace .vdj.
There is no current intent to replace the .vdj format with .vdx.

.vds Copy Chapter

.vds is a new, simple, multisheet format based on JSONL that includes some sheet metadata (mostly column definitions).
If you care only about the data and columns on the sheets, and not any provenance or exactitude or any of that stuff you get from going directly to the source, then you can save to .vds.
Opening a .vds pushes its sheets onto the stack in the order they were saved, so a file written with save-all (g Ctrl+S) reopens with the same sheet on top.
Each sheet retains the data and layout it was saved with, but without any history.
Use open-source (`) on any of them to see the index of all sheets in the file.
Each row is a JSON object keyed by column name; a column whose name repeats an earlier one is keyed name_2, name_3, etc., with that key saved as key in its column metadata.
A key missing from a row stays missing, so sparse JSON rows keep their shape; an explicit null is saved as null.

Join (`docs/join.md`) Copy Section

eleventyNavigation: key: Combining datasets order: 9 Update: 2020-10-27 Version: VisiData 2.0.1 Copy Chapter

How to join two datasets Copy Chapter

1. Open the datasets in VisiData.

   a. vd d1.tsv d2.tsv

   or

   b. Press o and enter a filepath for each file.

2. Press S to open up the Sheets Sheet. Through here, you can navigate to every sheet by pressing Enter on the row it is referenced in.

3. Navigate to the sheets you want the join, and set their shared columns as key columns with !.

4. Press S to return to the Sheets sheet. Select the sheets you want to merge with s.

5. Optional: If performing a left outer join, use Shift+J or Shift+K to reorder the sheets. The first sheet will be the one for whom all rows will be retained.

6. Type & to open the join-chooser, and select your desired jointype with Enter.

jointype description

inner keeps only rows which match keys on all sheets
outer keeps all rows from first selected sheet
full keeps all rows from all sheets (union)
diff keeps only rows NOT in all sheets
extend keeps all rows and retain SheetType from first selected sheet
merge Merges differences from other sheets into first sheet
append keeps all rows from all sheets; columns from all sheets
concat keeps all rows from all sheets; matches columns by position, with names and types from first sheet

concat requires the same sheet type and number of visible columns in each sheet. append matches columns by name.

How to append two datasets Copy Chapter

1. Open the datasets with VisiData.
2. Press Shift+S to open the Sheets sheet.
3. Use s or t to select the sheets to merge.
4. Type & and press Enter on append to concatenate the selected datasets.

Identifying source rows Copy Chapter

The append and concat join types add a hidden origin_sheet column that shows which source sheet each row came from. Unhide all hidden columns with gv.

Loading (`docs/loading.md`) Copy Section

eleventyNavigation: key: Loading data order: 2 parent: Getting Started update: 2018-12-18 version: VisiData 1.5.1 Copy Chapter

Specifying a source file Copy Chapter

The first way to open a dataset in VisiData, specify it directly after invoking vd.

vd filename.ext

Alternatively, you can pipe or redirect data in through stdin.

ps aux | vd
vd - < sample.tsv

Finally, you can launch the DirSheet, navigate to the file you wish to load, and press Enter.

vd .

In VisiData, a loader is a module which directs how VisiData structures and engages with a particular data source. These sources are currently supported.

On default, the file extension determines which loader is used. Unknown filetypes are loaded as Text sheets. To force a particular loader, pass -f with the filetype or format name.

vd -f sqlite bar.db
ls -l | vd -f fixed

To read an Arrow IPC stream from stdin (requires pyarrow):

vd -f arrows - < data.arrows

How CLI options apply to files Copy Section

-f/-if/--input-filetype applies to all subsequent input paths on the command line, not just the one immediately following. The output file given with -o has its own flag, -of/--output-filetype; without it the output is saved by its extension.

-f attaches only to input paths given on the command line (and to piped stdin). Files opened later from within the session use their own extensions; the open-file and save-sheet commands have their own filetype field (the as filetype: prompt) for overriding it.

You can reset back to extension-based detection for later files with -f "":

See the manpage for more details.

vd -f csv data.txt -f "" other.jsonl

This loads data.txt as csv, and other.jsonl using its file extension as usual.

Loading sources supported by pandas Copy Chapter

VisiData has an adapter for pandas. To load a file format which is supported by pandas, execute vd -f pandas data.foo. This will call pandas.read_foo().

For example:

vd -f pandas data.parquet

loads a parquet file. When using the pandas loader, the .fileformat file extension is mandatory.

Note that if you are using Python v3.7, then you will need to manually install pandas >=0.23.2 (our requirements.txt file installs v0.19.2 as the last version compatible with 3.4).

Opening an R data frame with VisiData Copy Chapter

@paulklemm has wonderfully developed a small R package which bridges the gap between R and VisiData.

To install rvisidata using devtools run:

devtools::install_github('paulklemm/rvisidata')

from within the R interpreter.

Any data frame can then be opened by VisiData:

vd(iris)

Please note, that this tool opens the data frame in readonly mode. Any changes made will be discarded.

For more more details, questions, and feedback, check out the rvisidata repository.

Loading multiple datasets simultaneously Copy Chapter

Multiple files can be passed as inputs through the commandline.

vd birdsdiet.tsv surveys.csv sunshinelist.html

Upon launching, the final dataset to load (in this case, sunshinelist.html) will be displayed on top.

To load files from within a VisiData session, press o and enter a filepath.

Accessing other loaded or derived sheets Copy Chapter

1. Press Shift+S to open the Sheets sheet.
2. Move the cursor to the row containing the desired sheet.
3. Press Enter to jump to the sheet referenced in that current cursor row.

[How to convert a dataset from one supported filetype into another](#convert) {#convert} Copy Chapter

vd -b countries.fixed -o countries.tsv

The output format follows the -o filename's extension. To save to a filename whose extension does not match the desired format, give -of/--output-filetype:

vd -b countries.fixed -of csv -o countries.txt

Note: Not all filetypes which are supported as loaders are also supported as savers. See the formats page for the supported output formats.

Macros (`docs/macros.md`) Copy Section

eleventyNavigation:
key: Macros
order: 5
update: 2023-11-18

version: VisiData 3.0 Copy Chapter

Macros

Macros allow you to bind a series of commands to a key and then replay those commands within a session by using that keystroke.

The basic usage is:

1. Press m (macro-record) to begin recording the macro.
2. Go through the commands you wish to record.
3. Then type m again to complete the recording, and prompt for the keystroke or longname to bind it to.

The macro will then be executed every time the provided keystroke is used. Note: the Alt+keys and the function keys are left unbound; overriding other keys may conflict with existing bindings, now or in the future.

Executing a macro will the series of commands starting on the current row and column on the current sheet.

Keystroke Binding Copy Chapter

When binding a macro to a longname (like happy-time), you will also be prompted for an optional keystroke shortcut.
Type a keystroke (like Alt+s) and press Enter to bind it, or just press Enter to skip.

The keystroke is shown on the Macros Sheet and can be edited there.

Input Parameters Copy Chapter

Macro command inputs can use {param} or {param=default} placeholders in the input or col fields.
When a parameterized macro is run, VisiData prompts for parameter values before executing.

To add parameters to an existing macro:

1. Open the Macros Sheet with gm.
2. Press Enter on the macro to open its command log.
3. Edit the input or col cells to replace literal values with {param} or {param=default} placeholders.
4. Save the macro with z Ctrl+S.

For example, recording a macro that adds a column with =name + " suffix", then editing the input to {col} + " {text=suffix}", will prompt for col and text values each time the macro runs.

Parameter names must be valid Python identifiers, so {3} in a regex is left alone.
To keep literal braces in a parameterized macro, double them: f"{{name}}-{suffix}" runs as f"{name}-..." and prompts only for suffix.

If a recorded command's input or column already contains {name} (for example an f-string expression like =f"{first} {last}"), the macro is recorded with braces set to literal, and its braces run as typed with no parameter prompts.
The braces column on the Macros Sheet shows this; set it to param to use parameters in such a macro, after doubling any braces that should stay literal.

The Macros Sheet

Use gm (macro-sheet) to open an index of existing macros.

To delete a macro, move the cursor to it and press d to mark it for deletion. Then press z Ctrl+S to commit the change. This removes the macro's key binding and deletes its source file.

The helpstr column can be edited to add a description for each macro. This description will show up in the command palette and the Commands Sheet. Save changes with z Ctrl+S.

The keystroke column can be edited to bind or change a keystroke shortcut for a macro.

Enter will open the macro in the current row, and you can view the series of commands composing it.

Menu (`docs/menu.md`) Copy Section

eleventyNavigation:
key: Command Menus
order: 12
Updated: 2021-09-02

Version: VisiData v2.6 Copy Chapter

At the top of the screen, there's a typical application toplevel menubar:

Sheet  Edit  View  Column  ...

You can press Ctrl+H to activate the Help menu (shown in the upper right).
Then you can use the arrow keys (or hjkl of course) to navigate the various commands and submenus.

Enter or Space will execute the command or dive into the submenu.

q or Esc or Ctrl+Q will leave the menu without taking an action.

Other ways of opening the menu Copy Chapter

Alt+S or Alt+E etc will open that specific submenu (see the underlined letter).
On Macos, Option+S etc should work the same (with the exception of Option+E, which needs to be Option+E+Space).

You can also click on the menu with the mouse to open it, as well as clicking on the various menu items to navigate to them.
Click on the already active menu item will execute the command or navigate into the submenu.

Click off the menu to deactivate it.

Menu annotations Copy Chapter

The … annotation, like in traditional GUI menus, indicates that the command requires further input.
The ⎘ annotation on a command is specific to VisiData, and means that the command will push a sheet onto the sheet stack, so your data context will change. You can always return to the previous sheet with q (or if you like taking the long way around, the "Sheet»Quit»Top sheet" command in the menu).

Configuration Options Copy Chapter

To make the top menu line disappear (as it looked pre-2.6)

options.disp_menu = False

The menu can still be activated with Ctrl+H or the Alt+ keys as above. The menu system cannot be entirely disabled.

Theme options Copy Section

As with most VisiData display elements, the menu can be altered to taste. These options are available to change the colors and displayed chars:

• color_menu
• color_menu_active
• color_menu_spec
• color_menu_help
• disp_menu_boxchars
• disp_menu_more
• disp_menu_input
• disp_menu_push

Plugin API for adding new menu items Copy Chapter

The menu can be extended with additional commands. See the plugin API.

Mouse (`docs/mouse.md`) Copy Section

eleventyNavigation:
key: Mouse
order: 15
Update: 2022-06-19

Version: VisiData v2.8 Copy Chapter

How to disable the mouse in VisiData? Copy Chapter

To disable for all sessions, add to your ~/.visidatarc:

options.mouse_interval = 0 # disables the mouse-click
options.scroll_incr = 0    # disables the scroll wheel

To disable in the current session, press SPACE, and type the longname mouse-disable.

Move (`docs/move.md`) Copy Section

eleventyNavigation: key: How can I move around and search? order: 99 Copy Chapter

• basic movement just like vim
  • 'G' 'gg' '^G' work as in vim, as do some of the 'z' scroll commands
• 'g' prefix goes all the way
  . 'gh' (or g Left) already goes to the first column, but maybe '0' should also.
• (call out no number prefix?)
• '/' and '?' search for regex, 'z/' and 'z?' search by Python expr. 'n' and 'N' continue the last search of either type. These should work just like in vim but only within a single column.
  • 'g' searches across all columns
• 'zr'/'zc' go to a specific row or column by number
• 'r' and 'c' go by row key or column name

i

• 'g' and 'z' are also command prefixes in vd, but in vd they generally means something like 'g'lobal/embi'g'gen and 'z'croll/'z'mallify.

Navigate (`docs/navigate.md`) Copy Section

eleventyNavigation: key: Navigation order: 3 updated: 2020-07-18 version: VisiData 2.0 Copy Chapter

How to rapidly scroll through a sheet Copy Chapter

Command(s) Operation

← ↑ → ↓ PgUp PgDn Home End move as expected
h j k l move cursor one cell left/down/up/right (like in vim)
gh gj gk gl move all the way to the left/bottom/top/right of sheet
< > move up/down the current column to the next value which differs from current cell
{ } move up/down the current column to the next selected row

Go to a numbered row or column Copy Chapter

z r (go-row-number) and z c (go-col-number) go to a 0-based row or column number.
Negative numbers count from the end: -1 is the last row or column.
Column numbers include visible columns first, followed by hidden columns.
Out-of-bounds numbers move to the nearest end and warn with the resulting position.

How to search within a sheet Copy Chapter

Command(s) Operation

/ ? regex search for regex matches up/down the current column
g/ g? regex search for regex matches up/down over all visible columns
n Shift+N move to next/previous match from last search
z/ z? expr search by Python expr up/down (with column names as variables)

By default, search matches are highlighted in the sheet (like vim's hlsearch).

Command(s) Operation

highlight-col highlight a regex in the current column (without moving)
highlight-sheet highlight a regex in all columns (without moving)
highlight-clear clear all highlight patterns

Undo of a search command does not undo the highlighting; use highlight-clear to remove all highlights.

Options

Option Default Description

highlight_search True whether to highlight search matches
color_highlight_search 21 blue on 15 white color for highlighted search matches

The following example uses sample.tsv.

Question Has there been a day where we sold more than 95 Items?

1. Set the type of the Units column to integer by moving to the Units column and pressing #.
2. Type z/ followed by Units > 95.

Question What is the longname for gk?

1. Press z Ctrl+H to open the Commands sheet.
2. Move to the keystrokes column and press /, followed by gk.
3. Press c followed by longname to move the cursor to the longname column.

Input history Copy Chapter

When entering a search regex or other input that has been used before, a history palette appears showing previous inputs.

• Up/Down cycle through previous inputs (like shell history).
• Down past the newest entry restores the original typed text.
• Type to fuzzy search through history.
• PgUp/PgDn to scroll through the history palette.

The history palette uses the same visual style as the command palette, and is enabled when cmdpalette is included in options.disp_help_flags.

How to move between sheets Copy Chapter

The Sheets Sheet is a list of all sheets that have ever been opened (in order of opening).

The Sheets Stack (z Shift+Sis the list of active sheets (most recently used at top).

Jumping to sheets

1. Press Shift+S to open the Sheets sheet.
2. Move the cursor to the row containing the desired sheet.
3. Press Enter to jump to the sheet referenced in that current cursor row.

Jumping away from sheets

Command(s) Operation

Ctrl+^ jump to the previous sheet, without closing the current one
q quit the current sheet (closes it)

Note: A quit/closed sheet is grayed out on the Sheets Sheet.
Closed sheets are removed from the active sheets stack, and the next sheet is then shown. When the active sheets stack is empty, VisiData exits.

Pipes (`docs/pipes.md`) Copy Section

eleventyNavigation: key: Pipes order: 14 Update: 2019-09-11 Version: VisiData 2.0 Copy Chapter

stdin/stdout pipe/redirect Copy Chapter

VisiData works with other tools in a pipeline. VisiData will read any piped input from stdin as a sheet, and save the
results of any remaining sheets to stdout Visdata allows interactive edits in the middle of a pipeline.

• Use it to manually update (sort, filter, edit) tabular data in a pipeline `mysql < query.sql | vd | awk 'awkity {awk}'
• Use it to interactively pick processes to kill ps -ef | vd | tail -n +2 | xargs --no-run-if-empty kill

When redirecting VisiData output:

• Ctrl+Q will output current sheet (as it quits with the top sheet still on the stack)
• q (or gq) will output nothing (as it quits by dropping all sheets from the stack)

Use vd -o- to send directly to the terminal when not redirecting stdout (it's not necessary if already redirected).

To output a single column without the column header, make sure only that column is visible and save as .txt. For example, vd . --save-filetype txt | lpr.

Plugins (`docs/plugins.md`) Copy Section

eleventyNavigation: key: Plugins order: 13 Updated: 2026-10-16 Version: VisiData 3.5 Copy Chapter

A plugin is an optional Python module to modify or extend VisiData functionality. Once installed, all plugins are automatically imported when vd is started.

How to install a plugin

Plugins can be installed in 3 places:

1. They can come packaged with the VisiData install.
2. They can be installed via pip.
3. They can be copied into the plugins/ subdirectory of the VisiData directory.

builtin plugins Copy Chapter

The VisiData release includes additional functionality that isn't enabled by default, but which can be made available by importing the module.
For example, there are some features available in visidata.experimental, like digit_autoedit, which starts editing the current cell when a numeric digit is pressed (like Excel).
VisiData intentionally does not work this way, but people who prefer their terminal applications to act more like Excel can add this to their .visidatarc:

import visidata.experimental.digit_autoedit

via pip Copy Chapter

Any package with a visidata.plugins entry point will be loaded automatically by default.
These plugins must be managed via pip (to upgrade or remove).
Plugins can be published to PyPI or downloaded as wheels or installed directly from source.

For example, DarkDraw is a VisiData plugin that allows drawing text-art.
Once the Python module is installed, you can use vd to open and save a .ddw file, which is a text image or animation specified in JSON format.
It also adds other commands and options.

copied manually into local plugins directory Copy Chapter

Python modules in the VisiData directory (<vd_dir> below) are available for import from .visidatarc.

To install a plugin manually:

1. Make a plugins directory: mkdir -p <vd_dir>/plugins
2. Copy the plugin Python file there: cp myplugin.py <vd_dir>/plugins
3. Add a line to ~/.visidatarc to import the plugin: import plugins.myplugin
4. Install the dependencies for the plugin (if any).

How to Disable Plugins

The nuclear option is --nothing or -N flag, which forces the base configuration, not loading any .visidatarc or plugins.
This is especially useful for debugging; when filing an issue, please make sure the bug reproduces with -N (if not, it is likely something in your .visidatarc)

To load .visidatarc but not autoload plugins installed with pip, set options.plugins_autoload on the command-line:

`vd --plugins-autoload=False`

Plugins explicitly imported in .visidatarc will still be installed.

The VisiData directory

The VisiData directory holds local plugins (plugins/) and their dependencies (plugins-deps/), saved macros, Google API tokens, the HTTP cache for the scrape and reddit loaders, and REPL history.
VisiData adds it to the Python import path at startup.

The default VisiData directory is:

• Linux: ~/.config/visidata
• macOS: ~/Library/Preferences/visidata
• Windows: %LOCALAPPDATA%\visidata\visidata

On Linux and macOS, XDG_CONFIG_HOME changes the default to $XDG_CONFIG_HOME/visidata.

To use another directory, set the VD_DIR environment variable before starting vd, or pass --visidata-dir=<vd_dir> on the command line.
The command-line option takes precedence over VD_DIR.
Setting options.visidata_dir in .visidatarc does not work, because the VisiData directory is set up before .visidatarc is loaded.
Before VisiData 3.2, the default was ~/.visidata.
To keep using it, add this to the shell startup file:

export VD_DIR="$HOME/.visidata"

Other storage locations Copy Chapter

VisiData stores some files outside the VisiData directory.
VD_DIR and --visidata-dir do not move these.

• The data directory holds input history (input_history.jsonl) and the list of saved macros (macros.jsonl).
  Default: ~/.local/share/visidata (Linux), ~/Library/Application Support/visidata (macOS), %LOCALAPPDATA%\visidata\visidata (Windows), or $XDG_DATA_HOME/visidata.
• The cache directory holds cached downloads, like the message of the day and currency rates.
  Default: ~/.cache/visidata (Linux), ~/Library/Caches/visidata (macOS), %LOCALAPPDATA%\visidata\visidata\Cache (Windows), or $XDG_CACHE_HOME/visidata.
• The config file is config.py in the default VisiData directory if it exists, otherwise ~/.visidatarc.
  Set VD_CONFIG or pass --config=<file> to use another file.

VisiData API reference

For help making plugins, see the VisiData API reference.

Rows (`docs/rows.md`) Copy Section

eleventyNavigation: key: Rows order: 4 Update: 2023-10-12 Version: VisiData 3.0 Copy Chapter

[How to perform operations on a subset of rows](#subset) {#subset} Copy Chapter

Many commands can be finetuned to operate on rows which are 'selected'.

How to specify the subset of rows which are selected

Command(s) Operation

s t u select/toggle/unselect current row
gs gt gu select/toggle/unselect all rows
| \ *regex* select/unselect rows matching *regex* in **current** column
g| g\ regex select/unselect rows matching regex in any visible column
z| z\ *expr* select/unselect rows matching Python *expr* in **any visible** column
, select rows matching current cell in **current** column
g, select rows matching entire current row in all visible columns

An example usage follows.

How to filter rows Copy Chapter

1. Press s or t on the rows to be filtered.

2. Press

   a. gd to delete the selected rows.

   or

   b. " to open a duplicate sheet which has references to the selected rows. Edits performed within the duplicate sheet will also propagate to the source sheet.

   or

   c. g" to open a duplicate sheet which has references to all rows, and keeping the selected rows selected. The selection can be modified on the parent sheet and the new sheet independently, but any changes will not be reflected on both sheets.

   or

   d. z" to open a copy of the sheet which has copies of the selected rows. Any changes will not affect the source sheet.

   or

   d. gz" to open a copy of the sheet which has copies of all rows. Any changes will be reflected on the source sheet

The following example uses the file sample.tsv.

Question On which days have we sold less than 10 Binders?

1. Scroll to the Units column. Set the type of the Units column by pressing # (int).
2. Type z| followed by Item == 'Binder' and Units < 10 to select all of the rows where the Item is Binder and the number of Units is less than 10.
3. Press " to open a duplicate sheet with only those selected rows.

How to filter a random subset of rows Copy Chapter

1. Type Space random-rows followed by the number of rows you wish included in your random population sample.

How to select rows where the current column is not null or empty? Copy Chapter

'Null' cells, by default, are cells which contain None. This can be changed with options.null_value. Null cells can be set with zd (set current cell to None) or gzd (set selected rows in current column to None). Null cells are distinguished with a yellow ∅' symbol on the right hand corner. They are distinct from empty cells (which are '' in columns of string type.)

1. Type | followed by . to select all rows without empty or null cells in the current column.

How to select rows where the current column is null? Copy Chapter

There are several different options:

• z, matches by typed value while , matches by display value. Move to an empty or None cell in the column of interest and press , to select all empty and None cells in that column. Type z, on a None cell to select all rows that contain None in that column. Type z. on an empty cell to select all rows that have an empty cell in that column.
• Open a DescribeSheet for the current sheet with Shift+I. Move to the nulls column, and then move to the row which references the source column of interest. Type zs to select all null rows for that column.
• z| is a command which allows selection by provided Python expression criteria (using column names as variables). For non-numerical columns z| followed by not ColumnName, will select all empty cells and None cells for that column. For numerical columns it will also select cells with 0.
• A typed value of Python None is always a TypedWrapper, which allows sorting, etc., on columns with None values. For this reason z| ColumnName is None will not work, thought z| ColumnName.val is None will work. This is not a recommended approach, though at the moment, it is available.

How to move, copy and remove rows Copy Chapter

Command(s) Operation

J K move cursor row down/up
gJ gK move cursor row all the way to the bottom/top of sheet

How to copy and paste a single row

1. Press y to copy the row to the clipboard.
2. Move the cursor to the desired location.
3. Press p/Shift+P to paste the row after/before current row.

How to copy and paste multiple rows

1. Press s/t on those rows to select them.
2. Press gy to copy all selected rows to the clipboard.
3. Move the cursor to the desired location.
4. Press p/Shift+P to paste those rows after/before current row.

Note

VisiData has a universal paste mechanism: it creates new rows on the target sheet and then fills them with values from the copied rows from the previous sheet.

This value-filling happens positionally, so if columns are missing or in a different order, the values will be in different columns,

How to sort rows Copy Chapter

Command(s) Operation

[ ] sorts ascending/descending by current column; replace any existing sort criteria
g[ g] sorts ascending/descending by all key columns; replace any existing sort criteria
z[ z] sorts ascending/descending by current column; add to existing sort criteria
gz[ gz] sorts ascending/descending by all key column; add to existing sort criteria

How to sort a numerical column from highest number to lowest:

1. Set the type of the column being sorted by pressing # (int) or % (float).
2. Press [ to sort the column from highest to lowest.

How to sort a date column in chronological order:

1. Set the type of the column being sorted by pressing @ (date).
2. Press ] to sort the column chronologically.

How to sort based on multiple columns

1. Press ! on those columns to designate them as key columns.
2. Press g[ or g] to sort.

or

1. Sort the first column with [ or ].
2. Sort the next column with z[ or z] to add sorting to the existing criteria.

How to increase row height

Press v on any TableSheet to toggle multi-line rows. This dynamically lengthens rows so that the full content of the column is visible.

Multi-line rows have some limitations; they can't be paged, for instance. The full contents of a cell can be viewed or edited in an external program like emacs or less:

• Press e on a cell to enter Editing mode. Then scroll left and right to explore its contents.
• Press Ctrl+O while in editing mode, to open the contents of the current cell in an external $EDITOR.

TextSheet-specific options
TextSheets are used for loading .txt files in VisiData. They are also the default loaders used for un-identified sources. They are notable for having a single column which has the name "text".

TextSheets have a bonus option wrap which will wrap the text into multiple rows, so that it fits the window width.

• On TextSheet, press Shift+O to open its OptionsSheet. Press Enter on the wrap option, to set it to True. Press q to return to your TextSheet. Reload it with Ctrl+R. Note that reload undoes an previous modifications you may have made to the TextSheet.
• On the commandline, the option --wrap will set the wrap option to True for all TextSheets in that session.
• In your ~.visidatarc, options.wrap = True will set the wrap option to True for all TextSheets in every session.

Save Restore (`docs/save-restore.md`) Copy Section

eleventyNavigation:
key: Save and Restore
order: 11
Updated: 2018-01-17

Version: VisiData 0.99 Copy Chapter

How to save and replay a VisiData session Copy Chapter

This example creates and then uses the file pivot.vdj.

<script type="text/javascript" src="/asciinema-player.js"></script>
<script type="text/javascript">
    AsciinemaPlayer.create('../casts/save-restore.cast', document.getElementById('player'), { poster: 'npt:0:41', rows: 27 });
</script>

To save and restore a session in VisiData:

1. Save the cmdlog using one of the following options:

   a. Press Ctrl+D to save the cmdlog to a fn.vdj file.

   or

   b. Press Shift+D to view the CommandLog Sheet, then press Ctrl+S and save it with a .vdj suffix.

2. Press gq to quit VisiData.

3. Replay the cmdlog, on the commandline: vd -p fn.vdj.

Environment and locale Copy Chapter

set-envvar sets an environment variable and records it in the command logs, like a global option.
Setting LANG or any LC_* variable also reapplies the locale, which affects type-floatlocale, number formatting, and date formatting.

To record a locale in every saved cmdlog, add this to .visidatarc:

vd.setenv('LC_NUMERIC', 'en_US.UTF-8')

The locale must be installed on the replaying machine (locale -a lists them), and a nonempty LC_ALL overrides the other LC_* variables.

How to replay a cmdlog on a different file Copy Chapter

Cmdlogs record the exact filename and sheet names from the original session.
To reuse a cmdlog with a different input file, you need to make the cmdlog filename-independent.

Method: Edit the `.vd`/`.vdj` to remove hardcoded references Copy Section

1. Save your cmdlog as usual (Shift+D to open the cmdlog, Ctrl+S to save).
2. Edit the .vdj file:
   • Remove the open-file line (the line with "longname": "open-file").
   • Erase all "sheet" values (for instance with gs gzd). A blank sheet name means "use the current sheet", so commands operate on whatever is on top of the stack.
3. Replay on a new file:

vd -b -p cmdlog.vdj newfile.csv -o output.tsv

The new file is passed as a positional argument, and -p replays the cmdlog against it. Use -b for batch (non-interactive) mode and -o to write the result.

Limitations: Blank sheet names only work for linear workflows where you never switch between sheets. If your analysis involves joining sheets or switching back to a previous sheet, you'll need to either keep the sheet names, or use stack-relative commands like jump-prev (Ctrl+^, jump to previous sheet) or jump-first (gCtrl+^, jump to first sheet) to navigate between sheets without relying on their names.

Cmdlog formats Copy Section

VisiData supports several cmdlog formats:

• .vd (TSV) — original format with tab-separated columns
• .vdj (JSONL) — JSON objects, one per line; fields include sheet, col, row, longname, input, keystrokes, comment
• .vdx (simplified) — human-readable, one command per line; only emits sheet/col/row lines when they change

When replaying from stdin (vd -p -), VisiData uses the .vdx loader, which accepts all three formats.

Shell (`docs/shell.md`) Copy Section

eleventyNavigation:
key: Shell
order: 10
update: 2022-08-16

version: VisiData v2.9 Copy Chapter

How to add ZSH Completion Copy Chapter

1. Clone the VisiData repo: git clone https://github.com/saulpw/visidata
2. cd visidata
3. Run the script that will create the list of autocompletes for visidata

dev/zsh_completion.py

4. echo $fpath | grep site-functions to learn where your zsh site-functions folder is located. (Possible locations /usr/share/zsh/site-functions, /usr/local/share/zsh/site-functions)
5. Move the visidata compdefs to where zsh can find them: sudo mv _visidata /path/to/site-functions
6. compinit
7. Check that the autocompletes are configured with grep -n visidata ~/.zcompdump

Split (`docs/split.md`) Copy Section

eleventyNavigation: key: Split view order: 99 Copy Chapter

Viewing two sheets simultaneously

VisiData supports the viewing of two sheets simultaneously with a feature called split screen.

Entering split screen mode Copy Chapter

Keystroke(s) Action

Shift+Z split screen in half, placing second sheet on stack visibly in the second pane
z Shift+Z n split screen, setting the height of the second screen to n

When there is a single pane, all currently-open sheets are situated within the same sheets stack. The more recent that sheet was viewed, the higher it is in the stack.

Split-pane opens a second window, and creates a second stack for it. This keeps the sheet stably within a specific window.

For instance, pressing Shift+C on the sample.tsv will give us a stack with two columns, the sample Sheet and its Columns Sheet.

![]

Pressing Shift+Z will now give us a view of them both, with the sample Sheet moved to the second window.

![]

Split screen mode specific commands Copy Chapter

While in split screen mode, you can move between windows, adjust the sheets that appear in the current window, and full-screen the current pane.

Keystroke(s) Action

z Shift+Z n adjust the height of the second window to n
g Shift+Z full screen the current window
Tab make the other window the active window
Ctrl+^ swap top two sheets in stack in current window, second sheet in stack is now visible
g Ctrl+^ cycle through sheets in stack in current window

For instance, we can use Tab to make the sample Sheet the active window.

![]

Moving around it, we can see the Columns Sheet in the top window continuing to be updated.

We can press Shift+F on the Item column to view its Frequency Table.

Now, the visible sheet of the second window has changed, while the visible sheet on the top window has stayed the same.

![]

We can use ^^, so that sample Sheet is once again visible, and full-screen it with g Shift+Z, effectively leaving split screen mode.

![]

Test (`docs/test.md`) Copy Section

eleventyNavigation:
key: Contributing Tests
order: 99
Update: 2024-01-10

Version: VisiData 3.0 Copy Chapter

The tests folder contains functional tests in the form of .vd scripts, each of which records a session of VisiData commands. These ensure that data processing works consistently and reliably.

dev/test.sh (run from the git root) will execute all tests. The final sheet of each test is saved as .tsv and compared to the respective expected output checked into the tests/golden directory.

As of January 2024, to pass all the tests, you will need to install the test extras:

git clone https://github.com/saulpw/visidata.git
cd visidata
pip3 install ".[test]"

To show each step of a test with a delay of 1 second between commands:

$ bin/vd -p tests/foo.vd -w 1

To build a .vd file:

1. Go through all of the steps of the workflow, ending on the sheet with the final result.

2. Press Shift+D to view the CommandLog Sheet.

3. Edit the commandlog to minimize the number of commands. Cells may be parameterized like {foo}, to be specified on the commandline:

   $ vd cmdlog.vd --foo=value

4. Save the cmdlog:

   Press Shift+D to open the commanDlog Sheet, and press Ctrl+S to save it with a .vd suffix.

There are also unit tests in visidata/tests. To run the unit tests:

pytest -sv visidata/tests

Usage (`docs/usage.md`) Copy Section

eleventyNavigation:
key: Usage
order: 99

How can I load files and exit the program?

If you use the command line, 'vd' should pretty much work as you expect:

$ vd foo.csv

You can pipe in tabular data:

$ find | vd

If the file is not plain ASCII .txt or tsv, you will have to specify the filetype with -f and/or other options:

$ ls -al | vd -f fixed --skip 1 --headers 0

Many data formats are supported, with more being added every day.

How do I quit?

You can just bang on 'q' until all the sheets are gone (there are usually only one or two anyway).

Note: If a prompt or editbox is waiting for input, you may have to press '^C' first.

It requires a bit more dexterity, but Control-Q (aka '^Q') will abort the program quite rudely.[1]

If you wish to be a bit gentler, 'gq' will quit all sheets ('global quit').

[1] Control-Q is the only builtin command (necessary sometimes during development). All other commands can be overridden.

How do I confirm before quitting?

• options.quitguard = True in .visidatarc: confirm before quitting a sheet with unsaved changes.
• vd --quitguard foo.csv: guard the sheets opened from the command line. Quitting a guarded sheet always asks, even with no changes.
• guard-sheet (File > Guard > on): guard the current sheet. guard-sheet-off removes the guard.

When exiting VisiData, one confirmation covers all sheets.

Viewtsv (`docs/viewtsv.md`) Copy Section

eleventyNavigation:
key: viewtsv Annotated
order: 99
Date: 2017-12-27

Version: 1.0 Copy Chapter

viewtsv

viewtsv.py
is a great example of a minimal VisiData application. This is an extremely functional utility in 25 lines of code, leveraging the essence of the VisiData architecture. Here it is in its entirety, with line by line annotations:

#!/usr/bin/env python3

VisiData 2.x requires Python 3.6+.

import sys
from visidata import Sheet, ColumnItem, options, asyncthread, run

Import the relevant components used below.

class TsvSheet(Sheet):
    rowtype = 'rows'  # rowdef: tuple of values

All tabular data sheets inherit from Sheet. The rowtype is displayed on the right status bar. The rowdef comment declares the structure of every row object, and should be present for every Sheet.

columns = [ColumnItem('tsv', 0)]

An initial column. Generally the class-level columns is set to the actual columns of the sheet, but in this case, the columns aren't known until the source is loaded.
(See the reload() function below where they are set from the contents of the first row.) This line is not strictly necessary but makes loading feel a bit more responsive.

@asyncthread

@asyncthread marks the function to spawn a new thread whenever it is called.

def reload(self):

The reload() function collects data from the source and puts it into rows. It's called once automatically when first pushed, and manually with ^R.

self.rows = []

rows is a list of Python objects. The row definition ('rowdef') for the TsvSheet is a tuple of values, with each position corresponding to one column.

with open(self.source, encoding=options.encoding) as fp:

source is the filename, which has been passed to the constructor (see the last line with run()).

for line in fp:
                line = line[:-1]
                if line:
                    self.addRow(line.split('\t'))

For each line, strip the included newline character, and filter out any blank lines. Add each split tuple to rows.

for i, colname in enumerate(self.rows[0]):
            self.addColumn(ColumnItem(colname, i))

The actual columns are set from the first (header) row.
ColumnItem is a builtin, which creates a column to use getitem/setitem with the given key/index.

self.rows = self.rows[1:]

The header row is removed from the list of rows. (Column names are displayed on the first row anyway).

run(*(TsvSheet(fn, source=fn) for fn in sys.argv[1:]))

run(*sheets) is the toplevel entry point for a VisiData application.

Guides MovementGuide (`guides/MovementGuide.md`) Copy Section

Movement and Search

The g prefix extends most movement commands to go all the way to the edge
of the sheet, and the z prefix gives a more precise or smaller variant.

Basic movement Copy Chapter

• {help.commands.go-left}
• {help.commands.go-down}
• {help.commands.go-up}
• {help.commands.go-right}
• {help.commands.go-leftmost}
• {help.commands.go-bottom}
• {help.commands.go-top}
• {help.commands.go-rightmost}
• {help.commands.scroll-middle}

Skip by value Copy Chapter

• {help.commands.go-next-value}
• {help.commands.go-prev-value}
• {help.commands.go-next-null}
• {help.commands.go-prev-null}
• {help.commands.go-next-selected}
• {help.commands.go-prev-selected}

Search Copy Chapter

Search operates on the current column's displayed values by default. Use the g prefix to search across all visible columns.

• {help.commands.go-col-regex}
• {help.commands.search-keys}
• {help.commands.search-col}
• {help.commands.searchr-col}
• {help.commands.search-cols}
• {help.commands.searchr-cols}
• {help.commands.search-expr}
• {help.commands.searchr-expr}
• {help.commands.search-next}
• {help.commands.searchr-next}

Sheets Copy Chapter

• {help.commands.sheets-stack}
• {help.commands.jump-prev}

Api Async (`docs/api/async.rst`) Copy Section

.. _threads:

========================= Threads

VisiData is not necessarily the fastest at processing large amounts of data, but great care has been taken to make sure that it remains responsive.
It is better to wait 10 seconds with active indication of progress, than 2 seconds with the program frozen and not accepting input (or worse; queueing up commands without ability to cancel).

To this end, VisiData has a main interface thread for display and input, and creates threads for any task that may be time-consuming.
In fact, anything that iterates through all rows (there may be millions) or all columns (there may be thousands), should probably be in its own thread.

@asyncthread

Use the ``@asyncthread`` decorator on a function to spawn a new thread for each call to that function.
The return value is the spawned thread (which can often be ignored by the caller), and the return value of the original function is effectively lost.

Cells which are being computed in a separate thread will have the Thread object as their value until the result is available.
[This will show the ``options.disp_pending`` cell note and allow the user to interact with the specific thread (via e.g. :kbd:`z Ctrl+Y` or :kbd:`z Ctrl+C`).]

Each thread is added to ``Sheet.currentThreads`` on the current sheet, and to ``vd.threads``.
Note that a thread spawned by calling a function on a different sheet will add the thread to the currentThreads for the topmost/current sheet instead.

.. autoclass:: visidata.asyncthread

.. data:: visidata.vd.threads

A list of all threads.

.. data:: visidata.BaseSheet.currentThreads

A list of all threads started from commands on this sheet.

.. autofunction:: visidata.vd.execAsync
.. autofunction:: visidata.vd.sync
.. autofunction:: visidata.vd.cancelThread

.. note::

    ``vd.cancelThread(*threads)`` sends each thread an ``EscapeException``, which percolates up the stack to be caught by the thread entry point.
    ``EscapeException`` inherits from Python's ``BaseException`` instead of ``Exception``, so that functions can still have catchall try-blocks with ``except Exception:``, without catching explicit user cancellations.

    An unqualified ``except:`` clause is bad practice (as always).
    When used in an async function, it will make the thread uncancelable.

.. _progress:

Progress counters

To provide the progress meter in the right status bar, every function that could iterate over a large collection of objects (usually rows or columns) or loop a large number of times, should use an Progress object somewhere to track its progress.

.. note::

Many internal functions will contribute their own ``Progress`` counters (including ``visidata.Path.read``), so an async function may not need its own, but it can still be helpful to provide a more precise *gerund* to show to the user.

.. autoclass:: visidata.vd.Progress

.. function:: visidata.vd.Progress.addProgress(n)

Increase the progress count by *n*.

Progress as iterable
^^^^^^^^^^^^^^^^^^^^^

When iterating over a potentially large sequence:

::

for item in Progress(iterable, gerund='working'):

This is equivalent to for item in iterable. The gerund specifies the action being performed.

• These Progress only display if used in another thread, and has no effect otherwise. So any iteration over rows and columns should have Progress, even if it is not @asyncthread.
• Use Progress around the innermost iterations for maximum precision in the progress meter.
• But this incurs a small amount of overhead, so if a tight loop needs every last optimization, use it with an outer iterator instead.
• Multiple Progress objects used in parallel will stack properly.
• Multiple Progress objects used serially will make the progress indicator reset (which is better than having no indicator at all).

If iterable does not know its own length, then pass the length or an approximation as the total keyword argument:

::

for item in Progress(iterable, gerund='working', total=approx_size):

Each iteration of the Progress object contributes 1 towards the total.
To contribute a different amount, use Progress.addProgress(n) (n-1 if being used as an iterable, as 1 will be added automatically).

Progress as context manager
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To manage Progress without wrapping an iterable, use it as a context manager with only a total keyword argument, and call addProgress as progress is made:

::

with Progress(gerund='working', total=amt) as prog:
    while amt > 0:
        some_amount = some_work()
        prog.addProgress(some_amount)
        amt -= some_amount

• Using Progress other than as an iterable or a context manager will have no effect.

Threads Sheet

The Threads Sheet is useful for analyzing the performance of long-running async commands.

All threads (active, aborted, and completed) are added to ``vd.threads``, which can be viewed as the ThreadsSheet via :kbd:`Ctrl+T`.
Threads which take very little time (currently less than 10ms) are removed, to reduce clutter.

- Press :kbd:`Ctrl+_` (*toggle-profile*) to toggle profiling of the main thread.
- Press :kbd:`Enter` (*open-row*) on a thread to view its performance profile (if ``options.profile`` was set when the thread started).

Profiling
^^^^^^^^^

The performance profile sheet in VisiData is the output from ``pstats.Stats.print_stats()``.

- :kbd:`z Ctrl+S` on the performance profile will call ``dump_stats()`` and save the profile data to the given filename, for analysis with e.g. `pyprof2calltree <https://libraries.io/pypi/pyprof2calltree>`__ and `kcachegrind <https://kcachegrind.github.io/html/Home.html>`__.

---

### Api Canvas (`docs/api/canvas.rst`)

Canvas
=========

.. autofunction visidata.BoundingBox

.. autofunction visidata.Canvas.point
.. autofunction visidata.Canvas.line
.. autofunction visidata.Canvas.polyline
.. autofunction visidata.Canvas.polygon
.. autofunction visidata.Canvas.qcurve
.. autofunction visidata.Canvas.label

.. autofunction visidata.Canvas.plotterCursorBox
.. autofunction visidata.Canvas.cursorBox
.. autofunction visidata.Canvas.plotterMouse
.. autofunction visidata.Canvas.canvasMouse
.. autofunction visidata.Canvas.zoomTo
.. autofunction visidata.Canvas.resetBounds
.. autofunction visidata.Canvas.rowsWithin
.. autofunction visidata.Canvas.fixPoint

.. autofunction visidata.Canvas.plotpixel
.. autofunction visidata.Canvas.plotlabel

Canvas
=========
.. autofunction visidata.GraphSheet.formatX
.. autofunction visidata.GraphSheet.formatY
.. autofunction visidata.GraphSheet.parseX
.. autofunction visidata.GraphSheet.parseY

---

### Api Columns (`docs/api/columns.rst`)

.. _columns:

Columns
====================================

Columns are the heart of the VisiData computation engine.

Each column can **calculate** a value from a row object; and it might also be able to **put** a different value into the row object (for a later calculate to re-derive).

A Column subclass can override ``calcValue`` and ``putValue`` to define its fundamental interaction with the row object.
This is often the only thing a Column subclass has to do.

``calcValue`` and ``putValue`` should generally not be called by application code.
Instead, apps and plugins should call ``getValue`` and ``setValue``, which provide appropriate layers of caching.

.. autoclass:: visidata.Column
.. autoattribute:: visidata.Column.name
.. autoattribute:: visidata.Column.type
.. autoattribute:: visidata.Column.width
.. autoattribute:: visidata.Column.visibleWidth
.. versionadded:: 2.1

.. autoattribute:: visidata.Column.fmtstr

.. autoattribute:: visidata.Column.hidden

.. autofunction:: visidata.Column.calcValue
.. autofunction:: visidata.Column.putValue

.. autofunction:: visidata.Column.getValue
.. autofunction:: visidata.Column.getTypedValue
.. autofunction:: visidata.Column.getDisplayValue
.. autofunction:: visidata.Column.formatValue

.. autofunction:: visidata.Column.getValueRows
.. autofunction:: visidata.Column.getValues


.. autofunction:: visidata.Column.setValue
.. autofunction:: visidata.Column.setValues
.. autofunction:: visidata.Column.setValuesTyped
.. autofunction:: visidata.Column.setValuesFromExpr

.. autofunction:: visidata.BaseSheet.evalExpr
.. autofunction:: visidata.Column.recalc
.. autofunction:: visidata.TableSheet.recalc
.. autofunction:: visidata.Column.isError

- ``calcValue`` may be arbitrarily expensive or even asynchronous, so once the value is calculated, it is cached until ``Column.recalc()`` is called.
- ``putValue`` may modify the source data directly (for instance, if the row object represents a row in a database table).  VisiData will *never* modify source data without an explicit ``save`` command.  So applications (and all other code) must call ``setValue`` to change the value of any cell.

- ``delete-cell`` actually just calls setValue with None.

Columns are added to Sheets with :ref:`addColumn <enumerate-columns>`.

Example
~~~~~~~~~

::

    class SpecialColumn(Column):
        'Column for special fields on special rows.'
        def calcValue(self, row):
            return row.special()[self.expr]

        def putValue(self, row, val):
            row.special()[self.expr] = val

    c = SpecialColumn('field1', expr='field1_key')
    sheet.addColumn(c)


Column Subclasses

.. autoclass:: visidata.ItemColumn
.. autoclass:: visidata.AttrColumn
.. autoclass:: visidata.ExprColumn
.. autoclass:: visidata.SettableColumn
.. autoclass:: visidata.SubColumnFunc

Key Columns

.. autofunction:: visidata.TableSheet.setKeys
.. autofunction:: visidata.TableSheet.unsetKeys
.. autofunction:: visidata.TableSheet.rowkey
.. autofunction:: visidata.TableSheet.keystr
.. autoattribute:: visidata.TableSheet.keyCols
.. autoattribute:: visidata.TableSheet.nonKeyVisibleCols

Types

The value returned by getValue could be many different things:

• a string
• numerically typed
• a list or dict
• None
• a null value (according to options.null_value)
• an Exception (error getting)
• a Thread (async pending)
• any python object

This value may need to be parsed and/or converted to a consistent type.
So, every column has a type attribute, which affects how it is parsed, displayed, grouped, sorted, and more.

The default column type is anytype, which lets the underlying value pass through unaltered.
This is the only type for which Column.getTypedValue can return arbitrary types.

The classic VisiData column types are:

============ ================== ========= ================= ============
type description numeric command keystrokes
============ ================== ========= ================= ============
anytype pass-through type-anytype :kbd:z~
str string type-str :kbd:~
date date/time Y type-date :kbd:@
int integer Y type-int :kbd:#
float decimal Y type-float :kbd:%
currency decimal with units Y type-currency :kbd:$
vlen sequence length Y type-vlen :kbd:z#
============ ================== ========= ================= ============

The default keybindings for setting types are all on the shifted top left keys on a US keyboard.

User-defined Types

Fundamentally, a type is a function which takes the underlying value and returns an object of a specific type.
This function should accept a string and do a reasonable conversion, like Python ``int`` and ``float`` do.
And like those builtin types, this function should produce a reasonable baseline zero (arithmetic identity) when passed no parameters or None.

Computations should generally call ``getTypedValue``, so that the values being used are consistently typed.

If the underlying value is None, the result will be a ``TypedWrapper``, which provides the baseline zero value for purposes of comparison, but a stringified version of the underlying value for display.
For a ``calcValue`` which raises or returns an Exception, ``getTypedValue`` will return a ``TypedExceptionWrapper`` with similar behavior.


In the following, ``TYPE`` is the type (like ``int``, ``date``, etc), and ``typedval`` is an instance of that type.

A VisiData type function or constructor must have certain properties:

    - ``TYPE()`` must return a reasonable default value for *TYPE*.
    - ``TYPE(typedval)`` must return an exact copy of *typedval*.
    - ``TYPE(str)`` must convert from a reasonable string representation into *TYPE*.
    - ``TYPE.__name__`` must be set to the official name of the type function or constructor.

Objects returned by ``TYPE(...)`` must be:

    - comparable (for sorting)
    - hashable
    - formattable
    - roundable (if numeric, for binning)
    - idempotent (``TYPE(TYPE(v)) == TYPE(v)``)

Types API
~~~~~~~~~~~

.. autoclass:: visidata.vd.addType
.. autofunction:: visidata.vd.isNumeric
.. versionadded:: 2.1
.. autofunction:: visidata.isNumeric
.. deprecated:: 2.1
    Use ``vd.isNumeric`` instead.

Examples
~~~~~~~~~

::
    # Add an ip_address type.
    vd.addType(ipaddress.ip_address, icon=':', formatter=lambda fmt,ip: str(ip))

    TableSheet.addCommand(None, 'type-ipaddr', 'cursorCol.type=ipaddress.ip_address', 'set type of current column to IP address'

Nulls
~~~~~~

VisiData has a crude concept of null values.  These interact with:
   a. aggregators: the denominator counts only non-null values
   b. the Describe Sheet: only null values count in the nulls column
   c. the ``fill-nulls`` command
   d. the ``melt`` command only keeps non-null values
   e. the ``prev-null`` and ``next-null`` commands (:kbd:`z<` and :kbd:`z>`)

- The null values are Python ``None`` and options.null_value if set.

.. note::

    ``options.null_value`` can be used to specify a null value in addition to Python ``None``.
    The CLI can only set this option to a string like ``''`` (empty string) or ``'NA'``.
    The option can be set to a different type in ``.visidatarc`` or other code, though:

    ::

        options.null_value = 0.0
        options.null_value = date("1980-01-01")

.. autofunction:: visidata.BaseSheet.isNullFunc

There is no direct isNull function, because the possible null values can change at runtime via the above option, and getting an option value is very expensive to do in a bulk operation.

Aggregators
===========

Aggregators allow you to gather the rows within a single column, and interpret them using descriptive statistics.
VisiData comes pre-loaded with a default set like mean, stdev, and sum.

.. autofunction:: visidata.TableSheet.addAggregators

.. autofunction:: visidata.vd.aggregator

The `type` parameter is optional. It allows you to define the default type of the aggregated column.

Example
~~~~~~~

Add aggregator for `numpy's internal rate of return <https://numpy.org/doc/stable/reference/generated/numpy.irr.html>`__ module:

::

    import numpy as np
    vd.aggregator('irr', np.irr, type=float)

---

### Api Commands (`docs/api/commands.rst`)

.. _commands:

Commands
--------

VisiData is **command-driven**, which means that it only does something when you tell it to.
Otherwise, it just sits there, waiting for your next command.

Every command is a discrete unit of behavior that does a defined task and runs to completion.
Functions that could take longer than a couple hundred milliseconds should run in their own thread (see :ref:`Threads <threads>`).

Every command should be **reproducible**: given the same sheet, cursor position, and :ref:`input <input>` string (if relevant), a command should yield identical output (with a few obvious exceptions, like :kbd:random-rows).

Any command which makes changes to a :ref:`saveable sheet<sheets>` is appended to that sheet's **Command Log**.
Since all state changes must be initiated by a reproducible command, this command log can be `replayed </docs/save-restore>`__.

Adding new commands is a natural way to extend VisiData's functionality.

Command Overview
~~~~~~~~~~~~~~~~

Commands and Keybindings are managed in a similar way to options.
The same precedence hierarchy is used, so that commands can be created or overridden for a specific type of sheet, or even a specific sheet itself.

Since all Sheets ultimately inherit from ``BaseSheet``, a command on ``BaseSheet`` is effectively a global command.

.. _command-context:

Command Context
^^^^^^^^^^^^^^^

The *execstr* is a string of Python code passed to ``exec()`` when the command is run.

``exec()`` looks up symbols in this order:

-  the current ``sheet``
-  the ``vd`` object
-  the ``visidata`` module (see ``addGlobals()`` and ``getGlobals()`` :ref:`below<other-commands>`)

The ``vd`` and ``sheet`` symbols are available to specify explicitly.

``currow`` is available on TableSheets; it provides access to column values for the current cursor row by name (e.g. ``currow.Price`` returns the typed value of the ``Price`` column for the cursor row).

.. note::

    Unqualified ``options`` in a command execstr will use the sheet-specific options context for the current sheet.

.. warning::

   In an *execstr*, while you can **get** an attribute on ``vd`` or ``sheet`` without specifying the object, to **set** an attribute does require an explicit object.  e.g. instead of ``cursorRowIndex=2``, it must be ``sheet.cursorRowIndex=2``.

Commands API
~~~~~~~~~~~~

.. autofunction:: visidata.BaseSheet.addCommand

Keybindings
~~~~~~~~~~~~

   - Use "``Ctrl+X``" for :kbd:`Ctrl+X`.
   - Primarily, plugin authors and users should use ``0-9``, function keys (``F2``), ``Alt+`` for custom keybindings; these are purposefully left available for user keybindings.
   - Consider not providing a default at all, for infrequently used commands.
   - Instead give it an easy and memorable longname, and/or a unique *helpstr* which can be searched for in the **Command Help** (:kbd:`g Ctrl+H`) with :kbd:`g/`.
   - Many other keycodes can be returned from the curses library as strings.
   - To discover what to use for some unknown key, press that key in VisiData and use the keystroke shown in the status bar.

.. autoattribute:: visidata.vd.allPrefixes

``vd.allPrefixes`` is a list of *prefixes* (keystrokes that don't trigger the end of a command sequence).
New prefixes can be added to this list, and then they can also be used as prefixes in keybindings.

.. note::

    Combinations of prefixes are allowed, but only in the specified order: ``g`` must come before ``z``, which must come before ``Alt+``.

.. _other-commands:

Other helpful functions

.. autofunction:: visidata.vd.bindkey
.. autofunction:: visidata.BaseSheet.bindkey
.. autofunction:: visidata.BaseSheet.execCommand
.. autofunction:: visidata.addGlobals
.. autofunction:: visidata.getGlobals

Rules for command longnames
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1. 3 words max, 2 words if possible. A longname should be short and fit on a keymap.

2. Command classes should be unique in their first 3 chars and ideally mostly in their first 2.

3. Command longnames should be intuitively understandable, or at least not jargony.

4. Longnames should evoke interface, when possible.

5. Use existing verb-object-input structure if possible:

Verbs:

• open: push new sheet
• jump: push existing sheet
• dup: push copy of sheet
• show: display on status
• go: move the cursor
• scroll: change the visible screen area without changing the cursor
• addcol: add new column to this sheet
• setcol: set selected cells in this column
• search: search one or more columns
• searchr: search reverse
• select/unselect/stoggle: add/remove rows from selected rows
• syscopy/syspaste: copy/paste to system clipboard
• sysopen: open with $EDITOR or other external program

Objects:

• -all: all sheets or all visible columns
• -col: cursorCol
• -cols: all visible columns
• -cells: this column, selected rows
• -selected: selected rows

Inputs:

• -expr: python expression
• -regex: python regex

Many others are used, see the full command list for inspiration.

Examples

::

    @Sheet.api
    def show_hello(sheet):
        vd.status(sheet.options.disp_hello)

    # `sheet` members and `vd` members are available in the execstr scope
    BaseSheet.addCommand(None, 'show-hello', 'show_hello()', 'show a warm greeting')

    # bind Shift+H, Ctrl+H, and Alt+H to this command
    BaseSheet.bindkey('H', 'show-hello')
    BaseSheet.bindkey('Ctrl+H', 'show-hello')
    BaseSheet.bindkey('Alt+h', 'show-hello')

---

### Api Data (`docs/api/data.rst`)

Selected Rows
-------------

Each TableSheet has a set of *selected rows*, which is a strict subset of the rows on the sheet.

.. autoattribute:: visidata.TableSheet.selectedRows
.. autoattribute:: visidata.TableSheet.someSelectedRows
.. versionchanged:: 2.1
.. autoattribute:: visidata.TableSheet.onlySelectedRows
.. versionchanged:: 2.1
.. autoattribute:: visidata.TableSheet.nSelectedRows

.. autofunction:: visidata.TableSheet.selectRow
.. autofunction:: visidata.TableSheet.unselectRow
.. autofunction:: visidata.TableSheet.isSelected

.. autofunction:: visidata.TableSheet.clearSelected

.. autofunction:: visidata.TableSheet.selectByIdx
.. autofunction:: visidata.TableSheet.unselectByIdx

.. autofunction:: visidata.TableSheet.select
.. autofunction:: visidata.TableSheet.unselect
.. autofunction:: visidata.TableSheet.deleteSelected


.. note::

    To clear the set of selected rows before any bulk selection, set ``options.bulk_select_clear`` to True.
    The status message will include "instead" as a reminder that the option is enabled.

.. warning::

    ``selectedRows`` feels like a list, but it's actually a property that iterates over all rows to generate the selected rows in sheet order.
    With large datasets, collecting the list of selected rows itself can take a large time, regardless of the number of rows that are actually selected.
    So instead of using selectedRows in the execstr, call an ``@asyncthread`` ``@TableSheet.api`` function which uses sheet.selectedRows.
    Use it as a parameter immediately or save it to a local variable on the first usage, to avoid unnecessary work.

Sorting
-------

.. autofunction:: visidata.TableSheet.orderBy
.. autofunction:: visidata.TableSheet.sort

---

### Api Extensible (`docs/api/extensible.rst`)

API Extensions
===============

One of VisiData's core design goals is **extensibility**.
Many of its features can exist in isolation, and can be enabled or disabled independently, without affecting other features.

So, VisiData encourages plugin authors to provide new features in a similarly modular form.
These features can be enabled by importing the module, or left disabled by not importing it.
Modules should degrade or fail gracefully if they depend on another module which is not available.

The ``Extensible`` base class
-----------------------------

Python classes usually implement all methods and members in the class definition, which exists in a single source file. A modular VisiData feature, by contrast, is self-contained within a separate source file, while providing methods on previously-defined classes.

Core functionality can be changed through 'monkey-patching', which is the ability for modules loaded after startup to add or change methods on existing classes.

To make this a bit easier, the core classes in VisiData inherit from the ``Extensible`` class, which provides some helper functions and decorators to make monkey-patching easier and more consistent.
All of their subclasses are then also naturally Extensible.

Extensible API
--------------

.. autoclass:: visidata.Extensible

.. autofunction:: visidata.Extensible.api

This decorator defines a member function on a specific class.

Because this is a member function, the first parameter is the instance itself.
If this function were defined in the class, the first parameter would be named ``self`` by Python convention.
When members are defined in other files, it is better to use a specific local object name instead of ``self``.
Use ``sheet`` for any Sheet type, ``col`` for any Column type, and ``vd`` for VisiData (which will shadow the global ``vd`` object, but as it is a singleton, they will be identical).

::

        @VisiData.api
        def vd_func(vd, ...):
            pass

        @Sheet.api
        def sheet_func(sheet, ...):
            pass

        @Column.api
        def col_func(col, ...):
            pass

``Extensible.api`` can be used either to add new member functions, or to
override existing members. To call the original function, use
``func.__wrapped__``:

::

        @Sheet.api
        def addRow(sheet, row):
            # do something first
            addRow.__wrapped__(sheet, row)

        ....

        sheet.addRow(row)

.. autofunction:: visidata.Extensible.class_api

.. autofunction:: visidata.Extensible.property

This acts just like the ``@property`` decorator, if it were defined inline to the class.

.. autofunction:: visidata.Extensible.lazy_property

This works like ``@property``, except it only computes the value on first access, and then caches it for every subsequent usage.

.. note::

    Because of how Python instantiates classes, extensions monkey-patched into a class are not also added to already-instantiated objects.
    So global sheets defined by a plugin should be added to the VisiData object as a ``@VisiData.lazy_property``.
    This way, they are not created until their first use, which allows them to take advantage of Sheet extensions that were loaded after the plugin.


.. autofunction:: visidata.Extensible.before

Runs *beforefunc* before the original method.  The return value of *beforefunc* is ignored.

::

    @Sheet.before
    def reload(sheet):
        vd.status('reloading...')

.. autofunction:: visidata.Extensible.after

Runs *afterfunc* after the original method.  The return value of *afterfunc* is ignored; the original's return value is preserved.

::

    @DrawablePane.after
    def execCommand2(sheet, cmd, *args, **kwargs):
        vd.usedInputs[cmd.longname] += 1

.. autofunction:: visidata.Extensible.around

Wraps the original method, passing it as the first argument (*oldfunc*) to *aroundfunc*.
*aroundfunc* can call *oldfunc* with modified arguments, return a different value, or skip calling it entirely.

::

    @VisiData.around
    def input(vd, oldfunc, prompt, **kwargs):
        if should_intercept(kwargs):
            return custom_input(vd, prompt, **kwargs)
        return oldfunc(vd, prompt, **kwargs)

.. autofunction:: visidata.Extensible.init

If a module wants to store some data on an Extensible class, it can add
a member with a call to that class' ``init()``:

::

    TableSheet.init('foo', dict)

This monkey-patches ``TableSheet.__init__`` to add the instance member
``foo`` to every TableSheet on construction, and to initialize it with an
empty dict. To provide an initial non-object value:

::

    TableSheet.init('bar', lambda: 42)

.. note::

    This will not work with the ``vd`` because it is instantiated very early.
    Instead, assign member variables directly on ``vd`` in the toplevel scope:

    ::

        vd.bar = []

    This member can then be used like any other member of the class.

By default, when an instance of the class is copied, a member specified with this ``init()`` is reset to a newly constructed value (by calling the constructor again).
If *copy* is ``True``, then a copy is made of the member for the new instance.

.. autofunction:: visidata.vd.global_api

All features (and plugins) should expose functions and classes to plugins in one of these ways:

1. with `@VisiData.api` (or @Sheet.api): for most methods

These can be used in an execstr as though they were global (attributes on `vd` and `sheet` are both implicitly in scope in an execstr).

Outside of an execstr, use `vd.funcname(...)` (or `sheet.funcname(...)`).

Note that classes can be annotated with `@VisiData.api` also.

2. with `addGlobals({"funcname": funcname})`: for classes and methods internal to VisiData

These can be used via direct import:

`from visidata import SomeInternalClass`
`from plugins.myplugin import HelperClass`

This is acceptable for commonly-used classes.

See :ref:`getGlobals() and addGlobals() <other-commands>`.

What to extend: ``Sheet``, ``Column``, ``VisiData``, or globals?

Look at what the function uses.
If it uses a specific column, use @Column.api with col as the first "self" argument, and if you need access to the sheet, use col.sheet.

If the function naturally takes a sheet, use @Sheet.api with first argument of sheet.

Otherwise, extend the VisiData object if neither Column nor Sheet are relevant.
vd is always available as a global regardless.

Classes and functions which don't use vd or sheet at all are candidates for the list of bare globals in __all__.

The ``vd`` singleton Copy Chapter

The VisiData class is a singleton object, containing all of VisiData's global functions and state for the current session.
This object should always be available as vd.

Calling conventions

When calling functions on ``vd`` or ``sheet`` outside of a Command *execstr*, they should be properly qualified:

::

    @Sheet.api
    def show_hello(sheet):
        vd.status(sheet.options.disp_hello)

    BaseSheet.addCommand(None, 'show-hello', 'show_hello()')

The current **Sheet** and the **VisiData** object are both in scope for :ref:`execstrs <command-context>`, so within an *execstr*, the ``sheet.`` or ``vd.`` may be omitted, as in the hello world example:

::

    BaseSheet.addCommand(None, 'show-hello', 'status(options.disp_hello)')

---

### Api Guides (`docs/api/guides.rst`)

Guides
=======

A Guide is an in-app writeup of a particular feature that gives a basic synopsis of how to use it and explains how it works together with other features.

Open the Guide Index with ``Space open-guide-index`` within VisiData or ``vd -P open-guide-index`` from the CLI.


.. note::

    Guides that have not been written yet are grayed out.
    We love to get help with documentation, please get in touch if you want to write one or have other suggestions!

Here's an outline for adding a guide, with our writing style preferences:

1. Launch **GuideIndex** and find the ``name`` of the guide.
2. Create a markdown file ``visidata/guides/{name}.md``.
3. Write the guide.


Hello Guide
------------

This same general structure and process should work for most guides.

Step 1. Launch **GuideIndex** and find the ``name`` of the guide

.. image:: ../assets/guide.png
:width: 1000

Within VisiData, Space open-guide-index to see the Table of Contents.

gv to show the name column. Choose a guide to work on, and note down its name.
For example: MacrosSheet.

::

Step 2. Create a markdown file visidata/guides/{name}.md

In this example, we would create a ``visidata/guides/MacrosSheet.md``.

At the top of the file, optionally place template front matter, using YAML syntax:

::

    ---
    sheettype: Sheet
    ---

This front matter will override the default configuration for guides.

At the moment, the main configurable option is ``sheettype``. ``sheettype`` is a string for the type of sheet that the guide involves. VisiData uses this to auto-complete command + options patterns (see "stylings of note").

Its default value is ``Sheet`` (aka TableSheet), the base class for every table sheet.

.. note::

    The vast majority of guides will not need to set the front matter or change the default sheet type. Only set it if command + option patterns are failing to auto-complete.

    Feel free to ask us if unsure which sheet type to use.



Step 3. Write the guide
~~~~~~~~~~~~~~~~~~~~~~~

Next, fill out the text for the guide:

::

    ---
    sheettype: Sheet
    ---
    # Macros
    Macros allow you to bind a command sequence to a keystroke or longname, to replay when that
    keystroke is pressed or the command is executed by longname.

    The basic usage is:
        1. {help.commands.macro-record}
        2. Execute a series of commands.
        3. `m` again to complete the recording, and prompt for the keystroke or longname to bind it to.

    # The Macros Sheet

    - {help.commands.macro-sheet}

    - `d` (`delete-row`) to mark macros for deletion.
    - {help.commands.commit_sheet}
    - `Enter` (`open-row`) to open the macro in the current row, and view the series of commands composing it.
    '''

Read the `style guide </docs/api/style>`_ for an overview of the stylings and markup.

---

### Api Index (`docs/api/index.rst`)

.. VisiData documentation master file, created by
   sphinx-quickstart on Wed Sep 23 19:53:19 2020.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

=====================================
VisiData Plugin Authors Guide (v2.0)
=====================================

VisiData is designed to be **extensible**.
Anything that can be done with Python (which is basically everything) can be exposed in a VisiData interface with a minimal amount of code.

VisiData is also designed to be **modular**.
Many of its features can exist in isolation, and can be enabled or disabled independently, without affecting other features.
Modules should degrade or fail gracefully if they depend on another module which is not loaded.

.. _hello-world:

Hello World
===========

As a simple example, this code can be placed in ``~/.visidatarc``:

::

    BaseSheet.addCommand('1', 'hello-world', 'status("Hello world!")')

This extends VisiData so that when :kbd:`1` is pressed on any sheet, the command called ``hello-world`` with the Python code ``status("Hello world!")`` is executed, and this prints the string "Hello world!" on the status bar.

----

.. toctree::
   :maxdepth: 1
   :caption: Table of contents
   :name: mastertoc

   options
   commands
   loaders
   extensible
   sheets
   columns
   data
   modify
   interface
   async
   guides
   style
   plugins

----

Toplevel functions
===================

.. autofunction:: visidata.vd.view
.. autofunction:: visidata.vd.view_pandas
.. autofunction:: visidata.vd.run

---

### Api Interface (`docs/api/interface.rst`)

Drawing
========

These functions should rarely be necessary to use.
``BaseSheet.draw`` must be overridden on non-tabular sheets.

.. autofunction:: visidata.BaseSheet.refresh
.. autofunction:: visidata.vd.redraw
.. autofunction:: visidata.BaseSheet.draw

.. autoclass:: visidata.SuspendCurses

Window
========

.. autoattribute:: visidata.BaseSheet.windowWidth
.. autoattribute:: visidata.BaseSheet.windowHeight

.. data:: visidata.TableSheet.topRowIndex

Top row in the window, as an index into ``TableSheet.rows``.  Settable.

.. data:: visidata.TableSheet.leftVisibleColIndex

Leftmost column in the window (after key columns), as an index into ``TableSheet.visibleCols``.  Settable.

Cursor
======

Every Sheet has a cursor, which makes it easy to interact individual elements and slices of data.

A ``TableSheet`` has a row cursor and a column cursor, which overlap on a single cell.

.. data:: visidata.TableSheet.cursorVisibleColIndex

Column cursor, as an index into ``TableSheet.visibleCols``.  Settable.

.. data:: visidata.TableSheet.cursorRowIndex

Row cursor as an index into ``TableSheet.rows``.  Settable.

.. autofunction:: visidata.BaseSheet.checkCursor

.. autoattribute:: visidata.TableSheet.cursorCol
.. autoattribute:: visidata.TableSheet.cursorValue
.. autoattribute:: visidata.TableSheet.cursorTypedValue
.. autoattribute:: visidata.TableSheet.cursorDisplay

.. autofunction:: visidata.TableSheet.cursorDown
.. autofunction:: visidata.TableSheet.cursorRight
.. autofunction:: visidata.TableSheet.moveToNextRow
.. autofunction:: visidata.TableSheet.moveToCol
.. autofunction:: visidata.TableSheet.moveToRow

.. _input:

Input
======

.. autofunction:: visidata.vd.input
.. autofunction:: visidata.vd.choose
.. autofunction:: visidata.vd.chooseOne
.. autofunction:: visidata.vd.chooseMany
.. autofunction:: visidata.vd.confirm
.. autofunction:: visidata.vd.launchEditor

.. _status:

Status
======

.. autofunction:: visidata.vd.status
.. autofunction:: visidata.vd.warning
.. autofunction:: visidata.vd.fail
.. autofunction:: visidata.vd.exceptionCaught
.. autoclass:: visidata.ExpectedException

Examples
^^^^^^^^

::

    with SuspendCurses():
        try:
            return subprocess.call('less', '/etc/passwd')
        except Exception as e:
            vd.exceptionCaught(e)

::

    passwd = vd.input("password: ", display=False)

    # initial value is the formatted value under the cursor
    vd.status(vd.input("text to show: ", value=cursorDisplay))

.. _colors:

Colors
======

The TableSheet allows rows, columns, or individual cells to be colorized according to a Python function.

See `Color Docs <../colors>`__ for information on how to configure specific colors and attributes.

.. data:: visidata.TableSheet.colorizers

List of Colorizers for this class.  Class member.

Do not manually update this list to add colorizers on a specific sheet.
Instead use addColorizer and removeColorizer.

.. autofunction:: visidata.TableSheet.addColorizer
.. autofunction:: visidata.TableSheet.removeColorizer

.. autoclass:: visidata.DisplayWrapper

.. autoclass:: visidata.RowColorizer
.. autoclass:: visidata.ColumnColorizer
.. autoclass:: visidata.CellColorizer

- *precedence*: use color from colorizer with the greatest *precedence* value.
- *coloropt*: name of color option, or ``None``.
- *func*: ``func(sheet, col, row, value)`` is called for each cell as it is being rendered. Return True when *coloropt* should be applied to the cell.  If *coloropt* is None, ``func(...)`` can return the relevant coloropt (or None) instead.

- Attributes (underline, bold, reverse) combine, regardless of *precedence*.  Only colors will override other.

- The *value* passed to the Colorizer *func* is a ``DisplayWrapper``, which encapsulates the entirety of a displayed cell.  Use ``val.value`` to get the typed value (or None), or ``val.display`` for the displayed string.

.. note::

    Colorizers are called for headers and separators, in addition to cells.
    For a header, *row* will be None; for a separator, *col* will be None.
    So a RowColorizer needs to include a check that *row* is valid, a ColumnColorizer needs to check that *col* is valid, and a cell colorizer needs to check both.
    A simple check for non-False (see Examples) is sufficient.



Examples
^^^^^^^^

::

    class DescribeSheet(ColumnsSheet):
        colorizers = [
            # colors key 'rows' on the like key columns
            RowColorizer(7, 'color_key_col', lambda s,c,r,v: r and r in r.sheet.keyCols),
        ]

    class OptionsSheet(TableSheet):
        colorizers = [
            # colors cells
            CellColorizer(3, None, lambda s,c,r,v: v.value if r and c in s.columns[1:3] and r.name.startswith('color_') else None)
        ]

Menu System
===========

Features and plugins can add commands to the hierarchical menu system (new in v2.6).

.. autofunction:: visidata.vd.runMenu
.. autofunction:: visidata.vd.addMenu
.. autofunction:: visidata.vd.addMenuItem

.. autoclass:: visidata.Menu
.. data:: visidata.vd.menus

`vd.menus` contains the hierarchical list all Menu() objects, for exploration.

Examples
^^^^^^^^

::

    # add a single command to an existing menusubmenu with multiple commands
    vd.addMenuItem('Column', 'Add column', 'enhance', 'enhance-col')

    # add a submenu with multiple commands
    vd.addMenu(Menu('Data', Menu('Enhance',
        Menu('this column', 'enhance-col'),
        Menu('all columns', 'enhance-cols'),
    )))

    # activate menu and navigate to Data»Enhance
    vd.runMenu('Data', 'Enhance')

---

### Api Loaders (`docs/api/loaders.rst`)

Loaders
=======

.. note::

    You are welcome to submit new loaders to core VisiData, or as plugins. Please, see our `checklists for contribution <https://visidata.org/docs/contributing>`__.

Creating a new loader for a data source is simple and straightforward.

1. ``open_filetype`` boilerplate
2. ``FooSheet`` subclass with rowtype and rowdef
3. ``FooSheet`` reload or iterload
4. ``FooSheet.columns``

Hello Loader
------------

Here's a step-by-line breakdown of a basic loader, which reads in a text file as a series of lines.
This same general structure and process should work for all loaders.

Step 1. ``open_<filetype>`` boilerplate
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    @VisiData.api
    def open_readme(vd, p):
        return ReadmeSheet(p.base_stem, source=p)

This is used for filetype ``readme``, which is used for files with extension ``.readme``, or when specified manually with the ``filetype`` option like ``--filetype=readme`` or ``-f readme`` on the command line.

The ``open_<filetype>`` function usually looks exactly like this, with only the type of :ref:`Sheet <sheets>` changed.

The *p* argument is a :ref:`visidata.Path<vd-path>`.

The actual loading happens in the Sheet.  An existing :ref:`sheet type<sheets>` can be used, or a new sheet type can be created.

::

Step 2. Create a Sheet subclass
-------------------------------

::

    class ReadmeSheet(TableSheet):
        rowtype = 'lines'   # rowdef: [str]

-  TableSheet (and its alias ``Sheet``) is the basic tabular sheet of rows
   and columns. Most loader sheets will inherit from TableSheet, but
   some might inherit from more specialized sheets if they share
   functionality, or from ``BaseSheet`` if they are not tabular (like
   the ``Canvas``).

-  The ``rowtype`` member is only displayed on the :ref:`right-hand
   status <status>`. It should be **plural**. If not given, it is
   "``rows``". It's helpful to give the user an subconscious check of
   the kind of sheet being shown.

-  The ``rowdef`` should be given for all loaders, even though it is
   only a comment. It specifies the expected Pythonic structure of the
   rows on this sheet. This is important because nearly every other
   component of the sheet depends on this structure.

Step 3. Load data into rows, and yield them one-by-one
------------------------------------------------------

``reload()`` is called when the Sheet is first pushed, and thereafter by the user with :kbd:`Ctrl+R`.
The default ``TableSheet.reload()`` iterates through the rows returned by ``TableSheet.iterload()``, and takes care of a few common tasks (like running async and resetting the ``rows`` member to a new list).

Each loader for a tabular sheet should overload ``iterload()``, which uses the Sheet ``source`` to populate and then yield each row one-by-one.

::

    class ReadmeSheet(TableSheet):
        rowtype = 'lines'   # rowdef: [str]

        def iterload(self):
            for line in self.source:
                yield [line]

.. warning::
   ``str`` by itself is not a valid rowdef.

   Each row must have a unique *rowid*, which by default is the Python ``id()`` of the row.
   Because Python interns common strings, strings with the same value will have the same *id*.
   This would break a lot of features, like row selection for instance.

   Also, as an immutable type, it would be annoying to not be able to modify it.

   So it needs to be wrapped in a Python ``list``, which is guaranteed to be unique, and also mutable.


-  ``sheet.source`` is the :ref:`visidata.Path<vd-path>` given as the *source* kwarg to ``ReadmeSheet()`` in ``open_readme``. 

  .. note::

     Any *kwarg* passed to a Sheet constructor will be stored on the sheet in an attribute of the same name.

  .. note::

    `visidata.Path <vd-path>` objects are Path-like but have some additional features, like being iterable (yielding their contents one line at a time).

   While there is a ``visidata.Path.read_text()`` function, do **not** use ``for line in p.read_text().splitlines()`` in a loader, as that will read the entire file before returning the first line.
   A loader must be able to handle arbitrary amounts of data (including data too large to fit in memory), so this will not work.

   ``Path.__iter__`` is optimized to read the file a small amount at a time, so ``for line in path`` is workable for a textual line-based file format.

- If the loader requires a third-party library, import it inside ``iterload()`` or ``reload()`` (or ``open_<filetype>`` if necessary).
  Do not import at the toplevel, or ``vd`` will fail to start when the library is not installed.
    - preferably, import it using ``modname = importExternal(modname, pythonPackageName``. If the user does not have the package installed, it will output instructions to ``pip3 install pythonPackageName``.

.. autofunction:: visidata.vd.importExternal

By default, a Sheet has one Column which just displays a string representation of the row.
So the above example is a good starting point for any loader; just get the rows however they come most easily from the source, and launch ``vd`` with a sample dataset in that format.
Then use :kbd:`Ctrl+Y` to explore the resulting Python object, to find what attributes to show on the sheet.

reload()
~~~~~~~~

For more control over the whole loading process, ``BaseSheet.reload()`` can be overridden instead of ``iterload()``:

::

        @asyncthread
        def reload(self):
            self.rows = []
            for line in self.source:
                self.addRow([line])

-  ``@asyncthread`` launches the decorated function in its own thread.  See :ref:`Threads`
-  ``sheet.rows`` must always be reset to a new list.  **Never** call ``sheet.rows.clear()``.
-  Always add rows using :ref:`addRow()<sheets>`.

Supporting asynchronous loaders
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Loading a large dataset in the main thread will cause the interface to freeze.
However, the basic TableSheet ``reload`` and ``iterload`` structure results in an :ref:`asynchronous ` loader by default.
Since rows are yielded **one at a time**, they become available as they are loaded, and ``reload`` itself is decorated with an ``@asyncthread``, which causes it to be launched in a new thread.

- All row iterators should be wrapped with :ref:`Progress`.
  This updates the progress percentage as it passes each element through.

- Do not depend on the order of ``rows`` after they are added; e.g. do not reference ``rows[-1]``.  The order of rows may change during an asynchronous loader.

- Catch any ``Exception`` that might be raised while handling a specific row, and add them as the row instead.  Uncaught exceptions will cause the loader thread to abort.

- Do not use a bare ``except:`` clause or the loader thread will not be cancelable with :kbd:`Ctrl+C`.

Progress and Exception example
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

        class FooSheet(Sheet):
            ...
            def iterload(self):
                for bar in Progress(foolib.iterfoo(self.source.open_text())):
                    try:
                        r = foolib.parse(bar)
                    except Exception as e:
                        r = e
                    yield r

Testing for Loader Performance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Test the loader with a very large dataset to make sure that:

-  the first rows appear immediately;
-  the progress percentage is being updated;
-  the loader can be cancelled (with :kbd:`Ctrl+C`).

.. _enumerate-columns:

Step 4. Enumerate the Columns
------------------------------

Each sheet has a ``columns`` attribute with a unique list of ``Column`` objects. Each ``Column`` provides a
different view into the row.

::

        class FooSheet(Sheet):
            rowtype = 'foobits'  # rowdef: foolib.Bar object

            columns = [
                ColumnAttr('name'),  # foolib.Bar.name
                Column('bar', getter=lambda col,row: row.inside[2],
                              setter=lambda col,row,val: row.set_bar(val)),
                Column('baz', type=int, getter=lambda col,row: row.inside[1]*100)
            ]

In general, set ``columns`` as a class member containing a list of
static columns. If the columns aren't known until data is loaded,
reload/iterload can add new columns using :ref:`addColumn() `.

If the rowdef is a ``list``, and the columns are dynamic, :ref:`SequenceSheet.reload() <other-sheets>` could handle the Column creation.

::

    class FooSheet(SequenceSheet):
        rowtype = 'foobits'  # rowdef: a list, which is a sequence of values

        def iterload(self):
            with foolib.iterfoo(self.source.open_text() as f:
                r = foolib.parse(bar)
                yield r


Column attributes
~~~~~~~~~~~~~~~~~

Columns have several attributes; all except *name* are **optional** arguments to the constructor:

-  *name*: should be a valid Python identifier and unique among the column names on the sheet.  (Otherwise the column cannot be used in an expression.)
-  *type*: can be ``str``, ``int``, ``float``, ``date``, ``currency``, or a custom type.  By default it is ``anytype``, which passes the original value through unmodified.
-  *width*: the initial width for the column. ``0`` means hidden; ``None`` (default) means calculate on first draw.

Column getters can be any function, but many loaders are satisfied with a static list of ``ItemColumn`` (for values in dict and list rowdefs) and/or ``AttrColumn`` (for a members or attributes directly on the row object).
This is dependent on the loader function; some loaders may prefer to do less parsing to load faster, and then the Columns will need to be correspondingly more complicated.

See the :ref:`Columns section <columns>` for a complete API.

Passthrough options
~~~~~~~~~~~~~~~~~~~

Loaders which use a Python library (internal or external) are encouraged to pass its kwargs using ``**options.getall("foo_")`` interface.
For modules like ``csv`` which expose them as kwargs to some function or constructor, this is very easy:

::

        rdr = csv.reader(fp, **csvoptions())

Full Example
~~~~~~~~~~~~

This is a completely functional loader for the ``sas7bdat`` (SAS dataset file) format, thanks to Jared Hobbs' `sas7bdat package <https://bitbucket.org/jaredhobbs/sas7bdat>`__.

::

    from visidata import Sheet, ItemColumn, Progress

    @VisiData.api
    def open_sas7bdat(vd, p):
        return SasSheet(p.base_stem, source=p)

    class SasSheet(Sheet):
        def iterload(self):
            import sas7bdat
            SASTypes = { 'string': str, 'number': float, }

            self.dat = sas7bdat.SAS7BDAT(str(self.source),
                                         skip_header=True,
                                         log_level=logging.CRITICAL)

            self.columns = []
            for col in self.dat.columns:
                self.addColumn(ItemColumn(col.name.decode('utf-8'),
                                         col.col_id,
                                         type=SASTypes.get(col.type, anytype)))

            with self.dat as fp:
                yield from Progress(fp, total=self.dat.properties.row_count)


Guessing Filetypes
==================

When loading a file, VisiData tries to infer its filetype by peeking at the initial lines of the file and guessing from its structure.

``vd.guess_<filetype>(path)`` contains this logic for checking whether a file might be ``<filetype``.

If those structures are not present, the function should return nothing. If they are, the function should return a dictionary with:

- ``filetype`` being the filetype they detect (corresponding to the ``vd.open_<filetype>``)
- ``_likelihood`` (optional) being a number from 0-10, 10 being most likely and 0 meaning a last ditch effort if nothing else will take it
- any other key/values will be set as options on the *Sheet* the ``open_<filetype>`` function returns

`Examples of guess_filetype functions <https://github.com/saulpw/visidata/commit/4743f92bb855cf931d896e65845c549ce6027e2f>`_


::

        @VisiData.api
        def guess_foo(vd, p):
            import foobar
            if p.open_text().read(8).startswith("#Foo"):
                enc = foobar.encoding(p)
                return dict(filetype='foo', foo_encoding=enc)


Savers
=======

A full-duplex loader requires a **saver**.
The saver iterates over all ``rows`` and ``visibleCols``, calling ``getValue``, ``getDisplayValue`` or ``getTypedValue`` as the saving format allows, and saves the results in its format to the given *path*.
Savers should be decorated with ``@VisiData.api`` in order to make them available through the ``vd`` object's scope.

.. autofunction:: visidata.vd.save_txt

-  *p* is a :ref:`visidata.Path <vd-path>` object referencing the file being written to.
-  *sheets* is a list of 1 or more sheets to be saved.

The saver should preserve the column names and translate their types into ``foolib`` semantics, but other attributes on the Columns are generally not saved.

Savers which can handle typed values should use ``Column.getTypedValue``, and displayable savers (like html, markdown, csv) should use ``Column.getDisplayValue`` (which takes into account the column's *fmtstr*).

Example
^^^^^^^

With this example, saving as filetype ``table`` will call the `tabulate library <https://github.com/astanin/python-tabulate>`__ to save the data in any number of text formats, specified by the ``tbl_tablefmt`` option.
(Several built-in savers use ``tabulate`` also, but those savers work a little differently, as each tablefmt is available as a direct save filetype.)

::

    vd.option('tbl_tablefmt', 'simple', 'file format to save with "table" filetype')

    def get_rows(sheet, cols):
        for row in Progress(sheet.rows):
            yield [ col.getDisplayValue(row) for col in cols ]

    @VisiData.api
    def save_table(path, *sheets):
        import tabulate

        with path.open_text(mode='w') as fp:
            for vs in sheets:
                fp.write(tabulate.tabulate(
                    get_rows(vs, vs.visibleCols),
                    headers=[ col.name for col in vs.visibleCols ],
                    **options.getall('tbl_')))

.. _vd-path:

visidata.Path
=============

``visidata.Path`` is a wrapper around Python's builtin ``pathlib.Path`` that can also handle non-filesystem files (URLs, stdin, files within archives).

The ``given`` attribute is new to ``visidata.Path``.
Other functions listed here are wrappers around the equivalent ``pathlib.Path`` functions, with specialized functionality as needed for non-filesystem files.
All other accesses are forwarded to the inner ``pathlib.Path`` object, but will probably not work for non-filesystem files.

.. autoattribute:: visidata.Path.given

.. autofunction:: visidata.Path.exists
.. autofunction:: visidata.Path.open
.. autofunction:: visidata.Path.open_text
.. autofunction:: visidata.Path.read_text
.. autofunction:: visidata.Path.open_bytes
.. autofunction:: visidata.Path.read_bytes
.. autofunction:: visidata.Path.stat
.. autofunction:: visidata.Path.with_name

URL Scheme Loaders
---------------------------------------

When VisiData tries to open a URL with schemetype of ``foo`` (i.e.
starting with ``foo://``), it calls ``openurl_foo(urlpath, filetype)``.
``urlpath`` is a ``UrlPath`` object, with attributes for each of the
elements of the parsed URL.

``openurl_foo`` should return a Sheet or call ``error()``. If the URL
indicates a particular type of Sheet (like ``magnet://``), then it
should construct that Sheet itself. If the URL is just a means to get to
another filetype, then it can call ``openSource`` with a Path-like
object that knows how to fetch the URL:

::

        def openurl_foo(p, filetype=None):
            return openSource(FooPath(p.url), filetype=filetype)

---

### Api Modify (`docs/api/modify.rst`)

===============
Modifying Data
===============


When adding rows, deleting rows, or modifying cell values, **always use the functions documented in this API**.
Use ``addRow()`` to add a row, ``deleteBy()`` or ``deleteSelected()`` to delete a row, and the ``setValue()`` family of functions to modify values.
Even though ``setValue`` will call ``putValue`` and that may be defined by the plugin itself, do not modify rows manually.
This also means that rows should not be added or removed from ``sheet.rows`` directly.

Always use these functions, as they do some important bookkeeping for features like deferred modifications.

.. autofunction:: visidata.TableSheet.newRow
.. autofunction:: visidata.TableSheet.addNewRows
.. autofunction:: visidata.TableSheet.editCell
.. autofunction:: visidata.TableSheet.deleteBy

Deferred Modification
======================

On sheets where ``putValue`` changes the source directly, like **SQLite** and **DirSheet**, it would be undesirable for modifications to happen immediately.
For these cases, the modifications are cached, shown visually on the sheet in a different color, and deferred until they are committed with an explicit ``commit-sheet`` (bound to :kbd:`z Ctrl+S`).

Sheet types can add this deferring behavior by setting their ``defer`` attribute to ``True`` (either on the class type or on the individual sheet instance), but they may have to implement their own ``Sheet.putChanges``.

.. autofunction:: visidata.Column.getSourceValue

.. autofunction:: visidata.TableSheet.commit
.. autofunction:: visidata.TableSheet.getDeferredChanges
.. autofunction:: visidata.TableSheet.putChanges

Undo
-------

When a function adds, removes, or modifies data--including metadata--as part of a user action, it needs to ensure that operation is **undoable**.
For many operations, this is already handled internally by existing functions.
For instance, if the user executes a command which modifies cell values via ``Column.setValues``, the ``undo`` command will undo those modifications automatically.

If a new operation changes the data or metadata and undo is not already handled by existing functionality, then the new operation needs to call ``addUndo()`` to add an *undo function* to the currently executing command.

For performance reasons, undo functions should generally be as large as possible.
For instance, ``Column.setValues`` adds a single undo function to undo all changes in one go.
By contrast, ``Column.setValue`` does not add an undo function at all.
Therefore, commands should always use ``Column.setValues``, unless for some special reason the modification should not be undoable.
Undo functions, however, should use ``Column.setValue``, so it doesn't add an undo function for the undo itself.
(Redo is handled by replaying the previous command, not by undoing the undo).

.. autofunction:: visidata.vd.addUndo
.. autofunction:: visidata.vd.addUndoSetValues


Example
--------

.. note::

    ``Sheet.editCell`` requires row and column *indexes* for its parameters.
    For example, ``cursorVisibleColIndex`` and ``cursorRowIndex`` are sheet attributes that can be passed to ``Sheet.editCell``.

::

    Sheet.addCommand('e', 'edit-cell',
                     'cursorCol.setValues([cursorRow], editCell(cursorVisibleColIndex))',
                     'edit contents of current cell')


::

    # equivalent to Sheet.deleteSelected but slower
    sheet.deleteBy(sheet.isSelected)

::

    sheet.addRow(row)

---

### Api Modules (`docs/api/modules.rst`)

# Module abilities

Internal modules and external plugins can:

- define new commands
- add new options
- set existing options
- maintain state on vd singleton or sheets or columns
- add arbitrary Python code before/after most functions
- replace functions
- add new sheet types
- new file formats and urls
- have lazy dependencies (with vd.importExternal)

# Module imports

By default, VisiData loads all installed plugins and all of its own internal modules.
Some modules (esp loaders) may have dependencies in PyPI that aren't installed.

Modules must lazily import any external dependencies only when the feature or loader is attempted to be used.  i.e. modules must not import external dependencies at file scope.  For loaders this should happen in reload/iterload() instead of vd.open_foo, if possible.

The import should be with vd.importExternal so that VisiData can give a command to install the library via pip.

Some modules are not compatible with each other; for instance, the traditional visidata sqlite loader which loads all data into memory, and the vdsql sqlite loader which uses Ibis to compose SQL queries that get executed against the sqlite database itself.

The last module to be imported will have its functions registered for the filetype or command.  This should work fine for most cases (latter modules are generally more advanced), but to use the former module, this module must not be loaded in the first place.

---

### Api Options (`docs/api/options.rst`)

Options
========

Adding to the :ref:`hello world<hello-world>` example from the intro, the displayed text could be made configurable with an option:

::

    vd.option('disp_hello', 'Hello world!', 'string to display for hello-world command')

    BaseSheet.addCommand('1', 'hello-world', 'status(options.disp_hello)')


Now the user can set the option to modify which text is displayed during their session when they press :kbd:`1`.
For example, on the command line (note that the underscores can be converted to hyphens):

::

    vd --disp-hello="¡Hola mundo!"

The user can override it persistently for every session by adding a line to their ``.visidatarc``:

::

    options.disp_hello = 'Bonjour monde!'


Options Context
~~~~~~~~~~~~~~~

Options can have different values depending on the context in which they're used.
For instance, one TSV sheet might need its ``delimiter`` set to "``|``", while another TSV sheet in the same session might need to use the default ("``\t``", or TAB) instead.

Options can be overridden globally, or for all sheets of a specific type, or only for one specific sheet.

The options context should be referenced directly when setting:

    - ``sheet.options`` to *set* an option on a specific sheet instance (**sheet override**).
    - ``<SheetType>.options`` to *set* a option default for a particular type of Sheet (**class override**).
    - ``vd.options`` (or plain ``options``) to *set* an option globally for all sheets, except for sheets that have a sheet override (**global override**).

.. note::

    Class overrides prior to v2.10
    ------------------------------

    In VisiData v2.10 and earlier, class overrides had to be specified using ``<SheetType>.class_options``.
    In subsequent versions, ``.options`` can be used for getting and setting options at all levels.

Use ``sheet.options`` to *get* an option within the context of a specific sheet.
This is strongly preferred, so the user can override the option setting on a sheet-specific basis.
However, some options and situations are truly sheet-agnostic, and so ``vd.options`` (or plain ``options``) will *get* an option using the context of the **top sheet**.

When getting an option value, VisiData will look for a sheet override first, then class overrides next (from most specific subclass all the way up to BaseSheet), then a global override, before returning the default value from the option definition itself.

In general, plugins should use ``FooSheet.options`` to override values for the plugin-specific sheet type.

.. note::

    Performance
    ------------------

    Because it allows so much flexibility, getting option values is a comparatively slow operation.  For tight loops, save the option into a local variable in an outer block.  An option value can only change between commands anyway.

Options API
~~~~~~~~~~~~~~~

.. autofunction:: visidata.vd.options.__getattr__

``x = sheet.options.hello_world`` is the preferred style for getting a single option value.

.. autofunction:: visidata.vd.options.__setattr__

``sheet.options.hello_world = "Привет мир"`` is the preferred style for setting a single option value.

.. autofunction:: visidata.vd.options.get
.. autofunction:: visidata.vd.options.set
.. autofunction:: visidata.vd.options.unset
.. versionadded:: 2.1
.. autofunction:: visidata.vd.options.getall

The dict returned by ``options.getall('foo_')`` is designed to be used as kwargs to other loaders, so that their options can be passed through VisiData transparently.
For example, to pass all csv options through to the builtin Python ``csv`` module:

::

    csv.reader(fp, **sheet.options.getall('csv_'))

.. autofunction:: visidata.vd.option

.. note::

    If the option affects loading, transforming, or saving, then set *replay* to True.

Rules for naming options
^^^^^^^^^^^^^^^^^^^^^^^^

- Options defined within a plugin should all start with the same short module abbreviation, like "``mod_``".
- Except for theme option names, which should start with "``disp_``" for a displayed string and "``color_``" for a color option (see :ref:`Colors<colors>`).
- Use common abbreviations instead of full words.
- Use "``_``" (underscore) to separate words.
- Keep the option name length under 20 characters.  Maximum of 3 words (2 separators).

.. note::

    * Consider whether some subset of options can be passed straight through to the underlying Python library via kwargs (maximum power with minimal effort).

Option type
^^^^^^^^^^^^

When setting an option, strings and other types will be coerced to the type of the *default*.  An ``Exception`` is raised if conversion fails.  A *default* of ``None`` allows any type.

Examples
~~~~~~~~~

::

    vd.option('disp_hello', 'Hello world!', 'string to display for hello-world')

    # option without regard to a current sheet (global override only)
    vd.options.disp_hello = 'こんにちは世界'

    # show the option value given the context of a specific sheet
    vd.status(sheet.options.disp_hello)

    # option set on specific sheet only
    sheet.options.color_current_row = 'bold blue'

    # option set for all DirSheets
    DirSheet.options.color_current_row = 'reverse green'

---

### Api Plugins (`docs/api/plugins.rst`)

Plugins
========


Plugin file structure
----------------------

Most features can be self-contained in their own .py file, so that the feature is enabled or disabled by ``import``-ing that .py file or not.
A plugin is usually a single .py file, installed on the same computer as visidata.
`Instructions on how to install a plugin </docs/plugins/>`__

Plugins often start as a small snippet in ``.visidatarc``, and then the code is migrated to a separate file when it gets too large to share with other people via a short code snippet.
The actual code in either case should be identical.

.. note::

    To quickly install a personal plugin, place the plugin in the ``$HOME/.visidata/plugins/`` directory, and add an ``import`` statement to ``$HOME/.visidata/plugins/__init__.py``.
    At startup, VisiData will automatically import this ``plugins`` package and all the included plugins.

To publish a plugin, create a public repo with a .py file. To package it within VisiData, `open a PR on the VisiData repo <https://github.com/saulpw/visidata/pulls>`_, placing the code in the `the experimental folder <https://github.com/saulpw/visidata/tree/develop/visidata/experimental>`_.


Complete "Hello world" plugin example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

hello.py
^^^^^^^^^^^^^^^^

::

    '''This plugin adds the ``hello-world`` command to all sheets.
    Press ``0`` to show display options.disp_hello on the status line.'''

    __author__ = 'Jo Baz <[email protected]>'

    vd.option('disp_hello', 'Hello world!', 'string to display for hello-world command')

    BaseSheet.addCommand('0', 'hello-world', 'status(options.disp_hello)')

.. note::

Notes:

- There should be a searchable docstring description of the core features.
- Optionally include author metadata.
- Options at the top, commands at the bottom.
- Avoid toplevel imports of non-stdlib Python extensions.

---

### Api Requirements (`docs/api/requirements.txt`)

recommonmark
sphinx
sphinx-markdown-tables
sphinx-argparse
git+https://github.com/saulpw/visidata.git@develop

---

### Api Sheets (`docs/api/sheets.rst`)

.. _sheets:

Sheets
====================================

A Sheet is a representation of some source, usually a :ref:`Path <vd-path>`.
A path can be opened, returning a sheet, and/or a sheet can be saved to path.

.. autofunction:: visidata.vd.openPath
.. autofunction:: visidata.vd.openSource

.. autofunction:: visidata.vd.saveSheets

Sheets API
-----------

.. autoclass:: visidata.BaseSheet

.. autoattribute:: visidata.BaseSheet.name

.. autofunction:: visidata.BaseSheet.__len__
.. function:: visidata.BaseSheet.__copy__(self)

    Return shallow copy of sheet.

.. autofunction:: visidata.BaseSheet.reload

.. _tablesheet:

TableSheet
^^^^^^^^^^^

Sheets are generally specialized for their rowtype, and TableSheet, used for sheets with rows and columns, is the most common base class.
(So common, that it was originally just called ``Sheet``.  For clarity, ``TableSheet`` is preferred, but ``Sheet`` is a valid alias that will never be deprecated.)

.. autoclass:: visidata.TableSheet

.. data:: visidata.TableSheet.rows

List of row objects on this sheet.

.. data:: visidata.TableSheet.columns

List of all Column objects on this sheet (including hidden columns).


.. autoattribute:: visidata.TableSheet.nRows
.. autoattribute:: visidata.TableSheet.nCols

.. autoattribute:: visidata.TableSheet.visibleCols
.. autoattribute:: visidata.TableSheet.nVisibleCols

.. autofunction:: visidata.TableSheet.addRow
.. autofunction:: visidata.TableSheet.addColumn

.. autofunction:: visidata.TableSheet.iterload

.. autofunction:: visidata.TableSheet.rowid
.. autofunction:: visidata.TableSheet.column

.. autofunction:: visidata.TableSheet.openRow
.. autofunction:: visidata.TableSheet.openCell
.. autofunction:: visidata.TableSheet.gatherBy

.. _other-sheets:

Other Common Sheet Types
^^^^^^^^^^^^^^^^^^^^^^^^^^

These can be used as base classes, or instantiated as they are, or :ref:`used as templates <enumerate-columns>` for rows with similar structure,

.. autoclass:: visidata.IndexSheet
.. autoclass:: visidata.TextSheet
.. autoclass:: visidata.SequenceSheet
.. autoclass:: visidata.PyobjSheet

The Sheet Stack
-----------------

.. data:: visidata.vd.sheets

The "sheet stack", a list of active sheets (available as the **Sheets Sheet** via :kbd:`Shift+S`).
The top sheet, or the displayed sheet, is the first item in the list (``vd.sheets[0]``).

.. data:: visidata.vd.sheet

The top sheet on the sheet stack.

.. autofunction:: visidata.vd.push
.. autofunction:: visidata.vd.replace
.. autofunction:: visidata.vd.remove
.. autofunction:: visidata.vd.quit

.. autofunction:: visidata.vd.getSheet

---

### Api Style (`docs/api/style.rst`)

Guide Style Guide
=================

General Notes
-------------

- Review first sentences `with this in mind <https://jamesg.blog/2023/12/03/first-sentences/>`_.
- In general, do not use second person (you/your) outside of tutorials.

Referenced from `Simplified Technical English <https://en.wikipedia.org/wiki/Simplified_Technical_English>`_:

- Make instructions as clear and specific as possible.
- Use the approved forms of the verb:
    - The infinitive
    - The imperative
    - The simple present tense
    - The simple past tense
    - The past participle (only as an adjective)
    - The future tense
- Do not use helping verbs to make `complex verb structures <https://www.utsa.edu/twc/documents/Complex%20Verb%20Phrases.pdf>`_.
- Use the "-ing" form of a verb only as a technical name or as a modifier in a technical name.
- Do not use passive voice in procedures.
- Use the active voice as much as possible in descriptive texts.
- Prefer short sentences and simpler word choices.
- Do not omit parts of the sentence (e.g. verb, subject, article) to make your text shorter.
- Use vertical lists for complex text.
- Write one instruction per sentence.
- Write only one topic per paragraph.

Color/Markup Standards
-----------------------

- VisiData documentation uses `basic markdown <https://www.markdownguide.org/basic-syntax/>`_ like # Headings, \*\*bold\*\*, \*italics\*, \`code snippets\`, and \_underscore\_.

- Internally, VisiData has its `own display attribute syntax </docs/colors#attrs>`_. Some example usage:
    - ``[:onclick ]<text>[/]`` formats ``<text>`` into a clickable url that will open in ``$BROWSER``.
    - ``[:red on black]<sentence>[/]`` changes the color of ``<sentence>`` to be red text on black background.
      Any VisiData color option name can be used after ``:``, like ``[:warning]``, ``[:error]``, ``[:menu]``.
      Preference is for using ``[:option_name]`` instead of hard-coded colors when possible.
    - e.g. Use ``[:keys]`` for keystrokes and longnames, and ``[:code]`` for Python and other actual code.

- List relevant commands with the general pattern: ``- [:keys]<keystroke>[/] ([:code]<longname>[/]) to <command helpstring>``.
    - Use ``{help.commands.<longname>}`` to get the **GuideSheet** to replace it with the properly formatted string above. You can write it out manually if that would result in more detail or clarity, but it's much more preferred to adjust the docstring so that this pattern would work. 

- List relevant options with ``[:onclick options-sheet <option name>][:code]option name>[/] to <option helpstring> (default: <option default value>)``.
  - Similarly, use ``{help.options.<option-name>}`` to expand into the above, and prefer to modify the helpstring instead of writing it out manually.

---

--- METRICS ---
- Files Extracted: 57
- Estimated Token Budget: ~60799 tokens
- Canonical Reference: https://codewiki.google/github.com/saulpw/visidata