# Repository: johnkerl/miller # Stars: 9839 ## CLAUDE.md # Claude Development Guide for Miller ## Project Overview Miller is a command-line data processing tool for working with CSV, TSV, JSON, and other data formats. It's written in Go (v1.18+) and provides SQL-like operations on data. ## Initial Setup ### Setting Up staticcheck The `make staticcheck` target requires the staticcheck tool. To set it up: ```bash go install honnef.co/go/tools/cmd/staticcheck@latest ``` This installs staticcheck to `~/go/bin/` (the default Go binaries directory). For `make staticcheck` to work, you need `~/go/bin` in your `PATH`. **Add to your shell profile** (`.bashrc`, `.zshrc`, or equivalent): ```bash export PATH="$PATH:$HOME/go/bin" ``` **Verify the installation:** ```bash staticcheck -version ``` If this works, you can use `make staticcheck` without any further setup. ## Build & Test ### Building ```bash make build # Build the mlr executable make quiet # Build silently (no output messages) ``` ### Testing ```bash make check # Run all tests (unit + regression) make unit-test # Run unit tests only make regression-test # Run regression tests only make bench # Run benchmarks ``` ### Code Quality ```bash make fmt # Format code with go fmt make staticcheck # Run static analysis (see Initial Setup section above) ``` ### Full Developer Workflow ```bash make dev # Format, build, test, generate docs (comprehensive check before pushing) ``` ## Project Structure - `cmd/mlr` - Main Miller executable entry point - `pkg/` - Core library code organized by functionality - `pkg/lib/` - Utility libraries - `pkg/scan/` - Input scanning - `pkg/mlrval/` - Miller value types - `pkg/bifs/` - Built-in functions - `pkg/input/` - Input format handlers - `regression_test.go` - Regression test suite - `docs/` - Documentation (Markdown with live code samples) - `man/` - Man page generation ## Development Conventions ### Code Style - Follow Go conventions and `go fmt` output - Use meaningful variable and function names - Keep functions focused and testable ### Testing - Add unit tests for new functionality in `pkg/*/` directories - Use `go test` for unit tests - Run `make regression-test` for integration testing - The `mlr regtest` command provides more control for interactive debugging ### Documentation - Update relevant `.md.in` files in `docs/src/` when adding features - These are processed into live documentation with actual code examples - Run `make dev` to rebuild documentation ### Git Workflow - Create descriptive commit messages - Reference issue numbers when fixing bugs or implementing features - Test locally with `make check` before committing - For major changes, run `make dev` to ensure docs and tests pass ## Key Dependencies Miller uses the Go standard library. Check `go.mod` for specific versions. ## Common Tasks ### Adding a New Built-in Function 1. Implement in appropriate package (likely `pkg/bifs/`) 2. Add unit tests alongside 3. Update documentation in `docs/src/` 4. Run `make check` to verify ### Fixing a Bug 1. Create a minimal test case (unit test or regression test) 2. Fix the code 3. Verify with `make check` 4. Commit with reference to issue number ### Performance Optimization 1. Run `make bench` to establish baseline 2. Make changes 3. Run benchmarks again to measure improvement 4. Consider adding permanent benchmark in test files ## Documentation Building Documentation is built from `.md.in` template files that contain live code samples executed via Miller itself. When you make changes that affect command output or behavior, you may need to update these templates and rebuild docs with `make -C docs/src forcebuild`. ## Before Pushing Always run: ```bash make dev ``` This ensures code formatting, builds successfully, passes all tests, and documentation is up to date. ## Additional Resources - [Full documentation](https://miller.readthedocs.io/) - [Contributing guidelines](https://miller.readthedocs.io/en/latest/contributing/) - [Issue labeling notes](https://github.com/johnkerl/miller/wiki/Issue-labeling) ## README.md # What is Miller? **Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.** # What can Miller do for me? With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically, pretty-print, and more. ![cover-art](./docs/src/coverart/cover-combined.png) * Miller operates on **key-value-pair data** while the familiar Unix tools operate on integer-indexed fields: if the natural data structure for the latter is the array, then Miller's natural data structure is the insertion-ordered hash map. * Miller handles a **variety of data formats**, including but not limited to the familiar **CSV**, **TSV**, and **JSON**/**JSON Lines**. (Miller can handle **positionally-indexed data** too!) In the above image you can see how Miller embraces the common themes of key-value-pair data in a variety of data formats. # Getting started [![deepwiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/johnkerl/miller) * [Miller in 10 minutes](https://miller.readthedocs.io/en/latest/10min) * [A Guide To Command-Line Data Manipulation](https://www.smashingmagazine.com/2022/12/guide-command-line-data-manipulation-cli-miller) * [A quick tutorial on Miller](https://www.ict4g.net/adolfo/notes/data-analysis/miller-quick-tutorial.html) * [Miller Exercises](https://github.com/GuilloteauQ/miller-exercises) * [Tools to manipulate CSV files from the Command Line](https://www.ict4g.net/adolfo/notes/data-analysis/tools-to-manipulate-csv.html) * [www.togaware.com/linux/survivor/CSV_Files.html](https://www.togaware.com/linux/survivor/CSV_Files.html) * [MLR for CSV manipulation](https://guillim.github.io/terminal/2018/06/19/MLR-for-CSV-manipulation.html) * [Linux Magazine: Process structured text files with Miller](https://www.linux-magazine.com/Issues/2016/187/Miller) * [Miller: Command Line CSV File Processing](https://onepointzero.app/posts/miller-command-line-csv-file-processing/) * [Miller - A Swiss Army Chainsaw for CSV Data, Data Science and Data Munging](https://fuzzyblog.io/blog/data_science/2022/05/13/miller-a-swiss-army-chainsaw-for-csv-data-data-science-and-data-munging.html) * [Pandas Killer: mlr, the Scientist](https://xvzftube.xyz/posts/pandas_killers/#mlr%3A-the-scientist) # More documentation links * [**Full documentation**](https://miller.readthedocs.io/) * [Miller's license is two-clause BSD](https://github.com/johnkerl/miller/blob/main/LICENSE.txt) * [Notes about issue-labeling in the Github repo](https://github.com/johnkerl/miller/wiki/Issue-labeling) * [Active issues](https://github.com/johnkerl/miller/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc) # Installing There's a good chance you can get Miller pre-built for your system: [![Ubuntu](https://img.shields.io/badge/distros-ubuntu-db4923.svg)](https://launchpad.net/ubuntu/+source/miller) [![Ubuntu 16.04 LTS](https://img.shields.io/badge/distros-ubuntu1604lts-db4923.svg)](https://launchpad.net/ubuntu/xenial/+package/miller) [![Fedora](https://img.shields.io/badge/distros-fedora-173b70.svg)](https://packages.fedoraproject.org/pkgs/miller/miller/) [![Debian](https://img.shields.io/badge/distros-debian-c70036.svg)](https://packages.debian.org/stable/miller) [![Gentoo](https://img.shields.io/badge/distros-gentoo-4e4371.svg)](https://packages.gentoo.org/packages/sys-apps/miller) [![Pro-Linux](https://img.shields.io/badge/distros-prolinux-3a679d.svg)](http://www.pro-linux.de/cgi-bin/DBApp/check.cgi?ShowApp..20427.100) [![Arch Linux](https://img.shields.io/badge/distros-archlinux-1792d0.svg)](https://aur.archlinux.org/packages/miller-git) [![NetBSD](https://img.shields.io/badge/distros-netbsd-f26711.svg)](http://pkgsrc.se/textproc/miller) [![FreeBSD](https://img.shields.io/badge/distros-freebsd-8c0707.svg)](https://www.freshports.org/textproc/miller/) [![Anaconda](https://img.shields.io/badge/distros-anaconda-63ad41.svg)](https://anaconda.org/conda-forge/miller/) [![Snap](https://img.shields.io/badge/distros-snap-d85f33.svg)](https://snapcraft.io/miller) [![Homebrew/MacOSX](https://img.shields.io/badge/distros-homebrew-ba832b.svg)](https://formulae.brew.sh/formula/miller) [![MacPorts/MacOSX](https://img.shields.io/badge/distros-macports-1376ec.svg)](https://www.macports.org/ports.php?by=name&substr=miller) [![Chocolatey](https://img.shields.io/badge/distros-chocolatey-red.svg)](https://chocolatey.org/packages/miller) [![WinGet](https://img.shields.io/badge/distros-winget-392f55.svg)](https://github.com/microsoft/winget-pkgs/tree/master/manifests/m/Miller/Miller) |OS|Installation command| |---|---| |Linux|`yum install miller`
`apt-get install miller`
`snap install miller`| |Mac|`brew install miller`
`port install miller`| |Windows|`choco install miller`
`winget install Miller.Miller`
`scoop install main/miller`| See also [README-versions.md](./README-versions.md) for a full list of package versions. Note that long-term-support (LtS) releases will likely be on older versions. See also [building from source](https://miller.readthedocs.io/en/latest/build.html). # Community [![GitHub stars](https://img.shields.io/github/stars/johnkerl/miller.svg?label=GitHub%20stars)](https://github.com/johnkerl/miller/stargazers) [![Homebrew downloads](https://badges.weareopensource.me:/homebrew/installs/dy/miller?label=Homebrew%20downloads)](https://formulae.brew.sh/formula/miller) [![Conda downloads](https://anaconda.org/conda-forge/miller/badges/downloads.svg?label=Conda%20downloads)](https://anaconda.org/conda-forge/miller) [![All Contributors](https://img.shields.io/badge/all_contributors-41-orange.svg?style=flat-square)](#contributors-) * Discussion forum: https://github.com/johnkerl/miller/discussions * Feature requests / bug reports: https://github.com/johnkerl/miller/issues * How to contribute: [https://miller.readthedocs.io/en/latest/contributing/](https://miller.readthedocs.io/en/latest/contributing/) # Build status [![Multi-platform build status](https://github.com/johnkerl/miller/actions/workflows/go.yml/badge.svg)](https://github.com/johnkerl/miller/actions/workflows/go.yml) [![CodeQL status](https://github.com/johnkerl/miller/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/johnkerl/miller/actions/workflows/codeql-analysis.yml) [![Codespell status](https://github.com/johnkerl/miller/actions/workflows/codespell.yml/badge.svg)](https://github.com/johnkerl/miller/actions/workflows/codespell.yml) [![πŸ§ͺ Snap Builds](https://github.com/johnkerl/miller/actions/workflows/test-snap-can-build.yml/badge.svg)](https://github.com/johnkerl/miller/actions/workflows/test-snap-can-build.yml) # Building from source * First: * `cd /where/you/want/to/put/the/source` * `git clone https://github.com/johnkerl/miller` * `cd miller` * With `make`: * To build: `make`. This takes just a few seconds and produces the Miller executable, which is `./mlr` (or `.\mlr.exe` on Windows). * To run tests: `make check`. * To install: `make install`. This installs the executable `/usr/local/bin/mlr` and manual page `/usr/local/share/man/man1/mlr.1` (so you can do `man mlr`). * You can do `./configure --prefix=/some/install/path` before `make install` if you want to install somewhere other than `/usr/local`. * Without `make`: * To build: `go build github.com/johnkerl/miller/v6/cmd/mlr`. * To run tests: `go test github.com/johnkerl/miller/v6/pkg/...` and `mlr regtest`. * To install: `go install github.com/johnkerl/miller/v6/cmd/mlr@latest` will install to _GOPATH_`/bin/mlr`. * See also the doc page on [building from source](https://miller.readthedocs.io/en/latest/build). * For more developer information please see [README-dev.md](./README-dev.md). # For developers * [README-dev.md](README-dev.md) * [How to contribute](https://miller.readthedocs.io/en/latest/contributing/) # License [License: BSD2](https://github.com/johnkerl/miller/blob/main/LICENSE.txt) # Features * Miller is **multi-purpose**: it's useful for **data cleaning**, **data reduction**, **statistical reporting**, **devops**, **system administration**, **log-file processing**, **format conversion**, and **database-query post-processing**. * You can use Miller to snarf and munge **log-file data**, including selecting out relevant substreams, then produce CSV format and load that into all-in-memory/data-frame utilities for further statistical and/or graphical processing. * Miller complements **data-analysis tools** such as **R**, **pandas**, etc.: you can use Miller to **clean** and **prepare** your data. While you can do **basic statistics** entirely in Miller, its streaming-data feature and single-pass algorithms enable you to **reduce very large data sets**. * Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database. You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry. * Miller also goes beyond the classic Unix tools by stepping fully into our modern, **no-SQL** world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved. * Miller is **streaming**: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output. For those operations which require deeper retention (`sort`, `tac`, `stats1`), Miller retains only as much data as needed. This means that whenever functionally possible, you can operate on files which are larger than your system’s available RAM, and you can use Miller in **tail -f** contexts. * Miller is **pipe-friendly** and interoperates with the Unix toolkit. * Miller's I/O formats include **tabular pretty-printing**, **positionally indexed** (Unix-toolkit style), CSV, TSV, JSON, JSON Lines, and others. * Miller does **conversion** between formats. * Miller's **processing is format-aware**: e.g. CSV `sort` and `tac` keep header lines first. * Miller has high-throughput **performance** on par with the Unix toolkit. * Miller is written in portable, modern Go, with **zero runtime dependencies**. You can download or compile a single binary, `scp` it to a faraway machine, and expect it to work. # What people are saying about Miller

Today I discovered Millerβ€”it's like jq but for CSV: https://t.co/pn5Ni241KM

Also, "Miller complements data-analysis tools such as R, pandas, etc.: you can use Miller to clean and prepare your data." @GreatBlueC @nfmcclure

— Adrien Trouillaud (@adrienjt) September 24, 2020

Underappreciated swiss-army command-line chainsaw.

"Miller is like awk, sed, cut, join, and sort for [...] CSV, TSV, and [...] JSON." https://t.co/TrQqSUK3KK

— Dirk Eddelbuettel (@eddelbuettel) February 28, 2017

Miller looks like a great command line tool for working with CSV data. Sed, awk, cut, join all rolled into one: http://t.co/9BBb6VCZ6Y

— Mike Loukides (@mikeloukides) August 16, 2015

Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV: http://t.co/1zPbfg6B2W - handy tool!

— Ilya Grigorik (@igrigorik) August 22, 2015

Btw, I think Miller is the best CLI tool to deal with CSV. I used to use this when I need to preprocess too big CSVs to load into R (now we have vroom, so such cases might be rare, though...)https://t.co/kUjrSSGJoT

— Hiroaki Yutani (@yutannihilat_en) April 21, 2020

Miller: a *format-aware* data munging tool By @__jo_ker__ to overcome limitations with *line-aware* workshorses like awk, sed et al https://t.co/LCyPkhYvt9

The project website is a fantastic example of good software documentation!!

— Donny Daniel (@dnnydnl) September 9, 2018

Holy holly data swiss army knife batman! How did no one suggest Miller https://t.co/JGQpmRAZLv for solving database cleaning / ETL issues to me before

Congrats to @__jo_ker__ for amazingly intuitive tool for critical data management tasks!#DataScienceandLaw #ComputationalLaw

— James Miller (@japanlawprof) June 12, 2018

🀯@__jo_ker__'s Miller easily reads, transforms, + writes all sorts of tabular data. It's standalone, fast, and built for streaming data (operating on one line at a time, so you can work on files larger than memory).

And the docs are dream. I've been reading them all morning! https://t.co/Be2pGPZK6t

— Benjamin Wolfe (he/him) (@BenjaminWolfe) September 9, 2021
## Contributors ✨ Thanks to all the fine people who help make Miller better ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

Andrea Borruso
πŸ€” 🎨
Shaun Jackman
πŸ€”
Fred Trotter
πŸ€” 🎨
komosa
πŸ€”
jungle-boogie
πŸ€”
Thomas Klausner
πŸš‡
Stephen Kitt
πŸ“¦

Leah Neukirchen
πŸ€”
Luigi Baldoni
πŸ“¦
Hiroaki Yutani
πŸ€”
Daniel M. Drucker
πŸ€”
Nikos Alexandris
πŸ€”
kundeng
πŸ“¦
Victor Sergienko
πŸ“¦

Adrian Ho
🎨
zachp
πŸ“¦
David Selassie
πŸ€”
Joel Parker Henderson
πŸ€”
Michel Ace
πŸ€”
Matus Goljer
πŸ€”
Richard Patel
πŸ“¦

Jakub Podlaha
🎨
Miodrag Milić
πŸ“¦
Derek Mahar
πŸ€”
spmundi
πŸ€”
Peter KΓΆrner
πŸ›‘οΈ
rubyFeedback
πŸ€”
rbolsius
πŸ“¦

awildturtok
πŸ€”
agguser
πŸ€”
jganong
πŸ€”
Fulvio Scapin
πŸ€”
Jordan Torbiak
πŸ€”
Andreas Weber
πŸ€”
vapniks
πŸ“¦

Zombo
πŸ“¦
Brian Fulton-Howard
πŸ“¦
ChCyrill
πŸ€”
Jauder Ho
πŸ’»
PaweΕ‚ Sacawa
πŸ›
schragge
πŸ“–
Jordi
πŸ“– πŸ€”
This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind are welcome!