toon Documentation

File: README.md Copy Chapter

Token-Oriented Object Notation (TOON)

Token-Oriented Object Notation is a compact, human-readable encoding of the JSON data model that minimizes tokens and makes structure easy for models to follow.

TOON combines YAML's indentation-based structure for nested objects with CSV-style tabular forms for uniform data. Its sweet spot is uniform objects – same fields across items, whether in an array or keyed by ID – reaching CSV-like compactness while adding explicit structure that helps LLMs parse and validate data reliably. For deeply nested or non-uniform data, JSON may be more efficient.

Think of it as a translation layer: use JSON programmatically, and encode it as TOON for LLM input – a drop-in, lossless representation of the JSON you already have.

[!TIP]
The TOON format is stable, but also an idea in progress. Nothing's set in stone – help shape where it goes by contributing to the spec or sharing feedback.

Table of Contents Copy Chapter

• Why TOON?
• Key Features
• When Not to Use TOON
• Benchmarks
• Installation & Quick Start
• CLI
• Using TOON with LLMs
• Ecosystem
• Documentation
• Media Type & File Extension
• Other Implementations
• 📋 Full Specification

Why TOON? Copy Chapter

LLM tokens cost money – and JSON spends a lot of them on structure. A weather forecast in TOON:

location:
  city: Berlin
  country: DE
  units: metric
alerts[2]: frost,wind
forecast[3]{day,temp{min,max},condition,rainChance}:
  Mon,-2,4,snow,80
  Tue,1,7,cloudy,20
  Wed,3,11,sunny,5

The same data as JSON – ~117 tokens against TOON's ~66:

{
  "location": {
    "city": "Berlin",
    "country": "DE",
    "units": "metric"
  },
  "alerts": [
    "frost",
    "wind"
  ],
  "forecast": [
    {
      "day": "Mon",
      "temp": {
        "min": -2,
        "max": 4
      },
      "condition": "snow",
      "rainChance": 80
    },
    {
      "day": "Tue",
      "temp": {
        "min": 1,
        "max": 7
      },
      "condition": "cloudy",
      "rainChance": 20
    },
    {
      "day": "Wed",
      "temp": {
        "min": 3,
        "max": 11
      },
      "condition": "sunny",
      "rainChance": 5
    }
  ]
}

Three things are happening at once in the TOON above. Two are forms – one rendering of a value, picked automatically from the data's shape – and the third is a header feature:

• alerts[2]: frost,wind is inline form: a primitive array on its header line.
• forecast[3]{day,…}: is tabular form: the field list is declared once in the header, then one row per element.
• temp{min,max} inside that header is a nested field group: the uniform nested temp objects fold into the header while rows stay flat.

The third form is keyed tabular, for objects whose values are uniform objects – config maps, feature flags, records by ID. The colon after the length ([2:]) marks it, and each row carries its own key:

"production": { "region": "eu-central-1", "replicas": 6, "debug": false },
"staging": { "region": "eu-central-1", "replicas": 2, "debug": true }

}
}
|toon
environments[2:]{region,replicas,debug}:
production: eu-central-1,6,false
staging: eu-central-1,2,true

Anything that fits none of these – mixed types, non-uniform objects – falls back to the fourth form, **list form**: one `- ` item per element, or a bare `-` for an empty object. Those four cover the shapes; the [Format Overview](https://toonformat.dev/guide/format-overview) covers the rest.

> [!TIP]
> Try it on your own data – no install required:
>
> ```bash
> cat data.json | npx @toon-format/cli --stats
> ```
>
> It prints the TOON alongside what the conversion saved – on the weather forecast above, that's:
>
> ```
> ℹ Token estimates: ~117 (JSON) → ~66 (TOON)
> ✔ Saved ~51 tokens (-43.6%)
> ```

## Key Features

- 📊 **Token-Efficient & Accurate:** Matches JSON's retrieval accuracy while using 42.6% fewer tokens – see [Benchmarks](#benchmarks).
- 🔁 **JSON Data Model:** Encodes the same objects, arrays, and primitives as JSON with deterministic, lossless round-trips.
- 🛤️ **LLM-Friendly Guardrails:** Explicit `[N]` lengths and `{fields}` field lists give models a clear schema to follow, improving parsing reliability.
- 📐 **Minimal Syntax:** Uses indentation instead of braces and minimizes quoting, giving YAML-like readability with CSV-style compactness.
- 🧺 **Tabular Forms:** Uniform arrays of objects – and objects of uniform objects – collapse into tables that declare the field list once and stream row values line by line.
- 🌐 **Multi-Language Ecosystem:** Spec-driven implementations in many languages.

## When Not to Use TOON

TOON excels with uniform arrays of objects. Reach for something else when:

- **Structures are deeply nested or non-uniform** (tabular eligibility ≈ 0%) – compact JSON often wins outright.
- **Arrays are semi-uniform** (~40–60% eligibility) – savings shrink; stay on JSON if your pipeline already speaks it.
- **Data is purely tabular** – CSV is smaller. TOON's ~5–10% overhead buys declared lengths, field lists, and delimiter scoping, which is a reliability trade, not a size one.
- **Latency dominates** – some deployments (notably local or quantized models) process compact JSON faster despite the higher token count. Measure TTFT and total time on your own setup.

[Benchmarks](#benchmarks) below quantify the token and accuracy trade-offs; latency is the one you have to measure yourself.

## Benchmarks

Two tracks, so every comparison is like-for-like:

- **Mixed-Structure Track**: Nested and semi-uniform datasets (TOON vs JSON, YAML, XML). CSV is excluded – it cannot represent these structures without lossy flattening.
- **Flat-Only Track**: Flat, fully tabular-eligible datasets, where CSV is a fair competitor.

### Retrieval Accuracy



Benchmarks test LLM comprehension across different input formats using 244 data retrieval questions on 4 models.


**Show Dataset Catalog**

#### Dataset Catalog

| Dataset | Rows | Structure | CSV Support | Eligibility |
| ------- | ---- | --------- | ----------- | ----------- |
| Uniform employee records | 100 | uniform | ✓ | 100% |
| E-commerce orders with nested structures | 50 | nested | ✗ | 33% |
| Time-series analytics data | 60 | uniform | ✓ | 100% |
| Top 100 GitHub repositories | 100 | uniform | ✓ | 100% |
| Semi-uniform event logs | 75 | semi-uniform | ✗ | 50% |
| Deeply nested configuration | 1 | deep | ✗ | 0% |
| Valid complete dataset (control) | 20 | uniform | ✓ | 100% |
| Array truncated: 3 rows removed from end | 20 | uniform | ✓ | 100% |
| Extra rows added beyond declared length | 20 | uniform | ✓ | 100% |
| Inconsistent field count (missing salary in row 10) | 20 | uniform | ✓ | 100% |
| Missing required fields (no email in multiple rows) | 20 | uniform | ✓ | 100% |
| Feature flags keyed by name | 40 | uniform | ✗ | 100% |
| Contacts with nested address and plan groups | 50 | nested | ✗ | 100% |

**Structure classes:**
- **uniform**: All objects have identical fields with primitive values
- **semi-uniform**: Mix of uniform and non-uniform structures
- **nested**: Objects with nested structures (nested objects or arrays)
- **deep**: Highly nested with minimal tabular eligibility

**CSV Support:** ✓ (supported), ✗ (not supported – would require lossy flattening)

**Eligibility:** Percentage of arrays and keyed maps that qualify for TOON's tabular forms (uniform records whose fields are primitives or uniform nested objects folded into nested field groups)



#### Efficiency Ranking (Accuracy per 1K Tokens)

Each format ranked by efficiency (accuracy percentage per 1,000 tokens):

TOON ████████████████████ 29.2 acc%/1K tok │ 72.2% ±2.8 acc │ 2,474 tokens
JSON compact ████████████████░░░░ 23.8 acc%/1K tok │ 69.0% ±2.9 acc │ 2,892 tokens
YAML ██████████████░░░░░░ 20.1 acc%/1K tok │ 70.1% ±2.9 acc │ 3,487 tokens
JSON ███████████░░░░░░░░░ 16.6 acc%/1K tok │ 71.4% ±2.8 acc │ 4,308 tokens
XML ██████████░░░░░░░░░░ 14.4 acc%/1K tok │ 70.7% ±2.9 acc │ 4,909 tokens

*Efficiency score = (Accuracy % ÷ Tokens) × 1,000. Higher is better.*

> [!TIP]
> TOON achieves **72.2%** accuracy (vs JSON's 71.4%) while using **42.6% fewer tokens**.

> [!NOTE]
> CSV is excluded from the ranking as it only supports 109 of 244 questions (flat tabular data only). While CSV is highly token-efficient for simple tabular data, it cannot represent nested structures that other formats handle.

#### Accuracy on Flat Datasets

Every format answers the same 109 flat-dataset questions per model, so CSV can be compared on equal footing here.

| Format | Accuracy | Correct/Total | Avg Tokens |
| ------ | -------- | ------------- | ---------- |
| `toon` | 63.1% ±4.5 | 275/436 | 1,994 |
| `csv` | 62.2% ±4.5 | 271/436 | 1,851 |
| `json-pretty` | 60.3% ±4.6 | 263/436 | 3,950 |
| `xml` | 60.1% ±4.6 | 262/436 | 4,516 |
| `yaml` | 59.9% ±4.6 | 261/436 | 3,270 |
| `json-compact` | 58.0% ±4.6 | 253/436 | 2,718 |

#### Per-Model Accuracy

Accuracy across 4 LLMs on 244 data retrieval questions:

claude-haiku-4-5-20251001
→ TOON █████████████░░░░░░░ 65.6% ±5.9 (160/244)
JSON █████████████░░░░░░░ 63.5% ±6.0 (155/244)
XML ████████████░░░░░░░░ 62.3% ±6.0 (152/244)
YAML ████████████░░░░░░░░ 62.3% ±6.0 (152/244)
JSON compact ████████████░░░░░░░░ 61.9% ±6.0 (151/244)
CSV ██████████░░░░░░░░░░ 49.5% ±9.2 (54/109)

gemini-3.6-flash
→ TOON ██████████████░░░░░░ 69.3% ±5.8 (169/244)
JSON ██████████████░░░░░░ 68.4% ±5.8 (167/244)
YAML ██████████████░░░░░░ 67.6% ±5.8 (165/244)
XML █████████████░░░░░░░ 65.2% ±5.9 (159/244)
JSON compact █████████████░░░░░░░ 63.5% ±6.0 (155/244)
CSV ████████████░░░░░░░░ 57.8% ±9.1 (63/109)

gpt-5.4-nano
XML ████████████░░░░░░░░ 59.4% ±6.1 (145/244)
JSON ███████████░░░░░░░░░ 57.4% ±6.2 (140/244)
→ TOON ███████████░░░░░░░░░ 57.0% ±6.2 (139/244)
JSON compact ███████████░░░░░░░░░ 54.9% ±6.2 (134/244)
YAML ███████████░░░░░░░░░ 54.5% ±6.2 (133/244)
CSV █████████░░░░░░░░░░░ 46.8% ±9.2 (51/109)

grok-4.5
→ TOON ███████████████████░ 97.1% ±2.2 (237/244)
JSON ███████████████████░ 96.3% ±2.5 (235/244)
XML ███████████████████░ 95.9% ±2.6 (234/244)
YAML ███████████████████░ 95.9% ±2.6 (234/244)
JSON compact ███████████████████░ 95.5% ±2.7 (233/244)
CSV ███████████████████░ 94.5% ±4.5 (103/109)

> [!NOTE]
> Accuracy figures include Wilson 95% confidence intervals (±); when two formats' intervals overlap, the difference between them is not statistically meaningful. CSV answers only the 109 flat-dataset questions, so its per-model cells cover a smaller, easier population than the other formats.


**Performance by dataset and question type**

#### Performance by Question Type

| Question Type | TOON | JSON | XML | YAML | JSON compact | CSV |
| ------------- | ---- | ---- | ---- | ---- | ---- | ---- |
| Field Retrieval | 97.8% | 99.2% | 99.2% | 99.7% | 98.9% | 100.0% |
| Aggregation | 48.4% | 48.4% | 46.0% | 46.0% | 45.2% | 32.8% |
| Filtering | 38.0% | 41.1% | 37.5% | 40.1% | 38.0% | 33.3% |
| Structure Awareness | 90.3% | 84.0% | 84.0% | 79.2% | 78.5% | 82.8% |
| Structural Validation | 100.0% | 50.0% | 80.0% | 50.0% | 45.0% | 80.0% |

#### Performance by Dataset

##### Uniform employee records

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `csv` | 64.6% | 2,336 | 106/164 |
| `toon` | 62.8% | 2,537 | 103/164 |
| `json-compact` | 62.2% | 3,919 | 102/164 |
| `yaml` | 64.0% | 4,982 | 105/164 |
| `json-pretty` | 62.2% | 6,326 | 102/164 |
| `xml` | 61.0% | 7,286 | 100/164 |

##### E-commerce orders with nested structures

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `json-compact` | 70.7% | 6,875 | 116/164 |
| `toon` | 71.3% | 7,344 | 117/164 |
| `yaml` | 72.0% | 8,456 | 118/164 |
| `json-pretty` | 71.3% | 10,842 | 117/164 |
| `xml` | 74.4% | 12,180 | 122/164 |

##### Time-series analytics data

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `csv` | 64.2% | 1,408 | 77/120 |
| `toon` | 63.3% | 1,595 | 76/120 |
| `json-compact` | 59.2% | 2,351 | 71/120 |
| `yaml` | 62.5% | 2,951 | 75/120 |
| `json-pretty` | 65.0% | 3,678 | 78/120 |
| `xml` | 62.5% | 4,386 | 75/120 |

##### Top 100 GitHub repositories

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `toon` | 57.6% | 9,017 | 76/132 |
| `csv` | 54.5% | 8,726 | 72/132 |
| `json-compact` | 53.8% | 11,650 | 71/132 |
| `yaml` | 53.8% | 13,350 | 71/132 |
| `json-pretty` | 55.3% | 15,350 | 73/132 |
| `xml` | 53.8% | 17,304 | 71/132 |

##### Semi-uniform event logs

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `json-compact` | 56.7% | 4,793 | 68/120 |
| `toon` | 60.8% | 5,814 | 73/120 |
| `json-pretty` | 60.0% | 6,759 | 72/120 |
| `yaml` | 55.0% | 5,798 | 66/120 |
| `xml` | 50.8% | 7,668 | 61/120 |

##### Deeply nested configuration

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `json-compact` | 91.4% | 562 | 106/116 |
| `yaml` | 93.1% | 675 | 108/116 |
| `toon` | 91.4% | 669 | 106/116 |
| `json-pretty` | 94.8% | 918 | 110/116 |
| `xml` | 94.0% | 1,007 | 109/116 |

##### Valid complete dataset (control)

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `toon` | 100.0% | 566 | 4/4 |
| `json-compact` | 100.0% | 772 | 4/4 |
| `yaml` | 100.0% | 984 | 4/4 |
| `json-pretty` | 100.0% | 1,259 | 4/4 |
| `xml` | 0.0% | 1,441 | 0/4 |
| `csv` | 0.0% | 473 | 0/4 |

##### Array truncated: 3 rows removed from end

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 408 | 4/4 |
| `toon` | 100.0% | 498 | 4/4 |
| `xml` | 100.0% | 1,229 | 4/4 |
| `json-pretty` | 0.0% | 1,075 | 0/4 |
| `yaml` | 0.0% | 841 | 0/4 |
| `json-compact` | 0.0% | 660 | 0/4 |

##### Extra rows added beyond declared length

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 547 | 4/4 |
| `toon` | 100.0% | 644 | 4/4 |
| `xml` | 100.0% | 1,663 | 4/4 |
| `json-pretty` | 0.0% | 1,452 | 0/4 |
| `yaml` | 0.0% | 1,135 | 0/4 |
| `json-compact` | 0.0% | 893 | 0/4 |

##### Inconsistent field count (missing salary in row 10)

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 470 | 4/4 |
| `toon` | 100.0% | 563 | 4/4 |
| `json-compact` | 75.0% | 767 | 3/4 |
| `xml` | 100.0% | 1,432 | 4/4 |
| `yaml` | 75.0% | 977 | 3/4 |
| `json-pretty` | 75.0% | 1,251 | 3/4 |

##### Missing required fields (no email in multiple rows)

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `csv` | 100.0% | 442 | 4/4 |
| `toon` | 100.0% | 535 | 4/4 |
| `xml` | 100.0% | 1,386 | 4/4 |
| `yaml` | 75.0% | 941 | 3/4 |
| `json-pretty` | 75.0% | 1,207 | 3/4 |
| `json-compact` | 50.0% | 732 | 2/4 |

##### Feature flags keyed by name

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `toon` | 97.1% | 931 | 66/68 |
| `json-compact` | 94.1% | 1,264 | 64/68 |
| `yaml` | 92.6% | 1,443 | 63/68 |
| `json-pretty` | 95.6% | 1,873 | 65/68 |
| `xml` | 95.6% | 2,306 | 65/68 |

##### Contacts with nested address and plan groups

| Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- |
| `toon` | 94.4% | 1,444 | 68/72 |
| `json-compact` | 91.7% | 2,357 | 66/72 |
| `yaml` | 94.4% | 2,797 | 68/72 |
| `json-pretty` | 97.2% | 4,014 | 70/72 |
| `xml` | 98.6% | 4,534 | 71/72 |



#### Run Configuration

- **Models tested**: `claude-haiku-4-5-20251001`, `gemini-3.6-flash`, `gpt-5.4-nano`, `grok-4.5`
- **Formats compared**: TOON, JSON, XML, YAML, JSON compact, CSV
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding (GPT-5 tokenizer). Other providers tokenize differently, so absolute counts are tokenizer-specific; relative differences between formats hold directionally.
- **Reasoning**: Disabled via the AI SDK's universal `reasoning: 'none'` (Gemini 3 floors at minimal thinking, `grok-4.5` at `low`)
- **Temperature**: Not set (models use their defaults)
- **Total evaluations**: 244 questions × 6 formats × 4 models = 5,856 LLM calls

What the datasets contain, how the questions are generated, and how answers are validated is documented in [the benchmark README](https://github.com/toon-format/toon/tree/main/benchmarks#retrieval-accuracy-benchmark).



### Token Efficiency

Token counts are measured using the GPT-5 `o200k_base` tokenizer via [`gpt-tokenizer`](https://github.com/niieani/gpt-tokenizer). Savings are calculated against formatted JSON (2-space indentation) as the primary baseline, with additional comparisons to compact JSON (minified), YAML, and XML. Actual savings vary by model and tokenizer.

The benchmarks test datasets across different structural patterns (uniform, semi-uniform, nested, deeply nested) to show where TOON excels and where other formats may be better.



#### Mixed-Structure Track

Datasets with nested or semi-uniform structures. CSV excluded as it cannot properly represent these structures.

/* Detailed source-code truncated for AI context efficiency. */

#### Flat-Only Track

Datasets with flat, fully tabular-eligible data where CSV is applicable.

👥 Uniform employee records ┊ Tabular: 100%
│
CSV ███████████████████░ 47,153 tokens
TOON ████████████████████ 49,978 tokens (+6.0% vs CSV)
├─ vs JSON (−60.7%) 127,061 tokens
├─ vs JSON compact (−36.8%) 79,057 tokens
├─ vs YAML (−50.0%) 100,054 tokens
└─ vs XML (−65.9%) 146,605 tokens

📈 Time-series analytics data ┊ Tabular: 100%
│
CSV ██████████████████░░ 8,383 tokens
TOON ████████████████████ 9,115 tokens (+8.7% vs CSV)
├─ vs JSON (−59.0%) 22,245 tokens
├─ vs JSON compact (−35.9%) 14,211 tokens
├─ vs YAML (−49.0%) 17,858 tokens
└─ vs XML (−65.8%) 26,616 tokens

⭐ Top 100 GitHub repositories ┊ Tabular: 100%
│
CSV ███████████████████░ 8,711 tokens
TOON ████████████████████ 8,937 tokens (+2.6% vs CSV)
├─ vs JSON (−41.7%) 15,337 tokens
├─ vs JSON compact (−23.2%) 11,640 tokens
├─ vs YAML (−33.0%) 13,337 tokens
└─ vs XML (−48.3%) 17,294 tokens

──────────────────────────────────── Total ────────────────────────────────────
CSV ███████████████████░ 64,247 tokens
TOON ████████████████████ 68,030 tokens (+5.9% vs CSV)
├─ vs JSON (−58.7%) 164,643 tokens
├─ vs JSON compact (−35.2%) 104,908 tokens
├─ vs YAML (−48.2%) 131,249 tokens
└─ vs XML (−64.3%) 190,515 tokens

Token counts use `gpt-tokenizer` with `o200k_base` encoding (GPT-5 tokenizer). Other providers tokenize differently, so absolute counts are tokenizer-specific; relative differences between formats hold directionally.


**Show detailed examples**

#### 📈 Time-series analytics data

**Savings:** 13,130 tokens (59.0% reduction vs JSON)

**JSON** (22,245 tokens):

```json
{
  "metrics": [
    {
      "date": "2025-01-01",
      "views": 6138,
      "clicks": 174,
      "conversions": 12,
      "revenue": 2712.49,
      "bounceRate": 0.35
    },
    {
      "date": "2025-01-02",
      "views": 4616,
      "clicks": 274,
      "conversions": 34,
      "revenue": 9156.29,
      "bounceRate": 0.56
    },
    {
      "date": "2025-01-03",
      "views": 4460,
      "clicks": 143,
      "conversions": 8,
      "revenue": 1317.98,
      "bounceRate": 0.59
    },
    {
      "date": "2025-01-04",
      "views": 4740,
      "clicks": 125,
      "conversions": 13,
      "revenue": 2934.77,
      "bounceRate": 0.37
    },
    {
      "date": "2025-01-05",
      "views": 6428,
      "clicks": 369,
      "conversions": 19,
      "revenue": 1317.24,
      "bounceRate": 0.3
    }
  ]
}

TOON (9,115 tokens):

metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:
  2025-01-01,6138,174,12,2712.49,0.35
  2025-01-02,4616,274,34,9156.29,0.56
  2025-01-03,4460,143,8,1317.98,0.59
  2025-01-04,4740,125,13,2934.77,0.37
  2025-01-05,6428,369,19,1317.24,0.3

⭐ Top 100 GitHub repositories

Savings: 6,400 tokens (41.7% reduction vs JSON)

JSON (15,337 tokens):

{
  "repositories": [
    {
      "id": 132750724,
      "name": "build-your-own-x",
      "repo": "codecrafters-io/build-your-own-x",
      "description": "Master programming by recreating your favorite technologies from scratch.",
      "createdAt": "2018-05-09T12:03:18Z",
      "updatedAt": "2026-07-23T18:57:15Z",
      "pushedAt": "2026-07-14T19:25:58Z",
      "stars": 530712,
      "watchers": 6778,
      "forks": 50205,
      "defaultBranch": "master"
    },
    {
      "id": 21737465,
      "name": "awesome",
      "repo": "sindresorhus/awesome",
      "description": "😎 Awesome lists about all kinds of interesting topics",
      "createdAt": "2014-07-11T13:42:37Z",
      "updatedAt": "2026-07-23T18:57:24Z",
      "pushedAt": "2026-06-30T18:21:16Z",
      "stars": 488074,
      "watchers": 8292,
      "forks": 36010,
      "defaultBranch": "main"
    },
    {
      "id": 28457823,
      "name": "freeCodeCamp",
      "repo": "freeCodeCamp/freeCodeCamp",
      "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
      "createdAt": "2014-12-24T17:49:19Z",
      "updatedAt": "2026-07-22T07:01:33Z",
      "pushedAt": "2026-07-21T18:00:51Z",
      "stars": 452380,
      "watchers": 8590,
      "forks": 45624,
      "defaultBranch": "main"
    }
  ]
}

TOON (8,937 tokens):

repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
  132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2026-07-23T18:57:15Z","2026-07-14T19:25:58Z",530712,6778,50205,master
  21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2026-07-23T18:57:24Z","2026-06-30T18:21:16Z",488074,8292,36010,main
  28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2026-07-22T07:01:33Z","2026-07-21T18:00:51Z",452380,8590,45624,main

Installation & Quick Start Copy Chapter

# npm
npm install @toon-format/toon

# pnpm
pnpm add @toon-format/toon

# yarn
yarn add @toon-format/toon

To keep the CLI around instead of invoking it through npx, install it globally:

npm install @toon-format/cli

Example usage:

import { encode } from '@toon-format/toon'

const data = {
  users: [
    { id: 1, name: 'Ada', role: 'admin' },
    { id: 2, name: 'Bob', role: 'user' }
  ]
}

console.log(encode(data))
// users[2]{id,name,role}:
//   1,Ada,admin
//   2,Bob,user

Streaming large datasets:

import { encodeLines } from '@toon-format/toon'

const largeData = await fetchThousandsOfRecords()

// Memory-efficient streaming for large data
for (const line of encodeLines(largeData)) {
  process.stdout.write(`${line}\n`)
}

[!TIP]
For streaming decode APIs, see decodeFromLines() and decodeStream().

Transforming values with replacer:

import { encode } from '@toon-format/toon'

// Remove sensitive fields
const user = { name: 'Ada', password: 'secret', email: '[email protected]' }
const safe = encode(user, {
  replacer: (key, value) => key === 'password' ? undefined : value
})
// name: Ada
// email: [email protected]

[!TIP]
The replacer function provides fine-grained control over encoding, similar to JSON.stringify's replacer but with path tracking. See the API Reference for more examples, including verbatim output with rawString.

CLI Copy Chapter

Command-line tool for quick JSON↔TOON conversions, token analysis, and pipeline integration. Auto-detects format from file extension, supports stdin/stdout workflows, and offers delimiter options (comma, tab, pipe) that trade readability for fewer tokens.

# Encode JSON to TOON (auto-detected)
npx @toon-format/cli input.json -o output.toon

# Decode TOON to JSON (auto-detected)
npx @toon-format/cli data.toon -o output.json

# Pipe from stdin (no argument needed)
cat data.json | npx @toon-format/cli
echo '{"name": "Ada"}' | npx @toon-format/cli

# Output to stdout
npx @toon-format/cli input.json

# Show token savings
npx @toon-format/cli data.json --stats

[!TIP]
See the full CLI documentation for all options, examples, and advanced usage.

Using TOON with LLMs Copy Chapter

TOON works best when you show the format instead of describing it. Once a model sees one tabular example, the header – [N] length plus {fields} field list – tells it how to read the rest. Wrap data in ```toon code blocks for input, and show the expected header template when asking models to generate TOON. Tab delimiters buy further token savings. Full-line # comments are stripped on decode, so hand-annotated prompt data – and model output with explainer lines – still decodes cleanly.

Follow the detailed LLM integration guide for strategies, examples, and validation techniques.

Ecosystem Copy Chapter

Playgrounds – the official playground converts JSON or YAML to TOON in real time, compares token counts, and shares experiments by URL. Community alternatives: Format Tokenization Playground, TOON Tools.

Editors – TOON Language Support for VS Code (code --install-extension vishalraut.vscode-toon) adds highlighting, validation, and token analysis. tree-sitter-toon covers Neovim, Helix, Emacs, and Zed; toon.nvim is a Lua-native alternative. Elsewhere, YAML highlighting is a close approximation.

Tooling – Tooner is an MCP proxy that converts JSON tool responses to TOON.

Documentation Copy Chapter

Getting Started Copy Section

• Introduction & Installation – What TOON is, when to use it, first steps
• Format Overview – Complete syntax with examples
• Benchmarks – Accuracy & token efficiency results

Tools & Integration Copy Section

• CLI – Command-line tool for JSON↔TOON conversions
• Playgrounds – Interactive tools
• Using TOON with LLMs – Prompting strategies & validation

References Copy Section

• API Reference – TypeScript/JavaScript encode/decode API
• Syntax Cheatsheet – Quick format lookup
• Specification – Normative rules for implementers
• Glossary – One name per concept, for contributors and tooling

Media Type & File Extension Copy Chapter

TOON files use the .toon extension and the provisional media type text/toon. Documents are always UTF-8; the charset=utf-8 parameter may be given but is assumed when absent. See SPEC.md §17 for normative details.

Other Implementations Copy Chapter

TOON has official and community implementations across multiple languages including Python, Rust, Go, Java, Swift, .NET, and many more.

See the full list of implementations in the documentation.

Credits Copy Chapter

• Logo design by 鈴木ックス(SZKX)

License Copy Chapter

MIT License © 2025-PRESENT Johann Schopplich

File: benchmarks/README.md Copy Chapter

TOON Benchmarks

Benchmarks measuring TOON's token efficiency and retrieval accuracy compared to JSON, XML, YAML, and CSV.

[!NOTE]
Results are automatically embedded in the main README. This guide focuses on running the benchmarks locally.

Quick Start Copy Chapter

# Run token efficiency benchmark
pnpm benchmark:tokens

# Run retrieval accuracy benchmark (requires API keys)
pnpm benchmark:accuracy

Token Efficiency Benchmark Copy Chapter

Measures token count reduction across JSON, XML, YAML, CSV, and TOON:

1. Generate datasets (GitHub repos, analytics, orders)
2. Convert to all formats (TOON, JSON, XML, YAML, CSV)
3. Tokenize using gpt-tokenizer (o200k_base encoding)
4. Calculate savings and generate report

pnpm benchmark:tokens

Results are saved to results/token-efficiency.md.

Retrieval Accuracy Benchmark Copy Chapter

Tests how well LLMs can answer questions about data in different formats (TOON, JSON, JSON compact, XML, YAML, CSV):

1. Generate 244 questions across 13 datasets (8 primary + 5 structural validation; CSV only included for flat, tabular-eligible datasets)
2. Convert each dataset to all supported formats
3. Query each LLM with formatted data + question
4. Validate answers deterministically using type-aware comparison (no LLM judge needed)
5. Aggregate metrics and generate report

This measures comprehension: each model reads formatted data and answers questions about it. It does not test a model's ability to generate TOON.

What the Datasets Cover Copy Section

Live row counts and per-dataset scores are in the generated dataset catalog; this is what each one is for.

Primary datasets – eight shapes, chosen so the tabular-eligibility axis is covered end to end:

Structural validation datasets – five variants of one valid 20-row dataset. The corruption is applied to the encoded text after it is emitted, so TOON's [N] length and field-list width still declare the original shape while the other formats render the lossy-pipeline outcome:

That contrast is the point of the structural-validation track: two of these corruptions cannot be detected in JSON, YAML, XML, or CSV at all, because those formats carry no declared length.

How Questions Are Generated Copy Section

244 questions across five categories, generated from the datasets rather than hand-written (see src/questions/):

• Field retrieval – direct value lookups, including booleans and simple counts such as array lengths. "What is Ada's salary?" → 75000
• Aggregation – dataset-level totals and averages plus single-condition filters. "How many employees work in Engineering?" → 17
• Filtering – multi-condition queries requiring compound logic. "How many employees in Sales have salary > 80000?" → 5
• Structure awareness – format-native structural affordances: TOON's [N] count and field list, CSV's header row. "List the field names for employees"
• Structural validation – detecting truncated or corrupted data from the encoded text alone. "Is this data complete and valid?" → YES / NO

With reasoning disabled, multi-row arithmetic is hard in every format – aggregation and filtering scores mostly measure computation under format friction and sit near the floor for all formats. The per-question-type table in the generated report makes this visible.

Answers are validated deterministically with type-aware comparison (50000 = $50,000, Engineering = engineering, 2025-01-01 = January 1, 2025), so no LLM judge is involved.

Setup Copy Section

1. Edit src/evaluate.ts and add models to the exported MODELS array:
   ts Copy

   export const MODELS: ModelDescriptor[] = [
     { id: 'gpt-5.4-nano', rpm: 50, create: () => openai('gpt-5.4-nano') },
     { id: 'claude-haiku-4-5-20251001', rpm: 50, create: () => anthropic('claude-haiku-4-5-20251001') },
     { id: 'gemini-3.6-flash', rpm: 25, create: () => google('gemini-3.6-flash') },
     { id: 'grok-4.5', rpm: 25, reasoning: 'low', create: () => xai('grok-4.5') },
     // Add your models here
   ]

2. Duplicate .env.example to .env and add your API keys:
   bash Copy

   cp .env.example .env

Usage Copy Section

# Full benchmark
pnpm benchmark:accuracy

# Dry run (10 questions only, for testing setup)
DRY_RUN=true pnpm benchmark:accuracy

Running the script will:

1. Prompt you to select which models to test.
2. Skip models with existing results (rerun to overwrite).
3. Show progress with rate limiting.
4. Save results to results/accuracy/models/{model-id}.json.
5. Generate report at results/retrieval-accuracy.md.

Configuration Copy Section

Edit src/constants.ts to adjust:

• DEFAULT_CONCURRENCY – Parallel tasks (default: 10)
• DRY_RUN_LIMITS – Questions per dry run (default: 10)

Rate limits now live on each src/evaluate.ts MODELS entry via its rpm field.

Project Structure Copy Chapter

scripts/
├── accuracy-benchmark.ts         # Retrieval accuracy benchmark
├── token-efficiency-benchmark.ts # Token counting benchmark
├── fetch-github-repos.ts         # Update GitHub dataset
├── verify-feature-datasets.ts    # Keyed/nested-group dataset guards
├── verify-structural-corruption.ts # Corruption invariant guards
└── verify-utils.ts               # Shared verify script plumbing
src/
├── constants.ts                  # Configuration
├── datasets.ts                   # Test data generators
├── evaluate.ts                   # LLM evaluation
├── formats.ts                    # Format registry (converters, primers, fences, labels)
├── normalize.ts                  # Answer normalization
├── report.ts                     # Markdown reports
├── storage.ts                    # Result caching
├── structural-corruption.ts      # Post-encode text corruption
├── types.ts                      # Type definitions
├── utils.ts                      # Helpers
└── questions/                    # Question generators
    ├── analytics.ts
    ├── event-logs.ts
    ├── github.ts
    ├── index.ts
    ├── keyed.ts
    ├── nested-config.ts
    ├── nested-group.ts
    ├── nested.ts
    ├── structural-validation.ts
    ├── structure.ts
    ├── tabular.ts
    └── utils.ts
data/
└── github-repos.json             # Top 100 GitHub repos
results/
├── token-efficiency.md           # Token savings report
├── retrieval-accuracy.md         # Accuracy report
└── accuracy/models/              # Per-model results (JSON)

File: docs/guide/benchmarks.md Copy Chapter

Benchmarks

The benchmarks on this page measure TOON's performance across two key dimensions:

• Retrieval Accuracy: How well LLMs understand and extract information from different input formats.
• Token Efficiency: How many tokens each format requires to represent the same data.

Benchmarks are organized into two tracks to ensure fair comparisons:

• Mixed-Structure Track: Nested and semi-uniform datasets (TOON vs JSON, YAML, XML). CSV is excluded – it cannot represent these structures without lossy flattening.
• Flat-Only Track: Flat, fully tabular-eligible datasets, where CSV is a fair competitor (CSV vs TOON vs JSON, YAML, XML).

Retrieval Accuracy Copy Chapter

Benchmarks test LLM comprehension across different input formats using 244 data retrieval questions on 4 models.

Show Dataset Catalog

Dataset Catalog

Structure classes:

• uniform: All objects have identical fields with primitive values
• semi-uniform: Mix of uniform and non-uniform structures
• nested: Objects with nested structures (nested objects or arrays)
• deep: Highly nested with minimal tabular eligibility

CSV Support: ✓ (supported), ✗ (not supported – would require lossy flattening)

Eligibility: Percentage of arrays and keyed maps that qualify for TOON's tabular forms (uniform records whose fields are primitives or uniform nested objects folded into nested field groups)

Efficiency Ranking (Accuracy per 1K Tokens)

Each format ranked by efficiency (accuracy percentage per 1,000 tokens):

TOON           ████████████████████   29.2 acc%/1K tok  │  72.2%  ±2.8 acc  │  2,474 tokens
JSON compact   ████████████████░░░░   23.8 acc%/1K tok  │  69.0%  ±2.9 acc  │  2,892 tokens
YAML           ██████████████░░░░░░   20.1 acc%/1K tok  │  70.1%  ±2.9 acc  │  3,487 tokens
JSON           ███████████░░░░░░░░░   16.6 acc%/1K tok  │  71.4%  ±2.8 acc  │  4,308 tokens
XML            ██████████░░░░░░░░░░   14.4 acc%/1K tok  │  70.7%  ±2.9 acc  │  4,909 tokens

Efficiency score = (Accuracy % ÷ Tokens) × 1,000. Higher is better.

[!TIP]
TOON achieves 72.2% accuracy (vs JSON's 71.4%) while using 42.6% fewer tokens.

[!NOTE]
CSV is excluded from the ranking as it only supports 109 of 244 questions (flat tabular data only). While CSV is highly token-efficient for simple tabular data, it cannot represent nested structures that other formats handle.

Accuracy on Flat Datasets

Every format answers the same 109 flat-dataset questions per model, so CSV can be compared on equal footing here.

Per-Model Accuracy

Accuracy across 4 LLMs on 244 data retrieval questions:

claude-haiku-4-5-20251001
→ TOON           █████████████░░░░░░░    65.6% ±5.9 (160/244)
  JSON           █████████████░░░░░░░    63.5% ±6.0 (155/244)
  XML            ████████████░░░░░░░░    62.3% ±6.0 (152/244)
  YAML           ████████████░░░░░░░░    62.3% ±6.0 (152/244)
  JSON compact   ████████████░░░░░░░░    61.9% ±6.0 (151/244)
  CSV            ██████████░░░░░░░░░░    49.5% ±9.2 (54/109)

gemini-3.6-flash
→ TOON           ██████████████░░░░░░    69.3% ±5.8 (169/244)
  JSON           ██████████████░░░░░░    68.4% ±5.8 (167/244)
  YAML           ██████████████░░░░░░    67.6% ±5.8 (165/244)
  XML            █████████████░░░░░░░    65.2% ±5.9 (159/244)
  JSON compact   █████████████░░░░░░░    63.5% ±6.0 (155/244)
  CSV            ████████████░░░░░░░░    57.8% ±9.1 (63/109)

gpt-5.4-nano
  XML            ████████████░░░░░░░░    59.4% ±6.1 (145/244)
  JSON           ███████████░░░░░░░░░    57.4% ±6.2 (140/244)
→ TOON           ███████████░░░░░░░░░    57.0% ±6.2 (139/244)
  JSON compact   ███████████░░░░░░░░░    54.9% ±6.2 (134/244)
  YAML           ███████████░░░░░░░░░    54.5% ±6.2 (133/244)
  CSV            █████████░░░░░░░░░░░    46.8% ±9.2 (51/109)

grok-4.5
→ TOON           ███████████████████░    97.1% ±2.2 (237/244)
  JSON           ███████████████████░    96.3% ±2.5 (235/244)
  XML            ███████████████████░    95.9% ±2.6 (234/244)
  YAML           ███████████████████░    95.9% ±2.6 (234/244)
  JSON compact   ███████████████████░    95.5% ±2.7 (233/244)
  CSV            ███████████████████░    94.5% ±4.5 (103/109)

[!NOTE]
Accuracy figures include Wilson 95% confidence intervals (±); when two formats' intervals overlap, the difference between them is not statistically meaningful. CSV answers only the 109 flat-dataset questions, so its per-model cells cover a smaller, easier population than the other formats.

Performance by dataset and question type

Performance by Question Type

Performance by Dataset

Uniform employee records

E-commerce orders with nested structures

Time-series analytics data

Top 100 GitHub repositories

Semi-uniform event logs

Deeply nested configuration

Valid complete dataset (control)

Array truncated: 3 rows removed from end

Extra rows added beyond declared length

Inconsistent field count (missing salary in row 10)

Missing required fields (no email in multiple rows)

Feature flags keyed by name

Contacts with nested address and plan groups

Run Configuration

• Models tested: claude-haiku-4-5-20251001, gemini-3.6-flash, gpt-5.4-nano, grok-4.5
• Formats compared: TOON, JSON, XML, YAML, JSON compact, CSV
• Token counting: Using gpt-tokenizer with o200k_base encoding (GPT-5 tokenizer). Other providers tokenize differently, so absolute counts are tokenizer-specific; relative differences between formats hold directionally.
• Reasoning: Disabled via the AI SDK's universal reasoning: 'none' (Gemini 3 floors at minimal thinking, grok-4.5 at low)
• Temperature: Not set (models use their defaults)
• Total evaluations: 244 questions × 6 formats × 4 models = 5,856 LLM calls

What the datasets contain, how the questions are generated, and how answers are validated is documented in the benchmark README.

Token Efficiency Copy Chapter

Token counts are measured using the GPT-5 o200k_base tokenizer via gpt-tokenizer. Savings are calculated against formatted JSON (2-space indentation) as the primary baseline, with additional comparisons to compact JSON (minified), YAML, and XML. Actual savings vary by model and tokenizer.

The benchmarks test datasets across different structural patterns (uniform, semi-uniform, nested, deeply nested) to show where TOON excels and where other formats may be better.

Mixed-Structure Track

Datasets with nested or semi-uniform structures. CSV excluded as it cannot properly represent these structures.

/* Detailed source-code truncated for AI context efficiency. */

Flat-Only Track

Datasets with flat, fully tabular-eligible data where CSV is applicable.

👥 Uniform employee records  ┊  Tabular: 100%
   │
   CSV                 ███████████████████░    47,153 tokens
   TOON                ████████████████████    49,978 tokens   (+6.0% vs CSV)
   ├─ vs JSON          (−60.7%)               127,061 tokens
   ├─ vs JSON compact  (−36.8%)                79,057 tokens
   ├─ vs YAML          (−50.0%)               100,054 tokens
   └─ vs XML           (−65.9%)               146,605 tokens

📈 Time-series analytics data  ┊  Tabular: 100%
   │
   CSV                 ██████████████████░░     8,383 tokens
   TOON                ████████████████████     9,115 tokens   (+8.7% vs CSV)
   ├─ vs JSON          (−59.0%)                22,245 tokens
   ├─ vs JSON compact  (−35.9%)                14,211 tokens
   ├─ vs YAML          (−49.0%)                17,858 tokens
   └─ vs XML           (−65.8%)                26,616 tokens

⭐ Top 100 GitHub repositories  ┊  Tabular: 100%
   │
   CSV                 ███████████████████░     8,711 tokens
   TOON                ████████████████████     8,937 tokens   (+2.6% vs CSV)
   ├─ vs JSON          (−41.7%)                15,337 tokens
   ├─ vs JSON compact  (−23.2%)                11,640 tokens
   ├─ vs YAML          (−33.0%)                13,337 tokens
   └─ vs XML           (−48.3%)                17,294 tokens

──────────────────────────────────── Total ────────────────────────────────────
   CSV                 ███████████████████░    64,247 tokens
   TOON                ████████████████████    68,030 tokens   (+5.9% vs CSV)
   ├─ vs JSON          (−58.7%)               164,643 tokens
   ├─ vs JSON compact  (−35.2%)               104,908 tokens
   ├─ vs YAML          (−48.2%)               131,249 tokens
   └─ vs XML           (−64.3%)               190,515 tokens

Token counts use gpt-tokenizer with o200k_base encoding (GPT-5 tokenizer). Other providers tokenize differently, so absolute counts are tokenizer-specific; relative differences between formats hold directionally.

Show detailed examples

📈 Time-series analytics data

Savings: 13,130 tokens (59.0% reduction vs JSON)

JSON (22,245 tokens):

{
  "metrics": [
    {
      "date": "2025-01-01",
      "views": 6138,
      "clicks": 174,
      "conversions": 12,
      "revenue": 2712.49,
      "bounceRate": 0.35
    },
    {
      "date": "2025-01-02",
      "views": 4616,
      "clicks": 274,
      "conversions": 34,
      "revenue": 9156.29,
      "bounceRate": 0.56
    },
    {
      "date": "2025-01-03",
      "views": 4460,
      "clicks": 143,
      "conversions": 8,
      "revenue": 1317.98,
      "bounceRate": 0.59
    },
    {
      "date": "2025-01-04",
      "views": 4740,
      "clicks": 125,
      "conversions": 13,
      "revenue": 2934.77,
      "bounceRate": 0.37
    },
    {
      "date": "2025-01-05",
      "views": 6428,
      "clicks": 369,
      "conversions": 19,
      "revenue": 1317.24,
      "bounceRate": 0.3
    }
  ]
}

TOON (9,115 tokens):

metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:
  2025-01-01,6138,174,12,2712.49,0.35
  2025-01-02,4616,274,34,9156.29,0.56
  2025-01-03,4460,143,8,1317.98,0.59
  2025-01-04,4740,125,13,2934.77,0.37
  2025-01-05,6428,369,19,1317.24,0.3

⭐ Top 100 GitHub repositories

Savings: 6,400 tokens (41.7% reduction vs JSON)

JSON (15,337 tokens):

{
  "repositories": [
    {
      "id": 132750724,
      "name": "build-your-own-x",
      "repo": "codecrafters-io/build-your-own-x",
      "description": "Master programming by recreating your favorite technologies from scratch.",
      "createdAt": "2018-05-09T12:03:18Z",
      "updatedAt": "2026-07-23T18:57:15Z",
      "pushedAt": "2026-07-14T19:25:58Z",
      "stars": 530712,
      "watchers": 6778,
      "forks": 50205,
      "defaultBranch": "master"
    },
    {
      "id": 21737465,
      "name": "awesome",
      "repo": "sindresorhus/awesome",
      "description": "😎 Awesome lists about all kinds of interesting topics",
      "createdAt": "2014-07-11T13:42:37Z",
      "updatedAt": "2026-07-23T18:57:24Z",
      "pushedAt": "2026-06-30T18:21:16Z",
      "stars": 488074,
      "watchers": 8292,
      "forks": 36010,
      "defaultBranch": "main"
    },
    {
      "id": 28457823,
      "name": "freeCodeCamp",
      "repo": "freeCodeCamp/freeCodeCamp",
      "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
      "createdAt": "2014-12-24T17:49:19Z",
      "updatedAt": "2026-07-22T07:01:33Z",
      "pushedAt": "2026-07-21T18:00:51Z",
      "stars": 452380,
      "watchers": 8590,
      "forks": 45624,
      "defaultBranch": "main"
    }
  ]
}

TOON (8,937 tokens):

repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
  132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2026-07-23T18:57:15Z","2026-07-14T19:25:58Z",530712,6778,50205,master
  21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2026-07-23T18:57:24Z","2026-06-30T18:21:16Z",488074,8292,36010,main
  28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2026-07-22T07:01:33Z","2026-07-21T18:00:51Z",452380,8590,45624,main

Related Resources Copy Chapter

• Formal Byte-Level Model – Mathematical analysis of byte efficiency compared to JSON
• Specification – Formal TOON specification

File: docs/guide/format-overview.md Copy Chapter

Format Overview

TOON syntax reference with concrete examples. See Getting Started for an introduction.

The Four Forms Copy Chapter

A form is one rendering of a value. Which form you get follows from the data's shape and where it sits – you never choose by hand. Everything below is a variation on these four:

"Form" is deliberate: these are shapes within TOON, not sibling formats to JSON or YAML.

Data Model Copy Chapter

TOON models data the same way as JSON:

• Primitives: strings, numbers, booleans, and null
• Objects: mappings from string keys to values
• Arrays: ordered sequences of values

Root Forms Copy Section

A TOON document can represent different root forms:

• Root object (most common): Fields appear at depth 0 with no parent key
• Root array: Begins with [N]: or [N]{fields}: at depth 0
• Root primitive: A single primitive value (string, number, boolean, or null)

Most examples in these docs use root objects, but the format supports all three forms equally (spec §5).

Objects Copy Chapter

Simple Objects Copy Section

Objects with primitive values use key: value syntax, with one field per line:

id: 123
name: Ada
active: true

Indentation replaces braces. One space follows the colon.

Nested Objects Copy Section

Nested objects add one indentation level (default: 2 spaces):

user:
  id: 123
  name: Ada

When a key ends with : and has no value on the same line, it opens a nested object. All lines at the next indentation level belong to that object.

Empty Objects Copy Section

An empty object at the root yields an empty document (no lines). A nested empty object is key: alone, with no children.

Keyed Tabular Objects Copy Section

When an object has at least two entries whose values are uniform objects (same keys, primitive or nested-uniform values), it collapses into a keyed tabular form: the shared field structure appears once in the header, and each entry becomes one row that carries its own key:

users[2:]{age,city}:
  alice: 30,Berlin
  bob: 25,Oslo

The colon immediately after the length ([2:]) marks the keyed header, and [N] declares the entry count. Each entry row is entrykey: cell,cell,… – the entry key followed by the entry value's leaf values in field order.

When the root object itself is eligible, the key is omitted:

[2:]{age,city}:
  alice: 30,Berlin
  bob: 25,Oslo

Objects that don't qualify keep the nested form unchanged: single-entry objects, objects whose values mix shapes or include primitives, arrays, or empty objects. In practice this leaves most configuration-style maps as they are (spec §9.5).

Arrays Copy Chapter

TOON detects array structure and chooses the most efficient representation. Arrays always declare their length in brackets: [N].

Primitive Arrays (Inline Form) Copy Section

Arrays of primitives (strings, numbers, booleans, null) are rendered inline:

tags[3]: admin,ops,dev

The delimiter (comma by default) separates values. Strings containing the active delimiter must be quoted.

Arrays of Objects (Tabular Form) Copy Section

When all objects in an array share the same set of primitive-valued keys, TOON uses tabular form:

::: code-group

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5

users[2]{id,name,role}:
  1,Ada Lovelace,admin
  2,"Smith, Bob",user

:::

The header items[2]{sku,qty,price}: declares:

• Array length: [2] means 2 rows
• Field names: {sku,qty,price} defines the columns
• Active delimiter: comma (default)

Each row contains values in the same order as the field list. Values are encoded as primitives (strings, numbers, booleans, null) and separated by the delimiter.

[!NOTE]
Tabular form requires identical field sets across all objects (same keys, order per object may vary), at least one key per object, and every column either primitive-valued or a uniform nested object (see below) – arrays that contain an empty {} element or mix value shapes within a column fall back to list form.

Nested Field Groups Copy Section

A column whose values are uniform sub-objects (same keys in every element, recursively primitive or nested-uniform) folds into the header as a nested field group, while rows stay flat:

orders[2]{id,customer{name,country},total}:
  1,Ada,DK,99
  2,Bob,UK,149

The header customer{name,country} declares a nested-object column; each row's cells follow a depth-first walk of the field list, so Ada,DK fills customer.name and customer.country of the first order. Nesting depth is unbounded (spec §9.3).

Mixed and Non-Uniform Arrays (List Form) Copy Section

Arrays that don't meet the tabular requirements use list form with hyphen markers:

items[3]:
  - 1
  - a: 1
  - text

Each element starts with - at one indentation level deeper than the parent array header.

Objects as List Items Copy Section

When an array element is an object, it appears as a list item:

items[2]:
  - id: 1
    name: First
  - id: 2
    name: Second
    extra: true

When a tabular array is the first field of a list-item object, the tabular header appears on the hyphen line, with rows indented two levels deeper and other fields indented one level deeper:

items[1]:
  - users[2]{id,name}:
      1,Ada
      2,Bob
    status: active

When the object has only a single tabular field, the same pattern applies:

items[1]:
  - users[2]{id,name}:
      1,Ada
      2,Bob

This is the canonical encoding for list-item objects whose first field is a tabular array.

Arrays of Arrays (List Form) Copy Section

When you have arrays containing primitive inner arrays:

pairs[2]:
  - [2]: 1,2
  - [2]: 3,4

Each inner array gets its own header on the list-item line.

When the inner arrays are themselves arrays of objects or non-uniform arrays, the same - [N]: header appears on the hyphen line and the nested items follow one indent deeper:

items[3]:
  - summary
  - id: 1
    name: Ada
  - [2]:
    - id: 2
    - status: draft

Empty Arrays Copy Section

Empty arrays render as key: [] for fields and [] at the root:

items: []

The legacy items[0]: form is still decoded for backward compatibility.

Array Headers Copy Chapter

Header Syntax Copy Section

Array headers follow this pattern:

key[N<delimiter?>]<{fields}>:

Where:

• N is the non-negative integer length
• delimiter (optional) explicitly declares the active delimiter:
  • Absent → comma (,)
  • \t (tab character) → tab delimiter
  • | → pipe delimiter
• fields (optional) for tabular arrays: {field1,field2,field3}

[!NOTE]
The array length [N] helps LLMs validate structure. If you ask a model to generate TOON output, explicit lengths let you detect truncation or malformed data.

Delimiter Options Copy Section

TOON supports three delimiters: comma (default), tab, and pipe. The delimiter is scoped to the array header that declares it.

::: code-group

items[2]{sku,name,qty,price}:
  A1,Widget,2,9.99
  B2,Gadget,1,14.5

items[2	]{sku	name	qty	price}:
  A1	Widget	2	9.99
  B2	Gadget	1	14.5

items[2|]{sku|name|qty|price}:
  A1|Widget|2|9.99
  B2|Gadget|1|14.5

:::

Tab and pipe delimiters are explicitly encoded in the header brackets and field braces. Inside an array scope, only the active delimiter triggers quoting – the others are literal data. Object field values (key: value) follow the document delimiter (§11.1) regardless of any surrounding array's active delimiter.

[!TIP]
Tab delimiters often tokenize more efficiently than commas, especially for data with few quoted strings. Use encode(data, { delimiter: '\t' }) for additional token savings.

Comments Copy Chapter

Decoders strip every line whose first non-space character is # in a lexical pre-pass, before anything else:

# Server configuration
host: example.com
port: 8080

Comments are full-line only – a # anywhere else on a line is ordinary content – and decode-side only: encoders never emit them, and string values starting with # are always quoted so encoder output never contains a line that reads as a comment. A comment between tabular rows or entry rows does not end them (spec §5.1).

Quoting and Types Copy Chapter

When Strings Need Quotes Copy Section

TOON quotes strings only when necessary to maximize token efficiency. A string must be quoted if:

• It's empty ("")
• It has leading or trailing whitespace
• It equals true, false, or null (case-sensitive)
• It looks like a number (e.g., "42", "-3.14", "1e-6", "05", "+1")
• It contains special characters: colon (:), quote ("), backslash (\), brackets, braces, or any control character in U+0000–U+001F
• It contains the relevant delimiter (the active delimiter inside an array scope, or the document delimiter elsewhere)
• It equals "-" or starts with "-" followed by any character
• It equals "#" or starts with "#" (the line would read as a comment)

Otherwise, strings can be unquoted. Unicode, emoji, and strings with internal (non-leading/trailing) spaces are safe unquoted:

message: Hello 世界 👋
note: This has inner spaces

Escape Sequences Copy Section

In quoted strings and keys, six escape sequences are valid:

Other escapes (e.g., \x, \0, \b) are always rejected, as are lone-surrogate \uXXXX values (U+D800–U+DFFF).

Type Conversions Copy Section

Numbers are emitted in canonical decimal form for values in the §2 carve-out range; exponent notation is permitted outside. Non-JSON types (NaN, Infinity, BigInt, Date, Set, Map, undefined, etc.) are normalized before encoding – see API Reference – Type Normalization for the full mapping.

Decoders accept both decimal and exponent forms on input (e.g., 42, -3.14, 1e-6), and treat tokens with forbidden leading zeros (e.g., "05") as strings, not numbers.

Custom Serialization with toJSON Copy Section

Objects with a toJSON() method are serialized by calling the method and normalizing its result before encoding, similar to JSON.stringify:

const obj = {
  data: 'example',
  toJSON() {
    return { info: this.data }
  }
}

encode(obj)
// info: example

The toJSON() method:

• Takes precedence over built-in normalization (Date, Array, Set, Map)
• Results are recursively normalized
• Is called for objects with toJSON in their prototype chain

For complete rules on quoting, escaping, type conversions, and strict-mode decoding, see spec §2–4 (data model), §7 (strings and keys), and §14 (strict mode).

File: docs/guide/getting-started.md Copy Chapter

Getting Started

What Is TOON? Copy Chapter

Token-Oriented Object Notation is a compact, human-readable encoding of the JSON data model that minimizes tokens and makes structure easy for models to follow.

TOON combines YAML's indentation-based structure for nested objects with CSV-style tabular forms for uniform data. Its sweet spot is uniform objects – same fields across items, whether in an array or keyed by ID – reaching CSV-like compactness while adding explicit structure that helps LLMs parse and validate data reliably.

Think of it as a translation layer: use JSON programmatically, and encode it as TOON for LLM input – a drop-in, lossless representation of the JSON you already have.

Why TOON? Copy Section

LLM tokens cost money – and standard JSON is verbose. A weather forecast in TOON:

location:
  city: Berlin
  country: DE
  units: metric
alerts[2]: frost,wind
forecast[3]{day,temp{min,max},condition,rainChance}:
  Mon,-2,4,snow,80
  Tue,1,7,cloudy,20
  Wed,3,11,sunny,5

The same data as JSON – ~117 tokens against TOON's ~66:

{
  "location": {
    "city": "Berlin",
    "country": "DE",
    "units": "metric"
  },
  "alerts": [
    "frost",
    "wind"
  ],
  "forecast": [
    {
      "day": "Mon",
      "temp": {
        "min": -2,
        "max": 4
      },
      "condition": "snow",
      "rainChance": 80
    },
    {
      "day": "Tue",
      "temp": {
        "min": 1,
        "max": 7
      },
      "condition": "cloudy",
      "rainChance": 20
    },
    {
      "day": "Wed",
      "temp": {
        "min": 3,
        "max": 11
      },
      "condition": "sunny",
      "rainChance": 5
    }
  ]
}

TOON combines YAML's indentation for the location object, inline form for the primitive alerts array, and tabular form for the forecast array: [3] declares the array length (letting LLMs answer dataset-size questions and detect truncation), {day,…} declares the field names once, and each row streams comma-separated values. The uniform nested temp objects fold into the header as a nested field group (temp{min,max}) while rows stay flat. Each form is chosen automatically from the data's shape.

The pattern is the same throughout TOON: declare structure once, stream data compactly – landing close to CSV density with explicit structure preserved.

Maps of uniform objects collapse as well: the keyed tabular form turns them into tables whose rows carry their own keys.

Design Goals Copy Section

TOON is optimized for specific use cases. It aims to:

• Make uniform arrays of objects as compact as possible by declaring structure once and streaming data.
• Stay fully lossless and deterministic – round-trips preserve all data and structure.
• Keep parsing simple and robust for both LLMs and humans through explicit structure markers.
• Provide validation guardrails (array lengths, field counts) that help detect truncation and malformed output.

When to Use TOON Copy Chapter

TOON excels with uniform arrays of objects – data with the same structure across items. For LLM prompts, the format produces deterministic, minimally quoted text with built-in validation. Explicit array lengths ([N]) and field lists ({fields}) help detect truncation and malformed data, while tabular form declares the field list once rather than repeating it in every row.

::: tip
The TOON format is stable, but also an idea in progress. Nothing's set in stone – help shape where it goes by contributing to the spec or sharing feedback.
:::

When Not to Use TOON Copy Chapter

TOON is not always the best choice. Consider alternatives when:

• Deeply nested or non-uniform structures (tabular eligibility ≈ 0%): JSON-compact often uses fewer tokens. Example: complex configuration objects with many nested levels.
• Semi-uniform arrays (~40–60% tabular eligibility): Token savings diminish. Prefer JSON if your pipelines already rely on it.
• Pure tabular data: CSV is smaller than TOON for flat tables. TOON adds minimal overhead (~5–10%) to provide structure (array length declarations, field lists, delimiter scoping) that improves LLM reliability.
• Latency-critical applications: Benchmark on your exact setup. Some deployments (especially local/quantized models) may process compact JSON faster despite TOON's lower token count.

::: info
For data-driven comparisons across different structures, see Benchmarks. When optimizing for latency, measure TTFT, tokens/sec, and total time for both TOON and JSON-compact, and use whichever is faster in your specific environment.
:::

Installation Copy Chapter

TypeScript Library Copy Section

Install the library via your preferred package manager:

::: code-group

npm install @toon-format/toon

pnpm add @toon-format/toon

yarn add @toon-format/toon

:::

CLI Copy Section

The CLI can be used without installation via npx, or installed globally:

::: code-group

npx @toon-format/cli input.json -o output.toon

npm install @toon-format/cli

pnpm add @toon-format/cli

yarn global add @toon-format/cli

:::

For full CLI documentation, see the CLI reference.

Media Type & File Extension Copy Chapter

TOON files conventionally use the .toon extension. For HTTP transmission, the provisional media type is text/toon, always with UTF-8 encoding. While you may specify charset=utf-8 explicitly, it's optional – UTF-8 is the default assumption. This follows the registration process outlined in spec §17.

Your First Example Copy Chapter

The examples below use the TypeScript library for demonstration, but the same operations work in any language with a TOON implementation.

Let's encode a simple dataset with the TypeScript library:

import { encode } from '@toon-format/toon'

const data = {
  users: [
    { id: 1, name: 'Ada', role: 'admin' },
    { id: 2, name: 'Bob', role: 'user' }
  ]
}

console.log(encode(data))

Output:

users[2]{id,name,role}:
  1,Ada,admin
  2,Bob,user

Decoding Back to JSON Copy Section

Decoding is just as simple:

import { decode } from '@toon-format/toon'

const toon = `
users[2]{id,name,role}:
  1,Ada,admin
  2,Bob,user
`

const data = decode(toon)
console.log(JSON.stringify(data, null, 2))

Output:

{
  "users": [
    { "id": 1, "name": "Ada", "role": "admin" },
    { "id": 2, "name": "Bob", "role": "user" }
  ]
}

Round-tripping is lossless: decode(encode(x)) always equals x (after normalization of non-JSON types like Date, NaN, etc.).

Where to Go Next Copy Chapter

Now that you've seen your first TOON document, read the Format Overview for complete syntax details (objects, arrays, tabular forms, quoting rules), then explore Using TOON with LLMs to see how to use it effectively in prompts. For implementation details, check the API Reference (TypeScript) or the Specification (language-agnostic normative rules).

File: docs/guide/llm-prompts.md Copy Chapter

Using TOON with LLMs

TOON is designed for passing structured data to Large Language Models with reduced token costs and improved reliability. This guide shows how to use TOON effectively in prompts, both for input (sending data to models) and output (getting models to generate TOON).

Why TOON for LLMs Copy Chapter

Beyond token savings (see Benchmarks), TOON adds structure guardrails: explicit [N] lengths and {fields} headers make it easier for models to track rows and for you to validate output. Strict mode helps detect truncation and malformed TOON when decoding model responses.

Sending TOON as Input Copy Chapter

TOON works best when you show the format instead of describing it. The structure is self-documenting – models parse it naturally once they see the pattern.

Wrap your encoded data in a fenced code block (label it ```toon for clarity):

Data is in TOON format (2-space indent, arrays show length and fields).

```toon
users[3]{id,name,role,lastLogin}:
  1,Ada,admin,"2025-01-15T10:30:00Z"
  2,Bob,user,"2025-01-14T15:22:00Z"
  3,Cleo,user,"2025-01-13T09:45:00Z"
```

Task: Summarize the user roles and their last activity.

The indentation and headers are usually enough – models treat TOON like familiar YAML or CSV. The explicit array lengths ([N]) and field lists ({fields}) help the model track structure, especially for large tables.

[!NOTE]
Most models don't have built-in TOON syntax highlighting, so ```toon or ```yaml both work fine. The structure is what matters.

Nested and Keyed Data Copy Section

Uniform nested objects don't break the tabular form: a nested-object column folds into the header as a nested field group, and rows stay flat:

orders[2]{id,customer{name,country},total}:
  1,Ada,DK,99
  2,Bob,UK,149

Maps of uniform objects – feature flags, users by ID, per-environment config – collapse into the keyed tabular form, where each entry row carries its own key:

environments[2:]{region,replicas,debug}:
  production: eu-central-1,6,false
  staging: eu-central-1,2,true

The same prompting rules apply: one example is enough, the header tells the model how to read the rows.

Annotating Data with Comments Copy Section

Decoders strip full-line # comment lines before parsing, so you can annotate prompt data by hand – and model output that includes # explainer lines still decodes cleanly. Encoders never emit comments, so round-trips stay canonical.

# Only active users, exported 2025-01-15
users[2]{id,name,role}:
  1,Ada,admin
  2,Bob,user

Generating TOON from LLMs Copy Chapter

For output, be more explicit. When you want the model to generate TOON:

• Show the expected header (e.g., users[N]{id,name,role}:). The model fills rows instead of repeating keys, reducing generation errors.
• State the rules: 2-space indent, no trailing spaces, [N] matches row count.

Here's a prompt that works for both reading and generating:

Data is in TOON format (2-space indent, arrays show length and fields).

```toon
users[3]{id,name,role,lastLogin}:
  1,Ada,admin,"2025-01-15T10:30:00Z"
  2,Bob,user,"2025-01-14T15:22:00Z"
  3,Cleo,user,"2025-01-13T09:45:00Z"
```

Task: Return only users with role "user" as TOON. Use the same header format. Set [N] to match the row count. Output only the code block.

Expected output:

users[2]{id,name,role,lastLogin}:
  2,Bob,user,"2025-01-14T15:22:00Z"
  3,Cleo,user,"2025-01-13T09:45:00Z"

The model adjusts [N] to 2 and generates two rows.

Validation with Strict Mode Copy Section

When decoding model-generated TOON, use strict mode (default) to catch errors:

import { decode } from '@toon-format/toon'

try {
  const data = decode(modelOutput, { strict: true })
  // Success – data is valid
}
catch (error) {
  // Model output was malformed (count mismatch, invalid escapes, etc.)
  console.error('Validation failed:', error.message)
}

Strict mode checks counts, indentation, and escaping so you can detect truncation or malformed TOON. For complete details, see the API Reference.

Delimiter Choices for Token Efficiency Copy Chapter

Use delimiter: '\t' for tab-separated tables if you want even fewer tokens. Tabs are single characters, often tokenize more efficiently than commas, and rarely appear in natural text (reducing quote-escaping).

const toon = encode(data, { delimiter: '\t' })

Tell the model "fields are tab-separated" when using tabs. For more on delimiters, see the Format Overview.

Streaming Large Outputs Copy Chapter

When working with large datasets (thousands of records or deeply nested structures), use encodeLines() to stream TOON output line-by-line instead of building the full string in memory.

import { encodeLines } from '@toon-format/toon'

const largeData = await fetchThousandsOfRecords()

// Stream large dataset without loading full string in memory
for (const line of encodeLines(largeData, { delimiter: '\t' })) {
  process.stdout.write(`${line}\n`)
}

The CLI also supports streaming for memory-efficient JSON-to-TOON conversion:

toon large-dataset.json -o output.toon

This streaming approach prevents out-of-memory errors when preparing large context windows for LLMs. For complete details on encodeLines(), see the API Reference.

Consuming streaming LLM outputs: If your LLM client exposes streaming text and you buffer by lines, you can decode TOON incrementally:

import { decodeFromLines } from '@toon-format/toon'

// Buffer streaming response into lines
const lines: string[] = []
let buffer = ''

for await (const chunk of modelStream) {
  buffer += chunk
  let index: number

  while ((index = buffer.indexOf('\n')) !== -1) {
    lines.push(buffer.slice(0, index))
    buffer = buffer.slice(index + 1)
  }
}

// Decode buffered lines
const data = decodeFromLines(lines)

For streaming decode APIs, see decodeFromLines() and decodeStream().

Tips and Pitfalls Copy Chapter

Show, don't describe. Don't explain TOON syntax in detail – just show an example. Models learn the pattern from context. A simple code block with 2–5 rows is more effective than paragraphs of explanation.

Keep examples small. Use 2–5 rows in your examples, not hundreds. The model generalizes from the pattern. Large examples waste tokens without improving accuracy.

Always validate output. Decode generated TOON with strict: true (default) to catch errors early. Don't assume model output is valid TOON without checking.

Real-World Example Copy Chapter

Here's a complete workflow: send data to a model and validate its TOON response.

Prompt with TOON input:

System logs in TOON format (tab-separated):

```toon
events[4	]{id	level	message	timestamp}:
  1	error	Connection timeout	"2025-01-15T10:00:00Z"
  2	warn	Slow query	"2025-01-15T10:05:00Z"
  3	info	User login	"2025-01-15T10:10:00Z"
  4	error	Database error	"2025-01-15T10:15:00Z"
```

Task: Return only error-level events as TOON. Use the same format.

Validate the response:

import { decode } from '@toon-format/toon'

const modelResponse = `
events[2	]{id	level	message	timestamp}:
  1	error	Connection timeout	"2025-01-15T10:00:00Z"
  4	error	Database error	"2025-01-15T10:15:00Z"
`

const filtered = decode(modelResponse, { strict: true })
// ✓ Validated – model correctly filtered and adjusted [N] to 2

File: docs/ecosystem/implementations.md Copy Chapter

Implementations

TOON has official and community implementations across multiple programming languages. All implementations are intended to conform to the same Specification to ensure compatibility and interoperability.

The code examples throughout this documentation site use the TypeScript implementation by default, but the format and concepts apply equally to all languages.

Official Implementations Copy Chapter

Community Implementations Copy Chapter

Community members have created implementations in additional languages:

Contributing an Implementation Copy Chapter

Building a TOON implementation for a new language? Here are the steps:

1. Follow the spec: Implement the latest specification.
2. Add tests: Run the reference test suite – its language-agnostic fixtures validate compatibility across implementations.
3. Document usage: Provide a clear README with installation and usage examples.
4. Share it: Open a PR to add your implementation to the README at github.com/toon-format/toon.

File: docs/ecosystem/tools-and-playgrounds.md Copy Chapter

Tools and Playgrounds

Experiment with TOON format interactively using these tools for token comparison, format conversion, and validation.

Playground Copy Chapter

The TOON Playground lets you convert JSON or YAML to TOON in real time, compare token counts, and share your experiments via URL.

CLI Tool Copy Chapter

The official TOON CLI provides command-line conversion, token statistics, and all encoding/decoding features. See the CLI reference for full documentation.

npx @toon-format/cli input.json --stats -o output.toon

Editor Support Copy Chapter

VS Code Copy Section

TOON Language Support – Syntax highlighting, validation, conversion, and token analysis.

Install from the VS Code Marketplace or via command line:

code --install-extension vishalraut.vscode-toon

Tree-sitter Grammar Copy Section

tree-sitter-toon – Grammar for Tree-sitter-compatible editors (Neovim, Helix, Emacs, Zed).

Neovim Copy Section

toon.nvim – Lua-based plugin for Neovim.

Other Editors Copy Section

Use YAML syntax highlighting as a close approximation. Most editors allow associating .toon files with YAML language mode.

Databases Copy Chapter

ToonStore Copy Section

ToonStore – Redis-compatible embedded database (Rust) that stores data in TOON format.

MCP Copy Chapter

Tooner Copy Section

Tooner – MCP proxy that converts JSON tool responses to TOON.

File: docs/cli/index.md Copy Chapter

Command Line Interface

The @toon-format/cli package converts JSON to TOON and TOON to JSON. Use it to measure token savings before integrating TOON into your application, or to pipe JSON through TOON in shell workflows alongside tools like curl and jq. The CLI supports stdin/stdout, token statistics, streaming for large datasets, and every encoding option in the library.

The CLI is built on the @toon-format/toon TypeScript implementation and follows the latest specification.

Usage Copy Chapter

Without Installation Copy Section

Use npx to run the CLI without installing:

::: code-group

npx @toon-format/cli input.json -o output.toon

npx @toon-format/cli data.toon -o output.json

echo '{"name": "Ada"}' | npx @toon-format/cli

:::

Global Installation Copy Section

Or install globally for repeated use:

::: code-group

npm install @toon-format/cli

pnpm add @toon-format/cli

yarn global add @toon-format/cli

:::

After global installation, use the toon command:

toon input.json -o output.toon

Basic Usage Copy Chapter

Auto-Detection Copy Section

The CLI automatically detects the operation based on file extension:

• .json files → encode (JSON to TOON)
• .toon files → decode (TOON to JSON)

When reading from stdin, use --encode or --decode flags to specify the operation (defaults to encode).

::: code-group

toon input.json -o output.toon

toon data.toon -o output.json

toon input.json

cat data.json | toon
echo '{"name": "Ada"}' | toon

cat data.toon | toon --decode

:::

By convention, TOON files use the .toon extension and the provisional media type text/toon (see spec §17).

Standard Input Copy Section

Omit the input argument or use - to read from stdin. This enables piping data directly from other commands:

# No argument needed
cat data.json | toon

# Explicit stdin with hyphen (equivalent)
cat data.json | toon -

# Decode from stdin
cat data.toon | toon --decode

Performance Copy Chapter

Streaming Output Copy Section

Both encoding and decoding operations use streaming output, writing incrementally without building the full output string in memory. This makes the CLI efficient for large datasets without requiring additional configuration.

JSON → TOON (Encode):

• Streams TOON lines to output.
• No full TOON string in memory.

TOON → JSON (Decode):

• Uses the same event-based streaming decoder as the decodeStream API in @toon-format/toon.
• Streams JSON tokens to output.
• No full JSON string in memory.

Process large files with minimal memory usage:

# Encode large JSON file
toon huge-dataset.json -o output.toon

# Decode large TOON file
toon huge-dataset.toon -o output.json

# Process millions of records efficiently via stdin
cat million-records.json | toon > output.toon
cat million-records.toon | toon --decode > output.json

Peak memory usage scales with data depth, not total size. This allows processing arbitrarily large files as long as individual nested structures fit in memory.

::: tip Token Statistics
When using the --stats flag with encode, the CLI builds the full TOON string once to compute accurate token counts. For maximum memory efficiency on very large files, omit --stats.
:::

Options Copy Chapter

Advanced Examples Copy Chapter

Token Statistics Copy Section

Show token savings when encoding:

toon data.json --stats -o output.toon

Example output:

✔ Encoded data.json → output.toon

ℹ Token estimates: ~15,145 (JSON) → ~8,745 (TOON)
✔ Saved ~6,400 tokens (-42.3%)

Alternative Delimiters Copy Section

TOON supports three delimiters: comma (default), tab, and pipe. Alternative delimiters can save additional tokens depending on the data.

::: code-group

toon data.json --delimiter $'\t' -o output.toon

toon data.json --delimiter "|" -o output.toon

:::

The --delimiter value must be the actual delimiter character. In bash/zsh, use $'\t' to pass a real tab; literal "\t" is rejected as an invalid delimiter.

Tab delimiter example:

::: code-group

items[2	]{id	name	qty	price}:
  A1	Widget	2	9.99
  B2	Gadget	1	14.5

items[2]{id,name,qty,price}:
  A1,Widget,2,9.99
  B2,Gadget,1,14.5

:::

::: tip
Tab delimiters often tokenize more efficiently than commas and reduce the need for quote-escaping. Use --delimiter $'\t' (bash/zsh) for maximum token savings on large tabular data. See Delimiter Strategies for full guidance.
:::

Lenient Decoding Copy Section

Skip validation for faster, more forgiving decoding:

toon data.toon --no-strict -o output.json

With --no-strict, the decoder stops enforcing array count matches, indentation multiples, and header delimiter mismatches. Duplicate sibling keys no longer throw – the last value wins. Malformed array headers fall back to plain key: value lines instead of erroring.

Decode Error Output Copy Section

When a TOON document fails to parse, the CLI renders the offending line with a caret pointing at the first non-whitespace character. Tabs are shown as → so the caret column reflects what the decoder actually saw.

For an input file that uses a tab to indent the second line (rendered here with →):

a:
→b: 1

The CLI prints:

ERROR  Failed to decode TOON at line 2: Tabs are not allowed in indentation in strict mode

  2 | →b: 1
      ^

The exit code is 1 on any error. Stack traces are suppressed by default. Pass --verbose to include the full stack and the underlying cause chain – useful when filing a bug report or diagnosing an unexpected error path:

cat broken.toon | toon --decode --verbose

::: tip Programmatic Access
Decode errors are thrown as ToonDecodeError instances by the library. The CLI's caret rendering is built on the structured line and source fields exposed on that class. See the Error Handling section of the API reference if you want the same diagnostic detail in your own code.
:::

Stdin Workflows Copy Section

The CLI integrates with Unix pipes and other command-line tools:

# Convert API response to TOON
curl https://api.example.com/data | toon --stats

# Process large dataset
cat large-dataset.json | toon --delimiter $'\t' > output.toon

# Chain with jq
jq '.results' data.json | toon > filtered.toon

File: docs/reference/api.md Copy Chapter

API Reference

TypeScript/JavaScript API documentation for the @toon-format/toon package. For format rules, see the Format Overview or the Specification. For other languages, see Implementations.

Installation Copy Chapter

::: code-group

npm install @toon-format/toon

pnpm add @toon-format/toon

yarn add @toon-format/toon

:::

Encoding Functions Copy Chapter

`encode(input, options?)` Copy Section

Converts any JSON-serializable value to TOON format.

import { encode } from '@toon-format/toon'

const toon = encode(data, {
  indentSize: 2,
  delimiter: ','
})

Parameters

Return Value

Returns a TOON-formatted string with no trailing newline or spaces.

Type Normalization

Non-JSON-serializable values are normalized before encoding:

::: info
TOON itself doesn't specify how Date should be encoded – the spec leaves this to implementations. This library emits an ISO 8601 string in quotes; other implementations may choose differently.
:::

Example

import { encode } from '@toon-format/toon'

const items = [
  { sku: 'A1', qty: 2, price: 9.99 },
  { sku: 'B2', qty: 1, price: 14.5 }
]

console.log(encode({ items }))

Output:

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5

`encodeLines(input, options?)` Copy Section

Preferred method for streaming TOON output. Converts any JSON-serializable value to TOON format as a sequence of lines, without building the full string in memory. Suitable for streaming large outputs to files, HTTP responses, or process stdout.

import { encodeLines } from '@toon-format/toon'

// Stream to stdout (Node.js)
for (const line of encodeLines(data)) {
  process.stdout.write(`${line}\n`)
}

// Write to file line-by-line
const lines = encodeLines(data, { indentSize: 2, delimiter: '\t' })
for (const line of lines) {
  await writeToStream(`${line}\n`)
}

// Collect to array
const lineArray = Array.from(encodeLines(data))

Parameters

Return Value

Returns an Iterable<string> that yields TOON lines one at a time. Each yielded string is a single line without a trailing newline character – you must add \n when writing to streams or stdout.

::: info Relationship to encode()
encode(value, options) is equivalent to:

Array.from(encodeLines(value, options)).join('\n')

:::

Example

import { createWriteStream } from 'node:fs'
import { encodeLines } from '@toon-format/toon'

const data = {
  items: Array.from({ length: 100000 }, (_, i) => ({
    id: i,
    name: `Item ${i}`,
    value: Math.random()
  }))
}

// Stream large dataset to file
const stream = createWriteStream('output.toon')
for (const line of encodeLines(data, { delimiter: '\t' })) {
  stream.write(`${line}\n`)
}
stream.end()

Replacer Function Copy Section

The replacer option allows you to transform or filter values during encoding. It works similarly to JSON.stringify's replacer parameter, but with path tracking for more precise control.

Type Signature

type EncodeReplacer = (
  key: string,
  value: JsonValue,
  path: readonly (string | number)[]
) => unknown

Parameters

Return Value

• Return the value unchanged to keep it
• Return a different value to replace it (will be normalized)
• Return undefined to omit properties/array elements
• For root value, undefined means "no change" (root cannot be omitted)

Examples

Filtering sensitive data:

import { encode } from '@toon-format/toon'

const data = {
  user: { name: 'Ada', password: 'secret123', email: '[email protected]' }
}

function replacer(key, value) {
  if (key === 'password')
    return undefined
  return value
}

console.log(encode(data, { replacer }))

Output:

user:
  name: Ada
  email: [email protected]

Transforming values:

const data = { user: 'alice', role: 'admin' }

function replacer(key, value) {
  if (typeof value === 'string')
    return value.toUpperCase()
  return value
}

console.log(encode(data, { replacer }))

Output:

user: ALICE
role: ADMIN

Path-based transformations:

const data = {
  metadata: { created: '2025-01-01' },
  user: { created: '2025-01-02' }
}

function replacer(key, value, path) {
  // Add timezone info only to top-level metadata
  if (path.length === 1 && path[0] === 'metadata' && key === 'created') {
    return `${value}T00:00:00Z`
  }
  return value
}

console.log(encode(data, { replacer }))

Output:

metadata:
  created: "2025-01-01T00:00:00Z"
user:
  created: 2025-01-02

::: info Replacer Execution Order
The replacer is called in a depth-first manner:

1. Root value first (key = '', path = [])
2. Then each property/element (with proper key and path)
3. Values are re-normalized after replacement
4. Children are processed after parent transformation
   :::

::: warning Array Indices as Strings
Following JSON.stringify behavior, array indices are passed as strings ('0', '1', '2', etc.) to the replacer, not as numbers.
:::

Raw String Output Copy Section

Return rawString(...) from a replacer to emit a string verbatim at the value position, bypassing TOON's quoting, escaping, and number/keyword detection. Compose it with escapeString to control quoting yourself without reimplementing escape handling.

import { encode, escapeString, rawString } from '@toon-format/toon'

const data = { name: 'Ada', age: 30 }

// Always-quote mode: wrap every leaf in quotes
console.log(encode(data, {
  replacer: (key, value) => rawString(`"${escapeString(String(value))}"`)
}))

Output:

name: "Ada"
age: "30"

Semantics

• A rawString is only honored where a primitive would go. Returned for an object or array value, it is ignored and the container is encoded normally – this lets "wrap every value" replacers recurse into containers instead of collapsing them.
• A value containing a line whose first non-space character is # throws at rawString(...) time – regardless of where the value would be emitted – since decoders silently strip such comment lines and the data would vanish without an error.

`escapeString(value)`

Escapes backslashes, quotes, and control characters for use inside a quoted TOON string. The decision whether a value needs quoting at all stays with the caller.

escapeString('a "quoted" value') // a \"quoted\" value
escapeString('line1\nline2') // line1\nline2 (escaped)

::: warning One-Way Escape Hatch
Raw emission bypasses the encoder's correctness guarantees: the output is not guaranteed to be valid TOON or to round-trip losslessly. In the example above, age decodes back as the string "30", not the number 30.
:::

Decoding Functions Copy Chapter

`decode(input, options?)` Copy Section

Converts a TOON-formatted string back to JavaScript values.

import { decode } from '@toon-format/toon'

const data = decode(toon, {
  indentSize: 2,
  strict: true
})

Parameters

Return Value

Returns a JavaScript value (object, array, or primitive) representing the parsed TOON data.

Numeric tokens decode to number and follow IEEE 754 double precision: values beyond it round silently (including integers outside the safe integer range), and tokens that overflow the finite range decode as strings – this is the decoder's documented out-of-range policy per spec §4. Values that must stay exact belong in quoted strings; encode() writes out-of-range BigInt values that way automatically.

Example

import { decode } from '@toon-format/toon'

const toon = `
items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5
`

const data = decode(toon)
console.log(data)

Output:

{
  "items": [
    { "sku": "A1", "qty": 2, "price": 9.99 },
    { "sku": "B2", "qty": 1, "price": 14.5 }
  ]
}

`decodeFromLines(lines, options?)` Copy Section

Decodes TOON format from pre-split lines into a JavaScript value. This is a streaming-friendly wrapper around the event-based decoder that builds the full value in memory.

Useful when you already have lines as an array or iterable (e.g., from file streams, readline interfaces, or network responses) and want the standard decode behavior.

Parameters

Return Value

Returns a JsonValue (the parsed JavaScript value: object, array, or primitive).

Example

Basic usage with arrays:

import { decodeFromLines } from '@toon-format/toon'

const lines = ['name: Ada', 'age: 30']
const value = decodeFromLines(lines)
// { name: 'Ada', age: 30 }

Streaming from Node.js readline:

import { createReadStream } from 'node:fs'
import { createInterface } from 'node:readline'
import { decodeFromLines } from '@toon-format/toon'

const rl = createInterface({
  input: createReadStream('data.toon'),
  crlfDelay: Infinity,
})

const value = decodeFromLines(rl)
console.log(value)

Choosing the Right Decoder Copy Section

::: info Key Differences

• Value vs. Events: Functions ending in Stream yield events without building the full value in memory.
• Async support: Only decodeStream() accepts async iterables (useful for file/network streams).
  :::

Streaming Decoders Copy Chapter

`decodeStreamSync(lines, options?)` Copy Section

Synchronously decodes TOON lines into a stream of JSON events. This function yields structured events that represent the JSON data model without building the full value tree.

Useful for streaming processing, custom transformations, or memory-efficient parsing of large datasets where you don't need the full value in memory.

::: tip Event Streaming
This is a low-level API that returns individual parse events. For most use cases, decodeFromLines() or decode() are more convenient.
:::

Parameters

Return Value

Returns an Iterable<JsonStreamEvent> that yields structured events (see TypeScript Types for event structure).

Example

Basic event streaming:

import { decodeStreamSync } from '@toon-format/toon'

const lines = ['name: Ada', 'age: 30']

for (const event of decodeStreamSync(lines)) {
  console.log(event)
}

// Output:
// { type: 'startObject' }
// { type: 'key', key: 'name' }
// { type: 'primitive', value: 'Ada' }
// { type: 'key', key: 'age' }
// { type: 'primitive', value: 30 }
// { type: 'endObject' }

Custom processing:

import { decodeStreamSync } from '@toon-format/toon'

const lines = ['users[2]{id,name}:', '  1,Ada', '  2,Bob']
let userCount = 0

for (const event of decodeStreamSync(lines)) {
  if (event.type === 'endObject' && userCount < 2) {
    userCount++
    console.log(`Processed user ${userCount}`)
  }
}

`decodeStream(source, options?)` Copy Section

Asynchronously decodes TOON lines into a stream of JSON events. This is the async version of decodeStreamSync(), supporting both synchronous and asynchronous iterables.

Useful for processing file streams, network responses, or other async sources where you want to handle data incrementally as it arrives.

Parameters

Return Value

Returns an AsyncIterable<JsonStreamEvent> that yields structured events asynchronously (see TypeScript Types for event structure).

Example

Streaming from file:

import { createReadStream } from 'node:fs'
import { createInterface } from 'node:readline'
import { decodeStream } from '@toon-format/toon'

const fileStream = createReadStream('data.toon', 'utf-8')
const rl = createInterface({ input: fileStream, crlfDelay: Infinity })

for await (const event of decodeStream(rl)) {
  console.log(event)
  // Process events as they arrive
}

Error Handling Copy Chapter

Decoding throws a ToonDecodeError when input cannot be parsed. The class extends SyntaxError, so existing error instanceof SyntaxError checks keep working without code changes.

`ToonDecodeError` Copy Section

import { ToonDecodeError } from '@toon-format/toon'

Fields

The line and source fields are populated for every error that has line context – essentially every parse error during normal decoding. The cause chain points back to the underlying SyntaxError or TypeError thrown by the token-level parser, so debuggers and verbose loggers can show the original frame.

Example

import { decode, ToonDecodeError } from '@toon-format/toon'

try {
  decode('a:\n\tb: 1')
}
catch (error) {
  if (error instanceof ToonDecodeError) {
    console.error(`Line ${error.line}:`, error.source)
    console.error(error.message)
    // Line 2: 	b: 1
    // Line 2: Tabs are not allowed in indentation in strict mode
  }
  else {
    throw error
  }
}

::: info Backwards Compatibility
ToonDecodeError extends SyntaxError. Code written against earlier versions that catches SyntaxError continues to match these errors. The class adds structured fields without removing anything.
:::

Configuration Reference Copy Chapter

`EncodeOptions` Copy Section

Configuration for encode() and encodeLines():

Delimiter options:

::: code-group

encode(data, { delimiter: ',' })

encode(data, { delimiter: '\t' })

encode(data, { delimiter: '|' })

:::

See Delimiter Strategies for guidance on choosing delimiters.

`DecodeOptions` Copy Section

Configuration for decode() and decodeFromLines():

By default (strict: true), the decoder validates input strictly:

• Invalid escape sequences: Throws on \x, unterminated strings, lone-surrogate \uXXXX
• Syntax errors: Throws on missing colons, malformed headers
• Array length mismatches: Throws when declared length doesn't match actual count
• Keyed tabular mismatches: Throws when the entry-row count doesn't match the declared count or a row's cell count doesn't match the header's leaf fields (§9.5)
• Header delimiter mismatch: Throws when the bracket-declared delimiter differs from the field-list delimiter (§14.2)
• Indentation errors: Throws when leading spaces aren't exact multiples of indentSize, on depth jumps of more than one level into a nested scope, and on over-indented lines that belong to no scope (§14.2) – strict decoding never silently discards input, including trailing content after a completed root array or keyed tabular root (§5)
• Header structure: Throws on leading-zero or non-integer array lengths, malformed keyed markers, and intervening content between bracket/fields/colon
• Duplicate sibling keys: Throws when an object has two children with the same key, including duplicate entry keys (§14.3)

All decode errors are thrown as ToonDecodeError instances with structured line and source fields.

Set strict: false to skip these checks. Duplicate sibling keys then resolve with last-write-wins in document order. A declared [N] never truncates a scope: every list item, tabular row, and entry row the scope actually contains is decoded, whether that is fewer or more than N (§14.1).

Five conditions are errors in both modes, because no recovery preserves the document's meaning (§14): a missing colon in key context, an invalid escape or unterminated quoted string, characters after a quoted token's closing quote, a bare token line inside an array or object scope, and a document whose depth-0 lines are neither headers nor key-value lines.

Documented decoder policies. The specification requires each implementation to state the choices it leaves open (§4, §12, §15):

• Numbers out of range: a token matching §4's number grammar whose magnitude exceeds the IEEE 754 double range decodes as a string; one that underflows decodes as numeric 0; one that fits but cannot be represented exactly decodes as the nearest double. Use a replacer or post-process the decoded value when exact decimals matter.
• Tab indentation: rejected in strict mode. With strict: false, leading tabs are indentation and are removed from the line's content; each leading tab contributes one level of depth.
• Object representation: decoded objects are plain JavaScript objects. __proto__, constructor, and prototype are materialized as ordinary own entries and never mutate the prototype chain (§15). JavaScript reorders integer-like keys ahead of string keys, so a document whose keys include integer-like tokens does not preserve document key order (§2).

`DecodeStreamOptions` Copy Section

Configuration for decodeStreamSync() and decodeStream():

TypeScript Types Copy Chapter

`JsonStreamEvent` Copy Section

Events emitted by decodeStreamSync() and decodeStream():

type JsonStreamEvent
  = | { type: 'startObject' }
    | { type: 'endObject' }
    | { type: 'startArray', length: number }
    | { type: 'endArray' }
    | { type: 'key', key: string }
    | { type: 'primitive', value: JsonPrimitive }

Delimiters Copy Section

import { DEFAULT_DELIMITER, DELIMITERS } from '@toon-format/toon'

DEFAULT_DELIMITER // ','
DELIMITERS // { comma: ',', tab: '\t', pipe: '|' }

Option Types Copy Section

Guides & Examples Copy Chapter

Round-Trip Compatibility Copy Section

TOON provides lossless round-trips after normalization:

import { decode, encode } from '@toon-format/toon'

const original = {
  users: [
    { id: 1, name: 'Ada', role: 'admin' },
    { id: 2, name: 'Bob', role: 'user' }
  ]
}

const toon = encode(original)
const restored = decode(toon)

console.log(JSON.stringify(original) === JSON.stringify(restored))
// true

Delimiter Strategies Copy Section

Tab delimiters (\t) often tokenize more efficiently than commas. Tabs are single characters that rarely appear in natural text, which reduces the need for quote-escaping and leads to smaller token counts in large datasets.

Example:

items[2	]{sku	name	qty	price}:
  A1	Widget	2	9.99
  B2	Gadget	1	14.5

For maximum token savings on large tabular data, use tab delimiters:

encode(data, { delimiter: '\t' })

Choosing a Delimiter:

• Comma (,): Default, widely understood, good for simple tabular data.
• Tab (\t): Best for LLM token efficiency, excellent for large datasets.
• Pipe (|): Alternative when commas appear frequently in data.