neon Documentation

1. Project Overview & Quickstart (neondatabase/neon) Copy Chapter

File: README.md Copy Chapter

Neon

Neon is an open-source serverless Postgres database platform. It separates storage and compute and substitutes the PostgreSQL storage layer by redistributing data across a cluster of nodes.

Quick start Copy Chapter

Try the Neon Free Tier to create a serverless Postgres instance. Then connect to it with your preferred Postgres client (psql, dbeaver, etc) or use the online SQL Editor. See Connect from any application for connection instructions.

Alternatively, compile and run the project locally.

Architecture overview Copy Chapter

A Neon installation consists of compute nodes and the Neon storage engine. Compute nodes are stateless PostgreSQL nodes backed by the Neon storage engine.

The Neon storage engine consists of two major components:

• Pageserver: Scalable storage backend for the compute nodes.
• Safekeepers: The safekeepers form a redundant WAL service that received WAL from the compute node, and stores it durably until it has been processed by the pageserver and uploaded to cloud storage.

See developer documentation in SUMMARY.md for more information.

Running a local development environment Copy Chapter

Neon can be run on a workstation for small experiments and to test code changes, by
following these instructions.

Installing dependencies on Linux

1. Install build dependencies and other applicable packages

• On Ubuntu or Debian, this set of packages should be sufficient to build the code:

apt install build-essential libtool libreadline-dev zlib1g-dev flex bison libseccomp-dev \
libssl-dev clang pkg-config libpq-dev cmake postgresql-client protobuf-compiler \
libprotobuf-dev libcurl4-openssl-dev openssl python3-poetry lsof libicu-dev

• On Fedora, these packages are needed:

dnf install flex bison readline-devel zlib-devel openssl-devel \
  libseccomp-devel perl clang cmake postgresql postgresql-contrib protobuf-compiler \
  protobuf-devel libcurl-devel openssl poetry lsof libicu-devel libpq-devel python3-devel \
  libffi-devel

• On Arch based systems, these packages are needed:

pacman -S base-devel readline zlib libseccomp openssl clang \
postgresql-libs cmake postgresql protobuf curl lsof

Building Neon requires 3.15+ version of protoc (protobuf-compiler). If your distribution provides an older version, you can install a newer version from here.

2. Install Rust

# recommended approach from https://www.rust-lang.org/tools/install
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Installing dependencies on macOS (12.3.1)

1. Install XCode and dependencies

xcode-select --install
brew install protobuf openssl flex bison icu4c pkg-config m4

# add openssl to PATH, required for ed25519 keys generation in neon_local
echo 'export PATH="$(brew --prefix openssl)/bin:$PATH"' >> ~/.zshrc

If you get errors about missing m4 you may have to install it manually:

brew install m4
brew link --force m4

2. Install Rust

# recommended approach from https://www.rust-lang.org/tools/install
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

3. Install PostgreSQL Client

# from https://stackoverflow.com/questions/44654216/correct-way-to-install-psql-without-full-postgres-on-macos
brew install libpq
brew link --force libpq

Rustc version

The project uses rust toolchain file to define the version it's built with in CI for testing and local builds.

This file is automatically picked up by rustup that installs (if absent) and uses the toolchain version pinned in the file.

rustup users who want to build with another toolchain can use the rustup override command to set a specific toolchain for the project's directory.

non-rustup users most probably are not getting the same toolchain automatically from the file, so are responsible to manually verify that their toolchain matches the version in the file.
Newer rustc versions most probably will work fine, yet older ones might not be supported due to some new features used by the project or the crates.

Building on Linux

1. Build neon and patched postgres

# Note: The path to the neon sources can not contain a space.

git clone --recursive https://github.com/neondatabase/neon.git
cd neon

# The preferred and default is to make a debug build. This will create a
# demonstrably slower build than a release build. For a release build,
# use "BUILD_TYPE=release make -j`nproc` -s"
# Remove -s for the verbose build log

make -j`nproc` -s

Building on OSX

1. Build neon and patched postgres

# Note: The path to the neon sources can not contain a space.

git clone --recursive https://github.com/neondatabase/neon.git
cd neon

# The preferred and default is to make a debug build. This will create a
# demonstrably slower build than a release build. For a release build,
# use "BUILD_TYPE=release make -j`sysctl -n hw.logicalcpu` -s"
# Remove -s for the verbose build log

make -j`sysctl -n hw.logicalcpu` -s

Dependency installation notes

To run the psql client, install the postgresql-client package or modify PATH and LD_LIBRARY_PATH to include pg_install/bin and pg_install/lib, respectively.

To run the integration tests or Python scripts (not required to use the code), install
Python (3.11 or higher), and install the python3 packages using ./scripts/pysync (requires poetry>=1.8) in the project directory.

Running neon database

1. Start pageserver and postgres on top of it (should be called from repo root):

# Create repository in .neon with proper paths to binaries and data
# Later that would be responsibility of a package install script
> cargo neon init
Initializing pageserver node 1 at '127.0.0.1:64000' in ".neon"

# start pageserver, safekeeper, and broker for their intercommunication
> cargo neon start
Starting neon broker at 127.0.0.1:50051.
storage_broker started, pid: 2918372
Starting pageserver node 1 at '127.0.0.1:64000' in ".neon".
pageserver started, pid: 2918386
Starting safekeeper at '127.0.0.1:5454' in '.neon/safekeepers/sk1'.
safekeeper 1 started, pid: 2918437

# create initial tenant and use it as a default for every future neon_local invocation
> cargo neon tenant create --set-default
tenant 9ef87a5bf0d92544f6fafeeb3239695c successfully created on the pageserver
Created an initial timeline 'de200bd42b49cc1814412c7e592dd6e9' at Lsn 0/16B5A50 for tenant: 9ef87a5bf0d92544f6fafeeb3239695c
Setting tenant 9ef87a5bf0d92544f6fafeeb3239695c as a default one

# create postgres compute node
> cargo neon endpoint create main

# start postgres compute node
> cargo neon endpoint start main
Starting new endpoint main (PostgreSQL v14) on timeline de200bd42b49cc1814412c7e592dd6e9 ...
Starting postgres at 'postgresql://[email protected]:55432/postgres'

# check list of running postgres instances
> cargo neon endpoint list
 ENDPOINT  ADDRESS          TIMELINE                          BRANCH NAME  LSN        STATUS
 main      127.0.0.1:55432  de200bd42b49cc1814412c7e592dd6e9  main         0/16B5BA8  running

2. Now, it is possible to connect to postgres and run some queries:

> psql -p 55432 -h 127.0.0.1 -U cloud_admin postgres
postgres=# CREATE TABLE t(key int primary key, value text);
CREATE TABLE
postgres=# insert into t values(1,1);
INSERT 0 1
postgres=# select * from t;
 key | value
-----+-------
   1 | 1
(1 row)

3. And create branches and run postgres on them:

# create branch named migration_check
> cargo neon timeline branch --branch-name migration_check
Created timeline 'b3b863fa45fa9e57e615f9f2d944e601' at Lsn 0/16F9A00 for tenant: 9ef87a5bf0d92544f6fafeeb3239695c. Ancestor timeline: 'main'

# check branches tree
> cargo neon timeline list
(L) main [de200bd42b49cc1814412c7e592dd6e9]
(L) ┗━ @0/16F9A00: migration_check [b3b863fa45fa9e57e615f9f2d944e601]

# create postgres on that branch
> cargo neon endpoint create migration_check --branch-name migration_check

# start postgres on that branch
> cargo neon endpoint start migration_check
Starting new endpoint migration_check (PostgreSQL v14) on timeline b3b863fa45fa9e57e615f9f2d944e601 ...
Starting postgres at 'postgresql://[email protected]:55434/postgres'

# check the new list of running postgres instances
> cargo neon endpoint list
 ENDPOINT         ADDRESS          TIMELINE                          BRANCH NAME      LSN        STATUS
 main             127.0.0.1:55432  de200bd42b49cc1814412c7e592dd6e9  main             0/16F9A38  running
 migration_check  127.0.0.1:55434  b3b863fa45fa9e57e615f9f2d944e601  migration_check  0/16F9A70  running

# this new postgres instance will have all the data from 'main' postgres,
# but all modifications would not affect data in original postgres
> psql -p 55434 -h 127.0.0.1 -U cloud_admin postgres
postgres=# select * from t;
 key | value
-----+-------
   1 | 1
(1 row)

postgres=# insert into t values(2,2);
INSERT 0 1

# check that the new change doesn't affect the 'main' postgres
> psql -p 55432 -h 127.0.0.1 -U cloud_admin postgres
postgres=# select * from t;
 key | value
-----+-------
   1 | 1
(1 row)

4. If you want to run tests afterwards (see below), you must stop all the running pageserver, safekeeper, and postgres instances
   you have just started. You can terminate them all with one command:

> cargo neon stop

More advanced usages can be found at Local Development Control Plane (neon_local)).

Handling build failures

If you encounter errors during setting up the initial tenant, it's best to stop everything (cargo neon stop) and remove the .neon directory. Then fix the problems, and start the setup again.

Running tests Copy Chapter

Rust unit tests Copy Section

We are using cargo-nextest to run the tests in Github Workflows.
Some crates do not support running plain cargo test anymore, prefer cargo nextest run instead.
You can install cargo-nextest with cargo install cargo-nextest.

Integration tests Copy Section

Ensure your dependencies are installed as described here.

git clone --recursive https://github.com/neondatabase/neon.git

CARGO_BUILD_FLAGS="--features=testing" make

./scripts/pytest

By default, this runs both debug and release modes, and all supported postgres versions. When
testing locally, it is convenient to run just one set of permutations, like this:

DEFAULT_PG_VERSION=17 BUILD_TYPE=release ./scripts/pytest

Flamegraphs Copy Chapter

You may find yourself in need of flamegraphs for software in this repository.
You can use flamegraph-rs or the original flamegraph.pl. Your choice!

[!IMPORTANT]
If you're using lld or mold, you need the --no-rosegment linker argument.
It's a general thing with Rust / lld / mold, not specific to this repository.
See this PR for further instructions.

Cleanup Copy Chapter

For cleaning up the source tree from build artifacts, run make clean in the source directory.

For removing every artifact from build and configure steps, run make distclean, and also consider removing the cargo binaries in the target directory, as well as the database in the .neon directory. Note that removing the .neon directory will remove your database, with all data in it. You have been warned!

Documentation Copy Chapter

docs Contains a top-level overview of all available markdown documentation.

• sourcetree.md contains overview of source tree layout.

To view your rustdoc documentation in a browser, try running cargo doc --no-deps --open

See also README files in some source directories, and rustdoc style documentation comments.

Other resources:

• SELECT 'Hello, World': Blog post by Nikita Shamgunov on the high level architecture
• Architecture decisions in Neon: Blog post by Heikki Linnakangas
• Neon: Serverless PostgreSQL!: Presentation on storage system by Heikki Linnakangas in the CMU Database Group seminar series

Postgres-specific terms Copy Section

Due to Neon's very close relation with PostgreSQL internals, numerous specific terms are used.
The same applies to certain spelling: i.e. we use MB to denote 1024 * 1024 bytes, while MiB would be technically more correct, it's inconsistent with what PostgreSQL code and its documentation use.

To get more familiar with this aspect, refer to:

• Neon glossary
• PostgreSQL glossary
• Other PostgreSQL documentation and sources (Neon fork sources can be found here)

Join the development Copy Chapter

• Read CONTRIBUTING.md to learn about project code style and practices.
• To get familiar with a source tree layout, use sourcetree.md.
• To learn more about PostgreSQL internals, check http://www.interdb.jp/pg/index.html

File: compute/etc/README.md Copy Chapter

Compute Configuration

These files are the configuration files for various other pieces of software
that will be running in the compute alongside Postgres.

`sql_exporter` Copy Chapter

Adding a `sql_exporter` Metric Copy Section

We use sql_exporter to export various metrics from Postgres. In order to add
a metric, you will need to create two files: a libsonnet and a sql file. You
will then import the libsonnet file in one of the collector files, and the
sql file will be imported in the libsonnet file.

In the event your statistic is an LSN, you may want to cast it to a float8
because Prometheus only supports floats. It's probably fine because float8 can
store integers from -2^53 to +2^53 exactly.

File: compute/README.md Copy Chapter

This directory contains files that are needed to build the compute
images, or included in the compute images.

compute-node.Dockerfile
To build the compute image

vm-image-spec.yaml
Instructions for vm-builder, to turn the compute-node image into
corresponding vm-compute-node image.

etc/
Configuration files included in /etc in the compute image

patches/
Some extensions need to be patched to work with Neon. This
directory contains such patches. They are applied to the extension
sources in compute-node.Dockerfile

In addition to these, postgres itself, the neon postgres extension,
and compute_ctl are built and copied into the compute image by
compute-node.Dockerfile.

File: compute_tools/README.md Copy Chapter

Compute node tools

Postgres wrapper (compute_ctl) is intended to be run as a Docker entrypoint or as a systemd
ExecStart option. It will handle all the Neon specifics during compute node
initialization:

• compute_ctl accepts cluster (compute node) specification as a JSON file.
• Every start is a fresh start, so the data directory is removed and
  initialized again on each run.
• Next it will put configuration files into the PGDATA directory.
• Sync safekeepers and get commit LSN.
• Get basebackup from pageserver using the returned on the previous step LSN.
• Try to start postgres and wait until it is ready to accept connections.
• Check and alter/drop/create roles and databases.
• Hang waiting on the postmaster process to exit.

Also compute_ctl spawns two separate service threads:

• compute-monitor checks the last Postgres activity timestamp and saves it
  into the shared ComputeNode;
• http-endpoint runs a Hyper HTTP API server, which serves readiness and the
  last activity requests.

If AUTOSCALING environment variable is set, compute_ctl will start the
vm-monitor located in [neon/libs/vm_monitor]. For VM compute nodes,
vm-monitor communicates with the VM autoscaling system. It coordinates
downscaling and requests immediate upscaling under resource pressure.

Usage example:

compute_ctl -D /var/db/postgres/compute \
            -C 'postgresql://cloud_admin@localhost/postgres' \
            -S /var/db/postgres/specs/current.json \
            -b /usr/local/bin/postgres

State Diagram Copy Chapter

Computes can be in various states. Below is a diagram that details how a
compute moves between states.

%% https://mermaid.js.org/syntax/stateDiagram.html
stateDiagram-v2
  [*] --> Empty : Compute spawned
  Empty --> ConfigurationPending : Waiting for compute spec
  ConfigurationPending --> Configuration : Received compute spec
  Configuration --> Failed : Failed to configure the compute
  Configuration --> Running : Compute has been configured
  Empty --> Init : Compute spec is immediately available
  Empty --> TerminationPendingFast : Requested termination
  Empty --> TerminationPendingImmediate : Requested termination
  Init --> Failed : Failed to start Postgres
  Init --> Running : Started Postgres
  Running --> TerminationPendingFast : Requested termination
  Running --> TerminationPendingImmediate : Requested termination
  Running --> ConfigurationPending : Received a /configure request with spec
  Running --> RefreshConfigurationPending : Received a /refresh_configuration request, compute node will pull a new spec and reconfigure
  RefreshConfigurationPending --> RefreshConfiguration: Received compute spec and started configuration
  RefreshConfiguration --> Running : Compute has been re-configured
  RefreshConfiguration --> RefreshConfigurationPending : Configuration failed and to be retried
  TerminationPendingFast --> Terminated compute with 30s delay for cplane to inspect status
  TerminationPendingImmediate --> Terminated : Terminated compute immediately
  Failed --> RefreshConfigurationPending : Received a /refresh_configuration request
  Failed --> [*] : Compute exited
  Terminated --> [*] : Compute exited

Tests Copy Chapter

Cargo formatter:

cargo fmt

Run tests:

cargo test

Clippy linter:

cargo clippy --all --all-targets -- -Dwarnings -Drust-2018-idioms

Cross-platform compilation Copy Chapter

Imaging that you are on macOS (x86) and you want a Linux GNU (x86_64-unknown-linux-gnu platform in rust terminology) executable.

Using docker Copy Section

You can use a throw-away Docker container (rustlang/rust image) for doing that:

docker run --rm \
    -v $(pwd):/compute_tools \
    -w /compute_tools \
    -t rustlang/rust:nightly cargo build --release --target=x86_64-unknown-linux-gnu

or one-line:

docker run --rm -v $(pwd):/compute_tools -w /compute_tools -t rust:latest cargo build --release --target=x86_64-unknown-linux-gnu

Using rust native cross-compilation Copy Section

Another way is to add x86_64-unknown-linux-gnu target on your host system:

rustup target add x86_64-unknown-linux-gnu

Install macOS cross-compiler toolchain:

brew tap SergioBenitez/osxct
brew install x86_64-unknown-linux-gnu

And finally run cargo build:

CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-unknown-linux-gnu-gcc cargo build --target=x86_64-unknown-linux-gnu --release

File: control_plane/README.md Copy Chapter

Local Development Control Plane (`neon_local`)

This crate contains tools to start a Neon development environment locally. This utility can be used with the cargo neon command. This is a convenience to invoke
the neon_local binary.

Note: this is a dev/test tool -- a minimal control plane suitable for testing
code changes locally, but not suitable for running production systems.

Example: Start with Postgres 16 Copy Chapter

To create and start a local development environment with Postgres 16, you will need to provide --pg-version flag to 2 of the start-up commands.

cargo neon init
cargo neon start
cargo neon tenant create --set-default --pg-version 16
cargo neon endpoint create main --pg-version 16
cargo neon endpoint start main

Example: Create Test User and Database Copy Chapter

By default, cargo neon starts an endpoint with cloud_admin and postgres database. If you want to have a role and a database similar to what we have on the cloud service, you can do it with the following commands when starting an endpoint.

cargo neon endpoint create main --pg-version 16 --update-catalog true
cargo neon endpoint start main --create-test-user true

The first command creates neon_superuser and necessary roles. The second command creates test user and neondb database. You will see a connection string that connects you to the test user after running the second command.

File: docker-compose/ext-src/README.md Copy Chapter

PostgreSQL Extensions for Testing

This directory contains PostgreSQL extensions used primarily for:

1. Testing extension upgrades between different Compute versions
2. Running regression tests with regular users (mostly for cloud instances)

Directory Structure Copy Chapter

Each extension directory follows a standard structure:

• extension-name-src/ - Directory containing test files for the extension
  • test-upgrade.sh - Script for testing upgrade scenarios
  • regular-test.sh - Script for testing with regular users
  • Additional test files depending on the extension

Available Extensions Copy Chapter

This directory includes the following extensions:

• hll-src - HyperLogLog, a fixed-size data structure for approximating cardinality
• hypopg-src - Extension to create hypothetical indexes
• ip4r-src - IPv4/v6 and subnet data types
• pg_cron-src - Run periodic jobs in PostgreSQL
• pg_graphql-src - GraphQL support for PostgreSQL
• pg_hint_plan-src - Execution plan hints
• pg_ivm-src - Incremental view maintenance
• pg_jsonschema-src - JSON Schema validation
• pg_repack-src - Reorganize tables with minimal locks
• pg_roaringbitmap-src - Roaring bitmap implementation
• pg_semver-src - Semantic version data type
• pg_session_jwt-src - JWT authentication for PostgreSQL
• pg_tiktoken-src - OpenAI Tiktoken tokenizer
• pg_uuidv7-src - UUIDv7 implementation for PostgreSQL
• pgjwt-src - JWT tokens for PostgreSQL
• pgrag-src - Retrieval Augmented Generation for PostgreSQL
• pgtap-src - Unit testing framework for PostgreSQL
• pgvector-src - Vector similarity search
• pgx_ulid-src - ULID data type
• plv8-src - JavaScript language for PostgreSQL stored procedures
• postgresql-unit-src - SI units for PostgreSQL
• prefix-src - Prefix matching for strings
• rag_bge_small_en_v15-src - BGE embedding model for RAG
• rag_jina_reranker_v1_tiny_en-src - Jina reranker model for RAG
• rum-src - RUM access method for text search

Usage Copy Chapter

Extension Upgrade Testing Copy Section

The extensions in this directory are used by the test-upgrade.sh script to test upgrading extensions between different versions of Neon Compute nodes. The script:

1. Creates a database with extensions installed on an old Compute version
2. Creates timelines for each extension
3. Switches to a new Compute version and tests the upgrade process
4. Verifies extension functionality after upgrade

Regular User Testing Copy Section

For testing with regular users (particularly for cloud instances), each extension directory typically contains a regular-test.sh script that:

1. Drops the database if it exists
2. Creates a fresh test database
3. Installs the extension
4. Runs regression tests

A note about pg_regress: Since pg_regress attempts to set lc_messages for the database by default, which is forbidden for regular users, we create databases manually and use the --use-existing option to bypass this limitation.

CI Workflows Copy Section

Two main workflows use these extensions:

1. Cloud Extensions Test - Tests extensions on Neon cloud projects
2. Force Test Upgrading of Extension - Tests upgrading extensions between different Compute versions

These workflows are integrated into the build-and-test pipeline through shell scripts:

• docker_compose_test.sh - Tests extensions in a Docker Compose environment

• test_extensions_upgrade.sh - Tests extension upgrades between different Compute versions

Adding New Extensions Copy Chapter

To add a new extension for testing:

1. Create a directory named extension-name-src in this directory
2. Add at minimum:
   • regular-test.sh for testing with regular users
   • If regular-test.sh doesn't exist, the system will look for neon-test.sh
   • If neither exists, it will try to run make installcheck
   • test-upgrade.sh is only needed if you want to test upgrade scenarios
3. Update the list of extensions in the test_extensions_upgrade.sh script if needed for upgrade testing

Patching Extension Sources Copy Section

If you need to patch the extension sources:

1. Place the patch file in the extension's directory
2. Apply the patch in the appropriate script (test-upgrade.sh, neon-test.sh, regular-test.sh, or Makefile)
3. The patch will be applied during the testing process

File: docker-compose/README.md Copy Chapter

Example docker compose configuration

The configuration in this directory is used for testing Neon docker images: it is
not intended for deploying a usable system. To run a development environment where
you can experiment with a miniature Neon system, use cargo neon rather than container images.

This configuration does not start the storage controller, because the controller
needs a way to reconfigure running computes, and no such thing exists in this setup.

Generating the JWKS for a compute Copy Chapter

openssl genpkey -algorithm Ed25519 -out private-key.pem
openssl pkey -in private-key.pem -pubout -out public-key.pem
openssl pkey -pubin -inform pem -in public-key.pem -pubout -outform der -out public-key.der
key="$(xxd -plain -cols 32 -s -32 public-key.der)"
key_id="$(printf '%s' "$key" | sha256sum | awk '{ print $1 }' | basenc --base64url --wrap=0)"
x="$(printf '%s' "$key" | basenc --base64url --wrap=0)"

File: docs/rfcs/001-cluster-size-limits.md Copy Chapter

Cluster size limits

Summary Copy Chapter

One of the resource consumption limits for free-tier users is a cluster size limit.

To enforce it, we need to calculate the timeline size and check if the limit is reached before relation create/extend operations.
If the limit is reached, the query must fail with some meaningful error/warning.
We may want to exempt some operations from the quota to allow users free space to fit back into the limit.

The stateless compute node that performs validation is separate from the storage that calculates the usage, so we need to exchange cluster size information between those components.

Motivation Copy Chapter

Limit the maximum size of a PostgreSQL instance to limit free tier users (and other tiers in the future).
First of all, this is needed to control our free tier production costs.
Another reason to limit resources is risk management — we haven't (fully) tested and optimized neon for big clusters,
so we don't want to give users access to the functionality that we don't think is ready.

Components Copy Chapter

• pageserver - calculate the size consumed by a timeline and add it to the feedback message.
• safekeeper - pass feedback message from pageserver to compute.
• compute - receive feedback message, enforce size limit based on GUC neon.max_cluster_size.
• console - set and update neon.max_cluster_size setting

Proposed implementation Copy Chapter

First of all, it's necessary to define timeline size.

The current approach is to count all data, including SLRUs. (not including WAL)
Here we think of it as a physical disk underneath the Postgres cluster.
This is how the LOGICAL_TIMELINE_SIZE metric is implemented in the pageserver.

Alternatively, we could count only relation data. As in pg_database_size().
This approach is somewhat more user-friendly because it is the data that is really affected by the user.
On the other hand, it puts us in a weaker position than other services, i.e., RDS.
We will need to refactor the timeline_size counter or add another counter to implement it.

Timeline size is updated during wal digestion. It is not versioned and is valid at the last_received_lsn moment.
Then this size should be reported to compute node.

current_timeline_size value is included in the walreceiver's custom feedback message: ReplicationFeedback.

(PR about protocol changes https://github.com/neondatabase/neon/pull/1037).

This message is received by the safekeeper and propagated to compute node as a part of AppendResponse.

Finally, when compute node receives the current_timeline_size from safekeeper (or from pageserver directly), it updates the global variable.

And then every neon_extend() operation checks if limit is reached (current_timeline_size > neon.max_cluster_size) and throws ERRCODE_DISK_FULL error if so.
(see Postgres error codes https://www.postgresql.org/docs/devel/errcodes-appendix.html)

TODO:
We can allow autovacuum processes to bypass this check, simply checking IsAutoVacuumWorkerProcess().
It would be nice to allow manual VACUUM and VACUUM FULL to bypass the check, but it's uneasy to distinguish these operations at the low level.
See issues https://github.com/neondatabase/neon/issues/1245
https://github.com/neondatabase/neon/issues/1445

TODO:
We should warn users if the limit is soon to be reached.

**Reliability, failure modes and corner cases** Copy Section

1. current_timeline_size is valid at the last received and digested by pageserver lsn.

   If pageserver lags behind compute node, current_timeline_size will lag too. This lag can be tuned using backpressure, but it is not expected to be 0 all the time.

   So transactions that happen in this lsn range may cause limit overflow. Especially operations that generate (i.e., CREATE DATABASE) or free (i.e., TRUNCATE) a lot of data pages while generating a small amount of WAL. Are there other operations like this?

   Currently, CREATE DATABASE operations are restricted in the console. So this is not an issue.

**Security implications** Copy Section

We treat compute as an untrusted component. That's why we try to isolate it with secure container runtime or a VM.
Malicious users may change the neon.max_cluster_size, so we need an extra size limit check.
To cover this case, we also monitor the compute node size in the console.

File: docs/rfcs/002-storage.md Copy Chapter

Neon storage node — alternative

**Design considerations** Copy Chapter

Simplify storage operations for people => Gain adoption/installs on laptops and small private installation => Attract customers to DBaaS by seamless integration between our tooling and cloud.

Proposed architecture addresses:

• High availability -- tolerates n/2 - 1 failures
• Multi-tenancy -- one storage for all databases
• Elasticity -- increase storage size on the go by adding nodes
• Snapshots / backups / PITR with S3 offload
• Compression

Minuses are:

• Quite a lot of work
• Single page access may touch few disk pages
• Some bloat in data — may slowdown sequential scans

**Summary** Copy Chapter

Storage cluster is sharded key-value store with ordered keys. Key (page_key) is a tuple of (pg_id, db_id, timeline_id, rel_id, forkno, segno, pageno, lsn). Value is either page or page diff/wal. Each chunk (chunk == shard) stores approx 50-100GB and automatically splits in half when grows bigger then soft 100GB limit. by having a fixed range of pageno's it is responsible for. Chunks placement on storage nodes is stored in a separate metadata service, so chunk can be freely moved around the cluster if it is need. Chunk itself is a filesystem directory with following sub directories:

|-chunk_42/
  |-store/ -- contains lsm with pages/pagediffs ranging from
  |	      page_key_lo to page_key_hi
  |-wal/
  |  |- db_1234/ db-specific wal files with pages from page_key_lo
  |		 to page_key_hi
  |
  |-chunk.meta -- small file with snapshot references
		  (page_key_prefix+lsn+name)
		  and PITR regions (page_key_start, page_key_end)

**Chunk** Copy Chapter

Chunk is responsible for storing pages potentially from different databases and relations. Each page is addressed by a lexicographically ordered tuple (page_key) with following fields:

• pg_id -- unique id of given postgres instance (or postgres cluster as it is called in postgres docs)
• db_id -- database that was created by 'CREATE DATABASE' in a given postgres instance
• db_timeline -- used to create Copy-on-Write instances from snapshots, described later
• rel_id -- tuple of (relation_id, 0) for tables and (indexed_relation_id, rel_id) for indices. Done this way so table indices were closer to table itself on our global key space.
• (forkno, segno, pageno) -- page coordinates in postgres data files
• lsn_timeline -- postgres feature, increments when PITR was done.
• lsn -- lsn of current page version.

Chunk stores pages and page diffs ranging from page_key_lo to page_key_hi. Processing node looks at page in wal record and sends record to a chunk responsible for this page range. When wal record arrives to a chunk it is initially stored in chunk_id/wal/db_id/wal_segno.wal. Then background process moves records from that wal files to the lsm tree in chunk_id/store. Or, more precisely, wal records would be materialized into lsm memtable and when that memtable is flushed to SSTable on disk we may trim the wal. That way some not durably (in the distributed sense) committed pages may enter the tree -- here we rely on processing node behavior: page request from processing node should contain proper lsm horizons so that storage node may respond with proper page version.

LSM here is a usual LSM for variable-length values: at first data is stored in memory (we hold incoming wal records to be able to regenerate it after restart) at some balanced tree. When this tree grows big enough we dump it into disk file (SSTable) sorting records by key. Then SStables are mergesorted in the background to a different files. All file operation are sequential and do not require WAL for durability.

Content of SSTable can be following:

(pg_id, db_id, ... , pageno=42, lsn=100) (full 8k page data)
(pg_id, db_id, ... , pageno=42, lsn=150) (per-page diff)
(pg_id, db_id, ... , pageno=42, lsn=180) (per-page diff)
(pg_id, db_id, ... , pageno=42, lsn=200) (per-page diff)
(pg_id, db_id, ... , pageno=42, lsn=220) (full 8k page data)
(pg_id, db_id, ... , pageno=42, lsn=250) (per-page diff)
(pg_id, db_id, ... , pageno=42, lsn=270) (per-page diff)
(pg_id, db_id, ... , pageno=5000, lsn=100) (full 8k page data)

So query for pageno=42 up to lsn=260 would need to find closest entry less then this key, iterate back to the latest full page and iterate forward to apply diffs. How often page is materialized in lsn-version sequence is up to us -- let's say each 5th version should be a full page.

**Page deletion** Copy Section

To delete old pages we insert blind deletion marker (pg_id, db_id, #trim_lsn < 150) into a lsm tree. During merges such marker would indicate that all pages with smaller lsn should be discarded. Delete marker will travel down the tree levels hierarchy until it reaches last level. In non-PITR scenario where old page version are not needed at all such deletion marker would (in average) prevent old page versions propagation down the tree -- so all bloat would concentrate at higher tree layers without affecting bigger bottom layers.

**Recovery** Copy Section

Upon storage node restart recent WAL files are applied to appropriate pages and resulting pages stored in lsm memtable. So this should be fast since we are not writing anything to disk.

**Checkpointing** Copy Section

No such mechanism is needed. Or we may look at the storage node as at kind of continuous checkpointer.

**Full page writes (torn page protection)** Copy Section

Storage node never updates individual pages, only merges SSTable, so torn pages is not an issue.

**Snapshot** Copy Section

That is the part that I like about this design -- snapshot creation is instant and cheap operation that can have flexible granularity level: whole instance, database, table. Snapshot creation inserts a record in chunk.meta file with lsn of this snapshot and key prefix (pg_id, db_id, db_timeline, rel_id, *) that prohibits pages deletion within this range. Storage node may not know anything about page internals, but by changing number of fields in our prefix we may change snapshot granularity.

It is again useful to remap rel_id to (indexed_relation_id, rel_id) so that snapshot of relation would include it's indices. Also table snapshot would trickily interact with catalog. Probably all table snapshots should hold also a catalog snapshot. And when node is started with such snapshot it should check that only tables from snapshot are queried. I assume here that for snapshot reading one need to start a new postgres instance.

Storage consumed by snapshot is proportional to the amount of data changed. We may have some heuristic (calculated based on cost of different storages) about when to offload old snapshot to s3. For example, if current database has more then 40% of changed pages with respect to previous snapshot then we may offload that snapshot to s3, and release this space.

Starting db from snapshot

When we are starting database from snapshot it can be done in two ways. First, we may create new db_id, move all the data from snapshot to a new db and start a database. Second option is to create Copy-on-Write (CoW) instance out of snapshot and read old pages from old snapshot and store new pages separately. That is why there is db_timeline key field near db_id -- CoW (🐮) database should create new db_timeline and remember old db_timeline. Such a database can have hashmap of pages that it is changed to query pages from proper snapshot on the first try. db_timeline is located near db_id so that new page versions generated by new instance would not bloat data of initial snapshot. It is not clear for whether it is possibly to effectively support "stacked" CoW snapshot, so we may disallow them. (Well, one way to support them is to move db_timeline close to lsn -- so we may scan neighboring pages and find right one. But again that way we bloat snapshot with unrelated data and may slowdown full scans that are happening in different database).

Snapshot export/import

Once we may start CoW instances it is easy to run auxiliary postgres instance on this snapshot and run COPY FROM (...) TO stdout or pg_dump and export data from the snapshot to some portable formats. Also we may start postgres on a new empty database and run COPY FROM stdin. This way we can initialize new non-CoW databases and transfer snapshots via network.

**PITR area** Copy Section

In described scheme PITR is just a prohibition to delete any versions within some key prefix, either it is a database or a table key prefix. So PITR may have different settings for different tables, databases, etc.

PITR is quite bloaty, so we may aggressively offload it to s3 -- we may push same (or bigger) SSTables to s3 and maintain lsm structure there.

**Compression** Copy Section

Since we are storing page diffs of variable sizes there is no structural dependency on a page size and we may compress it. Again that could be enabled only on pages with some key prefixes, so we may have this with db/table granularity.

**Chunk metadata** Copy Section

Chunk metadata is a file lies in chunk directory that stores info about current snapshots and PITR regions. Chunk should always consult this data when merging SSTables and applying delete markers.

**Chunk splitting** Copy Section

(NB: following paragraph is about how to avoid page splitting)

When chunks hits some soft storage limit (let's say 100Gb) it should be split in half and global metadata about chunk boundaries should be updated. Here i assume that chunk split is a local operation happening on single node. Process of chink splitting should look like following:

1. Find separation key and spawn two new chunks with [lo, mid) [mid, hi) boundaries.

2. Prohibit WAL deletion and old SSTables deletion on original chunk.

3. On each lsm layer we would need to split only one SSTable, all other would fit within left or right range. Symlink/split that files to new chunks.

4. Start WAL replay on new chunks.

5. Update global metadata about new chunk boundaries.

6. Eventually (metadata update should be pushed to processing node by metadata service) storage node will start sending WAL and page requests to the new nodes.

7. New chunk may start serving read queries when following conditions are met:

a) it receives at least on WAL record from processing node

b) it replayed all WAL up to the new received one

c) checked by downlinks that there were no WAL gaps.

Chunk split as it is described here is quite fast operation when it is happening on the local disk -- vast majority of files will be just moved without copying anything. I suggest to keep split always local and not to mix it with chunk moving around cluster. So if we want to split some chunk but there is small amount of free space left on the device, we should first move some chunks away from the node and then proceed with splitting.

Fixed chunks Copy Section

Alternative strategy is to not to split at all and have pageno-fixed chunk boundaries. When table is created we first materialize this chunk by storing first new pages only and chunks is small. Then chunk is growing while table is filled, but it can't grow substantially bigger then allowed pageno range, so at max it would be 1GB or whatever limit we want + some bloat due to snapshots and old page versions.

**Chunk lsm internals** Copy Section

So how to implement chunk's lsm?

• Write from scratch and use RocksDB to prototype/benchmark, then switch to own lsm implementation. RocksDB can provide some sanity check for performance of home-brewed implementation and it would be easier to prototype.
• Use postgres as lego constructor. We may model memtable with postgres B-tree referencing some in-memory log of incoming records. SSTable merging may reuse postgres external merging algorithm, etc. One thing that would definitely not fit (or I didn't came up with idea how to fit that) -- is multi-tenancy. If we are storing pages from different databases we can't use postgres buffer pool, since there is no db_id in the page header. We can add new field there but IMO it would be no go for committing that to vanilla.

Other possibility is to not to try to fit few databases in one storage node. But that way it is no go for multi-tenant cloud installation: we would need to run a lot of storage node instances on one physical storage node, all with it own local page cache. So that would be much closer to ordinary managed RDS.

Multi-tenant storage makes sense even on a laptop, when you work with different databases, running tests with temp database, etc. And when installation grows bigger it start to make more and more sense, so it seems important.

Storage fleet

**Storage fleet**

• When database is smaller then a chunk size we naturally can store them in one chunk (since their page_key would fit in some chunk's [hi, lo) range).

Few databases are stored in one chunk, replicated three times

• When database can't fit into one storage node it can occupy lots of chunks that were split while database was growing. Chunk placement on nodes is controlled by us with some automatization, but we always may manually move chunks around the cluster.

Here one big database occupies two set of nodes. Also some chunks were moved around to restore replication factor after disk failure. In this case we also have "sharded" storage for a big database and issue wal writes to different chunks in parallel.

**Chunk placement strategies** Copy Chapter

There are few scenarios where we may want to move chunks around the cluster:

• disk usage on some node is big
• some disk experienced a failure
• some node experienced a failure or need maintenance

**Chunk replication** Copy Chapter

Chunk replication may be done by cloning page ranges with respect to some lsn from peer nodes, updating global metadata, waiting for WAL to come, replaying previous WAL and becoming online -- more or less like during chunk split.

File: docs/rfcs/003-laptop-cli.md Copy Chapter

Command line interface (end-user)

Neon CLI as it is described here mostly resides on the same conceptual level as pg_ctl/initdb/pg_recvxlog/etc and replaces some of them in an opinionated way. I would also suggest bundling our patched postgres inside neon distribution at least at the start.

This proposal is focused on managing local installations. For cluster operations, different tooling would be needed. The point of integration between the two is storage URL: no matter how complex cluster setup is it may provide an endpoint where the user may push snapshots.

The most important concept here is a snapshot, which can be created/pushed/pulled/exported. Also, we may start temporary read-only postgres instance over any local snapshot. A more complex scenario would consist of several basic operations over snapshots.

Possible usage scenarios

Install neon, run a postgres Copy Chapter

> brew install pg-neon 
> neon pg create # creates pgdata with default pattern pgdata$i
> neon pg list
ID            PGDATA        USED    STORAGE            ENDPOINT
primary1      pgdata1       0G      neon-local       localhost:5432

Import standalone postgres to neon Copy Chapter

> neon snapshot import --from=basebackup://replication@localhost:5432/ oldpg
[====================------------] 60% | 20MB/s
> neon snapshot list
ID          SIZE        PARENT
oldpg       5G          -

> neon pg create --snapshot oldpg
Started postgres on localhost:5432

> neon pg list
ID            PGDATA        USED    STORAGE            ENDPOINT
primary1      pgdata1       5G      neon-local       localhost:5432

> neon snapshot destroy oldpg
Ok

Also, we may start snapshot import implicitly by looking at snapshot schema

> neon pg create --snapshot basebackup://replication@localhost:5432/
Downloading snapshot... Done.
Started postgres on localhost:5432
Destroying snapshot... Done.

Pull snapshot with some publicly shared database Copy Chapter

Since we may export the whole snapshot as one big file (tar of basebackup, maybe with some manifest) it may be shared over conventional means: http, ssh, git+lfs.

> neon pg create --snapshot http://learn-postgres.com/movies_db.neon movies

Create snapshot and push it to the cloud Copy Chapter

> neon snapshot create pgdata1@snap1
> neon snapshot push --to ssh://[email protected] pgdata1@snap1

Rollback database to the snapshot Copy Chapter

One way to rollback the database is just to init a new database from the snapshot and destroy the old one. But creating a new database from a snapshot would require a copy of that snapshot which is time consuming operation. Another option that would be cool to support is the ability to create the copy-on-write database from the snapshot without copying data, and store updated pages in a separate location, however that way would have performance implications. So to properly rollback the database to the older state we have neon pg checkout.

> neon pg list
ID            PGDATA        USED    STORAGE            ENDPOINT
primary1      pgdata1       5G      neon-local       localhost:5432

> neon snapshot create pgdata1@snap1

> neon snapshot list
ID                    SIZE        PARENT
oldpg                 5G          -
pgdata1@snap1         6G          -
pgdata1@CURRENT       6G          -

> neon pg checkout pgdata1@snap1
Stopping postgres on pgdata1.
Rolling back pgdata1@CURRENT to pgdata1@snap1.
Starting postgres on pgdata1.

> neon snapshot list
ID                    SIZE        PARENT
oldpg                 5G          -
pgdata1@snap1         6G          -
pgdata1@HEAD{0}       6G          -
pgdata1@CURRENT       6G          -

Some notes: pgdata1@CURRENT -- implicit snapshot representing the current state of the database in the data directory. When we are checking out some snapshot CURRENT will be set to this snapshot and the old CURRENT state will be named HEAD{0} (0 is the number of postgres timeline, it would be incremented after each such checkout).

Configure PITR area (Point In Time Recovery). Copy Chapter

PITR area acts like a continuous snapshot where you can reset the database to any point in time within this area (by area I mean some TTL period or some size limit, both possibly infinite).

> neon pitr create --storage s3tank --ttl 30d --name pitr_last_month

Resetting the database to some state in past would require creating a snapshot on some lsn / time in this pirt area.

Manual

storage Copy Chapter

Storage is either neon pagestore or s3. Users may create a database in a pagestore and create/move snapshots and pitr regions in both pagestore and s3. Storage is a concept similar to git remote. After installation, I imagine one local storage is available by default.

neon storage attach -t [native|s3] -c key=value -n name

Attaches/initializes storage. For --type=s3, user credentials and path should be provided. For --type=native we may support --path=/local/path and --url=neon.tech/stas/mystore. Other possible term for native is 'zstore'.

neon storage list

Show currently attached storages. For example:

> neon storage list
NAME            USED    TYPE                OPTIONS          PATH
local           5.1G    neon-local                         /opt/neon/store/local
local.compr     20.4G   neon-local        compression=on    /opt/neon/store/local.compr
zcloud          60G     neon-remote                        neon.tech/stas/mystore
s3tank          80G     S3

neon storage detach

neon storage show

pg Copy Chapter

Manages postgres data directories and can start postgres instances with proper configuration. An experienced user may avoid using that (except pg create) and configure/run postgres by themselves.

Pg is a term for a single postgres running on some data. I'm trying to avoid separation of datadir management and postgres instance management -- both that concepts bundled here together.

neon pg create [--no-start --snapshot --cow] -s storage-name -n pgdata

Creates (initializes) new data directory in given storage and starts postgres. I imagine that storage for this operation may be only local and data movement to remote location happens through snapshots/pitr.

--no-start: just init datadir without creating

--snapshot snap: init from the snapshot. Snap is a name or URL (neon.tech/stas/mystore/snap1)

--cow: initialize Copy-on-Write data directory on top of some snapshot (makes sense if it is a snapshot of currently running a database)

neon pg destroy

neon pg start [--replica] pgdata

Start postgres with proper extensions preloaded/installed.

neon pg checkout

Rollback data directory to some previous snapshot.

neon pg stop pg_id

neon pg list

ROLE                 PGDATA        USED    STORAGE            ENDPOINT
primary              my_pg         5.1G    local              localhost:5432
replica-1                                                     localhost:5433
replica-2                                                     localhost:5434
primary              my_pg2        3.2G    local.compr        localhost:5435
-                    my_pg3        9.2G    local.compr        -

neon pg show

my_pg:
    storage: local
    space used on local: 5.1G
    space used on all storages: 15.1G
    snapshots:
        on local:
            snap1: 1G
            snap2: 1G
        on zcloud:
            snap2: 1G
        on s3tank:
            snap5: 2G
    pitr:
        on s3tank:
            pitr_one_month: 45G

neon pg start-rest/graphql pgdata

Starts REST/GraphQL proxy on top of postgres master. Not sure we should do that, just an idea.

snapshot Copy Chapter

Snapshot creation is cheap -- no actual data is copied, we just start retaining old pages. Snapshot size means the amount of retained data, not all data. Snapshot name looks like pgdata_name@tag_name. tag_name is set by the user during snapshot creation. There are some reserved tag names: CURRENT represents the current state of the data directory; HEAD{i} represents the data directory state that resided in the database before i-th checkout.

neon snapshot create pgdata_name@snap_name

Creates a new snapshot in the same storage where pgdata_name exists.

neon snapshot push --to url pgdata_name@snap_name

Produces binary stream of a given snapshot. Under the hood starts temp read-only postgres over this snapshot and sends basebackup stream. Receiving side should start neon snapshot recv before push happens. If url has some special schema like neon:// receiving side may require auth start neon snapshot recv on the go.

neon snapshot recv

Starts a port listening for a basebackup stream, prints connection info to stdout (so that user may use that in push command), and expects data on that socket.

neon snapshot pull --from url or path

Connects to a remote neon/s3/file and pulls snapshot. The remote site should be neon service or files in our format.

neon snapshot import --from basebackup://<...> or path

Creates a new snapshot out of running postgres via basebackup protocol or basebackup files.

neon snapshot export

Starts read-only postgres over this snapshot and exports data in some format (pg_dump, or COPY TO on some/all tables). One of the options may be neon own format which is handy for us (but I think just tar of basebackup would be okay).

neon snapshot diff snap1 snap2

Shows size of data changed between two snapshots. We also may provide options to diff schema/data in tables. To do that start temp read-only postgreses.

neon snapshot destroy

pitr Copy Chapter

Pitr represents wal stream and ttl policy for that stream

XXX: any suggestions on a better name?

neon pitr create name

--ttl = inf | period

--size-limit = inf | limit

--storage = storage_name

neon pitr extract-snapshot pitr_name --lsn xxx

Creates a snapshot out of some lsn in PITR area. The obtained snapshot may be managed with snapshot routines (move/send/export)

neon pitr gc pitr_name

Force garbage collection on some PITR area.

neon pitr list

neon pitr destroy

console Copy Chapter

neon console

Opens browser targeted at web console with the more or less same functionality as described here.

2. Official Technical Reference & Guides (neondatabase/website) Copy Chapter

Neon Next.js

Table of Contents Copy Chapter

• Getting Started
• Usage
  • Learn more
  • Build the website
• Project Structure
• Code Style
  • ESLint
  • Prettier
  • VS Code
• Docs

Getting Started Copy Chapter

1. Clone this repository

git clone [email protected]:neondatabase/website.git

2. Install dependencies

npm install

3. Fill environment variables

cp .env.example .env

4. Check for environment variables

You can find this information on the following internal Neon Notion page: How to build the Neon website locally.

Usage Copy Chapter

Run the website Copy Section

npm run dev

Open http://localhost:3000 with your browser to see the result.

You can start editing the page by modifying src/app/page.jsx. The page auto-updates as you edit the file.

Build the website Copy Section

npm run build

Run the built website Copy Section

npm run start

Checks broken links Copy Section

npm run check:broken-links -- https://neon.com

The command may take time, be patient. You can also specify which part of the website you want to check by passing a specific URL, for example https://neon.com/docs for checking the Docs

N.B. The automatic check is done every Monday at midnight by GitHub CI. You can find the reports on the "Actions" tab

Learn More Copy Section

To learn more about Next.js, take a look at the following resources:

• Next.js Documentation - learn about Next.js features and API.
• Learn Next.js - an interactive Next.js tutorial.

You can check out the Next.js GitHub repository - your feedback and contributions are welcome!

Deploy on Vercel Copy Section

The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.

Check out our Next.js deployment documentation for more details.

Project Structure Copy Chapter

├── content — Documentation sources in `mdbook` format, see [Documentation](#docs) section
├── public
├── src
│   ├── app
│   ├── components
│   │  ├── pages — React components that are being used specifically on a certain page
│   │  └── shared — React components that are being used across the whole website
│   ├── hooks
│   ├── icons
│   ├── images
│   ├── lib
│   ├── scripts
│   ├── styles
│   ├── utils
├── next.config.js — Main configuration file for a Next.js site. Read more about it [here](https://nextjs.org/docs/api-reference/next.config.js/introduction)
├── postcss.config.js — Main configuration file of PostCSS. [Read more about it here](https://tailwindcss.com/docs/configuration#generating-a-post-css-configuration-file)
└── tailwind.config.js — Main configuration file for Tailwind CSS [Read more about it here](https://tailwindcss.com/docs/configuration)

Component Folder Structure Copy Chapter

Each component includes Copy Section

1. Main JavaScript file
2. Index file

Each component optionally may include Copy Section

1. Folder with images and icons
2. Folder with data

Also, each component may include another component that follows all above listed rules.

Example structure Copy Section

component
├── nested-component
│  ├── data
│  │  └── nested-component-lottie-data.json
│  ├── images
│  │  ├── nested-component-image.jpg
│  │  ├── nested-component-inline-svg.inline.svg
│  │  └── nested-component-url-svg.url.svg
│  ├── nested-component.js
│  └── index.js
├── data
│  └── component-lottie-data.json
├── images
│  ├── component-image.jpg
│  ├── component-inline-svg.inline.svg
│  └── component-url-svg.url.svg
├── component.js
└── index.js

Code Style Copy Chapter

ESLint Copy Section

ESLint helps find and fix code style issues and force developers to follow same rules. Current configuration is based on Airbnb style guide.

Additional commands:

npm run lint

Run it to check the current status of eslint issues across project.

npm run lint:fix

Run it to fix all possible issues.

Prettier Copy Section

Prettier helps to format code based on defined rules. Difference between Prettier and ESLint.

Additional commands:

npm run format

Run it to format all files across the project.

VS Code Copy Section

Following extensions required to simplify the process of keeping the same code style across the project:

• ESLint
• Prettier

After installation enable "ESLint on save" by adding to your VS Code settings.json the following line:

"editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
}

You can navigate to settings.json by using Command Pallete (CMD+Shift+P) and then type "Open settings.json".

To enable Prettier go to Preferences -> Settings -> type "Format". Then check that you have esbenp.prettier-vscode as default formatter, and also enable "Format On Save".

Reload VS Code and auto-format will work for you.

Docs Copy Chapter

Documentation for docs is described in content/docs/README.md