File: README.md Copy Chapter

Polars: Extremely fast Query Engine for DataFrames Copy Chapter

Polars is an analytical query engine for DataFrames, written in Rust. It is designed to be fast,
easy to use and expressive. Key features are:

• Fast: written from the ground up in Rust with multi-threaded, vectorized (SIMD) execution
• Lazy & eager execution: with query optimization out of the box
• Larger-than-RAM: the streaming engine processes datasets that don't fit in memory
• Expressive API: compose complex queries with powerful expressions
• Extensible: extend Polars natively with custom code through
  I/O and Expression plugins
• Multi-language: bindings for Python, Rust, Node.js, R, and SQL
• GPU support: optionally accelerate queries on NVIDIA GPUs
• Interoperable: uses the
  Apache Arrow Columnar Format for zero-copy
  data sharing

To learn more, read the user guide.

Polars in action Copy Chapter

Queries are composed from expressions. This lazy query gets optimized out of the box and runs in
parallel across all available cores:

import polars as pl

df = (
    pl.scan_parquet("orders.parquet")
    .filter(pl.col("status") == "shipped")
    .group_by("customer_id")
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.len().alias("n_orders"),
    )
    .sort("total", descending=True)
    .collect()
)

Performance Copy Chapter

Polars is very fast. In fact, it is one of the best performing Dataframe solutions available. See
the PDS-H benchmarks results.

Handles larger-than-RAM data Copy Section

If you have data that does not fit into memory, Polars' query engine is able to process your query
(or parts of your query) in a streaming fashion. This drastically reduces memory requirements, so
you might be able to process your 250GB dataset on your laptop. Collect with
collect(engine='streaming') to run the query streaming.

Installation Copy Chapter

Python Copy Section

Install the latest Polars version with:

pip install polars

See the User Guide for more details
on optional dependencies

Compile Polars from source

If you want a bleeding edge release you should compile Polars from source. Advanced users can also
compile for maximum performance for their architecture.

This can be done by going through the following steps in sequence:

1. Install the latest Rust compiler
2. Install maturin: pip install maturin
3. cd py-polars and choose one of the following:
   • make build, slow binary with debug assertions and limited symbols, fast compile times
   • make build-debug, same as make build, but with all symbols, produces large binaries
   • make build-release, fast binary without debug assertions, minimal debug symbols, long compile
     times
   • make build-nodebug-release, same as build-release but without any debug symbols, slightly
     faster to compile
   • make build-debug-release, same as build-release but with full debug symbols, slightly slower
     to compile
   • make build-dist-release, fastest binary, extreme compile times

By default the binary is compiled with optimizations turned on for a modern CPU. Specify LTS_CPU=1
with the command if your CPU is older and does not support e.g. AVX2.

Note that the Rust crate implementing the Python bindings is called py-polars to distinguish from
the wrapped Rust crate polars itself. However, both the Python package and the Python module are
named polars, so you can pip install polars and import polars.

Check the Installation guide for more advanced
installations. For example when you expect more than 2^32 (~4.2 billion) rows, run on an old CPU
(e.g. dating from before 2011), or on an x86-64 build of Python on Apple Silicon under Rosetta.

Contributing Copy Chapter

Want to contribute? Read our contributing guide
and check the issue tracker for accepted issues.

Contributors new to the codebase can look for the good first issue label to get familiar with the
project.

You can join the Polars Discord server for any help along the way.

Distributed Polars Copy Chapter

Running into hardware limitations executing your queries? Read how you can
horizontally scale your Polars query on a cluster.

License Copy Chapter

Polars is licensed under the MIT License (SPDX: MIT).

File: crates/polars-arrow/src/io/README.md Copy Chapter

IO module

This document describes the overall design of this module.

Rules: Copy Chapter

• Each directory in this module corresponds to a specific format such as csv and json.

• directories that depend on external dependencies MUST be feature gated, with a feature named with
  a prefix io_.

• modules MUST re-export any API of external dependencies they require as part of their public API.
  E.g.

  • if a module as an API write(writer: &mut csv:Writer<W>, ...), it MUST contain
    pub use csv::Writer;.

  The rational is that adding this crate to cargo.toml must be sufficient to use it.

• Each directory SHOULD contain two directories, read and write, corresponding to functionality
  about reading from the format and writing to the format respectively.

• The base module SHOULD contain use pub read; and use pub write;.

• Implementations SHOULD separate reading of "data" from reading of "metadata". Examples:

  • schema read or inference SHOULD be a separate function
  • functions that read "data" SHOULD consume a schema typically pre-read.

• Implementations SHOULD separate IO-bounded operations from CPU-bounded operations. I.e.
  implementations SHOULD:

  • contain functions that consume a Read implementor and output a "raw" struct, i.e. a struct
    that is e.g. compressed and serialized
  • contain functions that consume a "raw" struct and convert it into Arrow.
  • offer each of these functions as independent public APIs, so that consumers can decide how to
    balance CPU-bounds and IO-bounds.

File: crates/polars-arrow/src/doc/lib.md Copy Chapter

Welcome to polars_arrow's documentation. Thanks for checking it out!

This is a library for efficient in-memory data operations with
Arrow in-memory format. It is a re-write from
the bottom up of the official arrow crate with soundness and type safety in mind.

Check out the guide for an
introduction. Below is an example of some of the things you can do with it:

use std::sync::Arc;

use polars_arrow::array::*;
use polars_arrow::datatypes::{Field, DataType, Schema};
use polars_arrow::compute::arithmetics;
use polars_arrow::error::Result;
use polars_arrow::io::parquet::write::*;
use polars_arrow::chunk::Chunk;

fn main() -> Result<()> {
    // declare arrays
    let a = Int32Array::from(&[Some(1), None, Some(3)]);
    let b = Int32Array::from(&[Some(2), None, Some(6)]);

    // compute (probably the fastest implementation of a nullable op you can find out there)
    let c = arithmetics::basic::mul_scalar(&a, &2);
    assert_eq!(c, b);

    // declare a schema with fields
    let schema = Schema::from(vec![
        Field::new("c1", DataType::Int32, true),
        Field::new("c2", DataType::Int32, true),
    ]);

    // declare chunk
    let chunk = Chunk::new(vec![a.arced(), b.arced()]);

    // write to parquet (probably the fastest implementation of writing to parquet out there)

    let options = WriteOptions {
        write_statistics: true,
        compression: CompressionOptions::Snappy,
        version: Version::V1,
        data_page_size: None,
    };

    let row_groups = RowGroupIterator::try_new(
        vec![Ok(chunk)].into_iter(),
        &schema,
        options,
        vec![vec![Encoding::Plain], vec![Encoding::Plain]],
    )?;

    // anything implementing `std::io::Write` works
    let mut file = vec![];

    let mut writer = FileWriter::try_new(file, schema, options)?;

    // Write the file.
    for group in row_groups {
        writer.write(group?)?;
    }
    let _ = writer.end(None)?;
    Ok(())
}

Cargo features Copy Chapter

This crate has a significant number of cargo features to reduce compilation time and number of
dependencies. The feature "full" activates most functionality, such as:

• io_ipc: to interact with the Arrow IPC format
• io_ipc_compression: to read and write compressed Arrow IPC (v2)
• io_flight to read and write to Arrow's Flight protocol
• compute to operate on arrays (addition, sum, sort, etc.)

The feature simd (not part of full) produces more explicit SIMD instructions via
std::simd, but requires the nightly
channel.

File: crates/polars-arrow/src/compute/README.md Copy Chapter

Design

This document outlines the design guide lines of this module.

This module is composed by independent operations common in analytics. Below are some design of its
principles:

• APIs MUST return an error when either:

  • The arguments are incorrect
  • The execution results in a predictable error (e.g. divide by zero)

• APIs MAY error when an operation overflows (e.g. i32 + i32)

• kernels MUST NOT have side-effects

• kernels MUST NOT take ownership of any of its arguments (i.e. everything must be a reference).

• APIs SHOULD error when an operation on variable sized containers can overflow the maximum size of
  usize.

• Kernels SHOULD use the arrays' logical type to decide whether kernels can be applied on an array.
  For example, Date32 + Date32 is meaningless and SHOULD NOT be implemented.

• Kernels SHOULD be implemented via clone, slice or the iterator API provided by Buffer,
  Bitmap, Vec or MutableBitmap.

• Kernels MUST NOT use any API to read bits other than the ones provided by Bitmap.

• Implementations SHOULD aim for auto-vectorization, which is usually accomplished via
  from_trusted_len_iter.

• Implementations MUST feature-gate any implementation that requires external dependencies

• When a kernel accepts dynamically-typed arrays, it MUST expect them as &dyn Array.

• When an API returns &dyn Array, it MUST return Box<dyn Array>. The rational is that a Box is
  mutable, while an Arc is not. As such, Box offers the most flexible API to consumers and the
  compiler. Users can cast a Box into Arc via .into().

File: crates/polars-arrow/src/array/README.md Copy Chapter

Array module

This document describes the overall design of this module.

Notation: Copy Chapter

• "array" in this module denotes any struct that implements the trait Array.
• "mutable array" in this module denotes any struct that implements the trait MutableArray.
• words in code denote existing terms on this implementation.

Arrays: Copy Chapter

• Every arrow array with a different physical representation MUST be implemented as a struct or
  generic struct.

• An array MAY have its own module. E.g. primitive/mod.rs

• An array with a null bitmap MUST implement it as Option

• An array MUST be #[derive(Clone)]

• The trait Array MUST only be implemented by structs in this module.

• Every child array on the struct MUST be Box<dyn Array>.

• An array MUST implement try_new(...) -> Self. This method MUST error iff the data does not
  follow the arrow specification, including any sentinel types such as utf8.

• An array MAY implement unsafe try_new_unchecked that skips validation steps that are O(N).

• An array MUST implement either new_empty() or new_empty(DataType) that returns a zero-len of
  Self.

• An array MUST implement either new_null(length: usize) or new_null(DataType, length: usize)
  that returns a valid array of length length whose all elements are null.

• An array MAY implement value(i: usize) that returns the value at slot i ignoring the validity
  bitmap.

• functions to create new arrays from native Rust SHOULD be named as follows:

  • from: from a slice of optional values (e.g. AsRef<[Option] for BooleanArray)
  • from_slice: from a slice of values (e.g. AsRef<[bool]> for BooleanArray)
  • from_trusted_len_iter from an iterator of trusted len of optional values
  • from_trusted_len_values_iter from an iterator of trusted len of values
  • try_from_trusted_len_iter from an fallible iterator of trusted len of optional values

Slot offsets Copy Section

• An array MUST have a offset: usize measuring the number of slots that the array is currently
  offsetted by if the specification requires.

• An array MUST implement fn slice(&self, offset: usize, length: usize) -> Self that returns an
  offsetted and/or truncated clone of the array. This function MUST increase the array's offset if
  it exists.

• Conversely, offset MUST only be changed by slice.

The rational of the above is that it enable us to be fully interoperable with the offset logic
supported by the C data interface, while at the same time easily perform array slices within Rust's
type safety mechanism.

Mutable Arrays Copy Section

• An array MAY have a mutable counterpart. E.g. MutablePrimitiveArray<T> is the mutable
  counterpart of PrimitiveArray<T>.

• Arrays with mutable counterparts MUST have its own module, and have the mutable counterpart
  declared in {module}/mutable.rs.

• The trait MutableArray MUST only be implemented by mutable arrays in this module.

• A mutable array MUST be #[derive(Debug)]

• A mutable array with a null bitmap MUST implement it as Option<MutableBitmap>

• Converting a MutableArray to its immutable counterpart MUST be O(1). Specifically:

  • it must not allocate
  • it must not cause O(N) data transformations

  This is achieved by converting mutable versions to immutable counterparts (e.g.
  MutableBitmap -> Bitmap).

  The rational is that MutableArrays can be used to perform in-place operations under the arrow
  spec.

File: crates/polars-arrow/src/scalar/README.md Copy Chapter

Scalar API

Design choices:

`Scalar` is trait object Copy Section

There are three reasons:

• a scalar should have a small memory footprint, which an enum would not ensure given the different
  physical types available.
• forward-compatibility: a new entry on an enum is backward-incompatible
• do not expose implementation details to users (reduce the surface of the public API)

`Scalar` MUST contain nullability information Copy Section

This is to be aligned with the general notion of arrow's Array.

This API is a companion to the Array, and follows the same design as Array. Specifically, a
Scalar is a trait object that can be downcasted to concrete implementations.

Like Array, Scalar implements

• dtype, which is used to perform the correct downcast
• is_valid, to tell whether the scalar is null or not

There is one implementation per arrows' physical type Copy Section

• Reduces the number of match that users need to write
• Allows casting of logical types without changing the underlying physical type

File: crates/polars-arrow/src/README.md Copy Chapter

Crate's design

This document describes the design of this module, and thus the overall crate. Each module MAY have
its own design document, that concerns specifics of that module, and if yes, it MUST be on each
module's README.md.

Equality Copy Chapter

Array equality is not defined in the Arrow specification. This crate follows the intent of the
specification, but there is no guarantee that this no verification that this equals e.g. C++'s
definition.

There is a single source of truth about whether two arrays are equal, and that is via their equality
operators, defined on the module array/equal.

Implementation MUST use these operators for asserting equality, so that all testing follows the same
definition of array equality.

Error handling Copy Chapter

• Errors from an external dependency MUST be encapsulated on External.
• Errors from IO MUST be encapsulated on Io.
• This crate MAY return NotYetImplemented when the functionality does not exist, or it MAY panic
  with unimplemented!.

Logical and physical types Copy Chapter

There is a strict separation between physical and logical types:

• physical types MUST be implemented via generics
• logical types MUST be implemented via variables (whose value is e.g. an enum)
• logical types MUST be declared and implemented on the datatypes module

Source of undefined behavior Copy Chapter

There is one, and only one, acceptable source of undefined behavior: FFI. It is impossible to prove
that data passed via pointers are safe for consumption (only a promise from the specification).

File: crates/polars-compute/README.md Copy Chapter

polars-compute

polars-compute is an internal sub-crate of the Polars
library, supplying private compute kernels.

Important Note: This crate is not intended for external usage. Please refer to the main
Polars crate for intended usage.

File: crates/polars-core/README.md Copy Chapter

polars-core

polars-core is an internal sub-crate of the Polars library,
providing its core functionalities.

Important Note: This crate is not intended for external usage. Please refer to the main
Polars crate for intended usage.

File: crates/polars-dylib/README.md Copy Chapter

Polars dynamic library

# Cargo.toml
[workspace.dependencies.polars]
package = "polars-dylib"

# .cargo/config.toml
[build]
rustflags = [
  "-C",
  "prefer-dynamic",
]