## File: README.md

<h1 align="center">
  <a href="https://pola.rs">
    
  </a>
</h1>





## Polars: Extremely fast Query Engine for DataFrames

Polars is an analytical query engine for DataFrames, written in Rust. It is designed to be fast,
easy to use and expressive. Key features are:

- **Fast**: written from the ground up in Rust with multi-threaded, vectorized (SIMD) execution
- **Lazy & eager execution**: with query optimization out of the box
- **Larger-than-RAM**: the streaming engine processes datasets that don't fit in memory
- **Expressive API**: compose complex queries with powerful expressions
- **Extensible**: extend Polars natively with custom code through
  [I/O and Expression plugins](https://docs.pola.rs/user-guide/plugins/)
- **Multi-language**: bindings for Python, Rust, Node.js, R, and SQL
- **GPU support**: optionally accelerate queries on NVIDIA GPUs
- **Interoperable**: uses the
  [Apache Arrow Columnar Format](https://arrow.apache.org/docs/format/Columnar.html) for zero-copy
  data sharing

To learn more, read the [user guide](https://docs.pola.rs/).

## Polars in action

Queries are composed from expressions. This lazy query gets optimized out of the box and runs in
parallel across all available cores:

```python
import polars as pl

df = (
    pl.scan_parquet("orders.parquet")
    .filter(pl.col("status") == "shipped")
    .group_by("customer_id")
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.len().alias("n_orders"),
    )
    .sort("total", descending=True)
    .collect()
)
```

## Performance

Polars is very fast. In fact, it is one of the best performing Dataframe solutions available. See
the [PDS-H benchmarks](https://www.pola.rs/benchmarks.html) results.

### Handles larger-than-RAM data

If you have data that does not fit into memory, Polars' query engine is able to process your query
(or parts of your query) in a streaming fashion. This drastically reduces memory requirements, so
you might be able to process your 250GB dataset on your laptop. Collect with
`collect(engine='streaming')` to run the query streaming.

## Installation

### Python

Install the latest Polars version with:

```sh
pip install polars
```

See the [User Guide](https://docs.pola.rs/user-guide/installation/#feature-flags) for more details
on optional dependencies


**Compile Polars from source**

If you want a bleeding edge release you should compile Polars from source. Advanced users can also
compile for maximum performance for their architecture.

This can be done by going through the following steps in sequence:

1. Install the latest [Rust compiler](https://www.rust-lang.org/tools/install)
2. Install [maturin](https://maturin.rs/): `pip install maturin`
3. `cd py-polars` and choose one of the following:
   - `make build`, slow binary with debug assertions and limited symbols, fast compile times
   - `make build-debug`, same as `make build`, but with all symbols, produces large binaries
   - `make build-release`, fast binary without debug assertions, minimal debug symbols, long compile
     times
   - `make build-nodebug-release`, same as build-release but without any debug symbols, slightly
     faster to compile
   - `make build-debug-release`, same as build-release but with full debug symbols, slightly slower
     to compile
   - `make build-dist-release`, fastest binary, extreme compile times

By default the binary is compiled with optimizations turned on for a modern CPU. Specify `LTS_CPU=1`
with the command if your CPU is older and does not support e.g. AVX2.

Note that the Rust crate implementing the Python bindings is called `py-polars` to distinguish from
the wrapped Rust crate `polars` itself. However, both the Python package and the Python module are
named `polars`, so you can `pip install polars` and `import polars`.



Check the [Installation guide](https://docs.pola.rs/user-guide/installation/) for more advanced
installations. For example when you expect more than 2^32 (~4.2 billion) rows, run on an old CPU
(e.g. dating from before 2011), or on an `x86-64` build of Python on Apple Silicon under Rosetta.

## Contributing

Want to contribute? Read our [contributing guide](https://docs.pola.rs/development/contributing/)
and check the issue tracker for accepted issues.

Contributors new to the codebase can look for the `good first issue` label to get familiar with the
project.

You can [join the Polars Discord server](https://discord.gg/4UfP5cfBE7) for any help along the way.

## Distributed Polars

Running into hardware limitations executing your queries? Read how you can
[horizontally scale your Polars query on a cluster](https://docs.pola.rs/polars-cloud/).

## License

Polars is licensed under the [MIT License](LICENSE) (SPDX: `MIT`).

---

## File: crates/polars-arrow/src/io/README.md

# IO module

This document describes the overall design of this module.

## Rules:

- Each directory in this module corresponds to a specific format such as `csv` and `json`.
- directories that depend on external dependencies MUST be feature gated, with a feature named with
  a prefix `io_`.
- modules MUST re-export any API of external dependencies they require as part of their public API.
  E.g.
  - if a module as an API `write(writer: &mut csv:Writer<W>, ...)`, it MUST contain
    `pub use csv::Writer;`.

  The rational is that adding this crate to `cargo.toml` must be sufficient to use it.
- Each directory SHOULD contain two directories, `read` and `write`, corresponding to functionality
  about reading from the format and writing to the format respectively.
- The base module SHOULD contain `use pub read;` and `use pub write;`.
- Implementations SHOULD separate reading of "data" from reading of "metadata". Examples:
  - schema read or inference SHOULD be a separate function
  - functions that read "data" SHOULD consume a schema typically pre-read.
- Implementations SHOULD separate IO-bounded operations from CPU-bounded operations. I.e.
  implementations SHOULD:
  - contain functions that consume a `Read` implementor and output a "raw" struct, i.e. a struct
    that is e.g. compressed and serialized
  - contain functions that consume a "raw" struct and convert it into Arrow.
  - offer each of these functions as independent public APIs, so that consumers can decide how to
    balance CPU-bounds and IO-bounds.

---

## File: crates/polars-arrow/src/doc/lib.md

Welcome to polars_arrow's documentation. Thanks for checking it out!

This is a library for efficient in-memory data operations with
[Arrow in-memory format](https://arrow.apache.org/docs/format/Columnar.html). It is a re-write from
the bottom up of the official `arrow` crate with soundness and type safety in mind.

Check out [the guide](https://jorgecarleitao.github.io/polars_arrow/main/guide/) for an
introduction. Below is an example of some of the things you can do with it:

```rust
use std::sync::Arc;

use polars_arrow::array::*;
use polars_arrow::datatypes::{Field, DataType, Schema};
use polars_arrow::compute::arithmetics;
use polars_arrow::error::Result;
use polars_arrow::io::parquet::write::*;
use polars_arrow::chunk::Chunk;

fn main() -> Result<()> {
    // declare arrays
    let a = Int32Array::from(&[Some(1), None, Some(3)]);
    let b = Int32Array::from(&[Some(2), None, Some(6)]);

    // compute (probably the fastest implementation of a nullable op you can find out there)
    let c = arithmetics::basic::mul_scalar(&a, &2);
    assert_eq!(c, b);

    // declare a schema with fields
    let schema = Schema::from(vec![
        Field::new("c1", DataType::Int32, true),
        Field::new("c2", DataType::Int32, true),
    ]);

    // declare chunk
    let chunk = Chunk::new(vec![a.arced(), b.arced()]);

    // write to parquet (probably the fastest implementation of writing to parquet out there)

    let options = WriteOptions {
        write_statistics: true,
        compression: CompressionOptions::Snappy,
        version: Version::V1,
        data_page_size: None,
    };

    let row_groups = RowGroupIterator::try_new(
        vec![Ok(chunk)].into_iter(),
        &schema,
        options,
        vec![vec![Encoding::Plain], vec![Encoding::Plain]],
    )?;

    // anything implementing `std::io::Write` works
    let mut file = vec![];

    let mut writer = FileWriter::try_new(file, schema, options)?;

    // Write the file.
    for group in row_groups {
        writer.write(group?)?;
    }
    let _ = writer.end(None)?;
    Ok(())
}
```

## Cargo features

This crate has a significant number of cargo features to reduce compilation time and number of
dependencies. The feature `"full"` activates most functionality, such as:

- `io_ipc`: to interact with the Arrow IPC format
- `io_ipc_compression`: to read and write compressed Arrow IPC (v2)
- `io_flight` to read and write to Arrow's Flight protocol
- `compute` to operate on arrays (addition, sum, sort, etc.)

The feature `simd` (not part of `full`) produces more explicit SIMD instructions via
[`std::simd`](https://doc.rust-lang.org/nightly/std/simd/index.html), but requires the nightly
channel.

---

## File: crates/polars-arrow/src/compute/README.md

# Design

This document outlines the design guide lines of this module.

This module is composed by independent operations common in analytics. Below are some design of its
principles:

- APIs MUST return an error when either:
  - The arguments are incorrect
  - The execution results in a predictable error (e.g. divide by zero)

- APIs MAY error when an operation overflows (e.g. `i32 + i32`)

- kernels MUST NOT have side-effects

- kernels MUST NOT take ownership of any of its arguments (i.e. everything must be a reference).

- APIs SHOULD error when an operation on variable sized containers can overflow the maximum size of
  `usize`.

- Kernels SHOULD use the arrays' logical type to decide whether kernels can be applied on an array.
  For example, `Date32 + Date32` is meaningless and SHOULD NOT be implemented.

- Kernels SHOULD be implemented via `clone`, `slice` or the `iterator` API provided by `Buffer`,
  `Bitmap`, `Vec` or `MutableBitmap`.

- Kernels MUST NOT use any API to read bits other than the ones provided by `Bitmap`.

- Implementations SHOULD aim for auto-vectorization, which is usually accomplished via
  `from_trusted_len_iter`.

- Implementations MUST feature-gate any implementation that requires external dependencies

- When a kernel accepts dynamically-typed arrays, it MUST expect them as `&dyn Array`.

- When an API returns `&dyn Array`, it MUST return `Box<dyn Array>`. The rational is that a `Box` is
  mutable, while an `Arc` is not. As such, `Box` offers the most flexible API to consumers and the
  compiler. Users can cast a `Box` into `Arc` via `.into()`.

---

## File: crates/polars-arrow/src/array/README.md

# Array module

This document describes the overall design of this module.

## Notation:

- "array" in this module denotes any struct that implements the trait `Array`.
- "mutable array" in this module denotes any struct that implements the trait `MutableArray`.
- words in `code` denote existing terms on this implementation.

## Arrays:

- Every arrow array with a different physical representation MUST be implemented as a struct or
  generic struct.

- An array MAY have its own module. E.g. `primitive/mod.rs`

- An array with a null bitmap MUST implement it as `Option`

- An array MUST be `#[derive(Clone)]`

- The trait `Array` MUST only be implemented by structs in this module.

- Every child array on the struct MUST be `Box<dyn Array>`.

- An array MUST implement `try_new(...) -> Self`. This method MUST error iff the data does not
  follow the arrow specification, including any sentinel types such as utf8.

- An array MAY implement `unsafe try_new_unchecked` that skips validation steps that are `O(N)`.

- An array MUST implement either `new_empty()` or `new_empty(DataType)` that returns a zero-len of
  `Self`.

- An array MUST implement either `new_null(length: usize)` or `new_null(DataType, length: usize)`
  that returns a valid array of length `length` whose all elements are null.

- An array MAY implement `value(i: usize)` that returns the value at slot `i` ignoring the validity
  bitmap.

- functions to create new arrays from native Rust SHOULD be named as follows:
  - `from`: from a slice of optional values (e.g. `AsRef<[Option]` for `BooleanArray`)
  - `from_slice`: from a slice of values (e.g. `AsRef<[bool]>` for `BooleanArray`)
  - `from_trusted_len_iter` from an iterator of trusted len of optional values
  - `from_trusted_len_values_iter` from an iterator of trusted len of values
  - `try_from_trusted_len_iter` from an fallible iterator of trusted len of optional values

### Slot offsets

- An array MUST have a `offset: usize` measuring the number of slots that the array is currently
  offsetted by if the specification requires.

- An array MUST implement `fn slice(&self, offset: usize, length: usize) -> Self` that returns an
  offsetted and/or truncated clone of the array. This function MUST increase the array's offset if
  it exists.

- Conversely, `offset` MUST only be changed by `slice`.

The rational of the above is that it enable us to be fully interoperable with the offset logic
supported by the C data interface, while at the same time easily perform array slices within Rust's
type safety mechanism.

### Mutable Arrays

- An array MAY have a mutable counterpart. E.g. `MutablePrimitiveArray<T>` is the mutable
  counterpart of `PrimitiveArray<T>`.

- Arrays with mutable counterparts MUST have its own module, and have the mutable counterpart
  declared in `{module}/mutable.rs`.

- The trait `MutableArray` MUST only be implemented by mutable arrays in this module.

- A mutable array MUST be `#[derive(Debug)]`

- A mutable array with a null bitmap MUST implement it as `Option<MutableBitmap>`

- Converting a `MutableArray` to its immutable counterpart MUST be `O(1)`. Specifically:
  - it must not allocate
  - it must not cause `O(N)` data transformations

  This is achieved by converting mutable versions to immutable counterparts (e.g.
  `MutableBitmap -> Bitmap`).

  The rational is that `MutableArray`s can be used to perform in-place operations under the arrow
  spec.

---

## File: crates/polars-arrow/src/scalar/README.md

# Scalar API

Design choices:

### `Scalar` is trait object

There are three reasons:

- a scalar should have a small memory footprint, which an enum would not ensure given the different
  physical types available.
- forward-compatibility: a new entry on an `enum` is backward-incompatible
- do not expose implementation details to users (reduce the surface of the public API)

### `Scalar` MUST contain nullability information

This is to be aligned with the general notion of arrow's `Array`.

This API is a companion to the `Array`, and follows the same design as `Array`. Specifically, a
`Scalar` is a trait object that can be downcasted to concrete implementations.

Like `Array`, `Scalar` implements

- `dtype`, which is used to perform the correct downcast
- `is_valid`, to tell whether the scalar is null or not

### There is one implementation per arrows' physical type

- Reduces the number of `match` that users need to write
- Allows casting of logical types without changing the underlying physical type

---

## File: crates/polars-arrow/src/README.md

# Crate's design

This document describes the design of this module, and thus the overall crate. Each module MAY have
its own design document, that concerns specifics of that module, and if yes, it MUST be on each
module's `README.md`.

## Equality

Array equality is not defined in the Arrow specification. This crate follows the intent of the
specification, but there is no guarantee that this no verification that this equals e.g. C++'s
definition.

There is a single source of truth about whether two arrays are equal, and that is via their equality
operators, defined on the module [`array/equal`](array/equal/mod.rs).

Implementation MUST use these operators for asserting equality, so that all testing follows the same
definition of array equality.

## Error handling

- Errors from an external dependency MUST be encapsulated on `External`.
- Errors from IO MUST be encapsulated on `Io`.
- This crate MAY return `NotYetImplemented` when the functionality does not exist, or it MAY panic
  with `unimplemented!`.

## Logical and physical types

There is a strict separation between physical and logical types:

- physical types MUST be implemented via generics
- logical types MUST be implemented via variables (whose value is e.g. an `enum`)
- logical types MUST be declared and implemented on the `datatypes` module

## Source of undefined behavior

There is one, and only one, acceptable source of undefined behavior: FFI. It is impossible to prove
that data passed via pointers are safe for consumption (only a promise from the specification).

---

## File: crates/polars-compute/README.md

# polars-compute

`polars-compute` is an **internal sub-crate** of the [Polars](https://crates.io/crates/polars)
library, supplying private compute kernels.

**Important Note**: This crate is **not intended for external usage**. Please refer to the main
[Polars crate](https://crates.io/crates/polars) for intended usage.

---

## File: crates/polars-core/README.md

# polars-core

`polars-core` is an **internal sub-crate** of the [Polars](https://crates.io/crates/polars) library,
providing its core functionalities.

**Important Note**: This crate is **not intended for external usage**. Please refer to the main
[Polars crate](https://crates.io/crates/polars) for intended usage.

---

## File: crates/polars-dylib/README.md

# Polars dynamic library

```toml
# Cargo.toml
[workspace.dependencies.polars]
package = "polars-dylib"
```

```toml
# .cargo/config.toml
[build]
rustflags = [
  "-C",
  "prefer-dynamic",
]
```